ACM Multimedia 2020 Tutorial
Effective and Efficient: Toward Open-world Instance Re-identification

IS INSTANCE SEARCHA SOLVED PROBLEM?
Shin’ichi Satoh
National Institute of Informatics, Japan

What is Instance Search?
• Instance Search is different from image similarity search
NOT Instance Search• Instance Search is to search for instances of specific objects, such as…
• Particular objects, individuals,• Manufactured products whose appearance is indistinguishable,• Logos, etc.

Properties of Instance Search• Ground truth is “well-defined”: if images contain the same object to the query,the images are relevant• In contrast, image similarity is complicated to define• Well studied from the early stage of image semantic analysis• Many reasonable techniques work as expected: nice research topics• Request from society: primary target of users of image retrieval, e.g., search
for images of this person, search for images of this product, search for imagesof this landmark, etc.
• Good cue for multimedia analytics: market research of specific products bycompanies• Surveillance, safety, security

COIL-100: Early Dataset for Object Recognition• Columbia University Image Library(COIL-100) appeared in 1990s for object
recognition• The goal is to identify objects, despite theview angle differences• No background (simple)• Before Caltech, PASCAL VOC, ImageNet• “Easier” than these datasets• One reason: very small intra-classvariation"Columbia Object Image Library (COIL-100),"
S. A. Nene, S. K. Nayar and H. Murase,
Technical Report CUCS-006-96, February 1996.

Landmark Retrieval: Oxford and Paris• Landmark retrieval is very well studied• The definition of the ground truth is very clear: whether thetarget landmark appears in images or not (c.f., whether images
are similar or not)• 5k Oxford landmarks and 6k Paris landmarks• Oxford (CVPR07): 3k references, Paris (CVPR08): 1.5kreferences• Many important image retrieval techniques have been studied
based on these datasets• Query expansion, database-side augmentation, query-sideaugmentation, geometric verification, diffusion, etc…
• Recently very high performance reported (over 90% map):already solved?
Oxford: Object retrieval with large vocabularies and fast spatial matching, CVPR07Paris: Lost in quantization: Improving particular object retrieval in large scale image databases, CVPR08

Revisiting Oxford and Paris (CVPR18)• Cleansed annotations for Oxford and Paris withnew labels: Easy, Hard, Unclear, Negative• New, larger, cleaner distractors• There still remain many challenging situations:“image retrieval appears far from being solved”

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking, CVPR18

Google Landmark V2 (CVPR20)• Large-scale, long-tailed landmark

retrieval dataset• “ImageNet” of landmark retrieval• 5M images, 200k instance labels

the class distribution is very long-tailed, the dataset
contains a large number of lesser-known locallandmarks

Heatmap of the places in the GoogleLandmarks Dataset v2

Google Landmarks Dataset v2 A Large-Scale Benchmark for
Instance-Level Recognition and Retrieval, CVPR20

Google Landmark V2 (CVPR20)

Comparison against existing landmark datasets

Baseline results (% mAP@100) for theGoogle Landmarks Dataset v2 (GLDv2)
retrieval task.

Google Landmark V2: Noise problem

Large-scale Landmark Retrieval/Recognition under a Noisy and Diverse Dataset
https://arxiv.org/abs/1906.04087

Approaches• Deep local feature-based methods

are very high-performant in recent
landmark retrieval attempts• MAC, CroW, R-MAC, DELF, DIR…
• “Old” encoding techniques for local
features such as VLAD, SMK, ASMK
(originally proposed for SIFT) are stilleffective• Spatial verification, query expansion,diffusion, are also effective• SIFT has been replaced by deep, but
overall pipeline is kept: newbreakthrough?

DELF (Large-Scale Image Retrieval with Attentive Deep
Local Features, ICCV17)

New Challenge in Person Re-ID:
Guiding Robot w/ Follower

Face Recognition
Person Re-identification
The robot not only plans the routeand pilots the way (using the frontview camera), but also always
focuses the follower (using theback view camera), so that thefollower could catch up the robot
and keep in service.

ACMMM20 Demo
Progressive Domain Adaptation for
Robot Vision Person Re-identification

Security Vision vs. Robot Vision

Camera is fixed. Background of camera
view maintains. A stable illumination
condition.
The view of camera is from top to bottom.

The camera view changes. Person images
encounter shaking and changing
illuminations.
The view of camera is from bottom to top.

Persons have relatively stableappearance.

Persons have more variations, including
clothes change, frequently occlusion, half ofthe body.

Sufficient annotations

No annotation

Framework• Step 1: Re-ID models are pre-trained on a camera network dataset (Market1501).
• Step 2: The robot back-view video is input into the existing face recognitionand person tracking models, then a batch of tracklets will be generated.
• Step 3: In the feature space, the similarities of images from different tracklets
are calculated, and thus pseudo labels are generated.
• Step 4: The selected tracklets (some batches of images) and their pseudolabels are used to fine-tune the Re-ID model.

Example Videos
Face Recognition

W/ Mask

Different Poses

Person Re-identification

Far Distance

Illumination Change

Occlusion

Conclusion• Instance search is an old, well-defined, and well studied task
• But it’s still important and there still are many challenges remaining• Landmark retrieval faces now new and practical issues: large-scale, long-tail,noise issues• Approaches: deep local feature is used in place of SIFT, but overall pipeline:
local feature encoding, post processing, reranking, diffusion…
• New breakthrough is awaited• Person Re-Identification: viewpoint changes, severe illumination changes,
pose variations