TRECVID
INSTANCE SEARCH(INS)
Shin’ichi Satoh and Zheng Wang
National Institute of Informatics, JapanTRECVID (from TRECVID web site...)• Workshop series from 2001 to present• Large-scale laboratory testing for content-based video

analysis and retrieval• Forum for the• exchange of research ideas• discussion of approaches: what works, what doesn’t, and why• Aims for realistic system tasks and test collections• unfiltered data
• focus on relatively high-level functionality• Provides data, tasks, and uniform, appropriate scoringprocedures

TRECVID Instance Search (INS)• To find “instances” of some object, person, or location invideo• specific object, person, or location• e.g., search for this particular dog, search for Dot (a certain person)

appearing in Kitchen1 (a certain location)• different manufactured objects which are indistinguishable• the certain person and the certain location are not recognizable atthe same time

• Queries will be given as visual examples• There exist couple of related benchmark datasets• Oxford Building, Paris (landmarks)• Flickr Logos (logos)• UKBench, Stanford Mobile Visual Search (specific objects)• etc.

Comparison with other benchmarks• TRECVID INS determines data first: therefore very “wild”• Other benchmarks define queries first, and then collect

data: therefore objects clearly appear
UKB
Ox5k
FL32SMVS

Oxford 5K• 5062 images collected from Flickr for particular Oxford landmarks.
• Manually annotated to generate a comprehensive ground truth for

11 different landmarks, each represented by 5 possible queries.

For each image and landmark, one of four possible labels wasgenerated:• Good - A nice, clear picture of the object/building.
• OK - More than 25% of the object is clearly visible.
• Bad - The object is not present.
• Junk - Less than 25% of the object is visible, or there arevery high levels of occlusion or distortion.
• 7-220 good and ok images per query

Philbin, J. , et al., Object retrieval with large vocabularies and fast spatial matching, CVPR, 2007
http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/

The Stanford Mobile Visual Search
Dataset (SMVS)• The data set consists of images for many different

categories captured with a variety of cameraphones, and under widely varying lightingconditions. Database and query images alternate ineach category.

V. Chandrasekhar, et al., The Stanford Mobile Visual Search Data Set, MMSys, 2011
http://web.cs.wpi.edu/~claypool/mmsys-dataset/2011/stanford/

FlickrLogos-32• The dataset FlickrLogos-32 contains

photos showing brand logos and ismeant for the evaluation of logo
retrieval and multi-class logo
detection/recognition systems on
real-world images. They collectedlogos of 32 different logo brands bydownloading them from Flickr. All
logos have an approximately planar
surface.

http://www.multimedia-computing.de/flickrlogos/

University of Kentucky Benchmark (UKB)• The University of Kentucky retrieval

benchmark is a dataset which consist of2550 classes, each class with 4 images
with JPEG format.
• The pictures are from diverse categories

such as animals, plants, household objects,etc.

D. Nister, et al., Scalable recognition with a vocabulary tree. CVPR, 2006

Comparison with other benchmarks
TRECVID INS

The other benchmarks

Instance type

Different: object & person & location

Same: logo | landmark | …

Scale

Different scales in the images

Similar scale, main part of the image

TargetFrequency

A wide range from 10 to 2000

Similar number, stable

Data Source

TV videos

Images collected from internet

Data type

Video, audio, text

image

Characteristic determines data first: therefore very
“wild”

define queries first, and then collect
data: therefore objects clearly appear

TRECVID INS Data
• Collection of several hundreds hours of videos for each year
• Data should contain multiple occurrences of multiple specific objects.
• Search tasks should be reasonably difficult.

• Sound and Vision (2010): too difficult, too few repeatedinstances, otherwise too easy(copies)

TRECVID INS Data
• Collection of several hundreds hours of videos for each year
• Data should contain multiple occurrences of multiple specific objects.
• Search tasks should be reasonably difficult.

• BBC Rushes (2011): including retakes, artificial videotransformations

TRECVID INS Data
• Collection of several hundreds hours of videos for each year
• Data should contain multiple occurrences of multiple specific objects.
• Search tasks should be reasonably difficult.

• Flickr Creative Commons (2012): retrieved by text queries,reasonable, but still hard to find repeated instances

Data• Collection of several hundreds hours of videos for each year
• Data should contain multiple occurrences of multiple specific objects.
• Search tasks should be reasonably difficult.

• BBC EeastEnders (2013-present): drama series, “small world”

many repeated instances (person, location, objects, ...)•••The BBC and the AXES project made 464 hours of the BBC soap opera EastEnders available for research in MPEG-4
244 weekly “omnibus” files from 5 years of broadcasts• 471527 shots• Average shot length: 3.5 seconds• Transcripts from BBC• Per-file metadata
Represents a “small world” with a slowly changing set of:• People (several dozen)• Locales: homes, workplaces, pubs, cafes, open-air market, clubs• Objects: clothes, cars, household goods, personal possessions, pets, etc• Views: various camera positions, times of year, times of day

Frequency of Ground-truth
2000
1500

500
025
31
31
32
61
75
82
100
118
161
171
171
213
251
252
261
363
386
390
440
455
674
741
759782
831
880
1266
1605
2300

1000

2013

Ox5k5K, 55 query, 11 classes, 7-220 imgs

SMVS

1200, 3300 query, 1200 classes

FL328240, 32 classes, average

2000
1500

2

497

104

121

180

189

211

229

238

243

277

342

387

398

416

444

494

730

846

0

1568

500

Google Landmark Retrieval
1818

1000

2014

2000
1500

500
0

19
33
44
52
60
68
83
94
95
105
112
127
134
256
265
286
397
402
437
448
472
528
638
661
682
747
874
1103
1308
1735

1000

2015

Few images per class
23% with <= 5 images
44% with <= 10 images

Google Landmarks• 13k unique landmarks from across theworld• 1M images• 111k query images• Automatically generated ground truth(GPS and visual features)

Queries in 2013-2015• Couple of example images with masks• Original videos are also given (since 2014)

Task in 2013-2015• 2013-2015: specific object, person, or location

Task in 2013-2015• 2013-2015: specific object, person, or location

Task in 2013-2015• 2013-2015: specific object, person, or location

Task in 2013-2015• 2013-2015: specific object, person, or locationDifficulties
Easy topics● Simple visual context● Stationary target● Planar, rigid objectsDifficult topics● Small target● Moving target: differences incamera angle, location● Non-planar, non-rigid

Typical INS template system in 2013-2015
representation(local/global)

••
•••
•

HSV, LBP, HOG, CNN
Hesssian affine, Harris-Laplace, MSER
Detection methods, face classifier
RootSIFt + C-Sift
Cluster to BoVW (1M words, hard/soft etc)
VLAD quantization

0.453

0.45
0.4
0.35

0.313 0.325

0.3
0.25
0.2

matching

Post-processing••••••similarity function (asymmetric similarity)
normalization term in cosine similarity
BM25 enhancements for weighting
IDF adjusted for burstinessWeighting ROI vs. backgroundquery adaptive late fusion••••

spatial verification
Face/color filtering
Fusion of scores (Average pooling)
Use ROI features for reranking

0.15
0.1
0.05
0MAP
2013

2014

2015

2013

RootSIFt + C-Siftasymmetric similarity function

2014

larger BoVW vocabularies
combines BoVW and CNN
normalization term in cosine similarity
ROI features for reranking

2015

Use CNN for both local and global features
Query adaptive late fusion

Task in 2016-present• 2016-present: find a specific person in a specific location

The figure refers to PKU_ICST at TRECVID 2017: Instance Search Task

Comparison with task in 2013-2015
2013-20152016-present

Data Source

The same

Topicsobject / person / location

person + location

query

Image + mask

Person: image + mask
Location: 6-12 images
Related video shots

Characteristic One condition

Two conditions together

Difficulty

Persons / locations havedifferent views
Person and location influence toeach other, can not be searchedout simultaneously

Instance with different scalesand types

Topics in 2016
personTopics in 2016location

Topics in 2017
personTopics in 2017

locationTask in 2016-present• Example for person search 1
Query

Difficulties：
1. different camera conditions
2. different styles

Ranking
results

Task in 2016-present• Example for person search 2
QueryRanking
results

Difficulties：
3. incomplete or occluded
4. face is too small
5. appears in adjacent frameTask in 2016-present• Example for location search 1

Difficulties：
1. different views
2. blocked by persons

QueryRanking
results

Task in 2016-present• Example for location search 2
QueryRankingresults

Difficulties：
1. different views
2. blocked by persons

Task in 2016-present• Example for person + location search (true task)
QueryRanking
results

Task in 2016-present• additional difficulties for person + location : person searchand location search are always in a dilemma.

person faces are non-front or occluded

although it is a wide-angle view scene,
the person faces are very small

scenes are with low light or blur

scenes are blocked by persons

Task in 2016-present• additional difficulties for person + location : person searchand location search are always in a dilemma.

Topic 9170 in TRECVID INS 2016high scene score V.S. low person score

Topic 9210 in TRECVID INS 2017
low scene score V.S. high person score

Task in 2016-present

An example for consecutive shots in a time slice. Although the shots
contain the target person in the target location, the person andlocation scores are not always high simultaneously.
Neighbor shots will be helpful.

Systems Comparison
Location retrieval
Merge results
Person retrieval
Person retrieval

Location retrieval

Merge results

BUPTMCPRL

face retrieval (dlib)
person re-identification (Faster RCNN +fc layer feature)transcript-based

RootSIFT+AlexNet
VGG-16 Places365

Peron guide location+ location
guide person + random forest

NII-HitachiUIT

DPM+VGG-Face
SVM with RBF kernel

BOWscene tracking with person reidentificationIRIM

HOG detector + ResNet pre-trained on
FaceScrub & VGG-Face
Viola-Jones detector + FC7 of a VGG16network

Bow + Filter out person
Pretrained GoogLeNet Places365

Credits shots filtering
Indoor/Outdoor shots filtering
Shots threads filtering
Late fusion

PKU_ICSTVGG-Face + Cosine + SVM+
Progressive training

AKM-based (6 kinds of BoW)
DNN-based(VGGnet+GoogleNet+ResNet) +Progressive training

Peron guide location+ location
guide person + highlightcommon clues
Semi-supervised re-ranking

Typical Systems• IRIM at TRECVID 2017 (MAP = 0.4466)

Pers1： HOG detector + ResNet pre-trained on FaceScrub & VGG-Face
Pers2： Viola-Jones detector + FC7 of a VGG16 network
Loc1： Bow + Filter out person
Loc2： GoogLeNet Places365

Typical Systems• PKU_ICST at TRECVID 2017 (0.549)

Location-specific search: AKM-based (6 kinds of BoW) + DNN-based(VGGnet+GoogleNet+ResNet)
Person-specific search：VGG-Face + Cosine + SVM
Re-ranking：Semi-supervised re-ranking method (fusion)

AnalysisAnalysis“Ground truth” generation by pooling• Genuine ground truth is *not*maintained• Pool is composed of submitted
runs by multiple teams• Items in the pool are checked byhuman assessors• Inferred AP (infAP) is computed as
an unbiased estimate of AP• Extended AP (infxAP) is then usedbased on stratified random
sampling
infAP: J. A. Aslam et al., A Statistical Method for System Evaluation Using
Incomplete Judgments, SIGIR 2006
infxAP: E. Yilmaz et al. A Simple and Efﬁcient Sampling Method for Estimating APand NDCG, SIGIR 2008

Conclusion• Brief explanation of TRECVID Instance Search• Wild instance search benchmark because of “data first”

approach• Challenging task, while there still is a room to address, e.g.,• nature of video (consecutive shots, clips)• closed world information• New data and new query structure may also be considered:discussed among participants of TRECVID