Web Image Gathering withRegion-based Bag-of-features
and Multiple Instance Learning
IEEE International Conference on Multimedia
and Expo (IEEE ICME 2009) July 2nd, 2009

Keiji Yanai
The University of Electro-Communications, Tokyo, Japan
1.

2.
3.
4.

Objective & Background
Related Work
System & Methods
Experimental Results
11. Objective&BackgroundBackground3

Web is the largest image DB.
It is also a very noisy DB.
 To remove noise, image analysis is needed.
 Since 2001, we have been working on

Web Image Gathering with image analysis
Keiji Yanai: Image Collector: An Image-Gathering System from theWorld-Wide Web Employing Keyword-based Search Engines,
ICME 2001, Tokyo, Japan, pp.704-707 (2001/08). (ACMMM 2003,..)

 Non-interactive.

No feedback. Fully-automatic.
 To gather visual knowledge of many concepts
for object recognition from the Web

Objective of this paper4 Import region-based bag-of-features

to our Web image “gathering” system[Yanai et al. ICME01, ACM MM03, ACM MIR 05, ICME08]【Image representation】

new combination !

region-based bag-of-features[Ravinovich et al. ICCV 07]【Classifier】

mi-SVM (multiple instance learning)[Andrew et al. NIPS 03]

2. Related Work

General Framework: Web image

6search + Object Recognition Technique Firstly, gather images from the Web

using Web (image) search engines such
as Google, Ask.com and MSN searchby providing given keywords.
 Secondly, re-rank the results from theWeb search engines with object/scene
recognition methods
keywords

Gathering Web images

image analysis

rankedimages

Literature: Web image search + 7
Object Recognition Technique[Yanai ICME01]
 Color signature + EMD + k-NN [Yanai ACM MM03]
 Constellation model + RANSAC[Fergus ICCV04]
 JSEG + GMM （image-word translation model）[Yanai & Barnard ACM MIR 05]
 Bag-of-features (BoF)+ pLSA
[Fergus ECCV05]
 Bag-of-features + HDP(Hierarchical DirichletProcess) (OPTIMOL)[Li CVPR07]
 Bag-of-features + SVM [ICCV Schroff 07] [Yanai 07]
 【This paper】
JSEG + region-based bag-of-features+ mi-SVM (multiple instance learning) Color histogram + k-meansLiterature: Web image search + 8
Object Recognition Technique JSEG + GMM （image-word translation model）[Yanai & Barnard ACM MIR 05]
Prob.
+ Model(GMM)lion

 Bag-of-features + SVM [ICCV Schroff 07] [Yanai 07]+ SVM
 【This paper】

lion ornon-lion

codewords (visual words)

JSEG + region-based bag-of-features+ mi-SVM (multiple instance learning)

Contribution of this paper Import region-based bag-of-features

to our Web image “gathering” system【Image representation】

region-based bag-of-features[Ravinovich et al. ICCV 07]【Classifier】

mi-SVM (multiple instance learning)[Andrew et al. NIPS 03]

93. Methods

Basic framework of our system[Yanai ICME01]～
Collection stage :11Unchanged since [ICME01]

Gather image and HTML files using Web search engines.
Select pseudo-training images by HTML analysis

Selection Stage :Use supervised object rec. methods
Train a classifier and with pseudo-training images
rank images based on estimated relevancy“lion”
query
keywords

Collection stage

Selection stage（search by keywords ）（analyzing images）“lion”URLs

Web search engines

HTML files & images

WWWresults

12Image features
 Divide each image into regions by JSEG

(8 regions on the average)

Bag-of-features(BoF) histograms(1000-dim)

Segmentation （JSEG）

[image representation]
Bag-of-features13

 Represent an image as sets of features2.

3.

Densely-sample points along regular grids
Represent local patterns around sampled
points with SIFT descriptor
Vector-quantize SIFT vectors based on
pre-computed visual words (codewords)
frequency1.

codewords (visual words)

How to obtain visual words

14

 Extract many SIFT vectors from positive andnegative training samples
 Perform k-means clusteringcenter of clusters are “visual words”.

vi vi
vi vi
vi
vi vi
vi
vi v
i

SIFT vectors“Visual words” arerepresentative local patterns.

15

Multiple Instance Setting Positive bags / Negative bags

positive ins.
(foreground)negative ins.
(background)

Positive instances of “flower”

pseudo-training images

The rest of regions arenegative regions.

random images

mi-SVM[Andrew et al. NIPS 03]

16

 Apply soft-margin SVM iteratively
 Training → classifying → training →classifying → ……
During the iteration, the hyperplane is approaching the optimalplane to discriminate positive
instances from negative ones.
positive ins.
(foreground)negative ins.
(background)17

Final Image Re-ranking

 Regard the best SVM output score of theregions within an image as the score ofthe image
 An image having one positive region at leastis a positive image !
 Rank images based on the scores

flowerairplaneapple

4. Experimental
results

19

Experiments for 10+5 words
 sunset, mountain, waterfall, beach,(4scenes)
noodle, flower, lion, apple, baby, laptop-PC, (6objects)
airplane, guitar, leopard, motorbike, watch (5objects)

 Method:[raw data] raw (only HTML analysis) 39,143 images for 15 words[baseline1] GMM-based region probabilistic model [ACM MIR05][baseline2] BoF + SVM[proposed] region-based BoF + SVM

 Evaluation:

precision at 15% recall
the same as [ICCV Schroff 07]

Comparison of 4 methods(raw, GMM, BoK, reg-BoF)
120
prec.(%)
100
80
60
40
20021raw
GMM
BoF
reg-BoF

raw
62.2％
GMM
73.5％
BoF
88.17％
Reg-BoF
87.82%

Pseudo-training image sets
and results by perfect training
set (noise-free)
120

100
8060

training imgs

40

raw images

200

regBoFnoise-free23

Comparison with related work
 [Fergus ICCV05] Bag-of-features + pLSA

 [Schroff ICCV07] Bag-of-features + SVM
 [new] Region-based BoF + mi-SVM
120prec.(%)
1008060
40

200Fergus
Schroffreg BoF

26

Many result images
 Laptop-PC(positive and negative) Mountain Waterfall Flower Airplane ???
 As by-products, we can estimate

representative regions of images.

(different from standard BoF)

27

Conclusion29

 Import region-based bag-of-features(BoF) and mi-SVM
into the Web image gathering task.
 In spite of noisy training data,

the proposed method worked well.
 It was especially effective for objectconcepts.

Future work
 Large-scale experiments
 More than concept for 1000 concepts

 Improve the text analysis partto obtain more accurate psudotraining samples
 Use co-occurrency of tags
 Use taxonomy dic. (Wordnet, Wikipedia)

30

Thank you!

“sunset”

Rejected “sunset”“waterfall”

Rejected “waterfall”

32

“Chinese noodle”“notebook PC”“lion”“baby”

33