Region-based Automatic Web Image Selection
Keiji Yanai and Kobus Barnard
Univ. of Electro-Communications, Tokyo, JAPAN / Univ. of Arizona, USA
Overview
Web Image Re-ranking
with region-based BoF andML methods (LDA, pLSA, SVM, MIL)
“lion”

Generative
pLSA
LDA
Discrimina
-tive
SVM
MIL-SVM

Pseudopositive

Randomnegative[1] Provide
keywords[2] GatherWeb
Images
Web searchengines[3] Selectpseudopositive
samples[4] Segmentation& Extract features[5] Train &[6] Classify
by MLmethods[7] Re-ranked
results

1.Background & Objective
Web is the largest image DB.
It is also a very noisy DB.
To remove noise, we apply object
recognition methods.

“Web Image Re-ranking”
 We assume no feedback and fullyautomatic “Web Image Re-ranking”.
 It is desirable for gathering visualknowledge of many concepts forobject recognition from the Web

3. Methods

3. Methods (cont.)

processing steps :[1] Prepare keywords[2] Gather Web images using several
Web search engines[3] Select pseudo-positive images
by HTML analysis automatically[4] Extract region-based BOF[5] Train a model withpseudo-positive samples andrandom negative samples[6] Evaluate all the images
with the trained model[7] Re-rank all the images
according to the output values[5-3] pLSA (probabilistic Latent Semantic Analysis)[5-4] LDA (Latent Dirichlet Allocation)

import region-based bag-of-features(BoF)
to Web image re-ranking.
【Image representation】

region-based bag-of-features[Ravinovich et al. ICCV 07]【Classifier】 (2 kinds)• sMIL (sparse multiple instance learning)several search engines, andgather thousands of Web images(Google Image Search, Google Text Search,MSN, ask.com, Yahoo Image, Yahoo Search)• Evaluate HTMLs containing images
based on the simple heuristics[Yanai ACMMM03] and select pseudopositive images automatically[4] Extract region-based BOF vectors• Carry out region segmentation withJSEG and extract BoF from each region
Bag-of-features(BoF) histograms(1000-dim)• Probabilistic latent topic methods

with pLSA and LDA[Yanai et al. MIR05, Monay et al. PAMI 07]
Segmentation by JSEG

[5] Train a model• Two discriminative and two generative General Framework: Web imagesearch + object recognition methods
keywords

Gather images
from Web using
search engines

re-rankedimages

image analysis
ML methods

Literature•Color histogram + k-means[Yanai ICME01]•Color signature + EMD + k-NN [Yanai ACM MM03]•Constellation model + RANSAC [Fergus ICCV04]•JSEG + GMM （image-word translation model）[Yanai & Barnard ACM MIR 05]
Prob.
+ Model(GMM)

Regard all the regions from pseudopositive images as positive samples[5-2] sparse MIL [Benescu et al. ICML07]• Positive bags / Negative bags

lion ornon-lioncodewords (visual words)【This paper】
JSEG + Region-based BoF+ discriminative methods (SVM / sparse MIL)or generative methods (pLSA / LDA)

positive instance(foreground)negative instance(background)lion•Bag-of-features (BoF)+ pLSA [Fergus ECCV05]•Bag-of-features + HDP(Hierarchical Dirichlet Process)(OPTIMOL)[Li CVPR07]•Bag-of-features + SVM [ICCV Schroff 07] [Yanai 07]+ SVM[5-1] standard SVM

Sparse MIL is an extension of a SVM formultiple instance (MI) settings.

positive img. selection

0.661
0.167①D-:negative regions
p0i  (1 / D  )  P (d j | zi )d j D ② 0.186 p  (1/ D )  P(d | z )i1d j D ji

P ( pos | zi )  p0i /( p0i  p1i )0.407 P(neg | z )  1  P( pos | z )ii

0.023 P(positive | topic)
0.761
0.334

“mountain” images

latent topics

P(positive|topics)
0.987② Estimate P(positive | topic) for each topic
using positive regions from pseudo-positive
images and negative regions from negative
randomly-sampled images③ Compute P(positive | region) for each region:

Regard the maximum output values of regions
within an image as the output of the image(max strategy)[7] Re-rank all the images

4. Experiments for 15 words
 4 scenes + 6 objects + 5 objects
sunset, mountain, waterfall, beach,noodle, flower, lion, apple, baby, laptop-PC,
airplane, guitar, leopard, motorbike, watch

Same as[Yanai 05]
Same as[Fergus 05]

 Methods :[Bunescu et al. ICML 07]

2. Related Work

0.112 D+:positive regions[1] Prepare keywords
P(positive | region)
ke.g. “sunset”,“lion”and “apple fruit”
  P(positive | topic i ) P( topic i | region)[2] Gather Web imagesi 1• Send the given keyword to[6] Evaluate an image with the model[3] Select pseudo-positive images

In this paer, we① Do probabilistic topic clustering(PLSA/LDA)
for all the regions with the topic number k .
Obtain P(topic | region) for all the regions

Positive instances of “flower”

pseudo-training images[raw data] raw (only HTML analysis) 39,143img.
[baseline1] GMM-based region prob. Model [ACM MIR05][baseline2] whole-image-based BoF + SVM[SVM] region-based BoF + SVM[sMIL] region-based BoF + sparse MIL[pLSA] region-based BoF + pLSA
[LDA] region-based BoF + LDA
Evaluation: precision at 15% recall (same as [Schroff 07] ) Comparison of above 7 methods
10090807060
50403020100

The rest of the regions
are negative regions.

random negative images

5. Conclusions
We confirmed that region-based BoF
is effective for “object” words.
S-MIL outperfomed pLSA- and LDAbased probabilistic methods.
 We plan to use Folksonomy, more sophisticatedHTML analysis and various image features.

prec.(%)

sceneraw
B1
B2
SVM
sMIL
pLSA
LDAobjectComparison with related work
100
90
80
70
60
50
40
30
20
100prec.(%)

MethodScene / object

raw
67.5/60.8
GMM(B1)
88.3/74.3
BoF(B2)
98.4/81.3
SVM
91.3/73.7
sMIL
92.9/79.3
pLSA
94.1/77.4
LDA
94.2/75.6

Fergus 65.0
Schroff 73.4
Wunk 77.6
65.5
B285.5SVM
89.6sMIL
pLSA 81.0
80.6
LDA