Laconic:
Label Consistency for Image
Categorization
Samy Bengio, Googlewith Jeff Dean, Eugene Ie, Dumitru Erhan,
Quoc Le, Andrew Rabinovich, Jon Shlens,and Yoram Singer
1Motivation
WHAT IS THE
OCCLUDEDOBJECT ?

2

Motivation2

Motivation2

Speech Recognition Success• Modern speech recognition systems consist of twobuilding blocks:• Acoustic model(what is the most likely phoneme at this temporalwindow ?)• Language model(what is the most likely sequence of words thatwas uttered ?)• Both models are combined in the decoding phase• Eﬃcient because speech is approximately Markov
3

Language Models for Images?
• Can we apply a similar language modelingapproach to image recognition?

• More challenging since it is more diﬃcult toimpose “lexical” structure in an image• Simplify and consider a co-occurrence model:“a cobra on a desk” is a rare event• Use the web to collect A LOT OF co-occurrence
statistics

4

Related Work
• Plenty of work on using ontologies like WordNet• [Marszalek et al, CVPR2007; Weston et. al, NIPS2012;
Deng et. al, NIPS2011, ...]• Other Co-occurrence models ( see for instance [Rabinovitchet. al, ICCV2007], [Torralba et al, NIPS2005] [Ladicky et al,ECCV2007], etc)

• Not use a LARGE separate source of language model.
• No eﬃcient optimization over the potential subsets oflabels: often just try all of them... doesn’t scale!

• Here we focus on an eﬃcient scalable solution with a large
separate source of language model.

• Others (various graphical models, often not scalable)
5The “Laconic” Model• “Label Consistency for Image Categorization”• We look for a small set of labels that best describe
an image and are consistent w.r.t each other• “best describe an image”:

use a patch classifier(we use a deep learning approach)• “consistent with each other”:

use web co-occurrence statistics

6

Co-Occurrence Model• Harvest all web pages and gather statistics

about appearances of terms describing classes
P (i, j)s(i, j) = log
P (i)P (j)• Consider a co-occurrence if two terms appear
nearby (say K words apart) on a web page.

• Unbounded value: positive values imply correlation,negative values imply anti-correlation.
(1if s(i, j) > 01+exp( s(i,j))
S ij =0otherwise• Result is bounded between 0 and 1.
7

8

Label Voting Model
We use an already trained model (Google’s deeplearning model) to cast a score for each label atvarious positions and scales of an image
To simplify, we aggregate them:P
p2P w(p)v(i, p)Pµi =p2P w(p)w(p) is the weight of patch pv(i,p) is DeepNet score for the label-patch pair (i,p)
Patch weights can take into account position & size
9Inference as an Optimization Problem• We want to jointly maximize consistency and label
votes, under domain constraints:

10Inference as an Optimization Problem• We want to jointly maximize consistency and label

votes, under domain constraints:
min Q(↵|µ, S) = E(↵|µ) + C(↵|S) + ✏R(↵) s.t. ↵ 2 ⌦↵10

Inference as an Optimization Problem• We want to jointly maximize consistency and label

votes, under domain constraints:
min Q(↵|µ, S) = E(↵|µ) + C(↵|S) + ✏R(↵) s.t. ↵ 2 ⌦↵( Tµ ↵[linear]⇣⌘PE(↵|µ) =↵j[relative entropy]j ↵j log µj

10Inference as an Optimization Problem• We want to jointly maximize consistency and label

votes, under domain constraints:min Q(↵|µ, S) = E(↵|µ) + C(↵|S) + ✏R(↵) s.t. ↵ 2 ⌦↵( Tµ ↵[linear]⇣⌘PE(↵|µ) =↵j[relative entropy]j ↵j log µj
8
T[Ising]>< ↵ S↵ (C(↵|S) = P
d2ij /2|dij |  1>
[Huber]: i,j Si,j(|dij | 1/2) |dij | > 1
where dij = ↵i10↵jInference as an Optimization Problem• We want to jointly maximize consistency and label

votes, under domain constraints:min Q(↵|µ, S) = E(↵|µ) + C(↵|S) + ✏R(↵) s.t. ↵ 2 ⌦↵( Tµ ↵[linear]⇣⌘PE(↵|µ) =↵j[relative entropy]j ↵j log µj
8T[Ising]>< ↵ S↵ (C(↵|S) = P
d2ij /2|dij |  1>
[Huber]: i,j Si,j(|dij | 1/2) |dij | > 1
where dij = ↵iTR(↵) = ↵ ↵10↵jInference as an Optimization Problem• We want to jointly maximize consistency and label

votes, under domain constraints:min Q(↵|µ, S) = E(↵|µ) + C(↵|S) + ✏R(↵) s.t. ↵ 2 ⌦↵( Tµ ↵[linear]⇣⌘PE(↵|µ) =↵j[relative entropy]j ↵j log µj
8T[Ising]>< ↵ S↵ (C(↵|S) = P
d2ij /2|dij |  1>[Huber]: i,j Si,j(|dij | 1/2) |dij | > 1TR(↵) = ↵ ↵⌦=8<X:jwhere dij = ↵i↵j↵j  N , k↵k1  1 , 8j : ↵j
1009=;

Pseudocode
Until Approximation Accuracy Satisfied:
1. Apply gradient descent on the objective:✓◆@E(↵|µ)@C(↵|S)@R(↵)
v t = ↵t 1 ⌫ t++✏@↵@↵@↵
2. Project onto feasible set by solving:1↵t = arg min k↵↵ 2v t k s.t. 0

11↵

1 , k↵k1  NWhen To Use Laconic
Hypothesis: a label consistency model may help if
noise level of the predictions of the patch classifier
is higher than that of the co-occurrence model

12

Initial Experiments - Dataset• ImageNet 2012 - detection subtask• 3623 classes with bounding boxes• About 1M images in the test set.
• Sun’12• Multiple labels per image (total of 3765 labels)• About 1000 images in the test set.
• Also access to “training-set co-occurrence matrix”
13Comparing Laconic Approaches
Linear Linear Entropy
Statistics
Ising - Ising - IsingProjection

Entropy Linear Ising Huber Projection Projection

Prec@1 86.2% 86.3%

86.4%

85.8%

85.8%

Prec@5 53.1% 52.8%

52.8%

52.6%

50.1%

Average
76.3% 76.2%
Precision

76.2%

75.9%

74.9%

OUT OF 987 TEST IMAGES FROM SUN’12 DATASET

14

Results for Sun’12
Statistics Baseline Laconic-web

Laconictrain

LaconicMixture

Prec@1

85.7%

86.2%

85.5%

86.1%

Prec@5

50.6%

53.1%

50.8%

53.7%

Average
75.0%
Precision

76.3%

75.1%

76.6%

OUT OF 987 TEST IMAGES FROM SUN’12 DATASET

15

Results for ImageNet-3623 classes
Statistics

Baseline

Laconic

Relative
Improvement

Hit@1

24.8%

34.3%

38.3%

Hit@5

44.4%

52.4%

18.0%

Detection@1

13.7%

19.6%

43.1%

Detection@5

25.8%

28.4%

10.1%

OUT OF 965857 TEST IMAGES FROM IMAGENET
DETECTION TASK

16

Precision@K - Laconic vs Baseline

17

Examples (Sun’12)
CabinetWorktop
Floor

Cabinet

Trees

Door
Floor

Door

Trees

Door

Window
PersonFloor-black: ground truth-green: laconic
18-blue: baseline

TreesExamples (ImageNet)-red: ground truth-green: baseline-blue: laconic

19

Conclusion• Use external language model to regularize
image categorization eﬃciently.

• We need a good source of co-occurrences.
• Experiments on a subset of ImageNet and on
Sun’12 show promising results.

• More structure can be added to the model:• gather sentences like: “the sky is above themountain” , “the monitor is on the table”...
20