Automatic expansion of a food image
dataset leveraging existing categories
with domain adaptation
Yoshiyuki Kawano, Keiji Yanai
The University of Electro
Communications, Tokyo, JAPANBackground• Large-scale image dataset– up to over 1000
– Using crowd-sourcing• Semi-automatically build• The issue– Costly– Performance is not perfect
We aim to reduce cost and improve accuracy
2
by noise removal of data in advance automatically.

Objective• Expand an existing image dataset– Target domain: food image– 100 kinds of Japan foods -> 256 kinds of worldwide foods– Noise removaltiramisu• Annotation– Category label– Bounding box
3

Related work
• Food image dataset– UEC Food100 [Matsuda. ICME2013]• One country (japan) food• Expande this dataset.

• Large-scale database by using AMT– ImageNet (http://image-net.org) [CVPR2009]• Web image search + AMT• CV+ active learning + crowdsourcing– Object detector and AMT [Vijayanarasimhan. CVPR2012]4

Overview of the flow
1. Collect images from the Web– Web image search using keyword

2. Automatic noise removal by CV– Foodness classifier and adaptive SVM– Employing existing dataset

3. Manually noise removal by AMT– representative image selection task,
noise removal task, drawing BB task,
5

Detail of the flow
(1) pre-processing(2) Query: food name

Existing
food DBWeb (Flickr, Twitter, Bing)
Web API

Food group

HOG FV

SVMs

Color FV

SVMs

(3-a) Apply “Foodness” Classifier (FC)
Target images (Web)
VisualRank

HOG FV

A-SVM(3-b) Apply transfer learning (A-SVM)
Target images (CV)Color FV

A-SVM

(4-a) Select representative sample
Sample images(4-b) Noise removal
Target images(4-c) Draw bounding boxes
Food photos of the new categories

Annotated images6

1. Collect images from the Web• Prepare a new food name.
– e.g. “mango pudding”, ”tiramisu”• Collecting– Image search with a keyword using Web API(Flickr, Twitter, bing)– Local language and English

7

2．Noise removal by CV• Automatic noise removal by 2-step filtering
1. Foodness Classifier• Food or non food image• Training: using the existing food image dataset.

2．Adaptive SVM• the new category image or not.
• Training: Higher foodness images
8

2.1. Foodness Classifier (FC)• Objective– food / nonfood classification– use the existing dataset• Image features, Classifier– HOG and color FV, linear SVMs

13 groups• Grouping of the existing food categories– multi-class classification– Build a confusion matrix– Make some category groups based on the matrix.
(inspired by [Bergano et al. CVPR 2011])

9

2.1. Foodness Classifier (FC)• Train– positive： each 13 group.
– negative: non food web images• street stall, kitchen,• dinner party and restaurant

13train

SVM• Testtest– The maximum value as foodness

groups

train

train

SVM

SVM

test

test

max
Foodness score102.1 group and some food categories

112.1 group and some food categories

122.1 group and some food categories

132.1 group and some food categories

142.1 group and some food categories

15

2.2. adaptive SVM
Objective– Classify the new category or not– [issue] no training data of the new unknown category．
– Employ the existing dataset by transfer learning,• Use image recognition (same as 2.1 except A-SVM)– Image features + adaptive SVM– [pseudo-positive]: the images with higher VisualRank value(apply VisualRank for the remaining images)– [negative]: the images with lower foodness score– [source domain]: The most related food group16

Collecting Web images• tiramisu

17Apply Foodness Classifier• tiramisu

18Apply adaptive SVM• tiramisu

19Some examples of the pair oftarget foods and source food groups.
target the most related food group
Okinawasoba
Mango
puddingminestrone

nachos20

3. Manually noise removal by AMT• 3-step crowdsourcing– Representative select sample task
• 0.06$ per HIT, 5 workers– Noise removal task
• 0.03$ per HIT, 5 workers

tiramisu– Drawing bounding box task
• 0.05$ per HIT, 4 workers

21Experiment• Evaluation– Performance comparison on food image filtering– The final results after AMT and analysis of cost

• Setup– Web API (Flickr, twitter ,bing)– More than 600 images for each category– 5 countries, total 35 foods• abbreviation– Food Classifier: FC– Adaptive SVM: A-SVM– Noise removal task (AMT): NR Task

22

Experimental result
Average precision @300Method

America Japanese Chinese Thailand Indonesia Average

VisualRank (VR)58.47 54.95

60.66 62.19

58.71

59.00

Food
Classifier(FC)FC + SVM

78.00 75.33

77.61 82.85

78.61

78.48

85.57 83.38

85.09 89.23

82.23

85.10

FC +A-SVM

89.61 87.76

87.76 91.38

84.09

88.12

Experimental result
Precision of food image on dataset
FC
FC + A-SVM
FC + A-SVM + NR Task

precision[%]
91.10
94.19
97.83

gain[%]+ 3.09+ 3.64

24

Experimental result
Cost (annotated 100 images)
Noise removalFC
FC + A-SVM

FC + A-SVM +NR-Task

Draw BB

TotalRecoveryratio[%]cost[$]

Recoveryratio[%]

cost[$]

cost[$]

80.9

0.74

64.2
74.7
86.7

3.11
2.68
2.31

3.11
2.68
3.16

25

Conclusion• Proposed a novel framework to expand an
existing image dataset automatically– Generic classifiers– Domain adaptation• Feature Works– make further analysis on the difference between amanually collected dataset and an automaticallycollected dataset.
26

27

3.1. サンプル選択タスク• タスクの特徴– 30枚を与えて、最大7枚、一般的なサンプルを選択してもらう– リンクを与えるので勉強してもらう– $0.06、4人• 少し高め、外部リンクで勉強させる、また重要であるため– 質の向上• サンプル選択をしないとヒットを提出できない• 設置したリンクをたどらないとヒットを提出できない28

ワーカーの国別人口• 自由回答によるワーカーの国別人口– インド人が圧倒的に多い
india

usa

Canada French uk

Asia

1668

394

20433

• 今回対象にした食事は、世界の食事– インド人は未知の食事も多い
29

悪意のあるワーカー• 特定の一人の悪意のあるワーカーの結果

30

3.3. example of drawing BB task
Obtained representative images

Draw bounding box or mark irrelevant images

31

3.3. the real exapmle

32

3.3. the final result
Apple pie

33

3.1. the result of selecting representative task
• Obtain about 5 images– 100 categories100%• Worker’s evaluation– Ask workers in each HITNR TaskBB Taskusefulso souseless

89.59
91.68

7.90
7.02

2.52
1.31
34

Basic idea
• Food category– Look more similar to each other– Visual knowledge on food help to annotations
other food.

• Method– Effective framework by domain adaptation35

2.1. Foodness Classifier (FC)• objective– food / nonfood classification• Image– Image features + linear SVM– positive： make 13 group based on confusion matrix.
(originally 100 kinds) [Bergamo et al., CVPR2012]，– negative: gather images using web images• street stall, kitchen, dinner party and restaurant– Train SVMs for each 13 groups– The highest evaluation value is foodness36

2.1. Foodness Classifier (FC)• Objective– food / nonfood classificationtr– use the existing dataset• Image features, Classifier– HOG and color Patch FV, linear SVMs

• Train and Test
– positive： make 13 group based on confusion matrix.
[Bergamo et al., CVPR2012]，– negative: non food web images• street stall, kitchen, dinner party and restaurant– Train SVMs for each 13 groups– The highest evaluation value is foodness37

3.1. Selecting representative sample task
• issue
Worker dose not have knowledge about various kinds of foods.

• Objective– Obtain highly accurate representative samples for other task.

• Task– 1HIT 0.06$, 5 workers– Needs to study about the food category38

3.2. Noise removal task
• Objective– Manually noise removal• Task– 1 HIT $0.03, 5 workers– Majority voting– Cannot be submit if there are more than
4 unchecked images
39

3.3. Drawing BB task
• objective– Drawing bounding box– Manually noise removal• Task– 1 HIT $0.05, 4 workers– can mark irrelevant images

40