21st International Conference on Pattern Recognition (ICPR 2012)
November 11-15, 2012. Tsukuba, Japan

Multiple-Food Recognition Considering Co-occurrence
Employing Manifold Ranking
Yuji Matsuda and Keiji Yanai
Graduate School of Informatics, The University of Electro-Communications
Email: matsuda-y@mm.cs.uec.ac.jp, yanai@cs.uec.ac.jp

Abstract
In this paper, we propose a method to recognize food images which include multiple food items
considering co-occurrence statistics of food items.
The proposed method employs a manifold ranking method which has been applied to image retrieval successfully in the literature. In the experiments, we prepared co-occurrence matrices of 100
food items using various kinds of data sources including Web texts, Web food blogs and our own food
database, and evaluated the ﬁnal results obtained byapplying manifold ranking. As results, it has been
proved that co-occurrence statistics obtained from afood photo database is very helpful to improve theclassiﬁcation rate within the top ten candidates.

1

Figure 1. Examples of multiple-food photos.
search of scene recognition the targets of which usually contain multiple objects, relations between objects were considered as important cue for scene
recognition in some works [8, 9]. Inspired by theseworks, we introduce relation information between
food items for recognizing multiple-food meal photos. As relations which have been used in object
recognition research so far, co-occurrence [8] andrelative location [9] are common. In case of meal
photos, we think co-occurrence relation is more important than relative locations, because some combinations of foods such as “hamburger and frenchfries” are very common, while the way to place fooditems on the table is not strictly restricted in general.
In this paper, we propose a method to recognize multiple-food meal photos considering cooccurrence statistics by extending our previous
work [4]. To do that, we use manifold ranking [10]which have been used as a method for relevance
feedback of image retrieval [11]. Manifold ranking is a ranking method to consider similarities
between items. In the image retrieval research,
manifold ranking was used to rank images considering both user preference and visual similarities between images. In this paper, we use the
manifold ranking method to rank food candidates
considering both recognition results by our previous method [4] and co-occurrence statistics between
food items. That is, the method proposed in thispaper re-ranks the original candidate ranking obtained by the multiple-food recognition method,which does not consider co-occurrence statistics, by
taking into account co-occurrence statistics. To ourbest knowledge, this is the ﬁrst work to recognize
multiple-food items in one meal image taking into
account co-occurrence statistics.
The rest of this paper is organized as follows:
Section 2 explains the proposed method, and Sec-

Introduction

Recently, personal services to recode people’s
food habits by taking meal photos with mobile
phones have become popular. Currently, labelingfood names to meal photos requires human labor,
which is a quite troublesome task. Therefore, it
is desired to make recording of food items moreeasier and quickly. To this end, several methods
to recognize food images have been proposed so
far [1, 2, 3, 4].
Most of the existing works assumed that one
meal image contained only one food item. They
cannot handle a meal photo which contains two or
more food items such as a hamburger-and-frenchfries image. Then, in [4], we proposed a new
method for recognizing meal photos which contain two or more food items as shown in Figure
1. In our proposed method, ﬁrstly, we detect candidate regions with several methods including Felzenszwalb’s deformable part model (DPM) [5], a circle detector and the JSEG region segmentation[6]. Then, we extract various kinds of image features from each candidate region. After applying
the classiﬁcation models trained by multiple kernel
learning [7], we obtain the names of the top N food
item candidates over the given image.
In the proposed method [4], each food item isrecognized independently. Meanwhile, in the re-978-4-9906441-1-6 ©2012 IAPR

2017

tion 3 describes the experimental results. Finallyin Section 4 we conclude this paper.

2

photo. Then, we select the maximum SVM output score over all the candidate region regarding
each food category. Finally, we obtain the evaluation scores which express likelihood that the corresponding food items appear in the given photo. Asa system output, we obtain the names of the top
N food items over the given image regarding thedescending order of the evaluation score.

Proposed Method

The proposed method reﬁne the results obtainedby our previous method [4] with manifold ranking [10]. Before explaining the propose method, we
describe the previous method to detect food items
for multiple-food photos as shown in Figure 1.

2.2

Candidate Region Detection
DPM

CircleCoding Image Feature Vector
Color

SIFT

CSIFT

HOG

Classification
MKL-SVM

JSEG

Integration of Candidate Region

Ranking of 100 Foods
Re-Ranking by Manifold Ranking

Gabor

ResultFigure 2. Recognition Flow

2.1

with

Co-

Some common combinations of food items exist such as “hamburger and french-fries” and “riceand miso-soup”, while unlikely combinations exist such as “sushi and hamburger” or “sashimi
and french-fries”. From these observation, cooccurrence statistics is expected to be able toenhance the performance of multiple-food image
recognition. By considering co-occurrence statistics, we can reduce unlikely combinations and boost
possible combinations which are included in thehigher ranked candidates. As results, obtained
ranking of food item candidates become more precise.
For multi-food recognition with co-occurrence
statistics, we use manifold ranking [10], which is a
re-ranking method to consider similarities between
items. In this paper, we propose to re-rank thefood candidate ranking obtained by our previous
work [4] with the manifold ranking method.
The equation of the manifold ranking is as follows:(1)r ∗ = (I − αS)−1 r,

Query Image

Whole

Manifold
Rankingoccurrence Statistics

Previous Method

In [4], we proposed a food image recognition system which outputs the names of food items that are
expected to be shown in a given meal photo. We
show the overview of the processing ﬂow of the proposed system including the co-occurrence extension
proposed in this paper in Figure 2. Given an input
image, ﬁrst, the system detects candidate regionsof dishes. We use four types of detectors including the deformable part model (DPM) [5], a circle
detector, the JSEG region segmentation [6], andwhole image. Next, we integrate bounding boxesof the candidate regions detected by the four methods. Then, we check the aspect ratio of width andheight of the bounding boxes, and exclude irrelevant bounding boxes regarding their shapes fromthe candidate set.
Next, the system extracts various kinds of image
features including Bag-of-Features (BoF) of SIFT
and color SIFT with spatial pyramid, Histogram ofOriented Gradients (HoG), and Gabor texture features from the selected regions, and calculate SVM
scores by multiple-kernel learning (MKL) [7] withchi-square RBF kernels for each candidate region
in the one-vs-rest manner. MKL can estimate theoptimal weights for linear combination of kernels
each of which corresponds to one kind of a visualfeature.
After that, we obtain SVM output scores for
all the candidate regions and all the food categories. Note that our objective is not associating
extracted regions with possible names of food items
directly, but listing all the names of the food items
which are estimated to be shown in the given meal

where r and r ∗ represent the initial ranking vector
and the manifold ranking vector, and I and S represent an identity matrix and a similarity matrix,
respectively. Note that α is a constant varying from0 to 1, which adjusts the eﬀect of S for the initial
vector r.
The initial ranking vector r is calculated by applying a standard sigmoid function to the SVM output value vi of each category and L1-normalized asshown in the following equation:(1 + exp(−vi ))−1,
ri = −1
j (1 + exp(−vj ))(2)

As a similarity matrix to re-rank the initial ranking, we use a co-occurrence probability matrix,
which can be calculated by counting the number ofco-occurrence of two food item pairs in a training
dataset. Each element of the co-occurrence matrix
Si,j can be obtained with
Si,j = ci,jk∈F ∧k=j ck,j

2018

,

(3)

chicken-’n’- port cutlet
egg onon ricerice

beef currysushi

chickenrice

fried rice

tempurabowl

bibimbap

toastcroissantroll breadraisinbreadchip butty

hamburgersoba
noodleramen
noodlebeef noodle

tensinnoodlefried
noodlespaghetti

Japanesestylepancaketakoyakigratinsauteedvegetables

croquette

grilledeggplant

sauteedspinachsausageodenomeletganmodoki

jiaozistew

teriyakigrilled fish

fried fish

grilledsalmonsalmonmeunieresashimigrilledpacific
saurysukiyakisweet andsour pork

tempurafriedchickensirloincutletnanbanzuke boiled fishseasoned
beef withpotatoeshambarg
steak

steak

dried fishchiliginger pork spicy
flavored
saute
tofu

yakitoricabbagerollomeletegg
sunny-side
upcold tofu

egg roll

chillednoodlestir-fried
beef andpeppers

simmered
pork

boiledchickenandvegetables

sashimibowl

sushi bowl

fish-shaped
pancakewith beanjamshrimpwith chillsourceroast
chickensteamedmeat
dumpling

omeletwith friedricecutletcurry

spaghetti
meat sauce

friedshrimpgreen saladmacaroni
saladJapanese
tofu andvegetable
chowderpork miso
soupchinesesoup

beef bowl

rice ball

pizza toast

dipping
noodles

french fries

goyachanpuru

kinpirastylesauteedburdock

hot dog

mixed rice

riceeels on rice

pilaf

pizza

sandwiches

udon
noodletempuraudon

vegetable
tempuramiso soup

potagelightly
roastedfish

steamedegg
hotchpotch

natto

potatosalad

Figure 3. 100 food categories in ourdataset. Please see this ﬁgure on a PDF
viewer with magniﬁcation.
1

3

Experiments

In the experiments, we used our own food image dataset built for [4] as shown in Figure 3 which
includes 100 Japanese food categories with bounding boxes on each food item. It contains about one
hundred images for each category and 9132 images
totally. For the experiments, we selected 500 multiple food-item images from them which contain 1178
food items for test, and used the rest of all the images for training.
To evaluate the performance, we use a classiﬁcation rate within the top N candidates CR@N regarding food items, which is deﬁned in the following
equation:where ci,j represents the number of co-occurrence
num. of correctly-detected food items in top N
pairs of food category i and j over whole the train- CR@N = num. of all the food items in all the test image
ing dataset, and F represents a set of all the predeﬁned food categories. Note that all the diagonal
If the top N candidates include the names of theelements of S are deﬁned as 0.
food items appearing in the given food image, we
count them as the correctly-detected food items.
In addition to a food image database, we also
use World Wide Web as knowledge source to calFirstly, we compare the results before and after
culate co-occurrence matrix S. In [8], in adapplying manifold ranking using the co-occurrence
dition to co-occurrence statistics extracted frommatrix built from the food image dataset. Figure 4
image database, they proposed constructing coshows the classiﬁcation rates within the top N canoccurrence matrix with Google Web Search. Foldidates. In this ﬁgure, “Baseline” represents thelowing that, we construct co-occurrence matrix usinitial results obtained by the previous method [4],ing Web data. Note that Rabinovich et al. usedwhile “DB” represents the reﬁned results by manconditional random ﬁeld (CRF) [8], while we use
ifold ranking. Regarding classiﬁcation rate within
the manifold ranking method to re-rank the initial
top 10 (CR@10), “DB” outperformed “Baseline”
output.
by 7.67 points, which shows the eﬀectiveness of the
proposed co-occurrence-based reﬁnement method.
As a method to estimate co-occurrence of two
Figure 4 shows co-occurrence frequency among thewords from the Web, we use Normalized Google
top 15 frequent foods in our database. “Red boxes”
Distance (NGD), which is calculated based on themeans more high frequent co-occurrence pairs.
hit number of Web search results. NGD between
word x and word y is given by
In the previous experiment, we set the constantvalue α in Eq.1 as 0.1. This value was decided
by the experimental results varying α from 0.0 tomax(log f (x), log f (y)) − log f (x, y)
0.9 as shown in Figure 6. α is a constant adjust,
N GD(x, y) =log M − min(log f (x), log f (y))
ing how extent co-occurrence is taken into account(4)
in the manifold ranking computation. When α iswhere M is the total number of Web pages over
0.1, CR@10 achieved the maximum value, 63.50%.
whole the Web, and f (x) and f (y) represent theThat is why we set α as 0.1.
number of hit pages for the search word x and y,
Next, we carried out manifold ranking with corespectively. Then, a co-occurrence matrix can be
occurrence matrix obtained by the knowledge on
computed as follows:
the Web as well. As explained in the previous section, we used Normalized Google Distance which
can be calculated based on the hit numbers of givenexp(−N GD(Namei , Namej ),Si,j = words returned by Google Search. We built threeexp(−NGD(Name
,
Name
))
k
j
k∈F ∧k=jkinds of NGD-based co-occurrence matrices: the(5)ﬁrst one is computed using Google Search, the secwhere “Namei ” and “Namej ” means the name text
ond one is obtained using Google Image Search inof food category i and j, respectively.
stead of normal Google Search, and the third one
As a system output, we obtain the names of theis calculated using Google Search the search tartop N food items over the given image regarding
get sites of which were restricted to blogs mainly
the descending order of the elements of the manifold
related to foods. Figure 7 shows the results by
ranking vector r ∗ .
three kinds of matrices as well as baseline, and2019

Table 1. Classiﬁcation rate within the top
ten candidate (CR@10).
Baseline DB NGD(text) NGD(img) NGD(blog)
55.84 63.50
56.27
56.71
56.10
gain ⇒ +7.67
+0.43
+0.87
+0.26

Figure 6. Classiﬁcation rate varying thevalue of α

Figure 4. Classiﬁcation rate before and after applying manifold ranking(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15)rice (1)beef curry (2)Figure 7. Classiﬁcation rate with Web resources

tempura bowl (3)hamburger (4)miso soup (5)jiaozi (6)fried chicken (7)egg sunny-sidee up (8)external knowledge such as Web texts more deeply,
and compare other methods than manifold ranking
to consider co-occurrence statistics for multiple object recognition such as conditional random ﬁelds(CRF).

natto (9)cold tofu (10)green salad (11)Japanese chowder (12)beef bowl (13)french fries (14)
mixed rice (15)Figure 5. Co-occurrence matrix extracted
from our food photo database

References[1] T. Joutou and K. Yanai, “A food image recognition systemwith multiple kernel learning,” in ICIP, pp. 285–288, 2009.
[2] S. Yang, M. Chen, D. Pomerleau, and R. Sukthankar,“Food recognition using statistics of pairwise local features,” in CVPR, 2010.
[3] Z. Zong, D.T. Nguyen, P. Ogunbona, and W. Li, “On the
combination of local texture and global structure for foodclassiﬁcation,” in ISM, pp. 204–211, 2010.
[4] Y. Matsuda and K. Yanai, “Recognition of multiple-foodimages by detecting candidate regions,” in ICME, 2012.
[5] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and
D. Ramanan, “Object detection with discriminatively
trained part-based models,” PAMI, vol. 32, no. 9, pp.
1627–1645, 2010.
[6] Y. Deng and B. S. Manjunath, “Unsupervised segmentation of color-texture regions in images and video,” PAMI,
vol. 23, no. 8, pp. 800–810, 2001.
[7] S. Sonnenburg, G. Rätsch, C. Schäfer, and B. Schölkopf,“Large Scale Multiple Kernel Learning,” JMLR, vol. 7,
pp. 1531–1565, 2006.
[8] A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora,and S. Belongie, “Objects in context,” in ICCV, 2007.
[9] E. B. Sudderth, A. Torralba, W. T. Freeman, and A. S.
Willsky, “Learning hierarchical models of scenes, objects,
and parts,” in ICCV, pp. 1331–1338, 2005.
[10] D. Zhou, J. Weston, A. Gretton, O. Bousquet, and
B. Schölkopf, “Ranking on data manifolds,” NIPS, vol.
16, pp. 169–176, 2004.
[11] J. He, M. Li, H.J. Zhang, H. Tong, and C. Zhang,“Manifold-ranking based image retrieval,” in ACM MM,
pp. 9–16, 2004.

Table 1 shows the value of CR@10 of them. Although Google Search with food blogs achieved thebest results among the Web-based methods, its improvement was not so much. Overall, Web-based
co-occurrence was not eﬀective in this experiment.

4

Conclusions

In this paper, we proposed a method to detect
multiple food items from one food image considering co-occurrence statistics with the manifold ranking method. In the experiments, we could improved the result from 55.85% to 65.62% regarding the classiﬁcation rate within top ten using cooccurrence statistics estimated from a food image
dataset. For these results, it has been shown thatthe proposed method is very eﬀective. On the other
hand, Web-based co-occurrence does not improve
the initial results so much.
For future work, we need to examine how to use

2020