Learning Food Image Similarity for Food Image Retrieval
Wataru Shimoda Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi Tokyo 182-8585 JAPAN
Email: {shimoda-k,yanai}@mm.inf.uec.ac.jp
Abstract—For food application, recipe retrieval is an important task. However, many of them rely on only text query. Food
image retrieval has relation to recipe retrieval so that similarfood images are expected that they have similar recipes. Rising
image retrieval performance is desired for recipe retrieval.
On the other hand, to learn similarity by Siamese Networkor Triplet Network are known as an effective method forimage retrieval. However, there are no research for food image
retrieval using similarity learning with Convolutional Neural
Network as far as we know. Food recognition is known as
one of ﬁne-grained recognition tasks. Therefore it is unclearthat how effective similarity learning methods based on CNN
in food images. In our work, we trained some networks forfeature similarity, and evaluated their effectiveness in foodimage retrieval.
Keywords-deep learning, Siamese network, triplet network,
food image recognition

I. I NTRODUCTION
Food image recognition is one of the promising applications of visual object recognition, since it will help estimate
food calories and analyze people s eating habits for healthcare. Therefore, many works have been published so far [1]–[7]. To make food recognition more practical, increase of thenumber of recognizable food is crucial.
Meanwhile, recently the effectiveness of Convolutional
Neural Network (CNN) have been proved for large-scale
object recognition at ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) 2012. Krizhevsky et al. [8] won
ILSVRC2012 with a large margin to all the other teams
who employed a conventional hand-crafted feature approach.
Regrading food image recognition, the classiﬁcation accuracy on the UEC-FOOD100 dataset [4] was improved from
59.6% [3] to 72.26% [9] by replacing Fisher Vector and liner
SVM with DCNN.
In recent years, CNN have also achieved large progressin wide image recognition tasks. it was shown CNN is alsoeffective. Especially image retrieval methods which deal animage as query and search images by similarity from a query
image is receiving a lot of attention. In general, the similarity
is calculated by distance of image features. In this image
retrieval task, how generate features which are speciﬁc on
visual appearance plays an important role.

Siamese Network [10] and Triplet Network [11] are methods to learn feature similarity by optimizing feature distanceof image pairs or triplet. These learning methods provide
better features in similarity. Especially Triplet Network isknown as the state-of-the-art methods on image retrieval.
These network was used in some application ﬁelds such likeface authentication and clothes recognition and so on [12],[13].
For food application, recipe retrieval is importantparadigm. But many of them rely on only inputting text.
Food image retrieval has relation to recipe retrieval so thatsimilar food images are expected that they have similar
recipes. Rising image retrieval performance is desired forrecipe retrieval. However, there are no research for foodimage retrieval using similarity learning such like Siamese
Network and Triplet Network as far as we know. Foodrecognition is known as one of the ﬁne-grained recognition
task. Therefore it is unclear that how effective similarity
learning methods based on CNN in food images. In ourwork, we trained some network for feature similarity andevaluated the potential in image retrieval task.
II. M ETHOD
Our purpose is to obtain good features for image retrieval.
In this paper, we mainly tested three type networks asfollowing:• Simple Classiﬁcation Network• Siamese Network• Triplet Network
For simply, these network construction is based on Alex Net.
We illustrated the three type networks in Fig 1.
A. Similarity Learning Network
In this section we mention about Siamese Network andTriplet Network. These network goals are to learn image
similarity models. We deﬁne the similarity of two images
P and Q according to their Euclidean distance in the image
embedding space:
D (f (P ), f (Q)) = ||f (P ), f (Q)||22(1)where f (.) is the image embedding function that maps
an image to a point in an Euclidean space, andD(., .) is2) Triplet loss: Triplet Network is proposed by Wanget al. [11] in 2014. This network employs input as triplet
images and learns similarity in triplet images. As function
f (.) Triplet Network extracts features from each triplet
images by CNN which weights are shared. In triplet images
we regards a base image as query and other one as a positive
image and rest as a negative image. Suppose we have a setof images P and pi ∈ P, and r(i,j) = r(pi , pj ) is a pairwise
relevance score which states how similar the image pi ∈ Pand pj ∈ P are. We can deﬁne the following loss for a−
triplet:ti = (pi , p+i , pi ):
−+−l(pi , p+i , pi ) = max 0, D f (pi ), f (pi ) − D f (pi ), f (pi−+−∀pi , p+i , pi such that r(pi , pi ) > r(pi , pi )(3)

Figure 1. illustration for three types of networks. 1st network is general
simple network for classiﬁcation. 2nd network is Siamese Network. 3rd
network is Triplet Network. Green means to minimize distance of features,
while red means maximize distance of features.

Positive images should be same category, while negative
images are separated to two patterns. The ﬁrst pattern isdifferent categories from query image category pi . The
second type is in-class negative samples, which are thenegative samples that are in the same category as pi but
is less relevant to p+
i . In order to ensure robust ordering+−−
between pi and pi in a triplet ti = (pi , p+i , pi ), we
also require that the margin between the relevance score?
r(pi , p+i ) and r(pi , pi ) should be larger than the margin C
as following:−r(pi , p+i ) − r(pi , pi ) ≥ C

the Euclidean distance in this space. The distance is smallermeans image P and Q are more similar. These network solve
this minimization problem by optimizing CNN parameters
using Stochastic Gradient Decent (SGD).
1) Siamese Network: Siamese Network is proposed byBromley et al. [10] in 1994. This network employs input
as pair images and learns similarity in pair images. As
function f (.) Siamese Network extracts features from eachpair images by CNN which weights are shared. When pairimages are similar this network minimizes Euclid distancein pair images and when pair images aren’t similar thisnetwork maximize Euclid distance in pair images after
L2normalization. Suppose we have a set of images P andpi ∈ P. We can deﬁne the following loss function for a pairwith label Y :si = (p1i , p2i , Yi ):l(p1i , p2i , Yi ) = Yi · D f (p1i ), f (p2i +(2)(1 − Yi )max(0, C − D f (p1i ), f (p2i ) ,∀p1i , p2i , Yi ∈ {0, 1}

where C is a margin value and Y is a label for pair images
are similar or not. If images are similar Y is 1, while if
images are not similar Y is 0.

−∀ti = (pi , p+i , pi )(4)

B. Combine Classiﬁcation Loss Term
For image retrieval with a query image, results sometimes
include similar images to a query but different from query
image class. In recipe retrieval which is one of the purposeof food image retrieval, since recipes of different category
food are different, food category of retrieved images should
be same to query image category. To solve this problem,
we implement multitask network by combining similarity
learning loss and classiﬁcation loss. We illustrate our multitask network in Fig 2.
We suppose similarity loss term is Ls and classiﬁcation
loss term is Lcl . Over all loss L is expressed as following:
L = Ls + λ · Lcl(5)

λ is a parameter. In this paper, we simply set λ to 1.
III. E XPERIMENT
A. Dataset
In the experiments, we use the UEC-FOOD256
dataset [4]. UEC-FOOD256 dataset [4] consist of 256 classfood category and each category have round 100 images.

Multi Task Siamese Network with Fine-tunedmodel (MT-SN-FT)• Multi Task Triplet Network with Fine-tuned model(MTTN-FT)”Pre-trained model” is a trained CNN by ImageNet. Thisis a most simplest baseline. ”with Pre-trained model” means
parameters are initialized by pre-trained model. ”Finetuned model” is a trained CNN by UECFOOD256 andparameters are initialized by pre-trained model. ”with Finetuned model” means parameters are initialized by ﬁne-tuned
model. ”Multi Task” means containing classiﬁcation loss.
•Figure 2.
loss.

Illustration for combination of classiﬁcation loss and similarity

For testing, we use ”val 0” which is one of ﬁve validations
split list of UEC-FOOD256 dataset [4]. Of course weexclude validation images from training images.
B. Implementation DetailWe used the open source Chainer package to train ourmodels. The initial learning rate is set to 0.001, and minibatch size is 64. We multiply the learning rate by 0.1 atevery 10000 iterations.
We used Alex Network as base Network models. In ﬁnetuning we initialize parameters excepting of last layer by
publicly available pre-trained models.
C. Evaluation metricWe evaluate food image retrieval accuracy by Average
Precision which is commonly used in retrieval evaluation.
We deﬁne a correct image that is same class from a query
image. In below tables, Top K mAP means mean of 256
classes results of Average Precision(mAP) value for ranked
top K images.
D. Baseline and Options
For comparing we have three network options ”Simple
Network”,”Siamese Network” and ”Triplet Network” andsome branches for taking some learning metrics. I list ouroptions as below:Pre-trained model (PT)
Siamese Network with Pre-trained model (SN-PT)• Triplet Network with Pre-trained model (TN-PT)• Fine-tuned model (FT)• Siamese Network with Fine-tuned model (SN-FT)• Triplet Network with Fine-tuned model (TN-FT)• Multi
Task Siamese Network with Pre-trainedmodel (MT-SN-PT)• Multi
Task Triplet Network with Pre-trainedmodel (MT-TN-PT)••

E. Evaluation of Food Image Retrieval
First of all, we compare three types of network. For
simply, we pick up only three patterns which are initialized
by pre-trained model and we used last fully connectedlayer as features. The performance comparison is shownin Table I. Superiority for food image retrieval of three
types of network is clear. Siamese Network performance isbetter than a pre-trained model result, while Triplet Network
more over Siamese Network performance with large margin.
This results show similarity learning is effective but Triplet
Network is more powerful than Siamese Network.
In the next place, on the contrary to the previous comparison, we evaluate combination of similarity learningand classiﬁcation factor. Table II shows the results for thecomparison. Fine-tuned model achieved high mean averageprecision at food image retrieval even though being trainedfor classiﬁcation. On the other hand, we used ﬁne-tuned
model as initialization parameters. For example, at methodof ”SN-FT”, we initialized Siamese Network parameters.
However, the result is lower than ”FT” score. This meansthat even though start better parameters, similarity learning
hasn’t guarantee for getting better results. We also evaluate classiﬁcation factor by combining losses. As a results,
combining classiﬁcation loss and similarity loss is very
effective. ”MT-TN-FT” achieved a best result. AlthoughmAP of ”TN-PT” is already higher score than ”FT”, multitask loss showed further improvement. Classiﬁcation factor
is important and classiﬁcation task and similarity learning
can be well combined. For more detail, initialization with
ﬁne-tuned model showed better results than initialization
with pre-trained model even though combining classiﬁcation
loss and similarity loss. It is expected that training withcascade steps will be better. We show examples in Fig. 3.
Table IC OMPARISON OF THREE NETWORKS RESULTS
methodTop 10 mAPTop 20 mAPPT9.79.7

SN-PT
11.3
12.5

TN-PT
25.7
23.3

Table II
E VALUATION OF EFFECTIVENESS CLASSIFICATION FACTOR
methodTop 10 mAPTop 20 mAP

FT
24.8
22.6

SN-FT
18.0
16.8

MT-SN-PT
23.3
21.4

MT-SN-FT
26.4
24.1

TN-FT
27.1
24.4

MT-TN-PT
27.8
25.0

MT-TN-FT
31.7
28.8[3] Y. Kawano and K. Yanai, “Foodcam: A real-time foodrecognition system on a smartphone,” Multimedia Tools andApplications, pp. 1–25, 2014.
[4] Y. Matsuda, H. Hoashi, and K. Yanai, “Recognition ofmultiple-food images by detecting candidate regions,” inProc. of IEEE International Conference on Multimedia andExpo, 2012, pp. 1554–1564.
[5] M. Chen, Y. Yang, C. Ho, S. Wang, S. Liu, E. Chang, C. Yeh,and M. Ouhyoung, “Automatic chinese food identiﬁcation andquantity estimation,” in SIGGRAPH Asia, 2012.
[6] M. Bosch, F. Zhu, N. Khanna, C. J. Boushey, and E. J. Delp,“Combining global and local features for food identiﬁcationin dietary assessment,” in Proc. of IEEE International
Conference on Image Processing, 2011.
[7] S. Yang, M. Chen, D. Pomerleau, and R. Sukthankar, “Foodrecognition using statistics of pairwise local features,” inProc. of IEEE Computer Vision and Pattern Recognition,
2010.
[8] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet
classiﬁcation with deep convolutional neural networks.,” inAdvances in Neural Information Processing Systems, 2012.

Figure 3. Image retrieval result examples. (a) PT. (b) SN-PT. (c) TN-PT.
(d) FT. (e) SN-FT. (f) MT-SN-PT. (g) MT-SN-FT. (h) TN-FT. (i) MT-TNPT. (j) MT-TN-FT.

IV. C ONCLUSIONSIn this paper, we analyze effectiveness of similarity learning for food image retrieval. We tested three types of CNN
and it was turned out that Triplet Network was the mostpowerful network compared to others. We also showed
the performance of Triplet Network can be improved bycombining classiﬁcation task.
R EFERENCES[1] L. Bossard, M. Guillaumin, and L. V. Gool, “Food-101 mining discriminative components with random forests,” inProc. of European Conference on Computer Vision, 2014.
[2] H. Kagaya, K. Aizawa, and M. Ogawa, “Food detection andrecognition using convolutional neural network,” in Proc. ofACM International Conference Multimedia, 2014, pp. 1085–1088.

[9] Y. Kawano and K. Yanai, “Food image recognition with deep
convolutional features,” in Proc. of ACM UbiComp Workshop
on Workshop on Smart Technology for Cooking and Eating
Activities (CEA), 2014.
[10] J. Bromley, I. Guyon, E. Sckinger, and R Shah, “Signature
veriﬁcation using a siamese time delay neural network,”in Advances in Neural Information Processing Systems, 1994.
[11] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin,
B. Chen, and Y. Wu, “Learning ﬁne-grained image similarity
with deep ranking,” in Proc. of IEEE Computer Vision andPattern Recognition, 2014, pp. 1386–1393.
[12] Junlin. H, Jiwen. L, and Yap-Peng. T, “Discriminative deep
metric learning for face veriﬁcation in the wild,” in Proc. ofIEEE Computer Vision and Pattern Recognition, 2014.
[13] Qian. Y, Feng. L, Yi. S, Tao. X, Timothy. H, and Chen. L,“Sketch me that shoe,” in Proc. of IEEE Computer Vision
and Pattern Recognition, 2016.