Food Image Recognition with Deep
Convolutional Features
Yoshiyuki KAWANO
Department of Informatics
The University ofElectro-Communications,Tokyo
1-5-1 Chofugaoka, Chofu-shi,
Tokyo, 182-8585 Japankawano-y@mm.inf.uec.ac.jp

Keiji YANAI
Department of Informatics
The University ofElectro-Communications,Tokyo
1-5-1 Chofugaoka, Chofu-shi,
Tokyo, 182-8585 Japanyanai@mm.inf.uec.ac.jp

Abstract
In this paper, we report the feature obtained from theDeep Convolutional Neural Network boosts foodrecognition accuracy greatly by integrating it withconventional hand-crafted image features, Fisher Vectors
with HoG and Color patches. In the experiments, we haveachieved 72.26% as the top-1 accuracy and 92.00% as thetop-5 accuracy for the 100-class food dataset,
UEC-FOOD100, which outperforms the best classification
accuracy of this dataset reported so far, 59.6%, greatly.

Author Keywords
food recognition, Deep Convolutional Neural Network,
Fisher Vector

Introduction

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than the author(s) must be honored. Abstractingwith credit is permitted. To copy otherwise, or republish, to post on servers or
to redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from Permissions@acm.org.
UbiComp’14 , September 13–17 2014, Seattle, WA, USA.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3047-3/14/09 $15.00.
http://dx.doi.org/10.1145/2638728.2641339

Food image recognition is one of the promising
applications of object recognition technology, since it will
help estimate food calories and analyze people’s eating
habits for healthcare. Therefore, many works have been
published so far [2, 4, 7, 9, 11]. To make food recognition
more practical, increase of the number of recognizable
food is crucial. In [7,9], we created 100-class food dataset,
UEC-FOOD100, and made experiments with 100-classfood classification. The classification accuracy reported sofar was 59.6% [7], which was not enough for practical use.

Meanwhile, recently the effectiveness of Deep
Convolutional Neural Network (DCNN) have been proved
for large-scale object recognition at ImageNet Large-Scale
Visual Recognition Challenge (ILSVRC) 2012. Krizhevsky
et al. [8] won ILSVRC2012 with a large margin to all theother teams who employed a conventional hand-craftedfeature approach. In the DCNN approach, an input dataof DCNN is a resized image, and the output is a class-label
probability. That is, DCNN includes all the object
recognition steps such as local feature extraction, featurecoding, and learning. In general, the advantage of DCNN
is that it can estimate optimal feature representations fordatasets adaptively [8], the characteristics of which theconventional hand-crafted feature approach do not have.
In the conventional approach, we extract local features
such as SIFT and HoG first, and then code them into
bag-of-feature or Fisher Vector representations.
However, DCNN is not always applicable for any kinds ofdatasets, because it requires a lots of training images toachieve comparable or better performance to theconventional local-feature-based methods. In ourpreliminary experiments on DCNN-based food recognition
where we trained DCNN with the UEC-FOOD100 dataset,
we failed to confirm that the DCNN-based method
outperformed the conventional method. This is mainly
because the amount of training data is not enough. We
had only 100 images per food category, while ILSVRC
dataset has 1000 images per category. In general, DCNN
does not work well for a small-scale dataset, while DCNN
works surprisingly well for a large-scale dataset [6]. Then,
as a method to utilize DCNN for a small-scale dataset,using a pre-trained DCNN with a large-scale ILSVRC
dataset as a feature vector extractor has been
proposed [3]. DCNN features can be easily extracted fromthe output signals of the layer just before the last one ofthe pre-trained DCNN. Chatfield et al. madecomprehensive experiments employing both DCNN
features and conventional features such as SIFT andFisher Vectors on PASCAL VOC 2007 andCaltech-101/256 which can be regarded as small-scale
datasets where they have only about one hundred or lessimages per class [3]. They showed that the DCNN-feature
was effective for a small-scale dataset, and they achieved
the best performance for PASCAL VOC 2007 andCaltech-101/256 by combining DCNN features and FisherVectors.
Regarding food datasets, the effectiveness of the DCNN
features is still unclear, because food datasets are a kindof fine-grained datasets which is different from generic
datasets such as PASCAL VOC 2007 andCaltech-101/256. In food datasets, images belonging todifferent categories sometimes look very similar to eachother. Food image recognition is regarded as the moredifficult task than image recognition of generic categories.
Then, in this paper, we apply DCNN features for 100-classfood dataset and examine the effectiveness of DCNN
features for food photos by following Chatfield et al.’swork [3].

Methods
DCNN Features
Recently, it has been proved that Deep Convolutional
Neural Network (DCNN) is very effective for large-scale
object recognition. However, it needs a lot of training
images. In fact, one of the reasons why DCNN won theImageNet Large-Scale Visual Recognition Challenge(ILSVRC) 2012 is that the ILSVRC dataset contains one
thousand training images per category [8]. This situationdoes not fit food datasets most of which have only about
one hundred images a food category. Then, to make thebest use of DCNN for food recognition, we use thepre-trained DCNN with the ILSVRC 1000-class dataset asa feature extractor.

obtain a 32768-dim RootHOG FV and a 24576-dim Color
FV for each image. This setting is almost the same as [7]except for the number of spatial pyramid levels.

Following [3], we extract the network signals just before
the last layer of the pre-trained DCNN as a DCNN featurevector. Since we used the same network structure
proposed by Krizhevsky et al. [8], the number of elementsin the last layer is the same as the number of the classes,
1000, and the number of elements in the layer just before
the last one is 4096. Therefore, we obtain a 4096-dim
DCNN feature vector for a food image. As
implementation of DCNN, we used OverFeat 1 .

Classifiers
We use one-vs-rest linear classifiers for 100-class foodclassification. For integrating both DCNN features andconventional features, we adopt late fusion with noweighting. For lower-dimensional DCNN features, we usea standard linear SVM, while for higher-dimensional FVfeatures, we use an online learning method, AROW [5].
As their implementations, we use LIBLINEAR 2 andAROWPP 3 .

Conventional Features
As conventional features, we extract RootHoG patches
and color patches, and code them into Fisher Vector (FV)
representation with Spatial Pyramid with three levels(1x1+3x1+2x2). Fisher Vector is known as astate-of-the-art coding method [10].

Experiments

RootHoG is an element-wise square root of the L1
normalized HOG, which is inspired by “RootSIFT” [1].
The HOG we use consists of 2 × 2 blocks (totally fourblocks). We extract gradient histogram regarding eightorientations from each block. The total dimension of a
HOG Patch feature is 32. After extraction of HOG
patches, we convert each of them into a “RootHOG”.
As color patches, we extract mean and variance values ofRGB value of pixels from each of 2 × 2 blocks. Totally, weextract 24-dim Color Patch features.
After extracting RootHoG patches and color patches, weapply PCA and code them into Fisher Vectors (FV) with
the GMM consisting of 64 Gaussians. As results, weAs a food dataset for the experiments, we use theUEC-FOOD100 dataset [7, 9] which is an open 100-classfood image dataset 4 . Part of the food categories in theUEC-FOOD100 dataset is shown in Fig. 1. It includes
more than 100 images for each category and boundingbox information which indicates food location within eachfood photo. We extract features from the regions inside
the given bounding boxes following [7]. We evaluate theclassification accuracy within the top N candidates
employing 5-fold cross validation.
Figure 2 shows classification accuracy within the top-N
candidates with each of single features, RootHOG FV,Color FV and DCNN, the combination of RootHoG andColor FV, and the combination of all the three features.

2 http://www.csie.ntu.edu.tw/∼cjlin/liblinear/
3 https://code.google.com/p/arowpp/

1 http://cilvr.nyu.edu/doku.php?id=software:overfeat:start

4 http://foodcam.mobi/dataset/riceeels on rice

pilaf

chicken-’n’-egg on
rice

pork cutlet on rice

beef currysushi

chicken rice

fried rice

tempura bowl

bibimbap

toastcroissantroll breadraisin breadchip butty

hamburger

pizza

sandwiches

udon noodle

tempura udon

soba noodle

ramen noodle

beef noodle

tensin noodle

fried noodle

spaghetti

Japanese-stylepancaketakoyakigratinsauteed vegetables

croquette

grilled eggplantsauteed spinach

vegetable tempura

miso soup

potagesausageodenomeletganmodoki

jiaozistew

teriyaki grilled fish

fried fish

grilled salmon

salmon meunieresashimigrilled pacific saury

sukiyakisteamed egg
hotchpotch

tempurafried chicken

sirloin cutlet

nanbanzuke

boiled fishseasoned beef withpotatoes

hambarg steak

ginger pork saute

spicy chili-flavoredtofu

yakitoricabbage rollomelet

egg sunny-side up

natto

cold tofu

sweet and sour pork lightly roasted fish

steak

dried fish

Figure 1: 70 kinds of foods in the UEC-FOOD100 dataset.

Among the three single features, DCNN, RootHoG-FV,and Color-FV, the DCNN feature achieved the best
performance, 57.87%, in the top-1 accuracy, while
RootHoG-FV and Color-FV achieved 50.14% and 53.04%,
respectively. Although the combination of both FVsachieved 65.32% which was better than single DCNNfeatures, the total dimension of the FV combination was57,344, which 14 times as larger as the dimension ofDCNN features.

1Classification Rate

0.90.80.7

0.60.5

Color FV
RootHOG FVDCNNFVFV + DCNN

0.4
0.3

0.2
0.1
012345678910# of candidates
Figure 2: Classification accuracy within the top N candidate
on UEC-FOOD100 with DCNN, RootHoG-FV, Color-FV andtheir combinations.

The combination of all the three features achieved
72.26% in the top-1 accuracy and 92.00% in the top-5
accuracy, which were the best performance for theUEC-FOOD100 dataset, while the previous best was59.5% [7]. This indicates that DCNN features has
different characteristics from the conventional local
features and Fisher Vectors, and integration of them isimportant to achieve better performance rather than useof single ones. This is a very promising result for practicaluse of food image recognition technology.

Conclusions
In this work, we proposed introducing DCNN features
which are extracted from the pre-trained DCNN with theILSVRC 1000-class dataset into food photo recognition.
In the experimental results, we have achieved the bestclassification accuracy, 72.26%, for the UEC-FOOD100
dataset, which proved that that DCNN features canboosted the classification performance by integrating itwith the conventional features.
For future work, we will implement the proposed
framework on mobile devices. To do that, it is needed toreduce the amount of the pre-trained DCNN parameterswhich consist of about 60 million floating values.

References[1] Arandjelovic, R., and Zisserman, A. Three things
everyone should know to improve object retrieval. InProc. of IEEE Computer Vision and Pattern
Recognition (2012), 2911–2918.
[2] Bosch, M., Zhu, F., Khanna, N., Boushey, C. J., andDelp, E. J. Combining global and local features forfood identification in dietary assessment. In Proc. ofIEEE International Conference on Image Processing(2011).
[3] Chatfield, K., Simonyan, K., Vedaldi, A., andZisserman, A. Return of the devil in the details:
Delving deep into convolutional nets. arXiv preprint
arXiv:1405.3531 (2014).

[4] Chen, M., Yang, Y., Ho, C., Wang, S., Liu, S.,Chang, E., Yeh, C., and Ouhyoung, M. Automaticchinese food identification and quantity estimation.
In SIGGRAPH Asia 2012 Technical Briefs (2012).
[5] Crammer, K., Kulesza, A., and Dredze, M. Adaptive
regularization of weight vectors. In Advances in
Neural Information Processing Systems (2009),
414–422.
[6] Donahue, J., Jia, Y., Vinyals, O., Hoffman, J.,Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep
convolutional activation feature for generic visual
recognition. arXiv preprint arXiv:1310.1531 (2013).
[7] Kawano, Y., and Yanai, K. Foodcam: A real-time
food recognition system on a smartphone.
Multimedia Tools and Applications (2014), 1–25.
[8] Krizhevsky, A., Sutskever, I., and Hinton, G. E.
Imagenet classification with deep convolutional
neural networks. In Advances in Neural Information
Processing Systems (2012).
[9] Matsuda, Y., Hoashi, H., and Yanai, K. Recognitionof multiple-food images by detecting candidate
regions. In Proc. of IEEE International Conference on
Multimedia and Expo (2012), 1554–1564.
[10] Perronnin, F., Sánchez, J., and Mensink, T.
Improving the fisher kernel for large-scale imageclassification. In Proc. of European Conference onComputer Vision (2010).
[11] Yang, S., Chen, M., Pomerleau, D., and Sukthankar,
R. Food recognition using statistics of pairwise localfeatures. In Proc. of IEEE Computer Vision andPattern Recognition (2010).