Real-time Mobile Food Recognition System
Yoshiyuki Kawano and Keiji Yanai
The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan{kawano-y,yanai}@mm.inf.uec.ac.jp
Abstract

which is almost equivalent to a PC’s CPU released several
years ago in terms of performance. Old-style mobile systems with image recognition need to send images to highperformance servers, which must makes communication delay, requires communication costs, and the availability ofwhich depends on network conditions. In addition, in proportion to increase number of users, more computational resources of servers is also required, which makes it difficult
to recognize objects in a real-time way.
On the other hand, image recognition smartphone ismuch more promising in terms of availability, communication cost, delay, and server costs. Then, by taking advantage
of rich computational power of recent smartphones as well
as recent advanced object recognition techniques, in this paper, we propose a real-time food recognition system whichruns on a common smartphone. To do that, we adopt a linear
SVM and a fast χ2 kernel based on kernel feature maps [13]and implement a system as a multi-threaded system for using multiple CPU cores effectively.
Figure 1 shows the screenshot of the proposed system
which runs as an Android smartphone application. First auser point a smartphone camera to foods, draws a bounding box (represented in the yellow rectangular in the figure)by dragging on the screen, and then food image recognition is activated for the given bounding box. The top fivecandidates for the yellow bounding box are shown on theleft side of the screen. If a user touches one of the candidate
items, the food category name and the photo are recorded asa daily food record in the system. In addition, the proposedsystem has functions on automatic adjustment of boundingboxes based on the segmentation result by GrubCut [12],and estimation of the direction of the expected food regions
based on Efficient Sub-window Search (ESS) [5]. Since thisrecognition process is performed repeatedly about once asecond, a user can search for good position of a smartphone
camera to recognize foods accurately by moving it continuously without pushing a camera shutter button.
In the experiments, we have achieved the 81.55% classification rate for the top 5 category candidates when the
ground-truth bounding boxes are given. In addition, we
obtained positive evaluation by user study compared to thefood recording system without object recognition.
To summarize our contribution in this paper, it consistsof four folds: (1) implementing an interactive and real-time

We propose a mobile food recognition system the purposes of which are estimating calorie and nutritious offoods and recording a user’s eating habits. Since all theprocesses on image recognition performed on a smartphone, the system does not need to send images to a server
and runs on an ordinary smartphone in a real-time way.
To recognize food items, a user draws bounding boxes
by touching the screen first, and then the system starts food
item recognition within the indicated bounding boxes. To
recognize them more accurately, we segment each food item
region by GrubCut, extract a color histogram and SURFbased bag-of-features, and finally classify it into one of thefifty food categories with linear SVM and fast χ2 kernel. In
addition, the system estimates the direction of food regions
where the higher SVM output score is expected to be obtained, show it as an arrow on the screen in order to ask auser to move a smartphone camera. This recognition process is performed repeatedly about once a second. We implemented this system as an Android smartphone application so as to use multiple CPU cores effectively for real-time
recognition.
In the experiments, we have achieved the 81.55% classification rate for the top 5 category candidates when the
ground-truth bounding boxes are given. In addition, we obtained positive evaluation by user study compared to thefood recording system without object recognition.

1. Introduction
In recent years, food habit recording services for smartphones such as iPhone and Android phones have become
popular. They can awake users’ food habit problems suchas bad food balance and unhealthy food trend, which is useful for disease prevention and diet. However, most of suchservices require selecting eaten food items from hierarchicalmenus by hand, which is too time-consuming and troublesome for most of the people to continue using such services
for a long period.
Due to recent rapid progress of smartphones such as
iPhone and Android phones, they have obtained enoughcomputational power for real-time image recognition. Currently, a quad-core CPU is common as a smartphone’s CPU,1

Recognition Result

Confidence

Food Image

Suggest Direction

other hand, our system can recognize food items on a clientside in a real-time way, which requires no communication
to outside computational resources and enables user to use
it interactively. Note that our current system recognize only
50 food categories, and it requires user’s assistances to estimate food volumes by touching a slider on the system
screen.

2.2. Mobile Device and Computer Vision

Selected Food

Input Volume

Figure 1. The screenshot of the main screen of the proposed system.

food recognition and recording system running on a consumer smartphone, (2) using a liner SVM with a fast χ2 kernel for fast and accurate food recognition, (3) adjustment of
the given bounding box, and (4) estimation of the directionof the expected food region automatically.
The rest of this paper is organized as follows: Section
2 describes related work. In Section 3, we explain theoverview of the proposed system. In Section 4, we explain
the detailed method for food recognition. In Section 5 describes the experimental results and user study, and in Section 6 we conclude this paper.

2. Related Work2.1. Food Recognition
Food recognition is difficult task, since appearances offood items are various even within the same category. Thisis a kind of fine-grained image categorization problem. As
food recognition, Yang et al. [14] proposed pairwise local features which exploit positional relation of eight basic food ingredients. Matsuda et al. [9] proposed method
for recognition multiple-food images, which first detects
food put regions by several detectors, next recognizes byextracted color, texture, gradient, and SIFT using multiple
kernel learning(MKL).
As a food recording system with food recognition, Web
application FoodLog [3] estimates food balance. It divides
the food image into 300 blocks, from each blocks extracts
color and DCT coefficients, next classifies to five groups
such as staple, main dish, side dish, fruit, and non-food.
The TADA dietary assessment system [7] has food identification and quantity estimation, although it has some restriction that food must be put on white dishes and foodphotos must be taken with a checkerboard to food quantity
estimation.
In all the above-mentioned systems image recognition
processes perform on servers, which prevents systems frombeing interactive due to communication delays. On theWith the spread of smartphones, some mobile applications exploiting computer vision technique have been proposed. Google Goggles 1 is one of the most well-known
application which recognizes specific object, returns to user
about object information. Kumar et al. [4] proposed recognize 184 species application Leaf snap. For Leaf image
on the solid light-colored background segment and extract
curvature-based shape features. Maruyama et al. [8] proposed a system which extracts color feature and recognizes
30 kinds of food ingredients on a mobile device. As a mobile device application emphasizing real-time image recognition, Lee et al. [6] proposed a system which trains templates, then decomposes descriptors such as intensity andgradient orientation. At the time of testing, template matching at multiple scales enables the system to detect and track
objects in a real-time way. In the above-mentioned mobile
vision systems, local-feature-based specific object recognition was performed, while we tackle category-level object
recognition on foods on a mobile device.
As an interactive mobile vision system, Yu et al. [15]proposed Active Query Sensing (AQS) the objective of
which is localization of the current position by matchingof street photos. When the system fails in location search,
it suggests the best viewing direction for the next reference
photo to a user based on the pre-computed saliency on eachlocation. Our system is also built as an interactive system
which detects object regions based on the bounding boxes auser draws and suggests the direction of food regions where
the higher evaluation output score is expected.

3. System Overview
The final objective of the proposed system is to support users to record daily foods and check their food eating
habits. To do that easily, we built-in food image recognition
technique on the proposed system. In this paper, we mainly
describe a food image recognition part of the proposed system.
Processing flow of typical usage of the proposed system
is as follows (See Figure 2 as well):
1. A user points a smartphone camera toward food items
before eating them. The system is continuously acquiring frame images from the camera device in the
background.
2. A user draws bounding boxes over food items on thescreen. The bounding boxes will be automatically ad1 http://www.google.com/mobile/goggles/

Point a smartphone to dishes

Start recognition

Assesment
and
4-Gun

Input bounding box(a) Meal records on thescreen.

Select food items
Save food image

Register food record(b) Meal records on the
Web.

Figure 3. Meal records.

food items with the given bounding boxes, and (3) Estimation of the direction of the possible food region.

4.1. Bounding Box Adjustment

Figure 2. System process flow

justed to the food regions. More than two boundingboxes can be drawn at the same time. .
3. Food recognition is carried out for each of the regions
within the bounding boxes. At the same time, the direction of the region having the higher evaluation scoreis estimated for each bounding box.
4. As results of food recognition and direction estimation,
the top five food item candidates and the direction arrows are shown on the screen.
5. A user selects food items from the food candidate listby touching on the screen, if found. Before selecting
food items, a user can indicate relative rough volumeof selected food item by the slider on the right bottomon the screen. If not, user moves a smartphone slightly
and go back to 3.
6. The calorie and nutrition of each of the recognized
food items are shown on the screen.
In addition, a user can see his/her own meal record andits detail including calories and proteins of each food itemson the screen as shown in Figure 3(a). Meal records canbe sent to the server, and a user can see them on the Web(Figure 3(b)).

4. A Method of Food Recognition
Before starting to recognize food items for the frame images taken by a smartphone camera, first the system requiresthat a user draws bounding boxes which bounds food itemson the screen by dragging along the diagonal lines of theboxes.
In this section, we explain the following three processing: (1) Adjustment of bounding boxes, (2) Recognition ofFirst, a user draws bounding boxes roughly on the screenby dragging. The bounding boxes a user draws are not always accurate and sometimes they are too large for actual
food regions. Therefore, we apply well-known graph-cutbased segmentation algorithm GrabCut [12], and then modify the bounding boxes so as to fit them to food regions
segmented by GrabCut. GrabCut needs initial foregroundand background regions as seeds. Here, we provide GrubCut the regions within bounding boxes as foreground andthe areas out of the doubly-extended boxes of the original
bounding boxes as background. Since the computational
cost of GrabCut is relatively high in the real-time recognition system, the bounding box adjustment is performed only
once after it was drawn.

4.2. Food Recognition
Food recognition is performed for each of the window
images within the given bounding boxes. Firstly, image
features are extracted from each window, secondly feature
vectors are built based on the pre-computed codebook, andfinally we evaluate the feature vectors with the trained linear SVMs on 50 food categories. The top five candidates
which have the top five SVM scores over 50 categories andall the bounding boxes are shown on the screen as food item
candidates.

4.3. Image Features
Since we aims for implementing a mobile recognition
system which can run in the real-time way, image features tobe extracted should be minimum requisites. Then we evaluated and compared performance and computational costs ofvarious kinds of common image features including global
and local features such as color histogram, color moment,
color auto correlogram, Gabor texture feature, HoG, PHoG,
and Bag-of-SURF. Finally we chose the combination ofcolor histogram and Bag-of-SURF. To save computationalcost, if the longer side of a bounding box is more than 200,
the image extracted from the bounding box is resized so thatthe longer side becomes 200 preserving its aspect ratio.

Color Histogram: We divide an window image into 3 ×3 blocks, and extract a 64-bin RGB, color histogram fromeach block. Totally, we extract a 576-dim color histogram.
Note that we examined HSV and La*b* color histograms
as well, and RGB color histogram achieved the best amongthem.
Bag-of-SURF: As local features, we use dense-sampled
bag-of-SURF. SURF [1] is an invariant 64-dim local descriptor for scale, orientation and illumination change. We
sample points by dense sampling in scale 12 and 16 withevery 8 pixel with the window. To convert bag-of-features
vectors, we prepared a codebook where the number of codeword was 500 by k-means clustering in advance. We apply
soft assignment [11] by 3 nearest-neighbor assigned reciprocal number of Euclid distance to the codeword, also we
use fast approximated-nearest-neighbor search based kdtree to search the codebook for the corresponding codeword
for each sampled point. Finally, we create a 500-dim bagof-SURF vector.

4.4. Classification
As a classifier, we use a linear kernel SVM, and we adopt
the one-vs-rest strategy for multi-class classification. In
the experiment, since we prepared 50 food categories, wetrained 50 linear SVM classifiers.
Linear kernel is defined as the inner product of two vectors. In advance, we computed the inner product of a support vector and the weight of the corresponding support vector, then Linear SVM can be written as follows:
f (x) =MX

yi αi K(x, xi ) + b =

i=1

= h

M
X

M
X

yi αi hx, xi i + b

i=1

yi αi xi , xi + b = hw, xi + b

(1)

i=1

where x is an input vector, f (x) is an output SVM score,xi is support vector, yi ∈ {+1, −1} is a class label, αi is aweight of the corresponding support vector, b is a bias vector, and M is the number of support vector. By this transformation, we can save memory to store support vectors aswell as calculation of kernels. Therefore, when N is thedimension of feature vector, calculation of a SVM score requires O(N ) operations and O(N ) memory space. We train
SVMs with LIBLINEAR [2] in off-line.
Although a linear SVM is fast and can save memory,classification accuracy is not as good as a non-linear kernel SVM such as a SVM with χ2 -RBF kernel. To compensate weakness of a liner SVM, we use explicit embedding technique. In this work, we adopt kernel feature maps.
Vedaldi et al. [13] proposed homogeneous kernel maps forχ2 , intersection, Jansen-Shanon’s and Hellinger’s kernels,
which are represented in the closed-form expression. We
choose mapping for χ2 kernel and we set a parameter sothat the dimension of mapped feature vectors are 3 times
as many as the dimension of original feature vectors. This

mapping can be applied for L1-normalized histogram [13].
Then we apply this to L1-normalized color histograms andBag-of-SURF vectors. In the experiments, we comparedthis χ2 kernel mapping with the square-rooting feature vector which corresponds to an exact mapping for Hellinger’s
kernel the effectiveness of which was demonstrated by Perronnin et al. [10].
In the experiment, we use both a RGB color histogramand bag-of-SURF and integrity them by a linear weightingwith the weights estimated by cross-validation.

4.5. Estimation of the more reliable direction
In case that no categories with high SVM scores are obtained, camera position and viewing direction should bechanged slightly to obtain more reliable SVM evaluation
scores. To encourage a user to move a smartphone camera,
the proposed system has a function to estimate the direction
to get the more reliable SVM score and show the direction
as an arrow on the screen as shown in Figure 1.
To estimate the direction of the food regions withmore reliable SVM score, we adopt an effective window
search method for object detection, Efficient Sub-window
Search (ESS) [5], which can be directly applied for a combination of a liner SVM and bag-of-features.
The weighting factor w of an input vector of the SVM
classifier represented in Equation (1) can be decomposed
into a vector w+ including positive elements and a vector
w− including negative elements.
w = w+ + w−

(2)

Therefore, an SVM output score for one rectangular region
can be calculated in O(1) operation by making use of w+
and w− integral images according to the ESS method [5].
In case of the above-mentioned soft assignments to codewords, the output score can be calculated by a product of w
and assigned values to codewords. We search for the window which achieves the maximum SVM score by Efficient
Sliding Windows search so as to keep more than 50% area
being overlapped with the original window. Finally the relative direction to the window with the maximum score fromthe current window are shown as an arrow on the screen(See Figure 1).

5. Experiments
In this section, we describe experimental results regarding recognition accuracy and time. In addition, we also explain the evaluation result by user study.
In the experiments, we prepared a fifty-category fooddataset which has more than 100 training images per category all the food item in which are marked with bounding boxes. The total number of food images in the dataset
is 6,781. Figure 4 shows all the category names and theirsample photos.

5.1. Evaluation on classification accuracy
In this subsection, we evaluate the effectiveness of feature map-based fast χ2 kernel, bounding box adjustment,

Japanesestylepancakegratinsweet andsour pork

steamed egg
hotchpotch

yakitoricabbage rollegg
sunny-side
up

fried
shrimppotato salad green salad

rice

sushi

ramen
noodlefried noodle

fried fish

toast

hamburgerpizza

croquettesauteedspinachmiso soupsausageomeletjiaozifriedchickensirloincutletnanbanzuke

seasoned
beef withpotatoeshambarg
steak

ginger pork
saute

spicy chiliflavored
tofu

natto

egg roll

stir-fried
beef andpeppers

fish-shaped
pancakeshrimp with
with bean chill sourcejamsteamedmeat
dumpling

spaghetti
meat sauce

macaronisalad

pork miso
soup

beef bowl

french fries

hot dog

mixed rice

k-stylesauteedburdock

sandwiches udon noodle

Figure 4. 50 kinds of food images which are recognition targets inthe this paper.

and estimation of the expected food regions for mobile foodrecognition.
We evaluate food classification accuracy with the 50category food dataset by 5-fold cross-validation, whichcorresponds to food recognition under the condition thatground-truth bounding boxes are given.
Firstly, we evaluated and compared performance in foodrecognition with various kinds of common image features
including global and local features such as color histogram,color moment, color auto correlogram, Gabor texture feature, HoG, PHoG, and Bag-of-SURF. Figure 5 shows theclassification rate using single features. From this table,
RGB color histogram and Bag-of-SURF with fast χ2 areregarded as being effective. RGB color histogram with
fast χ2 achieved the classification rate 65.90% within the
top five, while bag-of-SURF with fast χ2 achieved 71.34%
within the top five, which outperformed the result with bagof-SURF with Hellinger’s kennel with the twice number ofcodewords. For all the cases, bag-of-SURF by soft assignment are better than by hard assignment. We also tried recent proposed local binary descriptor based BoF, but very
poor accuracy, consider cause by bigger quantization error
for little information. Finally we chose the combination ofcolor histogram and Bag-of-SURF as image features for the
proposed system.
Figure 6 shows the classification rate with RGB color
histogram, Bag-of-SURF and their combination with a fast
χ2 liner SVM. Regarding the combined feature, the rates
with a standard linear SVM and a non-linear RBF-χ2 SVM
are also shown in the Table. In case of two feature combination with a fast χ2 liner SVM and a standard linear
SVM, the classification rates were 53.5% and 39.1%, andthe classification rates within the top five candidate was
81.6% and 71.0%, respectively. Moreover, in case of a standard non-linear RBF-χ2 kernel SVM which is much moretime-consuming and is not appropriate for real-time recognition, the rate was 57.3%, and the rate within the top fivewas 83.2%. From these results, the results by a fast χ2 linear SVM outperformed the results by a standard linear SVM
by more than 10 points, and a fast χ2 SVM is almost com-

Classification Rate

fried rice

tempurabowl

beef curry

0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

top5
top1

Feature Type

Figure 5. Classification rates with single features within top 1 and
top 5.

parable to a non-linear SVM, which means that a fast χ2
linear SVM is very effective for both speed and accuracy
which are requirements for a mobile image recognition system. Since the proposed system shows the top five candidate on the screen, 81.6% can be regarded as the rate that atrue food name are shown in the candidate list.
Next, we made an experiments to examine effectivenessof bounding box adjustment. We magnified ground-truth
bounding boxes of test images with 25% in terms of bounding box size. In fact, we used only 1912 food images
as test images in the 5-fold cross-validation classification
experiment, since for some food photos their size are almost the same as the size of attached bounding boxes andthey do not includes 25% backgound regions. We compared groundtruth bounding boxes, 25%-magnified bounding boxes, and adjusted bounding boxes after 25% magnification in terms of classification rate under the same condition as the previous experiment. Figure7 shows the results, which indicates that 25% maginifation of groundtruth
bounding boxes degraded the classification rate within the
top five by 10.1%, while 25% maginification with bounding box adjustment degraded the rate by only 4.4%. From
this results, GrubCut-based bounding box adjustments canbe regarded as being effective.
Finally, we made an experiment on estimation of the direction of a food window. We evaluated error in the direction estimation in case of sifting the ground-truth boundingboxes by 10, 15, 20, and 25% to each of eight directions
around the original boxes. Figure 8 show cumulative classification rates of the estimated direction in case of using
soft and hard assignment, which proved that soft assignment
was better than hard assignment. Figure9 shows cumulativeclassification rates of the estimated direction with different
shifts. The rates with less than ±20◦ error and ±40◦ error were 31.81% and 50.34% in case of 15% shift, and are
34.54%, and 54.16% in case of 25% shift. From these results, when the difference between the ground-truth and thegiven bounding box is small, estimation of the direction of
the ground-truth bounding box is more difficult. This is because the difference of SVM scores between them is small
in case that the difference in the location of the boundingboxes is small.

Color(fast chi2)
SURF(fast chi2)
Color+SURF(fast chi2)
Color+SURF(linear)
Color+SURF(chi2-RBF)

1

2

3

4

5

6

7

8

9

10

Classification RateClassification Rate

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

10%
15%
20%
25%

±20° ±40° ±60° ±80° ±100° ±120° ±140° ±160° ±180°

allowable angular difference error

Number of Candidates

Classification Rate

Figure 6. Classification rates of RGB color histogram, bag-ofSURF and their combination with a fast χ2 linear SVM, and ratesof the combination with a standard linear SVM and a non-linear
RBF-χ2 SVM within the top n candidates. Note that the graphs ofsingle SURF with fast χ2 and combination with a standard linear
SVM are overlapped and difficult to distinguish.
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

0%
25%
25%, GrabCut

1

2

3

4

5

6

7

8

9

10

number of candidates

Classification Rate

Figure 7. Classification rates in case of the ground-truth boundingbox (BB), 25% magnified BB and adjusted BB after 25% magnification (shown as “25%, GrabCut”).
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

hard
soft

±20° ±40° ±60° ±80° ±100° ±120° ±140° ±160° ±180°

allowable angular difference error

Figure 8. Cumulative classification rate of the estimated orientation with soft assignment and hard assignment. The horizontal
axis expresses allowable angular difference error and the verticalaxis expresses classification rate in terms of estimated directions.

5.2. Evaluation of processing time
We implemented the proposed system as Android application assuming that quad core is available. High-cost

Figure 9. Cumulative rate of the estimated orientation in case of
10, 15, 20, and 25% shifted bounding boxes.
Table 1. Average processing time.

Bounding Box Adjustment
Recognition
Search Food Region
Recognition +Search Food Region

average time [sec]
0.70
0.26
0.091
0.34

SURF descriptor extraction, assignment to codewords, evaluation of 50 kinds of fast χ2 linear SVMs and direction estimation are carried out over four cores in parallel, while
low-cost color histogram extraction is performed on a single core.
We measured processing times on the latest smartphone,Samsung Galaxy Note II (1.6GHz Quad Core with Android
4.1). The results are shown in Table 1. Processing time forrecognition and direction estimation are 0.26 seconds, 0.09
seconds, while bounding box adjustment is relatively highcost, which takes 0.70 seconds. This is why bounding box
adjustment are carried out once after drawn. Among 0.26
seconds for recognition, color histogram extraction and linear SVM classification takes only 0.003 seconds, and mostof the time are taken for extraction of SURF descriptors andsearching of the codebook to create bag-of-features vectors.
Because the total time of recognition and estimation of thedirection is 0.34 seconds, three bounding boxes can be processed in around one second at the same time.

5.3. User Study
We asked five student subjects to evaluate quality ofthe proposed system in five step evaluation regarding foodrecognition, how easy to use, quality of direction estimation, and comparison of the proposed system with the baseline which has no food recognition and requires selecting
food names from hierarchical menus by touching. The evaluation score 5, 3 and 1 means good, so-so, and bad, respectively. At the same time, we measured time for selecting
food items with food recognition, and compared it with thetime for selecting food items from the hierarchical menu byhand.
Figure 10 shows the spent time for selecting each food• Improve accuracy of estimation of expected food regions or move the bounding boxes automatically instead of only showing the direction.

25
Count

20

15

proposed system
hand
median• Take into account additional information such as user’s
food history, GPS location data and time information.

10
5
0
䡚2.0
2.0䡚3.0
3.0䡚4.0
4.0䡚5.0
5.0䡚6.0
6.0䡚7.0
7.0䡚8.0
8.0䡚9.0
9.0䡚10.0
10.0䡚11.0
11.0䡚12.0
12.0䡚13.0
13.0䡚14.0
14.0䡚15.0
15䡚• Increase the number of food categories to make thesystem more practical.
Note that Android application of the proposed mobile
food recognition system can be downloaded from http://foodcam.mobi/ .

Spend Time[sec]Figure 10. Time comparison: food recognition vs. hierarchicalmenu.
Table 2. User study results which are the average of five-step evaluation scores.

Outcome Measure
Recognition quality
Facility to use
Quality of Direction Suggestion
Proposed System Quality(compared with hand-selection)

average score
3.4
4.2
2.4
3.8

item. The median time were 5.1 second with food recognition, and 5.7 second by hand. This means the proposedsystem can help a user select food names faster than from ahierarchical menu by hand. However, for some food items
which not able to be recognized, it spent long time to findfood names using food recognition.
Table 2 shows the system evaluation by the five grade
evaluation. Except for suggest direction, more than threepoints are obtained. Especially, usability of the system isgood, since recognition is carried out in a real-time way.
On the other hand, estimation of the expected food region
is not evaluated as being effective, since the classification
accuracy is not so good for practical use. We will improve
it as a future work.

6. Conclusions and Future WorkIn this paper, we propose a real-time food recognition
system on a smartphone. The system adopts a liner SVM
with a fast χ2 kernel, bounding box adjustment and estimation of the expected direction of a food region. In the experiment, we have achieved 81.55% classification rate withthe top five candidates when ground-truth bounding boxes
are given. In addition, we obtained positive evaluation byuser study compared to the food recording system without
object recognition.
As feature works, we plan to extend the system regarding
the following issues:• Touch just a point instead of drawing bounding boxes
to specify food regions.
• Use multiple images to improve accuracy of food item
recognition.

References[1] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-up robust features (SURF). Computer Vision and Image Understanding,
110(3):346–359, 2008. 4[2] R. E. Fan, K. W. Chang, C. J. Hsieh, X. R. Wang, and C. J. Lin.
LIBLINEAR: A library for large linear classification. The Journal of
Machine Learning Research, 9:1871–1874, 2008. 4[3] K. Kitamura, T. Yamasaki, and K. Aizawa. Food log by analyzing
food images. In Proc. of ACM International Conference Multimedia,
pages 999–1000, 2008. 2[4] N. Kumar, P. Belhumeur, A. Biswas, D. Jacobs, W. Kress, I. Lopez,
and J. Soares. Leafsnap: A computer vision system for automaticplant species identification. In Proc. of European Conference on
Computer Vision, 2012. 2[5] C. H. Lampert, M. B. Blaschko, and T. Hofmann. Beyond sliding
windows: Object localization by efficient subwindow search. In
Proc. of IEEE Computer Vision and Pattern Recognition, 2008. 1,
4[6] T. Lee and S. Soatto. Learning and matching multiscale templatedescriptors for real-time detection, localization and tracking. In Proc.
of IEEE Computer Vision and Pattern Recognition, 2011. 2[7] A. Mariappan, M. Bosch, F. Zhu, C. Boushey, D. Kerr, D. Ebert,
and E. Delp. Personal dietary assessment using mobile devices. In
Proc. of the IS&T/SPIE Conference on Computational Imaging VII,
volume 7246, pages 72460Z–1–72460Z–12, 2009. 2[8] T. Maruyama, Y. Kawano, and K. Yanai. Real-time mobile recipe
recommendation system using food ingredient recognition. In Proc.
of ACM Multimedia Workshop on Interactive Multimedia on Mobile
and Portable Devices, pages 27–34, 2012. 2[9] Y. Matsuda, H. Hoashi, and K. Yanai. Recognition of multiple-foodimages by detecting candidate regions. In Proc. of IEEE International Conference on Multimedia and Expo, pages 1554–1564, 2012.
2[10] F. Perronnin, J. Sánchez, and Y. Liu. Large-scale image categorization with explicit data embedding. In Proc. of IEEE Computer Vision
and Pattern Recognition, pages 2297–2304, 2010. 4[11] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Lost
in quantization: Improving particular object retrieval in large scale
image databases. In Proc. of IEEE Computer Vision and Pattern
Recognition, pages 1–8, 2008. 4[12] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. In SIGGRAPH, pages
309–314, 2004. 1, 3
[13] A. Vedaldi and A. Zisserman. Efficient additive kernels via explicit
feature maps. IEEE Transactions on Pattern Analysis and MachineIntelligence, 34(3):480–492, 2012. 1, 4
[14] S. Yang, M. Chen, D. Pomerleau, and R. Sukthankar. Food recognition using statistics of pairwise local features. In Proc. of IEEE
Computer Vision and Pattern Recognition, 2010. 2[15] F. Yu, R. Ji, and S. Chang. Active query sensing for mobile location
search. In Proc. of ACM International Conference Multimedia, pages
3–12, 2011. 2