Real-time Mobile Recipe Recommendation System
Using Food Ingredient Recognition
Takuma Maruyama

Yoshiyuki Kawano

Keiji Yanai

Department of Informatics, The University of Electro-Communications
Chofu, Tokyo 182-8585 JAPAN{maruyama-t,kawano-y,yanai}@mm.cs.uec.ac.jp

ABSTRACTIn this paper, we propose a mobile cooking recipe recommendation system employing object recognition for food ingredients such as vegetables and meats. The proposed system carries out object recognition on food ingredients in areal-time way on an Android-based smartphone, and recommends cooking recipes related to the recognized food ingredients. By only pointing a built-in camera on a mobile
device to food ingredients, the user can obtain a recipe list
instantly. As an object recognition method, we adopt bagof-features with SURF and color histogram extracted frommultiple images as image features and linear SVM with theone-vs-rest strategy as a classifier. We built 30 kinds of foodingredient short video database for experiments. With thisdatabase, we achieved the 83.93% recognition rate within
the top six candidates. In the experiment, we made user
study by comparing mobile recipe recommendation systems
with/without ingredient recognition.

Categories and Subject Descriptors
H.4 [Information Systems Applications]: Miscellaneous

General Terms
Algorithms, Design, Experimentation

Keywords
food ingredient, object recognition, generic object recognition, recipe recommendation, mobile, smartphone

1. INTRODUCTION
Recently, cooking recipe sites such as cooks.com and BBC
food search has become popular. Some of the people who
cook use such sites to obtain information on cooking recipes.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
IMMPD’12, November 2, 2012, Nara, Japan.
Copyright 2012 ACM 978-1-4503-1595-1/12/11 ...$15.00.

Figure 1: An image on the proposed system. A user points amobile phone camera to food ingredients at a grocery stores,
and then the system advises cooking recipes based on therecognized ingredients instantly.

Since these sites are accessible from mobile phones as well
as PCs, a user can access these sites at a grocery store aswell as at home. However, to use these sites, a user has to
input some keywords or select menu items to indicate his/her
preferences on cooking menus. This may cause to prevent
users from referring cooking recipe sites during shopping at
grocery stores.
On the other hand, object recognition technology has been
made much progress so far. Especially, generic object recognition, which is the technology that categories of the objects shown in an given image are recognized, have achieved
tremendous progress. At the same time, open source libraries on object recognition such as the Open Computer
Vision library (OpenCV) has spread widely. With such libraries, we can easily implement object recognition system
not only on PCs but also on mobile devices such as phones
and Android smartphones. Due to recent progress of mobile devices as well as their recent explosive spread, object
recognition on mobile devices in a real-time way becomes
possible.
Based on these situations, in this paper, we propose acooking recipe recommendation system on a mobile device
employing object recognition for food ingredients such as
vegetables and meats. The proposed system carries out
object recognition on food ingredients in a real-time way
on Android-based smartphones, and recommends cooking

recipes related to the recognized food ingredients. By pointing a mobile phone camera toward food ingredients, a user
can receive a recommendation recipe list instantly. We designed and implemented the system to be used easily andintuitively during shopping at grocery stores or supermarkets as well as before cooking at home.
To speed up object recognition for enabling the system torecommend cooking recipes in a real-time way, the system
uses color-histogram-based bag-of-features extracted frommultiple frames as image representation and linear kernel
SVM as a classifier. We built 30 kinds of food ingredient short video database for the experiments. With thisdatabase, we achieved the 83.93% recognition rate within
the top six candidates. In the experiment, we made user
study by comparing mobile recipe recommendation systems
with/without ingredient recognition.
In the rest of this paper, we describe related work in Section 2. In Section 3, we explain the overview and detail ofthe proposed system. The method to recognize food ingredients used in the proposed system is described in Section
4. Section 5 shows experimental results and user study. We
conclude this paper in Section 6.

2. RELATED WORKIn this section, we introduce related works in terms ofmobile object recognition, image recognition for food ingredients as well as cooking recipe recommendation.
As commercial services on image recognition for mobile
devices, Google Goggles1 is widely well-known. Google Goggles work as an application on both Android and iPhone,which can recognize letters, logos, famous art, cover pagesof books and famous landmarks in photos taken by users
with object recognition technology. Since it is mainly basedon specific object recognition method employing local feature matching, it is good at rigid objects such as landmarks
and logos. However, it cannot recognize generic objects suchas animals, plants and foods at all.
As a similar work, Lee et al. [6] proposed a mobile object
recognition system on a smartphone, which recognized registered objects in a real-time way. They devised descriptorsof local features and their matching method for real-time
object recognition. On the other hand, Yu et al. [10] propose a mobile location recognition system which recognizes acurrent location by local-feature-based matching with streetview images stored in a database. In their work, they proposed automated Active Query Sensing (AQS) method toautomatically determine the best view for visual sensing totake an additional visual query. All of these systems aimed
local-feature-based specific object matching, while we focus
on generic object recognition on food ingredients.
Next, we explain some works on image recognition on foodingredients. In Smart Kitchen Project [5] leaded by Minoh Lab, Kyoto University, which aims to realize a cookingassisted kitchen system, image recognition for food ingredients is used. This project includes image classification andtracking on food ingredients during cooking. While in thisproject food ingredient recognition is used for cooking assistance, in our work it is used for cooking recipe search.
Regarding works on cooking recipe recommendation, textbased methods have been studied so far. Ueda et al. proposed a method to recommend cooking recipe based on user’s1

http://www.google.com/mobile/goggles/

preference [9], and Shidochi et al. worked on finding replaceable ingredients in cooking recipe [8]. Akazawa et al. [1]proposed a method to search cooking recipes based on foodingredients left in a refrigerator. The recommended recipes
are ranked in the order considering consumption date andremaining amount of ingredients. In the current work ofours, we did not use the detail information on available ingredients. We plan to take into account the conditions ofingredients on amounts, nutrition and prices for future work.

3.

PROPOSED SYSTEM

In this section, we explain an overview and detail of the
proposed system.

3.1 Overview
The objective of this work is to propose a mobile system
which assists a user to decide what and how to cook using
generic object recognition technology. We assume that the
proposed system works on a smartphone which has built-in
cameras and Internet connection such as Android smartphones and iPhones. We intend a user to use our systemeasily and intuitively during shopping at grocery stores orsupermarkets as well as before cooking at home. By pointing food ingredients with a mobile phone built-in camera,a user can receive a recipe list which the system obtained
from online cooking recipe databases instantly. With oursystem, a user can get to know the cooking recipes related
to various kinds of food ingredients unexpectedly found in agrocery store including unfamiliar ones and bargain ones on
the spot.
To do that, the system recognizes food ingredients in thephotos taken by built-in cameras, and search online cooking
recipe databases for the recipes which need the recognized
food ingredients.
As an object recognition method, we adopt bag-of-featureswith SURF and color histogram extracted from not single
but multiple images as image features and linear kernel SVM
with the one-vs-rest strategy as a classifier.

3.2 Processing Flow
As mentioned before, our system aims to search for cooking recipes during shopping at grocery stores. In this subsection, we describe the flow of how to use the proposedsystem from taking photos of food ingredients until watching the recipe pages a user selected. Figure 2 shows theflow.
Step 1 Point a smartphone camera toward food ingredients at a grocery store or at a kitchen. The system iscontinuously acquiring frame images from the camera
device in the background.
Step 2 Recognize food ingredients in the acquired frame
images continuously. The top six candidates are shownon the top-right side of the screen of the mobile device.
Step 3 Search online cooking recipe databases with the nameof the recognized food ingredient as a search keyword,
and retrieve a menu list. If a user like to search forrecipes related to other candidates than the top one,the user can select one of the top six ingredients bytouching the screen.
Step 4 Display the obtained menu list on the left side.

3.3

Search Online Cooking Recipe Databases

Instead of preparing our own cooking recipe database, we
use Web APIs of commercial cooking recipe sites on the
Web such as CookPad2 and PunchFork3 . CookPad is aJapanese food recipe site where all the information is written in Japanese language, while PunchFork mainly focuseson Western food recipes which is operated by a US company.
Currently, we send the names of recognized food ingredients as search terms as they are, and obtain research resultson cooking recipes in which the recognized food ingredientsare needed to cook. Re-ranking of the returned results fromthe Web API of cooking recipe sites considering various elements including prices, amounts and user’s preferences is
our future work.

4.

IMAGE RECOGNITION METHOD

In this section, we explain a method on image-based foodingredient recognition, which is a core part of the proposedsystem, regarding image features, image representation, andimage classifier.

4.1 Image Features

Figure 2: Processing flow.
Step 5 Select one menu from the menu list. A user can see
other menus than ones shown on the screen initiallyby scrolling.
Step 6 For the selected menu, display the corresponding
cooking recipe including a list on necessary ingredientsand seasonings and a cooking procedure on the popup window. Basically, the recipe page in the original
recipe site will be shown.
Typically, an user uses the proposed system according to
the above steps from one to six. Figure 3 shows the system
screen.

In real-time object recognition on a mobile device, choiceof image features is important in terms of accuracy andspeed, both of which are trade-off in general. Recently,
new local features which are suitable for a mobile device
such as BRIEF [3] and ORB [7] are proposed. They require
small memory and run very fast because of binary-based descriptors. However, all the new features for a mobile device
intends instance-level specific object recognition based on
local feature matching. In our system, we need to carry out
category-level generic object recognition. To prevent information loss due to binarization of feature vectors, we use
SURF local feature [2].
SURF is an invariant local feature for scale, rotation andillumination change. When extracting SURF features, we
use dense sampling where all the local features are extracted
from multi-scale grids as well as a fast Hessian detector
which is a default keypoint detector of SURF.
In addition, for recognition of food ingredients, color isregarded as important information as well. We also extract
grid-based color histograms. As shown in Figure 4, we divide
an image into 12 × 12 grids, and extract a 64-bin color histogram from each block with dividing the space into 4×4×4
bins. Totally, we extract 144 64-dim color feature vectors
from one image. We regard each color feature vector as alocal color feature, and convert them into bag-of-feature representation in the same way as SURF features in the next
step. Note that we used three kinds of color spaces including
RGB, HSV and La*b* in the experiments.

4.2

Figure 3: The system screen.

Bag-of-Features Representation

Bag-of-Features (BoF) [4] is a standard feature representation to convert a set of local features into one feature vector,
since it has excellent ability in the context of category-level
generic object recognition in spite of its simplicity.
To convert a set of local feature vectors into a BoF vector,
we vector-quantize them against the pre-specified codebook.
After that, all the BoF vectors are L1-normalized. In the experiments, we built a 1000-dim codebook by k-means clus2
3

http://cookpad.com/
http://punchfork.com/

5.

EXPERIMENTS

5.1 Data Collection and Experimental Setting

Figure 4: Grid-based extraction of local color histograms.

As a data set for the experiments, we collected 10 shortvideos per ingredient category for 30 kinds of food ingredients (Figure 1) at grocery stores in Tokyo. Since we use
multiple frame object recognition, we collected short videos
instead of still images. Each of the videos was recored forabout 5 seconds in 25 fps with the VGA (640 × 480) resolution. In the experiments we carried out evaluation of object
classification performance with 10-fold cross validation.
Table 1: 30 kinds of food ingredients in the data set.

tering with local features sampled from training data offline
on a PC.
In this work, we can use multiple frames to build a BoF
vector, since we acquire frame images from the built-in camera continuously. Therefore, in the experiments, we aggregated local features extracted from five frames at most, andconvert them into one BoF vector.

types
fish (5)meat (6)vegetable(13)
fruit (6)ingredientstuna, squid, octopus, shrimp, salmon
beef, pork, chicken, minced meat, sausage, ham
mushroom, potato, eggplant, carrot, radish,tomato, cucumber, cabbage, green onions, onion,
Chinese cabbage, lettuce, Shiitake mushroom
apple, strawberry, pineapple, orange,
banana, grapefruit

4.3 Image Classifier
In this paper, we use a Support Vector Machine (SVM)
which is the most common classifier. It is common to use
non-linear kernels such as a RBF-χ2 kernel with a SVM
in category-level object recognition task, because of its highclassification performance. However, a non-linear kernel SVM
is computationally expensive compared to a linear kernel
SVM. In case of classification step, the computation cost ofa non-linear kernel SVM is O(dn), while that of a linear kernel SVM is O(d) where d and n represents the dimensionof feature vectors and the number of support vectors which
is typically proportional to the number of training samples,
respectively. Since we prioritize low computational cost for
real-time recognition, we adopt a liner kernel SVM.
A liner kernel K(x, y) is represented in the following function, which is equivalent to an inner product of two vectors.
K(x, y) = x · y=NX

wi K(x, xi ) + b

Multi-scale-grid-based SURF extraction
4 scales (12×12, 24×24, 48×48, 96×96pixels)
Grid-based color histogram
1 scales (dividing a image into 12 × 12grids)
Color space of color histogram
RGB, HSV, La*b*multi-frame feature extraction
1, 2, 3, 4, 5 frames





(1)

When x, y(x), N , xi , wi and b represents an input feature
vector, the output of a SVM classifier, the number of support
vectors, a support vector, the weight of the corresponding
support vector, and a bias scalar value, respectively, theequation of a linear SVM classifier can be transformed asfollows:y(x)In the experiments, we set the parameters as follows:Parameter setting in the experiments

(a)

(b)

(c)

(d)

(e)

(f)

(2)

i=1

=

N
X

wi x · xi + b

i=1
N
X

=

x·

=

x · v + b,

wi xi + b

i=1

PN

(3)

where v = i=1 wi xi . As shown above, we can evaluate oneclassifier with only the computation of one inner productbetween two vectors and addition of a bias scalar value. In
the experiments, to recognize 30 kinds of food ingredients,
we used the one-vs-rest strategy.

Figure 5: Typical situations: (a) The target is a “grapefruit”.
(b) A “grapefruit” is successfully recognized as the top candidate, and the cooking menus related to it are shown on
the screen. (c) The selected recipe related to “grapefruit”
is shown. (d) The target is a “salmon”. (e) A “salmon” isranked in the third. Then, ”salmon” is selected by touching
the screen. (f) The selected recipe related to “salmon” isshown.

5.2 Example of Usage of the System
Before showing the evaluation results, we show typicalcases of usage of the proposed system at a grocery store in
Figure 5. In these cases, we used the system on Samsung
Galaxy S2 (1.5Ghz dual core, Android 2.2). On this device,
it took 0.15 seconds to recognize an ingredient with the builtin camera in case of using only color features.

Figure 6: Classification rate using n frames (n = 1..5).

At first, we made experiments with single image features,
which are SURF with a fast Hessian detector (default detector), SURF with multi-scale grids, RGB, HSV and La*b*,
varying the number of frames to build a BoF vector. We
show the results in Figure 6, which indicates that the difference depending on the number of frames is limited, andthe best classification rate, 43.78%, is achieved in case of a
RGB color feature with three frames.
Figure 7 shows the results of single features with a single frame, a RGB color feature with three frames, and the
combination of SURF and RGB with three frames in a bar
graph. Although the combination of SURF and RGB withthree frames achieved the best result, 44.92%, the difference
to the result by only RGB feature is only 1.14%. The reason
that SURF feature does not work as well as RGB color isthat the dataset contains many blurred or reflected frames
as shown in Figure 9, from which it it difficult to extract
gradient-based features such as SURF effectively.
Figure 8 shows the classification rate within the top kcandidates in case of RGB, SURF and combination of RGB
and SURF with three frames. This shows that the result by
RGB and the result by combination of RGB and SURF arealmost equivalent. Therefore, in the release version of theapplication of the proposed system, we use only RGB color
features. Because the top six candidates can be shown on thescreen at the same time in the current implementation, theclassification rate within six is important, which is 83.93%.

(a)

(b)

(c)

(d)

Figure 9: Example photos in which recognition failed due toreflectance and blurring.

Figure 7: Classification rate by each method.

Figure 10: Precision rate for each of thirty kinds of the foodingredients.

Figure 8: Classification rate within the top k candidates(k = 1..10).

5.3 Evaluation on Object Classification Performance
We evaluated the classification accuracy with various settings in terms of image feature extraction.

Figure 10 shows the precision, the recall rates and theF-measure for each of the 30 kinds of ingredients by combination of RGB color and SURF with three frames, which
achieved the best result on the average. “Orange” achieved
the best result (Figure 11), since it has its unique and specific color. On the other hand, “shrimp” achieved the worst(Figure 12), since frozen “shrimps” have various colors andtheir appearance depends on how to pack greatly. This iscompletely different from “shrimp” images in the ImageNet
database4 .

(a)

(b)

(c)

(d)

Figure 11: An easy food ingredient to recognize: “orange”.

(a)

(b)

(c)

than fifteen seconds. However, the cases which took only
less than two seconds were twice by image recognition, but
none by hand. This shows that if image recognition works
successfully, image recognition is faster and more effective
than hand selection. We think this tendency gets more remarkable, if the number of food ingredients to be recognizedbecomes larger.
As shown in Figure 3, to select an ingredient from a 30kind list by touching the screen, a user has to select hierarchical menus and sometimes has to scroll the menus to findout the given ingredient. On the other hand, a user sometimes has to continue to change the build-in camera positionand direction until the correctly-recognized ingredient appears within the top six candidates on the screen, although
the rate within six candidates was 83.93%. For these reasons, both image-based method and hand-based methods
sometimes took more than ten seconds.

(d)Figure 12: A difficult food ingredient to recognize: “shrimp”.

5.4 User StudyIn this subsection, we show the results of user study employing five subjects.
At first, we recorded the times to search for the recipes
related to the given real food ingredients with the proposed
image-based method as well as by selecting ingredients byhand. Next, we asked them three questions on how easy to
use the system, how accurate ingredient recognition was, andwhich is easier to use, image recognition or selecting by hand.
We collected all the answers in the five-step evaluation. For
this study, we prepared three kinds of real food ingredients.

(a) Usability(b) Recognition accuracy(c) Which is better, image
recognition or by hand ?
Figure 14: The results of 5-step questions on usability, accuracy and comparison of both methods.

Figure 13: A graph on time (x-axis, seconds) vs frequency(y-axis) to select a recipe by hand and by image recognition.
Figure 13 shows the times to obtain the cooking recipes
related to the given food ingredients both in case of using
object recognition and in case of selecting ingredients bytouching the screen. The median of the times are 7.3 seconds
by hand and 8.5 seconds by image recognition, respectively.
This is because six cases by image recognition took more4

http://www.image-net.org/synset?wnid=n01986806

Next, we show the five-step evaluation results of the questions on usability, accuracy of image recognition, and comparison of both methods in Figure 14a, Figure 14b and Figure 14c, respectively. Overall, we did not obtained significant support for the image-based method at this moment.
This is mainly because the user interface is not sophisticated.
In fact, in this work, we gave priority to the image recognition part rather than the user interface. Improvement of theuser interface is part of our future work. We obtained somepositive comments from the subjects. “It is convenient tobe able to search for cooking recipes during shopping, whenbargaining ingredients are found unexpectedly.” “If the accuracy of recognition is improved, the system will be morepractical. ”

6. CONCLUSIONS AND FUTURE WORKIn this paper, we proposed a mobile cooking recipe recommendation system with food ingredient recognition on amobile device, which enables us to search for cooking recipes
only by pointing a built-in camera to food ingredients instantly. For 30 kinds of food ingredients, the proposed system has achieved the 83.93% classification rate within the
top six candidates. From the user study, it is turned out thatthe system is effective in case that food ingredient recognition works well.
For future work, we plan to improve the system in termsof object recognition, recipe recommendation and the system user interface. Regarding object recognition, we would
like to achieve 95% classification rate within the top six candidates for the 30 category food ingredients by adding otherimage features and segmenting food ingredient regions frombackground regions. Regarding recipe recommendation, weplan to implement recipe search considering combination ofmultiple food ingredients, nutrition and budgets.
Note that the application for Android smartphones of the
proposed system can be downloaded from http://mm.cs.
uec.ac.jp/mobile_recipe/.

7. REFERENCES[1] Y. Akazawa and K. Miyamori. Cooking recipe searchsystem considering food materials remained in arefrigerator. In Proc. the 3rd Forum on Data
Engineering and Infomation Management, 2011 (inJapanese).
[2] H. Bay, T. Tuytelaars, and L. Van Gool. SURF:
Speeded up robust features. In Proc. of European
Conference on Computer Vision, pages 404–415, 2006.
[3] M. Calonder, V. Lepetit, C. Strecha, and P. Fua.
BRIEF: Binary robust independent elementaryfeatures. In Proc. of European Conference on
Computer Vision, 2010.
[4] G. Csurka, C. Bray, C. Dance, and L. Fan. Visual
categorization with bags of keypoints. ECCV WS on
Statistical Learning in Computer Vision, pages 1–22,
2004.
[5] A. Hashimoto, N. Mori, T. Funatomi, Y. Yamakata,K. Kakusho, and M. Michihiko. Smart kitchen: A user
centric cooking support system. In Proc. ofInternational Conference on Information Processingand Management of Uncertainty in Knowledge-Based
Systems, pages 848–854, 2008.
[6] T. Lee and S. Soatto. Learning and matching
multiscale template descriptors for real-time detection,
localization and tracking. In Proc. of Computer Vision
and Pattern Recognition, pages 1457–1464, 2011.
[7] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski.
ORB: An efficient alternative to sift or surf. In Proc. ofInternational Conference on Computer Vision, 2011.
[8] Y. Shidochi, T. Takahashi, I. Ide, and H. Murase.
Finding replaceable materials in cooking recipe texts
considering characteristic cooking actions. In Proc. of
ACM MM WS on Multimedia for Cooking and Eating
Activities (CEA’09), pages 9–14, 2009.
[9] M. Ueda, M. Takahata, and S. Nakajima. User’s foodpreference extraction for cooking recipe
recommendation. In Proc. of the 2nd Workshop on

Semantic Personalized Information Management:
Retrieval and Recommendation, 2011.
[10] X. F. Yu, R. Ji, and S.-F. Chang. Active query sensing
for mobile location search. In Proc. of ACM
Multimedia, pages 3–12, 2011.