[DEMO PAPER] MIRURECIPE: A MOBILE COOKING RECIPE
RECOMMENDATION SYSTEM WITH FOOD INGREDIENT RECOGNITION
Yoshiyuki Kawano, Takanori Sato, Takuma Maruyama and Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan{kawano-t,sato-ta,maruya-t,yanai}@mm.inf.uec.ac.jp

ABSTRACTIn this demo, we demonstrate a cooking recipe recommendation system which runs on a consumer smartphone.
The proposed system carries out object recognition on foodingredients in a real-time way, and recommends cooking
recipes related to the recognized food ingredients. By only
pointing a built-in camera on a mobile device to food ingredients, the user can obtain a recipe list instantly. The objectiveof the proposed system is to assist people who cook to decide a cooking recipe at grocery stores or at a kitchen. In thecurrent implementation, the system can recognize 30 kinds offood ingredient in 0.15 seconds, and it achieved the 83.93%
recognition rate within the top six candidates.

smartphones, object recognition on smartphones in a realtime way becomes possible.
Based on these situations, in this demo paper, we propose
a cooking recipe recommendation system on a mobile device employing object recognition for food ingredients suchas vegetables and meats. The proposed system carries out
object recognition on food ingredients in a real-time way
on Android-based smartphones, and recommends cooking
recipes related to the recognized food ingredients. By pointing a mobile phone camera toward food ingredients, a user
can receive a recommendation recipe list instantly. We designed and implemented the system to be used easily and intuitively during shopping at grocery stores or supermarkets aswell as before cooking at home as shown in Figure 1.
To speed up object recognition for enabling the system
to recommend cooking recipes in a real-time way, the system uses bag-of-features with SURF and color-histogram extracted from multiple frames as image representation and linear kernel SVM as a classifier. We built 30 kinds of food ingredient short video database for the experiments. With thisdatabase, we achieved the 83.93% recognition rate within thetop six candidates.
2. PROPOSED SYSTEM

Fig. 1: Using the proposed system at a grocery store. A user
points a mobile phone camera to food ingredients on a display
stand, and then the system advises cooking recipes based on
the recognized ingredients instantly.
1. INTRODUCTION
Recently, cooking recipe sites such as cooks.com and BBC
food search has become popular. Some of the people who
cook use such sites to obtain information on cooking recipes.
Since these sites are accessible from mobile phones as well asPCs, a user can access these sites at a grocery store as well as
at home. However, to use these sites, a user has to input somekeywords or select menu items to indicate his/her preferenceson cooking menus. This may cause to prevent users from referring cooking recipe sites during shopping at grocery stores.
On the other hand, object recognition technology has been
made much progress so far. Especially, generic object recognition, which is the technology that categories of the objects
shown in an given image are recognized, have achieved
tremendous progress. In addition, due to recent progress of

2.1. Overview
The objective of this work is to propose a mobile system
which assists a user to decide what and how to cook using
generic object recognition technology. We assume that theproposed system works on a smartphone which has builtin cameras and Internet connection such as Android smartphones and iPhones. We intend a user to use our system easily and intuitively during shopping at grocery stores or supermarkets as well as before cooking at home. By pointing foodingredients with a mobile phone built-in camera, a user canreceive a recipe list which the system obtained from online
cooking recipe databases instantly. With our system, a user
can get to know the cooking recipes related to various kinds
of food ingredients unexpectedly found in a grocery store including unfamiliar ones and bargain ones on the spot.
To do that, the system recognizes food ingredients in thephotos taken by built-in cameras, and search online cooking
recipe databases for the recipes which need the recognized
food ingredients.
As an object recognition method, we adopt bag-offeatures with SURF and color histogram extracted from notsingle but multiple images as image features and linear kernel
SVM with the one-vs-rest strategy as a classifier.

Fig. 3: The system screen.

Fig. 2: Processing flow.

2.2. Processing Flow
As mentioned before, our system aims to search for cooking
recipes during shopping at grocery stores. In this subsection,
we describe the flow of how to use the proposed system fromtaking photos of food ingredients until watching the recipe
pages a user selected. Figure 2 shows the flow.
Step 1 Point a smartphone camera toward food ingredientsat a grocery store or at a kitchen. The system is continuously acquiring frame images from the camera device
in the background.
Step 2 Recognize food ingredients in the acquired frame
images continuously. The top six candidates are shownon the top-right side of the screen of the mobile device.
Step 3 Search online cooking recipe databases with thename of the recognized food ingredient as a search keyword, and retrieve a menu list. If a user like to search forrecipes related to other candidates than the top one, theuser can select one of the top six ingredients by touching the screen.
Step 4 Display the obtained menu list on the left side.
Step 5 Select one menu from the menu list. A user can see
other menus than ones shown on the screen initially byscrolling.
Step 6 For the selected menu, display the corresponding
cooking recipe including a list on necessary ingredientsand seasonings and a cooking procedure on the pop-up
window. Basically, the recipe page in the original recipesite will be shown.
Typically, an user uses the proposed system according tothe above steps from one to six. Figure 3 shows the system
screen.

2.3. Search Online Cooking Recipe Databases
Instead of preparing our own cooking recipe database, weuse Web APIs of commercial cooking recipe sites on theWeb such as CookPad and RecipePuppy. CookPad is aJapanese food recipe site where all the information is written in Japanese language, while RecipePuppy mainly focuseson Western food recipes which is operated by a US company.
Currently, we send the names of recognized food ingredients as search terms as they are, and obtain research resultson cooking recipes in which the recognized food ingredientsare needed to cook. Re-ranking of the returned results fromthe Web API of cooking recipe sites considering various elements including prices, amounts and user’s preferences is ourfuture work.
3. IMPLEMENTATION
We implemented the proposed system on Android-based
smartphones using the OpenCV library. In case of Samsung
Galaxy S2 (1.5Ghz dual core, Android 2.2), it took 0.15 seconds to recognize an ingredient once with the built-in camera.
For the evaluation of the 30 kinds of the trained linear SVMs
in the one-vs-rest way, it takes only 0.03 seconds of the total
recognition time, since we need to calculate inner products
only thirty times to do that.
As a data set for training, we collected 10 short videos per
ingredient category for 30 kinds of food ingredients (Table 1)at grocery stores. Since we use multiple frame object recognition, we collected short videos instead of still images. Eachof the videos was recored for about 5 seconds in 25 fps with
the VGA (640 × 480) resolution.
To evaluate recognition performance, we carried out evaluation of object classification performance with 10-fold crossvalidation. For 30 kinds of food ingredients, the proposedsystem has achieved the 83.93% classification rate within the
top six candidates.
Note that the application for Android smartphones of theproposed system can be downloaded fromhttp://mirurecipe.mobi/e/.
Table 1: 30 kinds of food ingredients in the data set.
types
fish (5)meat (6)vegetable(13)fruit (6)ingredientstuna, squid, octopus, shrimp, salmon
beef, pork, chicken, minced meat, sausage, ham
mushroom, potato, eggplant, carrot, radish,tomato, cucumber, cabbage, green onions, onion,
Chinese cabbage, lettuce, Shiitake mushroom
apple, strawberry, pineapple, orange,
banana, grapefruit