A system to help amateurs take pictures ofdelicious looking foodTakao Kakimori

Makoto Okabe∗

Keiji Yanai

Rikio Onai

kakimori@mm.inf.uec.ac.jp
m.o@acm.org
yanai@cs.uec.ac.jp
rikioonai@gmail.com
The University of Electro-Communications / JST, CREST∗

Abstract—Recently, many people have begun to take picturesof meals and food either at home or in restaurants. These pictures
are then uploaded to social networking services (SNS) where they
are shared with friends. People want to take pictures of food thatlooks delicious, but they often find this difficult. This is because
most people lack the knowledge required to take attractive
pictures. There are many photography techniques in use, e.g.,
composition [1], lighting, color, focus, etc. The techniques used
to take good pictures vary depending on the subject. Amateur
photographers find it difficult to choose techniques and apply
them appropriately. In this paper, we consider the compositionof food photographs and develop a system to support amateurs
taking pictures of meals and food to make the food look delicious.
Our target users are food photography amateurs. Our target
photographic subjects are food items on plates or dishes. Using
our system, there are four steps to food photography: 1) the user
provides information about the foods to be photographed, or oursystem automatically recognizes these food items, with the aid of acamera on a mobile phone; 2) our system suggests a compositionand camera tilt that will result in a picture that makes the foodlook delicious; 3) the user arranges the food and dishes on thetable, and sets the camera position and tilt; 4) finally, the user
takes the picture. If the user is not satisfied with the suggestion,
we allow the user to design a new composition quickly and easily
using their mobile phone. We performed a usability study for oursystem followed by a subjective evaluation of the quality of thepictures taken using our system.

the user (Figure 1).
If the user is not satisfied with the suggested composition,
the user can design a new composition and register it to the
database. This can be done by performing simple gestureson the mobile phone such as pinch-in and pinch-out. The
registered composition will then be shared by all users of oursystem.
We performed a usability study for our system. The results
indicated that our system can be quickly and easily used, evenby novice users. We also performed a subjective evaluation of
the quality of pictures taken using our system. Our evaluationsuggests that pictures taken using our system are of a higher
quality than pictures taken without using our system.

I. I NTRODUCTION
Most people have a mobile phone these days, and people
enjoy taking pictures of their food and immediately uploading
and sharing them with friends via SNS, e.g., Twitter and Facebook. Additionally, an owner of a restaurant can take picturesof their dishes to use on advertising materials or menus. These
people are often amateur photographers; however, since their
pictures will be seen and evaluated by the others, they want
the food to look as delicious as possible.
A professional photographer knows how to adjust composition [1], lighting, color, and focus, etc., so that theytake attractive pictures of food. In this paper, we consider
composition, and develop a system to suggest a desirable
composition to the user. The composition of a picture of foodis determined by three factors [2]: 1) the number of dishes,2) the shape of each dish, and 3) the ratio of the area offood to the area of the dish. Either the user provides thisinformation manually, or the system automatically recognizes
this information, having analyzed an image that the user hastaken on their mobile phone. The system then retrieves anappropriate composition from our database and shows it toFig. 1. Food photographs taken by users of our system. Each pair shows apicture taken by our system (left) and a suggested composition (right).

II. P REVIOUS WORK
A. Food recognition
Object recognition is a subject of active research in computer vision and pattern recognition. Before 2012, researchers
manually designed feature vectors for the recognition of
objects. For example, histograms of colors and image gradients remain popular and powerful features, e.g., SIFT [3],SURF [4], and ORB features [5]. These features are robustwith respect to variations in scale, rotation, and illumination,but often fail to recognize similar objects. Deep convolutional
neural networks (DCNN) were proposed in 2012 [6]. These automatically extract appropriate features from learning datasets.
By using these, we gain a 10% improvement in the accuracyof recognition. We use a network in network, a particular typeof DCNN, to implement our food recognizer [7]. This canrecognize a dish in less than 0.5 seconds. The accuracy is
74% in the case of the first-ranked dish, and 93.5% in thecase of the top-five-ranked dishes.
B. Evaluation of aesthetics
There are techniques to automatically evaluate the aesthetics
of a picture [8][9]. These techniques evaluate aesthetics using
DCNNs. These techniques have not been applied to theaesthetics of food pictures. In this paper we have not attempted
an automatic aesthetic evaluation; instead, we performed a
subjective evaluation of the quality of taken pictures.

oval, or square), and 3) the ratio of the area of the food to thearea of the dish.
We first observed and investigated pictures of delicious
looking food taken by skilled professional photographers. We
learned how they take such pictures [2]. We found that thecompositions of their pictures adhered to the following three
conditions:1.
The existence of side dishes.
2.
The shape of the main dish, which may be that of acircle, rectangle, oval, or square.
3.
The area of the main dish, which can be either greateror less than twice the size of the area of the food on
it. We call the former a ‘large’ dish and the latter a‘small’ dish.
Given the information about the dishes entered by the user
(Section III-A) or obtained by the automatic recognition (Section III-B), our system first determines the basic composition
based on the conditions described above, and then shows the
final composition to the user. First the situation is classified
according to the first condition: if there are no side dishes, we
use the rules defined in Table I; if there is at least one side dish,we use the rules defined in Table II. In both tables, a small
rectangular or oval dish has no basic composition, because wehave not found such a case through our observations.
TABLE IRULES WE USE IF THERE IS NO SIDE DISHES .

C. Image editing
Given an input image, an image retargeting technique rearranges objects within the image and modifies its composition.
Liu et al. retargeted the input image by evaluating its aesthetics
based on 1) the rule of thirds, 2) the diagonal composition, 3)
the balance of a picture, and 4) the size of each object [1].
This technique extracts parts from an input image but cannot
rearrange objects. Our system proposes rearrangements ofdishes to the user. The relighting technique allows the user
to modify the lighting conditions of an image. Ren et al. useda neural network for relighting [10], which requires several
hundreds of input images and is difficult for a novice user
to use. Depth of focus is also important for aesthetics. For
example, Liu et al. estimated the depth of a single image [11],which enabled them to later modify the depth of focus .
Relighting and modification of depth of focus are useful for
improving the aesthetics of a picture but are beyond the scopeof this paper and our future work.
III. TAKING A PICTURE OF FOODS
There are a number of guidelines for improving photographic composition to enhance the impact of a scene, e.g.,‘place your subject in the center’, ‘use the diagonals’ (Figure 1-a), ‘use the Fibonacci spirals’ (Figure 1-b), ‘balancing
with the rule of thirds’ (Figure 1-c), etc. We assume thatthese guidelines are also useful for food photography. We also
assume that the composition suitable for a scene is determinedby the following three parameters: 1) the number of side
dishes, 2) the shape of the dish (such as a circle, rectangle,

Shape of the main dishCircleRectangle or oval
Square

Size
Large
Small
Large
Small
Large
Small

Basic composition(a) Rule of thirds(b) Centering(c) Diagonal
N/A(d) Diagonal + centering(e) Diagonal + centering

TABLE II
RULES WE USE IF THERE IS AT LEAST ONE SIDE DISH .
Shape of the main dishCircleRectangle or oval
Square

Size
Large
Small
Large
Small
Large
Small

Basic composition(a) Diagonal + Fibonacci spiral(b) Diagonal + rule of thirds(c) Diagonal
N/A(d) Diagonal + centering(e) Diagonal

Figure 2 shows the final compositions in a case where there
are no side dishes. Figure 3 shows the final compositions in acase where there is at least one side dish. The system displays
the final composition on the screen of the user’s mobile phone.
We created a composition database to aid the specification of
the final composition. In this database the positions, rotations,and scales of the main dish and side dishes are registered forall possibilities admitted by the combination of items and thebasic compositions. In the case of a large circular main dishwithout any side dishes, we calculate the position and scale of

A. Manual input

Fig. 2. Each figure shows the final composition corresponding to the basic
composition in Table I. For example, when the main dish is large and circular,
the system determines the basic composition according to the rule of thirds,
and then makes a final composition like that shown in (a).

the main dish using the rule of thirds, as shown in Figure 2(a), which is registered in the database. Given a large circularmain dish with two side dishes, we calculate the positions andscales of the dishes based on the diagonal and the Fibonacci
spiral: we place side dishes along the diagonal and the spiral,and place the main dish at the end point of the spiral. In theFig. 3. Each figure shows the final composition corresponding to the basic
composition in Table II. For example, when the shape of the main dish is large
and circular, the system determines the basic composition to be a combinationof the diagonal and the Fibonacci spiral, and then suggests a final composition
similar to that shown in (a).

case of a rectangular side dish, we also take rotations into
account: the side dish rotates along the spiral (Figure 4). We

Before taking food pictures, the user can provide information about the dishes to be photographed manually by pushingbuttons (Figure 5). The user specifies the shape of the main
dish by pushing the ‘circle’, ‘rectangle or oval’, or ‘square’
button. The system also has buttons to specify the ratio of thearea of the dish to that of the food to indicate, for instance,
whether they have a ‘large’ or a ‘small’ dish. To provide
further information about each side dish, additional buttons
can be used; for example there are buttons that specify glasses
and cups as side dishes. (Figure 5).

Fig. 5. The user pushes the button to provide information about the dishes.
From left to right, the buttons to specify a circular shape, a rectangular oroval shape, a square shape, a glass as a side dish, and a cup as a side dish.

B. Automatic food recognition
Our system can recognize dishes automatically [7]. Whenusing this function, the user does not have to push buttons.
As this function recognizes the names of the items of foodon each dish, it enables our system to make more flexible
suggestions to the user.
When using this function, the user takes a shot of each dish,one by one. The system then shows the names of the top fiveranked dish names suggested by the automatic recognition.
The accuracy is 74% for the most highly ranked dish and93.5% for the top-five ranked dishes. The user then selects thecorrect name of the dish manually. The number of types offood that the system can recognize is 100. We also constructeda database containing the typical shape and size of each food
item and corresponding dish. For example, Chinese noodles
tend to be in a large circular dish and soup tends to be in asmall circular bowl. By querying the database with the nameof the food, we can determine the shape and size ratio ofthe dish. The drawback of this strategy is that we cannot takeunusual cases into account, for example, soup in a square dish.
In such a case, the user has to go to the manual input mode.
However, since such cases rarely happen, our simple strategy
based on this kind of mapping is still powerful in many of themore usual cases.
C. Taking a picture of foods

Fig. 4. The red line represents the Fibonacci spiral. The rectangular sidedish rotates along with the spiral.

also ascertain which camera angle will be best for taking thepicture. (Section III-C).

After providing information about the dishes, the systemshows the final composition on the screen of the user’s mobile
phone (Figure 6). The user arranges the dishes on the tableaccording to the composition so that each dish fits to thecorresponding object in the screen. As shown in Figure 1, thefinal composition overlays the image from the camera. Afterarrangement, the user adjusts the tilt of the camera and takes
the picture by pushing the shutter button. During this process,the ‘Tilt’ icon at the bottom right corner is useful: the color ofcould select the main dish from the two (Figure 8-right). Afterthe tutorial, we asked each subject to use our system to takepictures of these dishes. Figure 9 shows the pictures taken

Fig. 6.

The user interface to take the picture of foods.
Fig. 8.

the ‘Tilt’ icon changes to purple when the tilt of the camera
is close to the best tilt angle, as determined by the system.
IV. I NTERACTIVE DESIGN OF A NEW COMPOSITIONIn the case that the user wishes to photograph food items or
dishes that are not known to the system, our system allows theuser to interactively design a new composition. For example,
assume the user wants to design a composition of the dishes
shown in Figure 7 (a). The user first takes a picture of thedishes. The user then pushes the ‘rectangle or oval’ buttonof Figure 5, and the system enters the mode where the user
can design the rectangle by pinch-in and pinch-out gestures(Figure 7-b). After finishing the design of the rectangle forthe main dish, the user then designs the composition for theother dishes in a similar manner (Figure 7-c).

Fig. 7.

Dishes that we prepared for the subjective evaluation.

by the subjects using our system; even a novice user of oursystem could take pictures with a composition similar to thefinal composition (Figure 3) suggested by the system. Thisimplies that our system allows a novice user to quickly andeasily take a well-composed picture.

The user interface for designing a new composition.
Fig. 9.

Pictures taken by the subjects using our system.

V. R ESULTS AND DISCUSSION
We implemented our system on a mobile phone with theAndroid operating system (OS). Since the computational costsof our algorithms are low, our system works interactively andthe user can obtain immediate feedback from the system. We
performed subjective evaluations of the usability of our systemand the quality of the pictures taken using our system.
A. Usability study
The subjects were seven men in their twenties. They werealready familiar with operations on mobile phones with Android OS. No subjects had used our system previously. Firstof all, we gave each subject a short tutorial on our system.
We prepared two types of dishes for our experiments: one
consisted of a Japanese pork cutlet as the main dish, and rice,miso soup, and salad as side dishes (Figure 8-left). The otherconsisted of either Chinese fried rice or dumplings; the user

B. Quality assessment
In this experiment, we considered three groups of subjects:
groups A, B, and C. There were three subjects in group A,
three subjects in group B, and eighteen subjects in the group
C. We asked the subjects in group A to take pictures using oursystem while subjects in group B took pictures without using
our system. We told all the subjects to make the food in the
pictures look as delicious as possible whilst including all thedishes. Groups A and B used the same device to take pictures.
We then asked subjects in group C to evaluate the pictures
taken by groups A and B. Figure 10 shows the pictures. We
performed a subjective evaluation as follows: we showed eachrow of Figure 10 to the subject, and asked them to vote forthe picture with the most delicious looking food. Figure 10(a), 10 (c), and 10 (f) were taken by the group A, i.e., using

our system. Figure 10 (b), 10 (d), and 10 (e) were taken bygroup B, i.e., without using our system.
The results from the vote are shown in Figure 11. For eachrow of Figure 10, all the subjects voted for the picture taken bygroup A. We also asked each subject to vote for the best andworst pictures. The result is shown in Figure 12. Figure 10 (b)received 16 ‘worst’ votes. On the other hand, the ‘best’ votes
were distributed relatively uniformly among the pictures taken
by our system, which proves that our system enables users totake reasonably attractive pictures.
Fig. 12. The results of the vote for the best and worst pictures in Figure 10.

Fig. 13.

Fig. 10.

Pictures taken by subjects using our system.

Pictures of various foods taken by our system.

An interesting suggestion from one of the subjects regarding
the usability study was that they wanted to see multiple
composition candidates and then select their favorite. The extension of our system to show multiple candidate compositions
will be straightforward, so we will undertake this task in the
future.
We considered the composition of food items and disheswith regard to creating attractive pictures in this paper. Other
factors such as lighting and color correction will be consideredin future work. Also, we would like to implement a userinterface that enables depth of focus to be edited during postproduction.
R EFERENCES

Fig. 11.

The results of the vote for a picture from each row of Figure 10.

Finally, we show some other results: pictures of various
foods taken with the help of our system (Figure 13).
VI. C ONCLUSION
We proposed an interactive system to help people to takepictures of delicious looking food. Given information about
the dishes, such as the number, the shape, and the ratio of
dish to food, the system retrieves the best composition fromthe database and shows the suggestion to the user. Our system
also supports users in designing new compositions.

[1] L. Liu, R. Chen, L. Wolf, and D. Cohen-Or, “Optimizing photo composition,” Computer Graphics Forum, vol. 29, no. 2, pp. 469–478, 2010.
[2] N. S. Young, “Food photography: From snapshots to great shots,”
Peachpit Press, 2011.
[3] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
Int. J. Comput. Vision, vol. 60, no. 2, pp. 91–110, 2004.
[4] H. Bay, T. Tuytelaars, and L. V. Gool, “Surf: Speeded up robustfeatures,” in Proc. of ECCV, 2006, pp. 404–417.
[5] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “Orb: an efficient
alternative to sift or surf,” in Proc. of ICCV, 2011.
[6] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25, 2012, pp. 1097–1105.
[7] K. Yanai and Y. Kawano, “Food image recognition using deep convolutional network with pre-training and fine-tuning,” in Proc. of IEEE
International Conference on Multimedia Expo Workshops (ICMEW)
2015, 2015, pp. 1–6.
[8] X. Lu, Z. Lin, H. Jin, J. Yang, and J. Z. Wang, “Rapid: Rating pictorial
aesthetics using deep learning,” in Proc. of MM ’14, 2014, pp. 457–466.

[9] X. Lu, Z. Lin, X. Shen, R. Mech, and J. Z. Wang, “Deep multi-patch
aggregation network for image style, aesthetics, and quality estimation,”in Proc. of ICCV ’15, 2015.
[10] P. Ren, Y. Dong, S. Lin, X. Tong, and B. Guo, “Image based relighting
using neural networks,” ACM Trans. Graph., vol. 34, no. 4, pp. 111:1–
111:12, 2015.
[11] F. Liu, C. Shen, and G. Lin, “Deep convolutional neural fields for depth
estimation from a single image,” CoRR, vol. abs/1411.6387, 2014.