Shadow Cooking:Situated Guidance for a Fluid Cooking Experience
Ayaka Sato1, Keita Watanabe2, Jun Rekimoto1,3

2

1
The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan,
Meiji University, 4-21-1 Nakano, Nakano-ku, Tokyo 164-8525, 3 Sony Computer Science
Laboratories, 3-14-13 Higashigotanda, Shinagawa-ku, Tokyo 141-0022.

ayakasato@acm.org, watanabe@fms.meiji.ac.jp, rekimoto@acm.org

Abstract. Cooking is one of the most popular activities at home; however,preparing a new dish by reading a recipe is not a trivial task. People might lose
their current position in the recipe, misunderstand the required amount ofingredients, and generally become confused by the step that should be followed
next. Shadow Cooking guides users with situated, step-by-step information
projected on a kitchen counter. It consists of a depth camera and a projector,
which are installed above the kitchen counter. Shadow Cooking instructs theuser on the steps to follow by projecting information directly onto the utensils
and ingredients. The system also integrates a digital kitchen scale with therecipe such that the user is automatically prompted with the required weightbased on the ingredient currently being measured. In addition, we haveconnected the system with remote locations in order to enable a user tocommunicate with other cooks easily.

1Introduction

Cooking is a staple domestic activity around the world. Such activity provides afeeling of sufficiency; therefore, cooking is important beyond just providing
nourishment. However, there are many steps involved in cooking and it requires someknowledge, which makes cooking difficult, especially for beginners. Even people
with experience are likely to make mistakes when preparing a recipe for the first time.
Cooking by following a recipe is not a trivial task. A cook has to be aware of thesteps that have been completed, and be equally aware of the steps that follow next.
However, different cooking books will frequently list several steps within one
sentence or one block of text, such that the cook can sometimes lose their place in therecipe. Meanwhile, the cook might attempt to simultaneously focus on their current
activity and try to understand what to do next. This type of balancing can cause
mistakes, such as missing steps or inadvertently using the wrong amount of an
ingredient. Given that cooking is a real-time task, excessive efforts to understand therecipe can interrupt the cooking activity itself, which could result in an improperly
cooked dish.
 The other problem with current recipes is measurement. If the weight of anavailable ingredient (for example, meat) is different from the weight specified on arecipe, the amount of all the other ingredients must be recalculated accordingly.
However, recalculating ingredients is an extremely cumbersome task that can cause
mistakes when done incorrectly.
 We conducted a pilot study to observe the mistakes that are most likely to occur
with common written recipes. We used a muffin recipe, and both beginners andexperienced cooks participated. We found that beginners were not familiar withmeasuring units, such as grams, cups, spoons, and milliliters, and as a result, theyoften made mistakes by using the wrong measuring tools. Beginners were also
confused as to which utensil to use when the recipe only indicated to “mix” and useda spatula when a whisk was more appropriate. Both beginners and experienced cooks
became confused with units such as “tablespoon” and “teaspoon,” or inadvertently
missed steps, such as forgetting to use flour where the recipe indicated to “…mixsugar, flour, baking powder and oil…” These results illustrated to us that the current
style of recipes commonly used at homes can easily cause users to make mistakes.
In this paper, we propose a system that guides cooks by displaying a single step
directly onto the cooking environment; the step displayed varies according to theuser’s progress. This enables users to recognize immediately the step to follow, thus
helping the user to avoid mistakes and cook smoothly.

2

Related Work

Recently, several experimental systems have been developed to enhance the kitchen.
The method of integrating a recipe with a real kitchen is used in several systems [1,2, 3, 4, 5]. CounterActive [1] projects a recipe to a kitchen counter that the user canoperate by touching the counter. Panavi [4] is a sensor-embedded frying pan thatmanages user by recognizing the temperature of the frying pan. However, with thesesystems, users are still required to read instructions and be aware of their current stepin the recipe and the following step. Kitchen of the Future [6] installed several
monitors, cameras, and foot switches in an entire kitchen such that users were notrequired to see a recipe from a distance location. However, Kitchen of the Future does
not recognize user’s action. Cooking Navi [7] and Video CooKing [8] synthesize
multimedia such as videos and photographs in order to make text-based recipes more
understandable. These systems help users know detailed information. Some systems
specialize in recognizing user’s activities and objects within the kitchen [10, 11, 12].
Adding depth images to normal images enables such systems to recognize fine-grain
kitchen activities, such as mixing and the number of spoonful ingredients that havebeen poured into a cooking implement [11].
In contrast to these systems, our system is more focused on supporting the entire
cooking process according to the context of cooking, rather than merely recognizing
cooking activities.

Fig. 1. An operation scene from the Shadow Cooking system. Cooking instructions are directly
projected onto the kitchen counter and cooking objects. The instructions move forward
according to the user’s progress.

3

Shadow Cooking

As explained in the previous sections, the problem with written recipes lies in theseparation of the real cooking workspace and the recipe itself. Previous works thatmerged the kitchen with recipes did not attempt to solve this problem, and a cook is
still required to read the recipe, or the user must decide when to proceed to the next
step; therefore, it is still possible for the cook to commit mistakes. Our researchsupports a merged kitchen and recipes, and it guides users according to their situation.
Our system aims to decrease the possible mistakes and confusion in cooking in orderto facilitate a fluid cooking experience.
Our proposed system, called Shadow Cooking, is a system that projects a guidedirectly onto the work surface, in accordance with a real-time situation (Fig. 1). The
system recognizes the user’s current progress in the recipe and guides the next action
step-by-step. The real environment and the recipe are integrated in two ways. One isspatial (i.e., information is projected onto the position of actual ingredients) and theother is chronological (i.e., instruction information is projected according to thecurrent progress of actual cooking). Thus, we expect the cooking activity to proceed
uninterrupted, without the necessity to read a recipe. Our system aims to allow users
to place their full attention on the cooking activity.

3.1

System Configuration

Shadow Cooking consists of a computer, a depth camera, a projector, and a digital
kitchen scale (Fig. 2). The depth camera and the projector are installed above the
kitchen counter to detect objects and project a visual guide directly onto theingredients and utensils. A kitchen scale is installed with a Bluetooth chip that sends
weight information to the computer in real time. After a recipe is selected, a step-bystep guide is projected onto the objects according to the user’s current status in therecipe.

Fig. 2. Configuration of the system. The wireless digital kitchen scale tracks the quantity of an
ingredient that the user has poured into the scale.

3.2

Recognition of the User’s Cooking Context

The depth camera recognizes the existence of objects at specific locations on the
kitchen counter in order to determine whether the user is following the instructions.
The recognition is based on depth; therefore, once the user has placed an object on thespecified place and the depth has been recorded, the system would not fail torecognize objects because of hands moving over them. To distinguish objects, we set
the lower and higher height of each object; the system then recognizes the existenceof the objects when the depth value is within these two thresholds at the designatedposition on the kitchen counter. This is an extremely simple method, but it caneffectively detect typical incorrect situations, such as incorrectly placing a milk carton
instead of butter.
Fig. 3-(2) shows the recognition screen from the depth camera. The blue lines
indicate that an object exists in its designated position. Further, the digital kitchen
scale is used to recognize the currently used amount of ingredients. Steps move
forward by tracking the movement of objects and the currently used weight. For steps

that cannot be determined through these two actions, the user can simply place a handover the hand outline in order to move to the next step (Fig. 4-(2)).

Fig. 3. (1) A real scene from a kitchen counter, (2) a corresponding depth image forrecognition.

Fig. 4. Operation steps. (1) The user sets all the ingredients and utensils onto the projection.
(2) The steps move forward by detecting the movement of objects and the scale amount, or byplacing hands over an image as shown. (3) The user follows the instructions visualized withlines and some words. In this figure, the system is instructing the user to place 50 grams ofsugar into a bowl.

3.3

System Usage

The system follows the steps indicated in this section to guide the user.

Step 1: Preparation
After selecting a recipe, the user sets all the ingredients and utensils onto the shadows
as shown in Fig. 4-(1). After placing all the ingredients, the system will automaticallyproceed to the next step in the recipe.
Step 2: Adjustment (Optional)
Optionally, the user can adjust the quantity of the recipe according to the specificweight of an ingredient. To adjust, the user would pick up the ingredient from thecounter and place it on the scale; the amounts for the remaining ingredients areadjusted automatically.
Step 3: Cooking along with Shadow Cooking
After step 2, the user can place a hand over the hand outline to start cooking. The user
cooks based on arrows, numbers, and short instructions to progress. Steps proceed
according to the movements of objects. For example, if the step is to “Put 50 grams offlour into a bowl,” an arrow is directed from the flour to the bowl and a circle image
shows the remaining amount that needs to be added and the amount that has been
added already, as shown in Fig. 4-(3). When the user picks up the wrong ingredient oradds too much, an alert is displayed. When an ingredient is not needed anymore, its
shadow disappears from the counter, or the user is guided to wash the utensil to use it
again.
3.3

Trial

We conducted a trial to observe the performance of the system and to verify whether
the user interface is appropriate. Two subjects (a male and a female) used the system
to cook muffins. The male subject did not have any experience in cooking pastries;
the female subject cooks several times a year. We explained only the usage of astandard digital kitchen scale: it has two buttons, one is to power the apparatus on andoff; the other is to reset to zero grams any object on the scale. We did not explain how
to use the system. Moreover, we recorded videos from a camera above the kitchencounter and to the side in order to observe the participants’ handling of the utensils.
Additionally, we interviewed the subjects after the test.
Consequently, both subjects did not make any mistakes and both answered that there
was no difficulty understanding the instructions. However, the female participant
responded that when the instructions indicated to “Mix well,” she felt unsure as to
how well to mix. In addition, both subjects had an inclination to pour more liquid,
such as milk, than was necessary.

4

Remote Instruction

Novice cooks have a poor knowledge regarding cooking, and the information
provided in recipes is frequently inadequate to guide beginners; for example,
generally, recipes have no indication as to the proper heat level at which to cookcertain dishes, or how to cut ingredients. Further, beginners might be unaware of how
to manage unexpected circumstances, such as overheating, or a mixture not appearing
as expected. On the other hand, experienced cooks, who cook daily, have sufficientknowledge such that they can arrange recipes in their own way, and adjust the heatlevel and taste according to their experience. Novice cooks cannot perform suchchanges without experience, and experienced cooks lack the opportunity to share theirknowledge and sense of cooking.
With Shadow Cooking, we present another feature: experienced cooks can advise
other cooks who are cooking in real time and from remote locations. A USB camera
is attached to the ceiling of a cook’s kitchen, and the adviser can observe the actionsof the cook through video displayed from a touchscreen tablet (Fig. 5). The adviser
can then guide the cook in two ways: (1) through voice chat, and (2) throughannotation of hand-written messages using a conductive pen. For (1), users can
simply chat about cooking; for (2), when the adviser writes a message directly on thevideo using the tablet, the written contents are projected immediately onto the same
position on the cook’s kitchen counter. This annotation enables users to communicate
directly, and even demonstrate instructions that might be difficult to express using
words only. Moreover, the adviser can ensure that the cook is following instructions
correctly. We expect this feature to help novice cooks in acquiring more cooking
knowledge than their present experience. We also expect this feature to aid users ininheriting home cooking skills; for example, a mother can teach a child who lives
away from home.

Fig. 5. An adviser from a remote location advises a cook using live video from the camera
placed at the ceiling of the cook’s kitchen. The adviser can then speak and write instructions;
the written messages are then projected directly onto the kitchen counter.

5

Discussions

In this section, we discuss present problems and future work.
5.1

Supporting Other Cooking Processes

Our work assists the complete flow of cooking using weights and depth changes. For
the trial, we used a recipe for muffins, which mainly consisted of measuring andmixing ingredients. There are several other processes involved in cooking, such as
frying, steaming, and cutting. The processes that necessitate heat require temperature
management; however, it is difficult to determine the appropriate temperature, and theprocess cannot be reversed once a given dish has been overcooked. We consider that
previous studies, which visualize heat temperature by numerical methods [4] or heatmaps [14], are valid and users can evaluate visually whether they are applying thecorrect temperature. The process of cutting is also difficult, especially for novice
cooks, because there are several methods for cutting ingredients, for example,
chopping, slicing, and mincing; moreover, cutting requires practice. We consider that
it is effective to combine a video database, similar to the database that VideoCooking[8] incorporates into its system, or videos recorded by other cooks, such as CookTab
[15].
5.2

Combination of Other Measuring Tools

In the trial, we observed that the subjects poured too much liquid because pouring
speed is fast and it is difficult to control small amounts, compared with flour. The
authors of this work have previously developed a system called “smoon” [13], ameasuring spoon that automatically adjusts its capacity according to recipe data
(Fig. 6). We consider that this type of measuring tool is more suitable for liquids.

Fig. 6. Our previous work developed a utensil called “smoon,” which is ameasuring spoon that automatically adjusts its capacity according to recipe data.

5.3

Miniaturizing the System and Applying to Other Activities

The current system requires the installation of several devices in the kitchen.
Recently, USB cameras, projectors, and depth cameras can be miniaturized and canbe connected to a tablet. If the entire system is miniaturized and becomes portable, it
can be used for other situations, for example, to perform other activities on a table,
such as knitting, sewing, and soldering circuits.

6

Conclusion

In this paper, we proposed a cooking support system called “Shadow Cooking” thataims to integrate recipes with real-time cooking. Because the recipe steps areprogressively shown, and the position of ingredients and utensils spatially correspondto real ingredients and utensils placed on a kitchen counter, a user can concentrate on
the activity of cooking, instead of diverting their attention on attempting to understandthe recipe. Our user evaluation confirmed that this system helps users, and both
beginner as well as experienced cooks can prepare a dish equally correctly, without
any prior knowledge of the recipe.

References
1. Ju, W. Hurwitz, R., Judd, T., Lee, B. Counteractive: An Interactive Cookbook for the KitchenCounter. In Proc. CHI EA 2001, (2011), 269-270.
2. Suzuki, Y., Morioka, S. and Ueda, H, Cooking Support with Information Projection onto
Ingredient. In Proc. APCHI 2012, ACM Press (2012), 193-198.
3. Ikeda, S., Asghar, Z., Hyry, J., Pulli, P., Pitkanen, A., Kato, H. Remote assistance using
visual prompts for demented elderly in cooking. In Proc. ISABEL 2011. ACM Press (2011),
1–5.
4. Uriu, D., Namai, M., Tokuhisa, S., Kashiwagi, S., Inami, M. and Okude, N. Experience"panavi,": challenge to master professional culinary arts. In Proc. CHI 2012, ACM Press(2012), 1445-1446.
5. Mennicken, S., Karrer, T., Russell, P., and Borchers, J. First-person cooking: a dualperspective interactive kitchen counter. In CHI EA 2010, (2010), 3403–3408.
6. Siio, I., Mima, N., Frank, I., Ono, T. and Weintraub, H. Making recipes in the kitchen of the
future. In Proc. CHI EA 2001, ACM Press (2004), 1554-1554.
7. Hamaoka, R., Okabe, J., Ide, I., Satoh, S., Sakai, S., Tanaka, H. Cooking navi: assistant fordaily cooking in kitchen. In Proc.ACM Multimedia (2005), 371-374.
8. K. Doman, C. Y. Kuai, T. Takahashi, I. Ide, and H. Murase. Video CooKing: Towards thesynthesis of multimedia cooking recipes. In Advances in Multimedia Modeling, LNCS 6524,(2011), 135-145.
9. Chi, P.-Y. P., Chen, J.-H., Chu, H.-H., and Lo, J.-L. Enabling calorie-aware cooking in asmart kitchen. In Proc. PERSUASIVE ’08, (2008), 116–127.
10.Hooper C, J. Preston, A., Balaam, M., Seedhouse, P., Jackson, D., Pham, C., Ladha, C.,
Ladha, K., Plötz, T., Olivier, P. The French Kitchen: Task-Based Learning in an
Instrumented Kitchen, In Proc. UbiComp 2012, ACM Press (2012), 193-202.

11.Lei, J., Ren, X. and Fox, D. Fine-grained kitchen activity recognition using RGB-D, In
Proc. UbiComp 2012, AMC Press (2012), 208-211.
12. Ziola, R. Grampurohit, S., Nate L, Fogarty, J., and Harrison, B. OASIS: Creating Smart
Objects with Dynamic Digital Behavior. In Interacting with Smart Objects, Workshop at
ACM IUI 2011, (2011).
13.Watanabe, K., Sato, A., Matsuda, S., Inami, M., and Igarashi, T. smoon: A Spoon with
Automatic Capacity Adjustment. VRIC 2012 proceedings, Laval Virtual 2012 France,(2012).
14.Kita, Y., and Rekimoto, J. Thermal Visualization on Cooking. Proceedings of the 23rd
International Conference on Artificial Reality and Telexistence, (2013).
15.Sato, A., Tsukada, K., Siio, I. CookTab: smart cutting board for creating recipe with realtime feedback, In Proc. of UbiComp '12, ACM Press, 543-544(2012).