eat2pic: Food-tech Design as a Healthy Nudge
with Smart Chopsticks and Canvas
Rei Nakaoka1? , Yugo Nakamura1? , Yuki Matsuda1 , Shinya Misaki1 , Keiichi Yasumoto1
1

Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan
Email: {y-nakamura, yasumoto}@is.naist.jp

Abstract—In this paper, we introduce a food-tech design asa healthy nudge called eat2pic that encourages people to slow
down their eating pace and encourage healthier eating habits. The
eat2pic system is composed of a sensor-equipped chopstick (one ofa pair) and components that use context-aware digital canvases.
The eat2pic system achieves (1) automatic tracking of what andhow fast the user consumed each mouthful of food, (2) real-timevisual feedback that indicates good or bad eating behavior. The
key concept of eat2pic involves extending the relationship between
humans and paintings in order to establish a closed-loop in whichdaily eating behavior is reflected in pictures on the canvas near
us and feedback from the canvas leads our lifestyles towards
healthier diets. The eat2pic system provides a novel experience inwhich users can come to see eating as a playful task that involves
coloring landscape pictures, and not simply taking nutrition into
their bodies.
Index Terms—Nudge, Eating activity recognition, Persuasive
technology, Internet of things, Food technology

I. I NTRODUCTION
Recently, the World Health Organization (WHO) have highlighted the importance of healthy diets in relation to maintaining strong immune systems and avoiding or minimizing
chronic diseases and infections [1]. In particular, eating too
fast is a bad habit that should be discouraged since it can cause
both indigestion and overeating. In contrast, eating slowly
improves both digestion and hydration, facilitates weight loss
and maintenance, and increases food satisfaction. However,establishing healthy eating habits is not easy for many people,
which means that formulating solutions that encourage healthyeating habits such as eating slower and choosing a balanced
diet is essential.
To address these issues, various approaches have been proposed in the research areas of pervasive computing (PerCom),ubiquitous computing (UbiComp), and human-computer interactions (HCI). Currently, the most common approach involves
eating activity recognition using wearable or environmental
sensors and visualizing the results through smartphone applications. However, existing detailed diet monitoring methods
require users to wear special devices on their heads, necks,
or chests, which may interfere with their eating behaviors[2]–[5]. Separately, there are prior works [6]–[8] that focus
on smart table and mat, but none of those have been able
to detailed diet monitoring that recognizes both what andhow each bite was consumed. Other studies have found thatquantitative visualization methods centered on charts, graphs,* Rei Nakaoka and Yugo Nakamura are co-first authors.





・Eat slow
・Balanced diet


Change・Eat fast
・Unbalanced diet   

Eating

Painting

Chopsticks

Brushes

DishesPaletteFoodstuffs

ColorsFig. 1. An overview of the eat2pic concept.

and statistical reports tend to reduce user motivation levels [9]–[12].
In this paper, we introduce a food-tech design as a healthy
nudge called eat2pic that aimed to help people to slow
down their eating pace and encourage healthier eating habits.
The eat2pic system is composed of a sensing component
in the form of a sensor-equipped chopstick and intervention
components that use context-aware digital canvases as shownin Fig. 1. The eat2pic system provides (1) automatic trackingof what and how fast the user consumed each mouthful, (2)real-time visual feedback that indicates good or bad eating
behavior. The key concept behind our eat2pic system involves
extending the relationship between humans and paintings,
establishing a closed-loop in which daily eating behaviors
are reflected in those paintings, and permitting those reflectedbehaviors to “nudge” our lifestyles towards healthier diets.
II. OVERVIEW OF EAT 2 PIC
A. Approach
We considered the best approach for establishing healthy
human eating habits based on the Fogg behavior model [13],[14], in which three elements must meet at the same moment
for the desired behavior to occur. These elements are motivation, ability, and prompts. When an intended behavior does
not occur, the model states that at least one of those threeelements is missing. We also presume that most people alreadyhave the ability to establish healthy eating habits because thetask of preparing a balanced diet and eating it slowly is notparticularly onerous. Therefore, we hypothesize that the lackof healthy eating habits displayed by most people may be
due to a shortfall of personal motivation and/or insufficient
prompts within their personal living space. For these reasons,we adopted an approach that influences user motivation levels
by reframing eating tasks into other playful tasks and drawusers’ attention to behavior at appropriate times (e.g., whendetected speed eating) to encourage healthy eating behaviors.
Specifically, we employed two types of nudge mechanisms
called just-in-time prompts and ambient feedback [15].

  ­C. Implementation
The architecture of the eat2pic system is shown in Fig. 2,
where it can be seen that the system consists of a sensorequipped chopstick (one of a pair), a gateway device, and thedigital canvases. The sensor-equipped chopstick is equippedwith IMU sensors (MetaMotionR+* : sensors — 100 Hz
quaternion three-axis accelerometer/gyroscope; dimensions —L29×W18×H6 mm; weight — 5.7 g) for eating behavior
sensing, and a small camera (endoscope inspection camera† :resolution — 640×480 px; frame rate — 4–5 fps) to capture
food imagery before eating, as shown in Fig. 3-A.
The IMU sensor signals and camera data (image sequences)are streamed to the gateway device via Bluetooth Low Energy(BLE) and cable links, respectively. The gateway device thenanalyzes the IMU sensor signal and camera data simultaneously in order to track the user’s eating speed, the type(color) of the foods eaten, and the amount consumed in one
bite. More specifically, the gateway device performs the eating
activity recognition pipeline that consists of a time-series
signal processing part and an image recognition part basedon deep learning.
In the time-series signal processing part, the posture angleof the chopsticks is first calculated from the acceleration andgyroscope signals obtained from the IMU sensor, and theamount of rotation during eating. Then, the pipeline applies
peak detection processing to this data to identify the user’s* MbientLabs: https://mbientlab.com/metamotionr/† KKmoon: https://www.kkmoon.com/

­  ­ 

B. System Design
Based on the above consideration, we designed a nudging
system that monitors their “eating” behaviors in the physicalworld and reflecting them in a digital “painting” world, i.e.,“eat” to “pic” (eat2pic). The Fig. 1 shows an overview of theeat2pic concept design, in which we focused on an analogy
between eating and painting behaviors as a means to nudge
users toward improved eating habits. The root of our concept
is based on the realization that both eating and painting
behaviors possess similar elements; in this case, chopsticks andpaintbrushes, dishes and palettes, and food types and colors.
To design a technology that integrates smoothly into thedining setting, we decided to use chopsticks (which are
typically used during meals) as our sensor and employ pictures(which typically adorn living spaces) as actuators. The sensorequipped chopstick recognizes the eating speed of the user,
the type (color) of food selected, and the amount of foodconsumed simultaneously using a time-series signal processing
and deep learning approach. The eat2pic system encourages
the user’s healthy diet by representing the progress of a healthy
diet with gradually coloring to a landscape picture dependingon the user how to eat rather than showing quantitativevisualizations such as graphs.

  ­





    ­    ­ ­­ ­   ­ 
 

­­        







­
 ­
­

Fig. 2. System overview of eat2pic.
A

IMU sensor

B

C

Small camera

Fig. 3. Real setup of eat2pic system.

food eating behavior. In addition, the pipeline applies a deeplearning-based image recognition process. This process isperformed on every frame of the imagery from the camera(4-5 fps). For each frame, the object recognition model usedin this process (which we developed based on the VGG16 [16]model from ImageNet [17] and YOLO v3 [18] recognizes thetype and color of food, the size of the portion (small, medium,or large), and whether or not the food portion has been placedin the user’s mouth.
The pipeline determines the point at which the user consumed the food when a peak is detected in the time-series
signal processing part and when the food and mouth are recognized simultaneously in the image recognition part. The user’s
eating interval and speed are calculated from the detected
intake timing. In addition, the pipeline recognizes the type offood and the size of the portion in the view angle of the frame
before the detected timing. As a result, the pipeline outputs (1)eating interval/speed, (2) type/color of food, and (3) amountconsumed in one bite.
The gateway device also has a web server that controls
modifications to the landscape picture (canvas view) displayedon the digital canvas based on the recognized user’s eating
behavior. In the canvas view, the color of the food consumed
is reflected by the color of the applied paint, while eating speed
is reflected in the color mixture, and the amount in each biteis reflected in the painting speed.
III. EAT 2 PIC DEMONSTRATIONIn the demonstration, we present examples of scenarios thatshow how eat2pic helps people to slow down their eating paceand encourage healthier eating habits.
Fig. 4, 5 shows an example of how real-time visual feedbackof the eat2pic system works. In our scenario, the digital canvasof eat2pic is mounted on the wall next to the dining table, andusers can enjoy their mealtime while looking at a landscapepainting as shown in Fig. 3-B,C. In the eat2pic system, thecolors of the foodstuffs eaten by the user are reflected as parts

ACKNOWLEDGMENT

Healthy eating(w/ ideal speed)This work was supported by Japan Science and Technology
Agency (JST) ACT-I (JPMJPR18UK).
R EFERENCES

time
Unhealthy eating(w/ fast speed)... painting is not completed, and color is mixed

Fig. 4. Example of coloring applied based on eating speed. If the user eats
too quickly, multiple colors will be mixed and applied, resulting in a poor
visual appearance.
Short-term eat2pic

Healthy meal

(w/ various colors)

Unhealthy meal
(w/ few colors)

Fig. 5. Examples of coloring results on the canvas after different meal types.
If the user chooses a healthy (balanced) meal, the canvas will be filled withbeautiful colors. In contrast, there will be many unfilled places on the canvas
after an unhealthy (unbalanced) meal.

of the landscape paintings on the digital canvas. At this point,
the way the colors are applied will change depending on theway the user eats. For example, if a user rushes to eat a meal,
multiple colors will mix on one piece and the appearance ofthe landscape will degrade, as shown in Fig. 4. In order topaint clean colors, the user will need to take sufficient time(from 10 to 30 seconds) for each bite, which will motivate
him or her to eat slowly. The eat2pic system determines thewaiting time per intake based on the amount of food brought
to the mouth at one time. These interactions also encourage
users to eat more slowly.
In the eat2pic system, completing landscape picture par
one meal is designed as goals for a healthy diet. Since alandscape picture is composed of a variety of colors, the user
is required to eat meals that contain various foods in order tofill the color of all pieces. Fig. 5 shows examples of painting
results on the canvas after different meal types. If the user
chooses a healthy (balanced) meal, the canvas will be filledwith beautiful colors. In contrast, there will be many unfilledplaces on the canvas after the meal if the user chooses anunhealthy (unbalanced) meal. Herein, we intend that seeing
unfinished landscape paintings placed in the living space ona daily basis will motivate the user to color in the missing
pieces. For example, if a user looks at a picture with unfilledgreen areas and notices that low intake of green foods he or
she can choose to add a salad to his or her dinner.

[1] “World
healthorganization
nutrition
adviceforadultsduringthecovid-19
outbreak.”[Online].
Available: http://www.emro.who.int/nutrition/nutrition-infocus/nutritionadvice-for-adults-during-the-covid-19-outbreak.html[2] M. Mirtchouk, C. Merck, and S. Kleinberg, “Automated estimation offood type and amount consumed from body-worn audio and motion sensors,” in Proceedings of the 2016 ACM International Joint Conference
on Pervasive and Ubiquitous Computing, 2016, pp. 451–462.
[3] A. Bedri, R. Li, M. Haynes, R. P. Kosaraju, I. Grover, T. Prioleau, M. Y.
Beh, M. Goel, T. Starner, and G. Abowd, “Earbit: using wearable sensors
to detect eating episodes in unconstrained environments,” Proceedings of
the ACM on interactive, mobile, wearable and ubiquitous technologies,
vol. 1, no. 3, pp. 1–20, 2017.
[4] R. Zhang and O. Amft, “Monitoring chewing and eating in freeliving using smart eyeglasses,” IEEE journal of biomedical and healthinformatics, vol. 22, no. 1, pp. 23–32, 2017.
[5] A. Bedri, D. Li, R. Khurana, K. Bhuwalka, and M. Goel, “Fitbyte:
Automatic diet monitoring in unconstrained situations using multimodal
sensing on eyeglasses,” in Proceedings of the 2020 CHI Conference on
Human Factors in Computing Systems, 2020, pp. 1–12.
[6] B. Zhou, J. Cheng, M. Sundholm, A. Reiss, W. Huang, O. Amft,and P. Lukowicz, “Smart table surface: A novel approach to pervasive
dining monitoring,” in 2015 IEEE International Conference on Pervasive
Computing and Communications (PerCom). IEEE, 2015, pp. 155–162.
[7] B. Zhou, J. Cheng, P. Lukowicz, A. Reiss, and O. Amft, “Monitoring
dietary behavior with a smart dining tray,” IEEE Pervasive Computing,vol. 14, no. 4, pp. 46–56, 2015.
[8] G. Mertes, L. Ding, W. Chen, H. Hallez, J. Jia, and B. Vanrumste,“Measuring and localizing individual bites using a sensor augmented
plate during unrestricted eating for the aging population,” IEEE Journalof Biomedical and Health Informatics, vol. 24, no. 5, pp. 1509–1518,
2019.
[9] G. L. Cohen and D. K. Sherman, “The psychology of change: Selfaffirmation and social psychological intervention,” Annual review ofpsychology, vol. 65, 2014.
[10] E. L. Murnane, X. Jiang, A. Kong, M. Park, W. Shi, C. Soohoo, L. Vink,
I. Xia, X. Yu, J. Yang-Sammataro et al., “Designing ambient narrativebased interfaces to reflect and motivate physical activity,” in Proceedingsof the 2020 CHI Conference on Human Factors in Computing Systems,
2020, pp. 1–14.
[11] N. Daskalova, K. Desingh, A. Papoutsaki, D. Schulze, H. Sha,and J. Huang, “Lessons learned from two cohorts of personal
informatics self-experiments,” Proc. ACM Interact. Mob. Wearable
Ubiquitous Technol., vol. 1, no. 3, Sep. 2017. [Online]. Available:
https://doi.org/10.1145/3130911[12] L. J. Hinyard and M. W. Kreuter, “Using narrative communication
as a tool for health behavior change: A conceptual, theoretical,
and empirical overview,” Health Education & Behavior, vol. 34,
no. 5, pp. 777–792, 2007, pMID: 17200094. [Online]. Available:
https://doi.org/10.1177/1090198106291963[13] B. J. Fogg, “A behavior model for persuasive design,” in Proceedings ofthe 4th international Conference on Persuasive Technology, 2009, pp.
1–7.
[14] “Foggbehavior
model.”[Online].
Available:
https://www.behaviormodel.org[15] A. Caraban, E. Karapanos, D. Gonçalves, and P. Campos, “23 ways tonudge: A review of technology-mediated nudging in human-computerinteraction,” in Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems, 2019, pp. 1–15.
[16] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[17] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet:
A large-scale hierarchical image database,” in 2009 IEEE conference on
computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
[18] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,”arXiv preprint arXiv:1804.02767, 2018.