First Bite/Chew: distinguish typical allergic food by two IMUs
Juling Li∗

lijuling@mail.ustc.edu.cn
Keio University
Japan

Thad Starnerthadstarner@gmail.com
Georgia Institute of Technology
United States

Yifei Huang

hyf015@gmail.com
The University of TokyoJapan

Xiongqi Wang∗

aidan108wang@gmail.com
Keio UniversityJapan

George Chernyshov

send.letters.for.me.here@gmail.com
Keio University
Japan

Kai Kunze

kai@kmd.keio.ac.jp
Keio University
Japan

Junyu Chen∗

924342513@qq.com
The University of Tokyo
Japan

Jing Huang

hkoukenj@gmail.com
Keio UniversityJapan

Qing Zhang

qzkiyoshi@gmail.com
Keio University
Japan

Figure 1: First bite/chew are like the red, and yellow rectangles indicate; they are significantly different from the following
bites/chews, which are relatively the same among different food types.

ABSTRACT
Eating or overtaking allergic foods may cause fatal symptoms oreven death for people with food allergies. Most current food intake
tracking methods are camera-based, on-body sensor-based, microphone based, and self-reported. However, challenges that remain
are allergic food detection, social acceptance, lightweight, easy touse, and inexpensive.
Our approach leverages the first bite/chew and the corresponding hand movement as an indicator to distinguish typical typesof the allergic food. Our initial feasibility study shows that ourapproach can distinguish six types of food at an accuracy of 89.7%

over all four participants’ mixed data. Particularly, our method successfully detected and distinguished typical allergic foods such as
burgers (wheat), instant noodles (wheat), peanuts, egg fried rice,and edamame, which can be expected to contribute to not only
personal use but also medical usage.

CCS CONCEPTS• Human-centered computing → Interaction devices; Userinterface toolkits; • Hardware;

KEYWORDS∗ Three authors contributed equally to this research.

smart eyewear, food intake, diet monitoring

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
AHs ’23, March 12–14, 2023, Glasgow, United Kingdom© 2023 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9984-5/23/03.
https://doi.org/10.1145/3582700.3583708

326

ACM Reference Format:
Juling Li, Xiongqi Wang, Junyu Chen, Thad Starner, George Chernyshov,Jing Huang, Yifei Huang, Kai Kunze, and Qing Zhang. 2023. First Bite/Chew:
distinguish typical allergic food by two IMUs. In Augmented Humans Conference (AHs ’23), March 12–14, 2023, Glasgow, United Kingdom. ACM, NewYork, NY, USA, 4 pages. https://doi.org/10.1145/3582700.3583708

AHs ’23, March 12–14, 2023, Glasgow, United Kingdom

1Li, Wang, Chen and et al.

INTRODUCTION

Food intake monitoring is intensively explored, in which manual
food intake estimation methods contribute to poor accuracy [6, 15].
Manual methods also require significant time and effort and areprone to be forgotten or abandoned [11]. Smartphone methods arealso inconvenient as the user must stop eating to interact withthe phone. Such wearable agents [17] can significantly reduce theamount of time between a user’s first intention to do a task andtheir first action to do it [16], which greatly increases the likelihoodthat the interface will be used [18]. Another constraint for manyusers is that they do not want to wear devices that call unnecessary
attention to themselves or might cause onlookers to believe theuser has a disability [12].
This work’s key contribution is the concept of First Bite/Chewbased allergic food detection system. Our approach has several
benefits: (1) computational simplicity: our system only contains
two IMUs (one is built-in the following MCU) and an inexpensive
machine-learning capable MCU1 as the main parts, (2) easy-to-use:
to monitoring the different types of food intake, our approach does
not require extra manual practice and heavy load of learning, (3)replicable: our approach can be easily replicated concerning costand design, (4) socially acceptable appearance: as our method does
not require camera nor bulky computational heavy design, butmaintained an ordinary glasses-like appearance.

2

RELATED WORKS

Current food intake monitoring methods can be roughly categorized as IMU-based, IMU combined with other sensors, imageprocess-based, sound-based, and wearable on-body sensor based.
IMU sensor-based. Kim et al. [8] provided a smartwatch-based
method to address different eating patterns and food types and only
handled rice and noodle in their tests. Sound based. Sound-based
food intake detection has two main methods. The most studied one
is using microphones from hearing aids, earphones, or headsets
to capture users’ chewing noise and use it as an indicator of foodintake action[11]. Image-based. Regarding image-analysis-based
methods, processes like image segmentation, food recognition, andportion size estimation are required to complete food intake estimation [6]. Glasses-based chewing detection. Mertes et al. [9, 10]
developed a glasses-based method that detects the chewing motionof elderly people. Chung et al. [4] designed a pair of smart glasses
that can classify food intake motions from physical activities.
However, to our knowledge, none of those existing smart glassesbased studies could distinguish different types of food, especiallythose allergic to food that could be fatal threats.

3OUR APPROACH: FIRST BITE/CHEW BASEDFOODS INTAKE MONITORING

Our approach leverages the first bite/chew and the corresponding
hand movement as an indicator to distinguish the typical allergic
types of food. As Figure 2 shows, the concept of first bite/chew
contains two parts of signals. The first part (red rectangle) is thedata generated by biting/chewing as well as the head and muscles’
activities responding to the food during the first biting or chewing.

The other part (orange rectangle) is the movement data of thehand that grabs and moves the food during the first biting/chewing
period.
Hardware Design. To design the device that can obtain the firstbiting and/or chewing vibration via skull and muscles, we attachedan IMU2 on the inner side (near the head) of the glasses’ rightleg – close to the area of superior auricular muscle and temporalis
muscle referring [5]. See Figure 3. To also obtain the corresponding
movements of the hand during the biting/chewing, we leveraged
an IMU3 -embedded MCU4 and designed a wristband connected tothe IMU on the glasses’ leg with four wires via IIC-Bus. The device
connects to a MacBook pro laptop (i7 16gb) with a type-c cable.

4INITIAL FEASIBILITY STUDY

To evaluate the concept of first bite/chew, we conducted an initial
feasibility study having four participants (two male and two female),aged from 23 to 32, MEAN = 26.5, SD = 3.4. They were compensatedby gift cards for their time. Their eating activities data were recorded
during their self-reported general meal time. In total, we recorded
five meals for each participant in three days. Participants wererequired to confirm and sign a consent form and allergic foodchecking form, and they were also asked food and drink prohibitions
to make sure no one takes inappropriate foods.
Foods Selection. To examine the feasibility of our approach while
distinguishing allergic food from daily non-allergic food, we selected five types of the common allergic food. They are peanuts,edamame, burger, instant noodle, and egg (fried rice), among which
burgers and noodles both stand for the allergen of gluten. We also
selected one type of commonly non-allergic food, which is apple.
The food selection was based on Hefle et al. [7] and other related
works on food allergy [1–3, 13, 14, 14].

4.1

Result & ML Model Structure

For the food category recognition, a single-layer Neural Networkwith 20 fully connected neurons was adopted. We trained and tested
the classifier over four participants’ mixed data by using 80% as
training data and the remaining 20% as testing data. As the figure 5shows, the classifier achieves an overall accuracy of 89.7%. In detail,82.4% accuracy for distinguishing Burgers, 84.8% for Edamame,
96.6% for Egg fried rice, 87.7% for Noodles, 92.9% for Peanuts, and94.7% for Unknown food (apple).

5DISCUSSION & LIMITATION

Currently, the system is at risk of distinguishing allergic food mixed
in other materials, for the food texture can be somewhat different
from the original. Distinguishing foods with similar textures andeating manners is also challenging. To minimize the contact between allergic food and actual patients, as well as in terms of reallife usage, advanced machine learning designs such as Few-shotlearning could contribute to building ready-to-use by using the datagenerated from people who experience no or few food allergies.
2 MPU6050 based IMU modular module.
3 LSM6DS3

4 Seeeduino Xiao BLE sense. https://www.seeedstudio.com/Seeed-XIAO-BLE-Sense-

1 Micro Controller UnitnRF52840-p-5253.html

327

First Bite/Chew: distinguish typical allergic food by two IMUs

AHs ’23, March 12–14, 2023, Glasgow, United Kingdom

Figure 2: Examples of defining first bite/chew of different types of food.

Figure 3: Hardware design and the sensors placement.

Figure 4: Six types of the selected food.

Figure 5: Confusion matrix, F1 score and on-device performance.

6CONCLUSION

In this work, we proposed a computationally simple and easyrepeatable approach for automatic food intake monitoring by leveraging the First Bite/Chew as the indicator. The initial feasibility
evaluation reveals that our approach could distinguish five typicaltypes of allergic food (burger, noodle, peanut, egg fried rice, and328

edamame) during common eating styles with an average accuracyof 89.7%.

AHs ’23, March 12–14, 2023, Glasgow, United Kingdom

Li, Wang, Chen and et al.

ACKNOWLEDGMENTS
This work is partly supported by IoT Accessibility Toolkit JST
Presto Grant Number JPMJPR2132; JST SPRING, Grant Number
JPMJSP2123.

REFERENCES[1] Barbara K Ballmer-Weber and Stefan Vieths. 2008. Soy allergy in perspective.
Current opinion in allergy and clinical immunology 8, 3 (2008), 270–275.
[2] S Allan Bock and FM Atkins. 1989. The natural history of peanut allergy. Journalof Allergy and Clinical Immunology 83, 5 (1989), 900–904.
[3] Jean-Christoph Caubet and Julie Wang. 2011. Current understanding of egg
allergy. Pediatric Clinics 58, 2 (2011), 427–443.
[4] Jungman Chung, Wonjoon Oh, Dongyoub Baek, Sunwoong Ryu, Won Gu Lee,and Hyunwoo Bang. 2018. Design and evaluation of smart glasses for food intake
and physical activity classification. JoVE (Journal of Visualized Experiments) 132(2018), e56633.
[5] Abul Doulah, Tonmoy Ghosh, Delwar Hossain, Masudul H Imtiaz, and Edward
Sazonov. 2020. “Automatic ingestion monitor version 2”–a novel wearable device
for automatic food intake detection and passive capture of food images. IEEE
Journal of Biomedical and Health Informatics 25, 2 (2020), 568–576.
[6] Hamid Hassannejad, Guido Matrella, Paolo Ciampolini, Ilaria De Munari, MonicaMordonini, and Stefano Cagnoni. 2017. Automatic diet monitoring: a review ofcomputer vision and wearable sensor-based methods. International journal offood sciences and nutrition 68, 6 (2017), 656–670.
[7] Susan L Hefle, Julie A Nordlee, and Steve L Taylor. 1996. Allergenic foods. CriticalReviews in Food Science & Nutrition 36, S1 (1996), 69–89.
[8] Hyun-Jun Kim, Mira Kim, Sun-Jae Lee, and Young Sang Choi. 2012. An analysisof eating activities for automatic food type recognition. In Proceedings of the2012 asia pacific signal and information processing association annual summit and329

conference. IEEE, 1–5.
[9] Gert Mertes, Hans Hallez, Tom Croonenborghs, and Bart Vanrumste. 2015. Detection of chewing motion using a glasses mounted accelerometer towards monitoring of food intake events in the elderly. In International Conference on Biomedicaland Health Informatics. Springer, 73–77.
[10] Gert Mertes, Hans Hallez, Bart Vanrumste, and Tom Croonenborghs. 2017. Detection of chewing motion in the elderly using a glasses mounted accelerometerin a real-life environment. In 2017 39th Annual International Conference of theIEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 4521–4524.
[11] Sebastian Päßler, Matthias Wolff, and Wolf-Joachim Fischer. 2011. Food intake
recognition conception for wearable devices. In Proceedings of the First ACM
MobiHoc Workshop on Pervasive Wireless Healthcare. 1–4.
[12] Halley Profita, Reem Albaghli, Leah Findlater, Paul Jaeger, and Shaun K Kane.
2016. The AT effect: how disability affects the perceived social acceptability ofhead-mounted display use. In proceedings of the 2016 CHI conference on humanfactors in computing systems. 4884–4895.
[13] Hugh A Sampson, Louis Mendelson, and James P Rosen. 1992. Fatal and near-fatal
anaphylactic reactions to food in children and adolescents. New England Journalof Medicine 327, 6 (1992), 380–384.
[14] Jessica H Savage, Allison J Kaeding, Elizabeth C Matsui, and Robert A Wood. 2010.
The natural history of soy allergy. Journal of Allergy and Clinical Immunology
125, 3 (2010), 683–686.
[15] Dale A Schoeller. 1995. Limitations in the assessment of dietary energy intake
by self-report. Metabolism 44 (1995), 18–22.
[16] Thad Starner. 2013. Project glass: An extension of the self. IEEE Pervasive
Computing 12, 2 (2013), 14–16.
[17] Thad E Starner. 2002. Wearable agents. IEEE pervasive computing 1, 2 (2002),90–92.
[18] Thad E Starner, Cornelis M Snoeck, Benjamin A Wong, and R Martin McGuire.
2004. Use of mobile appointment scheduling devices. In CHI’04 Extended Abstracts
on Human Factors in Computing Systems. 1501–1504.