Automatic Detection of the Chewing Side Using
Two-channel Recordings under the Ear
Akihiro Nakamura,
Hiroshi Mineno, Masafumi Nishimura
Shizuoka University
3-5-1 Johoku, Naka-ku, Hamamatsu, Shizuoka, 432-8011, Japancs16067@s.inf.shizuoka.ac.jp
mineno@inf.shizuoka.ac.jp
nisimura@inf.shizuoka.ac.jp

Takato Saito, Daizo Ikeda, Ken Ohta
Research Laboratories, NTT DOCOMO, Inc.
3-5 Hikari-no-oka, Yokosuka, Kanagawa, 239-8536, Japan
takato.saitou.bu@nttdocomo.com
ikedad@nttdocomo.com
ootaken@nttdocomo.com

Abstract—Eating behavior is an important parameter of thestate of health. A previous study confirmed that the method ofrecording eating sounds under the ear along with the long shortterm memory-connectionist temporal classification (LSTM-CTC)
were effective in detecting chewing events. This study examined
the possibility of identifying the left and right sides of chewing to
improve the analytical ability of eating behavior. More accurate
detection was achieved through the utilization of the two-channel
recordings and their cross-correlation as a new feature thanthrough the conventional mel-frequency cepstral coefficients(MFCC) features.

chewing, and others. The last category included front chewing
and swallowing.
Fig. 1 shows the flow of the automatic detection of left andright chewing proposed in this paper.

Keywords— Cross-correlation, LSTM-CTC, Event Detection,
Chewing

I. INTRODUCTION
A decrease in the quality of eating behavior may adversely
affect health. The reduction of meal quantity is associated withthe lowering of a person’s immunity and overeating that causes
conditions such as obesity and lifestyle illness. Chewing
behavior is particularly important, and people who chew lessfrequently and eat quickly are more prone to obesity. Therefore,
the study of a system that can monitor a series of eating
behaviors in detail is vital from the viewpoint of healthmaintenance.
Many studies have examined the monitoring of eating
behavior. Ando et al. [1] classified dietary behavior using theGaussian Mixture Model, but the model used a small amount ofdata and did not show high performance. The correct label isnecessary for each frame of the input in conventional machinelearning, and the cost of labeling is a problem. Graves et al. [2]proposed a model that can learn input and output sequences ofdifferent lengths using long short-term memory (LSTM) withconnectionist temporal classification (CTC) as the loss function.
This method does not require an accurate label for each frame.
Billah et al. [3] confirmed that the combination of LSTM-CTC
and masticatory sounds recorded through a single-channel
condenser microphone under the ear was effective in detecting
chewing behavior.
However, no effective method could be devised toautomatically detect the balance of the chewing. Partial chewing
can cause tooth loss, facial distortion, etc. Shogetsu et al. [4]investigated the possibility of detecting the chewing side using amyoelectric potential sensor and a microphone and confirmed
the existence of a difference in sounds between left and rightchewing. To analyze the quality of eating behavior in terms of
the balance of the chewing, this paper proposes a method thatcan detect the chewing side automatically and examines itsperformance.
II. PROPOSED METHOD
The collected eating sounds were utilized to classify each
event into the following three divisions: left chewing, rightFig. 1. Flow of automatic detection of left and right chewing

First, the eating sound was recorded using a two-channel
condenser microphone under the ear (16 bit, 22 KHz sampling).
The installation example is shown in Fig. 2. In addition, theLSTM-CTC was trained with weak labels for left and rightchewing. An online application was developed to reduce the costof labeling. The subjects generated an event log by pressing keys
on the keyboard corresponding to left/right chewing andswallowing at the same time as eating, and the log was used todenote the weak labels. The weak labels did not carry the correcttime information.

Fig. 2. Installation of the two-channel condenser microphone under the ear

Next, the feature extraction was accomplished through theuse of two-channel signals recorded under the ear to detect thechewing side. The MFCC of the sum signal of the left and rightchannels was estimated as an enhancement processing of theobservation signal. The MFCC was estimated through thewindow width of 80 ms and the frame shift of 40 ms, and it wascombined with the cross-correlation of two-channel signals.
The features were then inputted into the LSTM-CTC, anautomatic detector, and the left and right chewing sequenceswere estimated. Since the length of the input/output sequence
was different, learning through the weak labels without the timeseries information became possible in the LSTM-CTC.
Finally, the LSTM-CTC output was smoothed. If there wasa change of chewing side in less than one second, it was modified

to record chewing only on one side and frequent changes inchewing sides within short spans were avoided.

TABLE III.

FOOD TYPE USING THE PROPOSED METHOD

Food TypeCrackers
Chewing Gum
Cabbage

III. EVALUATION
A. Experimental Condition
The eating sounds of chewing gum, crackers (Ritz), andcabbage (shredded) were recorded from 18 subjects in their 20s.
The number of collected weak labels was 17691 for chewing(left 8871 times, right 7718 times, front 1102 times) and 1982
for swallowing.
The proposed method was compared with the MFCC of theleft signal, concatenation of the two MFCCs of two-channelsignals, and MFCC of the sum of the two-channel signals. The
MFCC consisted of 12 units with 1 dimensional Root Mean
Square (RMS), 13 dimensional Δ and 13 dimensional ΔΔ.
Additionally, the cross-correlation was estimated at 3, 7, and 15
point shifts of the two-channel input.
The detection performance was evaluated through theperforming of a ninefold cross validation with 16 subjects oflearning data and 2 subjects of test data. The mean absolute
percentage error (MAPE) of left and right chewing performed in
the eating of one cracker was used as an evaluation metric.
MAPE is calculated by the following equation, where 𝐴𝑘 is thenumber of correct answers, 𝐹𝑘 is the number of estimates, and 𝑁
is the number of frames for evaluation.

DETECTION PERFORMANCE OF LEFT AND RIGHT CHEWING BY
Recall
0.72
0.78
0.73

Precision
0.79
0.80
0.88

F-measure
0.75
0.79
0.80

Fig. 3 illustrates the changes recorded in the F-measure overtime in left and right chewing of specific food items. The
chewing time was normalized for each food by dividing thewhole chewing time into five equal parts. Then, the average ofthe F-measure for left and right chewing was estimated. It can besaid that the chewing of gum evidenced little change in the Fmeasure with time, and the chewing sound was stable.
Conversely, the F-measure for crackers and cabbage tended tobe higher in the first half of chewing and lower in the latter half.
Although the left or right correct labels were accorded to eachchewing side according to the subject’s declaration, it wasconsidered that the latter half of the chewing was affected by thefact that the chewing position became unclear due to theexpansion of the food.

𝑁MAPE =100𝐴𝑘 − 𝐹𝑘∑|
|𝑁𝐴𝑘𝑘The Recall, Precision, and F-measure of the event unit were also
employed for evaluation.
B. Experiment Results and Discussion
Table Ⅰ demonstrates the average error value per frame of thechewing side when one cracker was eaten. The MFCC of the left
signal evinced a large error because it did not include theinformation of the chewing side. The proposed method (acombination of the MFCC of the sum of the signals and a crosscorrelation at the 7-point shift) exhibited the highest
performance.
TABLE I.
AVERAGE ERROR VALUE PER FRAME OF CHEWING SIDE:
THREE-CLASS (LEFT CHEWING/RIGHT CHEWING/OTHERS) DETECTION
MAPE (%)

Feature Type

BeforeSmoothing

After
Smoothing

MFCC of the Left Signal
Concatenation of 2 MFCCs of Two-ch Signals
MFCC of Sum of Two-ch Signals
Proposed* Method (15 Point)
Proposed* Method (7 Point)
Proposed* Method (3 Point)

97.51
70.07
65.57
48.13
43.24
55.02

81.26
59.99
55.43
27.81
26.64
48.53

Fig. 3. Relationship between normalized elapsed time and F-measureusing the proposed method

IV. CONCLUSION
The present study posits a mechanism for automatic
detection of left and right chewing. The results of the experiment
reveal that the method using the combination of the MFCC of
the sum of signals and cross-correlation obtained high accuracy,
and the objective of the automatic detection of left and rightchewing was achieved to some extent.
Future investigations will be undertaken to verify whether
the joint use of attention in CTC could enable detection
reflecting a complicated action history that includes front
chewing and swallowing. Furthermore, the practicability will be
examined and the evaluation will be conducted using the actual
eating sounds without specifying the food item.
ACKNOWLEDGMENT* Proposed: MFCC of sum of two-ch + Cross-correlation

Table Ⅱ displays the detection performance of left and rightchewing after smoothing. The highest performance was recorded
using the proposed method.
TABLE II.

DETECTION PERFORMANCE OF LEFT AND RIGHT CHEWING
AFTER SMOOTHING:
THREE-CLASS (LEFT CHEWING/RIGHT CHEWING/OTHERS) DETECTION
Feature Type

MFCC of the Left Signal
MFCC of Sum of Two-ch SignalsProposed* Method (7 Point)

Recall
0.47
0.63
0.74

Precision
0.62
0.80
0.82

F-measure
0.49
0.72
0.78

This work was partially supported by JSPS KAKENHI Grant
Number 18H03260.
REFERENCES
[1]

[2]

* Proposed: MFCC of sum of two-ch + Cross-correlation[3]

Table Ⅲ indicates the detection performance of left and rightchewing according to food items in terms of the proposed
method. The difference of detection accuracy vis-à-vis the food
item variation was small.

[4]

Jumpei Ando, Takato Saito, Satoshi Kawasaki, Masaji Katagiri, Daizo
Ikeda, Hiroshi Mineno, Takashi Tsunakawa, Masafumi Nishida,Masafumi Nishimura, “Dietary and Conversational Behavior Monitoring
by Using Sound Information,” NCSP 2018, pp.675-678, 2018.
A. Graves, S. Fernandez, F. Gomez, J. Schmid huber, “Connectionist
Temporal Classication:Labelling Unsegmented Sequence Data with Re
current Neural Networks,” Proc. Int. Conf. on Machine Learning, pp. 369376, 2006.
Muhammad Mehedi Billah,Taiju Abe, Akihiro Nakamura, Takato Saito,Daizo Ikeda, Hiroshi Mineno, Masafumi Nishimura, “Estimation ofNumber of Chewing Strokes and Swallowing Events by Using LSTMCTC and Throat Microphone,” Proc. of GCCE 2019, pp.944-945, 2019.
Ryosuke Shogetsu, Tsutomu Terada, Masahiko Tsukamoto,“Consideration to Prevent Bias Chews and Increse Chewing Count withSensor,” IEICE technical report, pp.1-6, 2018. (in Japanese)