AR tooth brushing game to promote children’s oral
care habits through object tracking
Haruki Kondo

Kenji Funahashi

Shinji Mizuno

Nagoya Inst. of Tech., Japanh.kondo.801@stn.nitech.ac.jp

Nagoya Inst. of Tech., Japankenji@nitech.ac.jp

Aichi Inst. of Tech., Japans mizuno@aitech.ac.jp

Index Terms—Tooth brushing, Children’s oral care, Machinelearning

I. I NTRODUCTION
We should brush tooth while considering easily soiled areas
and areas with a lot of polish leftovers [1]. Easily soiled areameans that it often have bacteria that is called food residue orplaque. People should brush tooth every day since the tooth
started to grow. That is because if we do not brush tooth,
our milk tooth may fall out because of tooth decay before
permanent tooth grow and change. That phenomenon makes us
get crooked tooth because there is a possibility that permanent
tooth grows bent or grows in the different place than it shouldhave. Further, the tooth decay in milk tooth may make theenvironment where tooth decay prone to occurrence. So weshould not leave the tooth decay in milk tooth [2]. However
Annual report of school health statistics research reminds
that the account for 31.16 percent of kindergarten children
have tooth decay and that the account for 44.82 percent ofelementary school student have them and the account for 19.15
percent of kindergarten children have tooth where no action
has been taken and the account for 21.74 percent of elementary
school student have them [3]. There account is larger thanthe one of junior high school students and the one of highschool students. The result reminds that the small children
do not brush their tooth enough. Also, the account of those
who brush their tooth with awareness of periodontal disease,
the account of those who brush their tooth with awarenessof caries prevention, and those interested in insurance news
in dental hygienist second year are larger than ones in dental
hygienist ﬁrst year and students from other departments [4].
The result shows that those who have received education indental health are increased interest and knowledge about dental
health. That means that helping people learn to brush their
tooth correctly makes them brush tooth regularly. So we thinkthat the reason that children do not brush enough is that they
do not learn how to brush their tooth. However it is difﬁcultthat children understand why they should brush tooth because
germs are invisible to the naked eye. So we make it a goalthat we make children learn correct toothbrushing habits byteaching them why they should brush tooth to interest them inbrushing tooth. We have proposed AR tooth brushing systemthat promotes oral care habits of children [5]. This system
detects tooth areas easily soiled from PC camera image in realtime using machine learning technique ﬁrst, and superimposes
germs illustrations to show easily soiled areas. By the way,
tooth areas easily soiled do not mean tooth soiled in fact.
This system enables children to see germs that can not been
seen with the naked eye in fact to increase motivation forchildren to brush their tooth. However even if children brushtheir tooth by toothbrush while using this system, the germs
illustrations can not be erased. The phenomenon may not givethe feeling of brushing their tooth. So it is difﬁcult to improvetheir toothbrushing habits by this system. We improve thesystem so that it changes the display of germs following themovement of the toothbrush that children have in this study.
The improved system also takes PC camera image as previoussystem, and detects tooth areas easily soiled using machinelearning technique ﬁrst. Then it shows easily soiled areas
superimposing germs illustrations same as previous system.
Further, easily soiled areas are tracked and if the area of
toothbrush overlap them, the illustrations are erased. This
improvement is expected to enable children to enjoy brushing
tooth by using this sysetm and to increase toothbrushingguidance efﬁciency.
II. P REVIOUS AR TOOTH BRUSHING SYSTEM
In our previous study, we have proposed the system that
displays the germs illustrations in tooth areas easily soiled
to improve their toothbrushing habits. This system detects
tooth areas easily soiled from PC camera image in real timeusing machine learning technique, and superimposes germs
illustrations (Fig. 1) to show easily soiled areas. It takes face
image (Fig. 2) by a front camera of PC ﬁrst. Then teeth aredetected and extracted, especially molars, the areas between atooth and gum, the areas between a tooth and another tooth,and the area behind a front tooth. We conducted following
three machine learning using library YOLOv3 [6].
1stTo learn easily soiled areas using the images that
include other than mouth
2nd To learn easily soiled areas using the images that
include only mouth
3rd To learn mouth areas using the images that include
other than mouth
Table I reminds that f-value of 2st learning is larger than one of
1st learning. So the system detects tooth by the result of 2ndlearning after detecting mouth by the result of 3rd learning
because that is better than detecting tooth from face image

directly by the result of 1st learning. However if mouth area
(Fig. 3) can not be detected from face image by the resultof 3rd learning, tooth are tried to be detected directly by theresult of 1st learning. Also the height size of superimposed
germs illustration is twice as large as the height of the detected
tooth area, and it dose not overlap with each other. Also
If the system detects too large area to be tooth area, thendoes not show illustrations because if detected area is toolarge, that detecting is appears to be a false positive. Thissystem can interest children in brushing their tooth. We got
that in the experiment for the family with children (Fig. 4).
However while we conducted the experiment for a long time,children were losing interest in the system because the systemthat just displayed illustrations was boring for children. And
children could not really feel it this system because the germs
illustrations were not changed even if children brush theirtooth. So we improved the system in this study.

Fig. 2: Face image taken by a PC camera

TABLE I: Learn for toothLearning
1st2nd3rdPrecision
0.25
0.35
0.98

Recall rate
0.53
0.65
0.98

f-value
0.34
0.45
0.98

Fig. 3: The area of mouth

Fig. 1: Germs illustrations

III. I MPROVED AR TOOTH BRUSHING GAME

Fig. 4: Previous experiment

We aim for that the system interests children in brushing
their tooth by showing that germs illustrations exist in theirmouth and by erasing germs illustrations depending on themovement of the toothbrush that they have in hand. Thesystem takes face image by a front camera of PC ﬁrst. And
it detects mouth and tooth areas easily soiled as same asprevious method. We conducted machine learning for noseusing library YOLOv3 and 256 face images (Fig. 5). The result
is Table II. The system detects nose using this result (Fig. 6).

Each detected area is tracked according to the algorithm ﬂow(Fig. 7). The same thinning process is also used. The positionof brush part of toothbrush is also obtained. Also if toothbrushexists in the area that is detected as mouth area, the system
may not be able to track the toothbrush because toothbrushprevent tooth from being visible. So the system calculated
the position and size of the tooth areas easily soiled fromthe position and size of the nose that is not prevented frombeing visible by the toothbrush that children have in hand. The
position of the tooth areas easily soiled if toothbrush exists inthe area that is detected as mouth area are as follows.
P t = P n × Sn

P pt − P pn
P sn(1)

their tooth and reminds them that brushing can remove germs.
This makes children brush their tooth regularly.
TABLE II: Learning for nosePrecision
0.94

Recall rate
0.95

f-value
0.95

The size of the tooth areas easily soiled if toothbrush exists
in the area that is detected as mouth area are as follows.
St =P st × Sn
P sn(2)

Where Pt is the position of the tooth, Pn is the position ofnose, Sn is the size of nose, Ppt is the position of tooth
before brush of toothbrush enters the area of mouth, Ppn isthe position of nose before brush of toothbrush enters the areaof mouth, Psn is the size of nose before brush of toothbrushenters the area of mouth, St is the size of the tooth, and
Pst is the size of tooth before brush of toothbrush enters thearea of mouth. Next, the system displays germs illustrations
on the tooth areas easily soiled that are tracked. The systemprocesses to germs illustration depending on the positional
relationship between the toothbrush and germs illustration(Fig. 8). Then the system records the number of times thatbrush of toothbrush overlaps the germs illustrations on thetooth areas easily soiled that are tracked. And if the numberof times recorded exceeds a certain level, the system records
the germs illustrations as the illustration that should be erased.
Also when brush of toothbrush overlaps the germs illustrations
on the tooth areas easily soiled that are tracked, the germs
illstration that is overlaped is permeance. And the germs
illustrations that are not recorded as the germs illustraion thatshould be erased are superimposed in real time (Fig. 9). Also
if when a certain amount of time has passed since the systemstarts tracking the tooth areas easily soiled, mouth, and nose,
the system records the informations of germs illstrations thatare not erased then. The informations of germs illstrations thatare not erased are the position, the germs illstrations, and thenumber of times that brush of toothbrush overlaps the germs
illustrations on the tooth areas easily soiled that are tracked.
When the system detects the tooth areas easily soiled next,
the system records the information of germs illstrations thatare not erased. If each position is recorded, the system record
the position and do not detect the positon newly. Also whena certain amount of time has passed since the system starts
tracking the tooth areas easily soiled, mouth, and nose, thesystem records the postion of nose and one of mouth. We used
the library named OpenCV for object tracking of the tootharea easily soiled, the area of mouth, and the area of nose. To
measure the position of brush part of toothbrush, we used AR
marker (Fig. 10) that is one of the image recognition technique.
Also if the system can not track the areas when the mouth isclosed, etc. and if the illustrations that should be displayed, thesystem obtains face image from PC camera image and detect
their tooth areas easily soiled, mouth, and nose from the image.
We expected that the sytem enables children to enjoy brushing(a) Nose image(b) LabelingFig. 5: Nose image

Fig. 6: Detecting nose

IV. EXPERIMENTThis system displayed the illustration of germs and erasedthem following the movement of the toothbrush that children
have in hand (Fig. 11(a)). Also the system can trace tooth areas
easily soiled even if user moves face position (Fig. 11(b)).
The refresh rate is 22.11fps. The refresh rate is enough topromote children’s oral care habits. Also we experimented fora elementary school student and her parent to make sure thatthe system can promote children’s oral care habits. First, weexplained them the way to use this system. Next they used
this system and answered a questionnaire (Fig. 12, 13). We
asked a elementary school student the following questions.
• Were you interested in this system?
• Did you feel like brushing your teeth in thefuture?

Fig. 7: System behavior from getting face image to tracking object

Fig. 9: System behavior from processing germs illustration to displaying it

Fig. 10: Toothbrush with AR marker

V. CONCLUSION
Fig. 8: System behavior from starting to track object to processing
germs illustration

We asked her parent the following questions.
•Can this system be used for children’s toothbrushing education?

As a result, she told us that this system made her interest
bushing her tooth and she wanted to brush tooth every day
and her parent told us that this system can be used to teachchildren how to brush their tooth. So the result showed that thesystem was so valid that children learn good tooth brushinghabit.

In this study, we proposed the improved system that looks
like video game to detect tooth areas easily soiled in realtime using machine learning technique, superimpose germs
illustrations to show easily soiled areas, and change the displayof germs following the movement of the toothbrush. Also weexperimented that the system is so useful that it promotes
oral care habits of children. As future works, we would liketo improve the system as considering the time and frequencyof brushing until the germs disappear. Additional experiments
are also expected for more homes or kindergarten to conﬁrm
usefulness.
ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant
Number JP20K11918 and JP22K12337.

https://www.mext.go.jp/b_menu/toukei/chousa05/
hoken/kekka/k_detail/1411711_00003.htm(last Viewed Date: June ﬁfteenth, 2022)[4] Kakuhiro Fukai, MAKI Yoshinobu, TAKAESU Yoshinori, Effect of Oral
Health Education on Health Behavior, Journal of Dental Health, Vol.45,
pp.7-13 (1995)[5] Haruki Kondo, Kenji Funahashi, AR tooth brushing system to promoteoral care habits of children, Proc. NICOGRAPH Int’l 2021, p.115 (2021)[6] Joseph Redmon, Ali Farhadi, YOLOv3: An Incremental Improvement,
arXiv:1804.02767 (2018)(a) Before moving(b) After moving

Fig. 11: Tooth area tracking

Fig. 12: Appearance of experimentFig. 13: Display with test labels

R EFERENCES[1] Kozue Matsubara, Method for Tooth Brushing Oral Health Exercise,
Dental Medicine Research, Vol.31, No.3, pp.258-260 (2011) .
[2] YUZ DENTAL, Is it okay if milk tooth change tooth decay because itcomes out?
https://www.mext.go.jp/b_menu/toukei/chousa05/
hoken/kekka/k_detail/1411711_00003.htm(last Viewed Date: August 2, 2022)[3] MEXT, Annual report of school health statistics research in the 1styear of Reiwa era (deﬁnite value), 2 Summary of Survey Results (inJapanese),