The 12th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications
7-9 September, 2023, Dortmund, Germany

Evaluating an In-Home Exercise Program Using
Vision-Based Edge AI for Elderly Healthcare
Sinan Chen1 , Masahide Nakamura2 , Kiyoshi Yasuda31, 2, 3 Kobe University, 1-1 Rokkodai-cho, Nada, Kobe, 657-8501, Japan, chensinan@gold.kobe-u.ac.jp
2 RIKEN Center for Advanced Intelligence Project, Tokyo, 103-0027, Japan, masa-n@cs.kobe-u.ac.jp
Evaluating in-home exercise

Abstract — In Japan, many in-home exercise programs
have appeared to improve the motor function of the elderly
through self-care. However, a challenging point always exists
in evaluating the process of exercise programs quantitively
and non-invasively for further reﬂecting the health status.
As a preliminary progress, we present and discuss a visionbased novel method on edge that aims to evaluate an
in-home exercise program in this paper. Our key idea
is to integrate multiple different vision-based pre-trainedmodels. In the proposed method, we mainly integrate andcall multiple pre-trained models to recognize human facial,skeletal, and hand movements with edge computing. Also,
we propose to evaluate an in-home exercise program processfrom the Euclidean distance, velocity, and angle of the twodimensional coordinates of feature points in the time series.
Through experiments, we estimate the physical health statusfrom the analysis of body movements in the elderly.

SoftwareHardwareModelsComputerApproach 1

ServerDigital twin

USB camera
Approach 2

Database

Raspberry Pi

Older adult

Figure 1. Overall architecture of the proposed method.
Keywords — exercise program; elderly healthcare; visionbased edge AI; edge computing

I. I NTRODUCTION
Evaluating the exercise of elderly people at home for
their healthcare is always a challenging research topic.
Such as the home-based rehabilitation (PR) evaluation
lacks direct supervision, and the existing complex natureof the rehabilitation model, most in-home exercises for
elderly people often require being encouraged to achieve,and so on. Hawley-Hague et al. [1] developed motivational smartphone apps to support patients in performing
exercises proven to aid fall reduction. However, someproblems require improvement, including the complexityof operation and difﬁculty for individual elderly peopleof general households.
The invasiveness to the daily life of the monitoring
device for evaluating the exercise of elderly people is alsoa task. Especially for elderly people requiring a long-term
exercise program at home, devices with invasiveness andunique environments increase the evaluating complexity.
Chan et al. [2] presented a wearable activity monitor system for patients with intermittent claudication. However,
we consider that these wearable devices only apply to theshort experiment to evaluate for a special purpose. Forthis, we must consider and study how to make a noninvasive evaluating process for elderly people at home,whether to retrieve or save data.

The goal of this study is to propose a non-invasive
and ﬁne-grained evaluation method for in-home exercise
programs. Figure 1 shows the overall architecture of the
proposed method. As an approach, we ﬁrst collect live
images at regular intervals using a ﬁxed-point camera.
Next, we install an edge-executable image recognition
model on a general-purpose computer or Raspberry Pi.
We subdivide and deﬁne the contents of in-home exercise
programs and extract feature data from selected representative time-series images. In addition, we calculate
from the Euclidean distance, velocity, and angle of thecoordinate data (i.e., face, ﬁngers, skeleton) as feature
data and perform visualization analysis. In this way, it ispossible to grasp the exercise status of the elderly at home,and estimating the physical health condition is promising.
II. P RELIMINARIES
A. In-Home Exercise Program
In recent years, many exercise programs preventing
falls in the home for the elderly were introduced into
people’s life in Japan. Decreased muscle strength isthought to be one of the causes of tumbles and falls. As an
exercise that can ﬁnish at home, it puts less strain on thejoints of the legs and lower back and trains the muscles
around the knees and waist that can ﬁnish quickly. The
existing exercise programs have become popular with theelderly, such as exercising while watching TV or listening1017

to music as they can ﬁnish1 . In this paper, we mainly focus
on an exercise program while brushing teeth for elderly
healthcare 2 .
B. Technical Challenge
To evaluate an exercise program for elderly healthcare,
we consider that the challenging point is the difﬁculty ofretrieving continuous and ﬁne-grained feature data witha non-invasive approach. Although some studies often
rest on the subjective evaluation with questionnaires (e.g.,Calculate the metabolic equivalents (METs) by asking
the action contents and time), it is hard to avoid theindividual differs from one household to another. Somerelated works also monitor elderly activity using wearable
devices described in Section I. The invasiveness of dailylife to elderly people from wearable devices exists, suchas forgetting to wear the special device sometimes, being
difﬁcult to apply it to the physical body due to no good
health status or suitable environments.
C. Vision-Based Edge AI
Vision-Based Edge Artiﬁcial Intelligence (AI) is one of
the newest research topics. It uses computer vision-based
pre-trained models with transfer learning to recognize
images or video streams on edge (see Figure 2). For
example, if we input an image into the target modelthat downloads from the cloud, the 2D coordinates orlabels of the feature point will be returned locally. The
typical recognition models include face recognition, poseestimation, and hand tracking, driven by the TensorFlow
platform [3]. We consider that these models can improveusers’ security and privacy issues and communication
overhead in real-time. In the previous study, although a
non-invasive ﬁne-grained home care monitoring system isproposed [4], it focuses on the pose estimation but lacks
further descriptions of the process. As a preliminary study,
this paper mainly integrates multiple models to evaluate
an exercise program for elderly health at home.
III. P ROPOSED M ETHOD
In this section, we present a method to evaluate inhome exercise programs quantitatively for older adults.
To acquire and analyze feature data (i.e., human facial,skeletal, and hand movements) changes with time series
on edge, our key idea is to integrate multiple vision-based
pre-trained models for a given set of exercise programs.
More speciﬁcally, the proposed method consists of thefollowing ﬁve steps.
A. Step 1: Acquiring Images on Time Series
In this step, the user ﬁrst selects a speciﬁc space inthe home for performing exercise programs. Next, theuser connects one of the following user-selectable devices
to the USB ﬁxed-point camera: (1) a general-purpose
1 http://www.city.chichibu.lg.jp/secure/8465/44gou.pdf (in Japanese)
2 https://www.youtube.com/watch?v=u MW0ai2oPI (in Japanese)

Figure 2. Overview of vision-based pre-trained models withtransfer learning.

computer browser and (2) a single-board computer (i.e.,
Raspberry Pi series). Then, we develop and execute a
program that acquires live images in a ﬁxed period (e.g.,ﬁve seconds).
B. Step 2: Deﬁning Exercise Programs to Recognize
In this step, the user deﬁnes the exercise programs
P = {p1 , p2 , .., pm } to recognize for evaluating the healthstatus. Generally, the exercise program often consists ofslightly different body movements and postures. Also, thefocus of strength training changes depending on the bodypart’s orientation and the direction of movement.
C. Step 3: Selecting Continuous Representative Images
In this step, for each program pi ∈ P , the user manually
selects continuous representative n images IM G(pi ) ={imgi1 , imgi2 , ..., imgin } that well expose pi , from all
images obtained in Step 1. Note that the user must choosea representative, consecutively acquired image for theprogram pi to analyze feature changes before and aftertime-series images (see Step 5).
D. Step 4: Calling Vision-Based Pre-Trained Models
In this step, the user integrates a set M ODEL ={model1 , model2 , · · · , modelq } of vision-based pretrained models to perform the image recognition onedge, using the Web browser in a general computer, or the unique program in a Raspberry Pi device. For every pi ∈ P , imgij ∈ IM G(ci ), andmodelk ∈ M ODEL, modelk (imgij ) is invoked, and a
set F eature(imgij , modelk ) = {w1 , w2 , w3 , ...} of output feature values is obtained. F eature(imgij , modelk )represents a recognition result for vision-based pre-trainedmodels modelk for an image imgij belonging to a program pi . The size of F eature(imgij , modelk ) varies for
imgij and modelk . Since there are m programs, n images
for each program, and q MODELs, this step creates totallym × n × q sets of output feature values.

1018

(a)

(b)

(c)

(d)

Figure 3. Experiment Setup: (a∼c) Representative images of every exercise programs. (d) A Raspberry Pi and a USB camera.

E. Step 5: Analyzing Output Feature Values
In this step, the user ﬁrst expands each point wl of
the feature data on the time series of n images obtained
from each modelk to two-dimensional coordinates wl ={‘x′ : ..., ‘y ′ : ...} for each program pi . The user thenquantitatively evaluates the exercise program from thefollowing three computational approaches:
Euclidean distance: The user calculates the Euclidean
distance dt at time t using the coordinate data (xt , yt ) attime t and the coordinate data (xt−1 , yt−1 ) at time t − 1
as follows:√(1)
dt = (xt − xt−1 )2 + (yt − yt−1 )2

Velocity: Since the images selected in Step 3 are representative and continuous with a constant time interval, theuser calculates the velocity vt by dividing with the timeinterval Tt from t − 1 to t as follows:dt(2)TtAngle: To convert to angular velocity ω, the user selects
coordinates of three feature points at time t, two vectors
are obtained, and cosθ(t) is calculated. For example,
suppose the user wants to check the degree of bendingof the left leg. In that case, the user can calculate theangle with the following formula by selecting the left hip→−
coordinate vector d1 = (x1 , y1 ), the left knee coordinate→−
vector d2 = (x2 , y2 ), and the left leg coordinate vector−→
d3 = (x3 , y3 ).
vt =−→ −→d21 · d31
d21 · d31
IV. E XPERIMENTAL E VALUATIONcosθ =(3)In this experiment, we set a single room as the targetspace. We aim to record and analyze the process of thespeciﬁc exercise programs of an aged man (60’s). Figure
3 shows representative images and experimental tools.
A. Experimental Setup
In step 1, we install a Raspberry Pi and a USB camera
with a Node.js standalone program. It can collect live
images automatically every ﬁve seconds. Moreover, weset the total image accumulation period to ten minutes andthe image resolution to 320 × 240. In step 2, we deﬁnethree kinds of in-home exercise programs while brushing
the teeth (see Figure 3 (a∼c)): (p1 ) Raising legs forward.
(p2 ) Raising legs left and right. (p3 ) Raising legs behind.
In step 3, we select 50 continuous representative images
for each program p. Considering that raising the left or
right legs differs, the number of images for raising thesame leg also split into half of 50. In step 4, we integrate
the Node.js standalone program of step 1 with the pretrained model PoseNet [5] on edge. Hence, it also canoutput the 17 feature points of the human body from every
image. In step 5, we ﬁrst extract the output data (i.e., xand y coordinates) of the six speciﬁc feature points of thelower part of the body. Then, we calculate and visualize
the Euclidean distance of the left knee, right knee, left
ankle, and right ankle in the time series. Finally, we also
calculate and visualize changes in the angle of raisingdifferent legs for each exercise program p.
B. Results
Figure 4 shows the representative results regarding
Euclidean distance and angle for the three exercise programs. In the graph of results for Euclidean distance,on the one hand, the horizontal axis represents 250
seconds combined with 50 results at 5-second intervals.
The vertical axis means the distance difference in thetime-series images’ pixel units. On the other hand, inthe graph of the angular results, the horizontal axis is
the same as above, and the vertical axis represents theangle in radians. These results show that in Program 1,
the exercise movement of the left leg is more irregularthan that of the right leg. In program 2, the lift distanceof the left leg is lower, but the angle is similar to that ofthe right leg. Furthermore, in Program 3, the right foothad better dexterity than the left foot, and the range andangle of movement were more excellent.
V. C ONCLUSION
To scientiﬁcally understand the motor function of older
adults at home, this paper mainly focused on quantitativeand non-invasive evaluation methods for in-home exercise programs. The main outcomes are as follows: (1)Proposing a method that integrates multiple vision-based
pre-trained models and extracting features of human faces,

1019

Figure 4. Representative results regarding Euclidean distance and angle for the three exercise programs.

skeletons, and ﬁngers simultaneously on edge. (2) Experimenting with a method to evaluate the process of in-home
exercise by calculating the Euclidean distance, velocity,and angle for the extracted two-dimensional coordinates
on the time series. Though the experiment, we found thatthe movement frequency and ﬂexibility of the subjects’right leg were better than that of the left leg. Meanwhile,
by visualizing the irregular movement process of the left
leg, it was presumed that there was a lack of motor
function. In future work, we will develop a modeled
home exercise evaluation system and further clarify its
application’s limitations and technical challenges (e.g.,refer to studies [6] [7]).
ACKNOWLEDGMENT
This research was partially supported by JSPS
KAKENHI Grant Numbers JP19H01138, JP20H05706,
JP20H04014, JP20K11059, JP22H03699, JP19K02973,
and Young Scientists (No.23K17006).
R EFERENCES[1] H. Hawley-Hague, C. Tacconi, S. Mellone, E. Martinez, C. Ford,
L. Chiari, J. Helbostad, C. Todd et al., “Smartphone apps to support

falls rehabilitation exercise: App development and usability andacceptability study,” JMIR mHealth and uHealth, vol. 8, no. 9, p.
e15460, 2020.
[2] C. Chan, V. Sounderajah, P. Normahani, A. Acharya, S. R. Markar,
A. Darzi, C. Bicknell, and C. Riga, “Wearable activity monitors
in home based exercise therapy for patients with intermittent
claudication: a systematic review,” European Journal of Vascularand Endovascular Surgery, vol. 61, no. 4, pp. 676–687, 2021.
[3] C. Carl, F. Schönfeld, I. Profft, A. Klamm, and D. Landgraf, “Automated detection of european wild mammal species in camera trap
images with an existing and pre-trained computer vision model,”European journal of wildlife research, vol. 66, no. 4, pp. 1–7, 2020.
[4] S. Chen, S. Saiki, and M. Nakamura, “Nonintrusive ﬁne-grainedhome care monitoring: Characterizing quality of in-home posturalchanges using bone-based human sensing,” Sensors, vol. 20, no. 20,
p. 5894, 2020.
[5] B. Jo and S. Kim, “Comparative analysis of openpose, posenet, andmovenet models for pose estimation in mobile devices,” Traitement
du Signal, vol. 39, no. 1, pp. 119–124, 2022.
[6] A. Saranya and R. Subhashini, “A systematic review of explainable
artiﬁcial intelligence models and applications: Recent developments
and future trends,” Decision analytics journal, p. 100230, 2023.
[7] R. I. Mukhamediev, Y. Popova, Y. Kuchin, E. Zaitseva, A. Kalimoldayev, A. Symagulov, V. Levashenko, F. Abdoldina, V. Gopejenko,
K. Yakunin et al., “Review of artiﬁcial intelligence and machinelearning technologies: Classiﬁcation, restrictions, opportunities andchallenges,” Mathematics, vol. 10, no. 15, p. 2552, 2022.

1020