Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 18–21___________________________________________________________________________________________________________Exercise classification using CNN with image frames
produced from time-series motion dataHajime ItohDivision of Production Systems Engineering, Muroran Institute of Technology, 27-1 Mizumoto-cho
Muroran, 050-8585, Japan
Naohiko Hanajima

College of Design Manufacturing Technology, Muroran Institute of Technology, 27-1 Mizumoto-cho
Muroran, 050-8585, Japan
Yohei Muraoka, Makoto Ohata

Steel Memorial Muroran Hospital, 1-45 Chiribetsu-cho
Muroran, 050-0076, Japan
Masato Mizukami, and Yoshinori Fujihira

College of Design Manufacturing Technology, Muroran Institute of Technology, 27-1 Mizumoto-cho
Muroran, 050-8585, Japan
E-mail: 15042009@mmm.muroran-it.ac.jp, hana@mondo.mech.muroran-it.ac.jp

Abstract
Exercise support systems for the elderly have been developed and some were equipped with a motion sensor toevaluate their exercise motion. Normally, it provides three-dimensional time-series data of over 20 joints. In thisstudy, we propose to apply Convolutional Neural Network (CNN) methodology to the motion evaluation. The
method converts the motion data of one exercise interval into one gray scale image. From simulation results, theCNN was possible to classify the images into specified motions.
Keywords: CNN, Gray scale image, Exercises classification, Time-series data.

Some exercise support systems have been equippedwith a motion sensor to evaluate the elderly’ exercise.
Normally, the motion sensor provides three-dimensional
time-series data of over 20 joints. In previous studies,
the motion was judged by human-induced conditions,such as the height of a hand position or the angle of thearm motion5. However, the human-induced conditions
tended to be experiential and intuition in some case.
Sometimes, the judgement did not fit the humanjudgement6.
In this study, we propose applying Convolutional
Neural-Network (CNN) methodology to the motion

1. Introduction
Population of the elderly is increasing compared withthat of the young in Japan1. It is estimated that thenumber of elderly care staff will be short ofapproximately 680,000 in 20352, 3. In order to contributefor solving this problem, exercise support systems has
been developed4. It is expected that the elderly canbecome healthy by strengthening their muscles withgymnastics, and less elderly care staffs will benecessary.

Copyright © 2017, the Authors. Published by Atlantis Press.
This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
18

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 18–21___________________________________________________________________________________________________________evaluation. The deep learning has succeeded in
integrating big data. CNNs is the oldest deep neuralnetwork architecture and is widely used as a means ofhandwriting recognition7. In recent years, image
classification has been highly succeeded with CNN8.
However, there are few examples that are applied totime-series data, and approaches to speech recognition
have been proposed9. In this paper, we propose to apply
CNN methodology to the motion evaluation. The
method converts all joints of the motion data into one
gray scale image. Then, CNN learns and classifies theconverted images. We first explain a method to create
motion data for CNN learning in Section 2. Section 3
proposes a conversion method from the motion data to
the gray scale image. After that, some case studies areexplained. Finally, we show the result of learning using
CNN and discuss its ability of classification.

Fig. 1. The image converted from a motion data. One of thestripes represents time-series of the X-Y-Z change of eachjoint and it is represented in rows in orderto 255 tone gray scale. Time is ordered from left to rightfor the column. The origin of the X-Y-Z coordinate
system is placed 1200 mm downward from the joint at
the center of the body, the luminance value changes
every 10 mm and can express up to 2550 mm.

2. Creation of motion data for training data
Learning with CNN requires a lot of training data.
However, exercise by humans cannot always do theright motion every time. Therefore, we created motion
data by simulating human exercises using simple
formulas. They were prepared with respect to what wasrecorded by a motion sensor in advance4. The motion
sensor can acquire three-dimensional position data of 20
joints. To imitate the position coordinate, we utilizedsinusoidal functions like Eq. (1). Eq. (1) is an exampleof the coordinate in the vertical direction of a hand
position P.
𝑃 = 𝑅 + 𝑙 × sin(𝜃) + rand

3.2. Arrangement of order of rows
We can define several arrangements of order of the row.
In this paper, we try two kinds of arrangement. First one
is a joint oriented order as shown in Fig. 1. Second one
is a coordinate oriented order, that is 𝑥1 , 𝑥2 , 𝑥3 , …, 𝑦1 ,𝑦2 , 𝑦3 , …, 𝑧1 , 𝑧2 , 𝑧3 , …, where subscript is the numberof the joints.
4. Case study for learning with CNN(0 < 𝜃 < 2𝜋) (1)We chose five kinds of motion for learning experiment.
They are shown in Fig.2 and their descriptions areshown as follows4.

Where R is the coordinate of the rotating joint, l is thelength of the arm from the rotating joint to the hand.
The function rand is a uniform random number
generator with a certain value.

i. Raise and drop arms from sideways to top.
ii. No actioniii. Raise and drop your arms from front to top.
iv. Raise the left leg 90 degrees forward.
v. Bend trunk 90 degrees forward

3. Conversion of motion data to image
CNN cannot process time-series data because there is noinput unit related to producing image frame from thetime. In this section, we propose the conversion methodfrom the time-series motion data to the gray scale
image. Then they are learned with CNN.

We used Caffe to learn CNN. It is a deep learning
framework developed by BVCL. The number of layersof CNN is six, the first layer has kernel size 5 andoutput is 32, the second layer has kernel size 5 andoutput is 32, the third layer has kernel size 5 and output
is 64. Also, each pooling layer has kernel size 3. The
number of output units varies depending on learning

3.1. Representation of pixelsFig. 1 shows the image converted from a time-series
motion data. Each row expresses X-Y-Z coordinate
value for each joint, and the value is converted thereby

19Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 18–21___________________________________________________________________________________________________________

Table 1.
Results when processing 100 images of the motion i. As the image being processed approaches 1, the possibility ishigher, and as it is closer to 0, it indicates that there is a high possibility that the image is different.
Comparative item
Confirmation of Average
exercise image Standard deviation
Learning for big Averagemotion
Standard deviation
Learning forAveragesmall changeStandard deviation

i.

ii.

iii.

iv.

v.

vi.

vii.

12.58E-09
01.06E-11
11.87E-11 2.94E-19 1.39E-23 4.2E-15
01.65E-13 2.80E-21 1.00E-25 3.3E-17
0.999781 5.54E-17
01.88E-29 2.76E-44 0.000219 2.34E-22
7.07E-06 1.66E-18
05.75E-31 8.49E-46 7.07E-06 6.9E-24

problem setting. We prepared 100 images for each kindof motion using Eq. (1). The random number
contributes the variety of data set. The number oftraining iterations was set to 4000. We performed threekinds of case study to confirm the ability ofclassification of the motions using proposed method.

viii.

ix.

x.

00

1.1E-14
2.98E-1600

4.3. Classifying for small difference
There is a possibility that the arm does not rise to thetop because of a disorder of the body. It may cause avery small difference in the actions. To investigate
ability of classification for such small difference, we
prepare five additional motions, vi to x with respect tothe motions, i to v. The motions, vi. and viii raise thearms to the lower position from the top by 10 degrees
than the motions, i and iii. The motions, ix and x raise
the legs up to 80 degrees other than the montions, iv andv. Also, the motion vii raises the arms lower by 10degrees than the motion i. There are ten inputs / outputs
in total. The small change of 10 degrees set here is an
exercise that cannot actually be judged by the exercise
support system. According to the exercise supervisor,
the small change of 10 degrees is judged to be a badpractice, so we will verify whether it can be judged ornot in this study.

4.1. Detection of exercise
In order to confirm whether the exercise movement canbe detected correctly, two kinds motions, i and ii werelearned with CNN. Their images are greatly different.
There are two inputs / outputs in the CNN composition.
4.2. Classifying for large difference
There is much difference over the prepared exercises. In
order to confirm the ability to classify completelydifferent actions, 5 kinds of exercises, i to v werelearned with CNN for the classification of the exercises.

5. Experimental result and discussion
After learning, classification test was performed. We
prepare data set for the classification separately from thelearning data, and the number of data is 100. Table 1
shows the probability and standard deviation ofclassification for result of each motion with test data. At
first, the row was arranged for the joint oriented order.
5.1. Detection of exercise
It can be seen from first 2 rows in Table 1 that they arecorrectly classified, that is average of i is 1, and ii isalmost zero. The average value of 1 shows possibility ofexcessive learning. The reason is considered that thekind of the motions is just two and the images used forlearning are very similar.

Fig. 2. Exercise to learn at CNN. The direction of the arrow
indicates the movement direction, the top is the front view andthe bottom is the horizontal.

There are five inputs / outputs in the CNN composition.

20

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 18–21___________________________________________________________________________________________________________

5.2. Classification for large difference

6. Conclusions

From middle 2 rows in Table 1, the motion i is detected
correctly. An average value of 1 was output for themotion i. It is considered that the reason is same as inSection 5.1. And, since other average values are verysmall, it is considered that the correct classification isachieved.

In this paper, we proposed a method to apply time-series
motion data to CNN. We showed that learning andclassification by CNN are possible by converting themotion data of one exercise interval into one gray scale
image. It was found that not only large difference but
also small difference in the motion data was correctlyclassified. Arrangement for rows did not affect theresults of classification in this case. We will apply theproposed method to analyze the human exercise data in
our future work.

5.3. Classification for small difference
In classification including the small change, it isobserved from bottom 2 rows Table 1 that the averagevalue of the motion i is almost 1 and the others are quite
small values. Therefore, we should say that theclassification was achieved. The average of the motion iis smaller than the previous section. The average valueof the motion vi is second largest in the motions. It
suggests that there is some correlation between themotion i and vi. Actually, the motion vi is a variant of
the motion i. These values seem to represent theirrelationship. Because of this, small changes inluminance values can also be regarded as feature
amounts, therefore there is a high possibility of judging
the difference in small change accurately.

References
1. Cabinet office, government of Japan, Heisei 28th edition
Aged Society White Paper (Japan, 2016), pp. 2-8.
2. Ministry of Economy, Trade and Industry, "Study Groupon Providing Long-Term Care Service According toFuture Nursing Care Demand" Report (Japan, 2016), pp.
22-39.
3. M. Kasahara et al., Relationship between One Leg
Standing Time and Knee Extension Strength in ElderlyPatients, Tairyoku Kagaku. 50(3) (2001), pp. 369–373.
4. Y. Takahashi, Study on exercise support robots providing
individualized instruction, Presentation of Graduation
Theses. (2014).
5. M. Hirano et al., Development of an Exercise Support
System for the Elderly Which Use a small Humanoid
Robot, Journal of Robotics and Mechatronics . 25(6)(2013) 939–948.
6. K. Urita et al., Precision Analysis of Center-Of-Gravity
Points Estimation Based on a Motion Sensor Comparison with a Force Plate, SICE annual conference
2014 conference proceedings, In Ternational Conferenceon Instrumentation, Information Technology and System
Integration (Japan, Sapporo, 2014), pp. 2020–2024.
7. D. Ciresan et al., Multi-column Deep Neural Networks
for Image Classification, In Computer Vision and Pattern
Recognition (CVPR), 2012 IEEE Conference on (2012),
pp. 3642-3649.
8. A. Krizhevsky et al., ImageNet Classification with Deep
Convolutional Neural Networks, In Advances in NeuralInformation Processing Systems 25 (2012).
9. O. A. Hamid et al., Convolutional Neural Networks for
Speech Recognition, IEEE/ACM Transactions on Audio,Speech, and Language Processing. 22(10) (2014) 1533–1545.

5.4. Rearrangement for rows in the coordinate
oriented order
Next, we show the result when the row was arranged in
the coordinate oriented order. Table 2 shows a part of
the result of the rearrangement for the rows order andlearning with CNN.
Table 2. Result of rearranging rows of pixels.
Except for the motions, i and vi, since it was lessthan E - 24, it is excluded in this table.
AverageStandard deviationi.
0.999892
3.15E-06vi.
0.000108
3.15E-06

It is observed that there is no significant difference
between two kinds of row orders. It was found that theaverage value of the motion i in Table 2 was slightly
larger than that in Table 1. It indicates that thepossibility of obtaining the feature is higher for themotion i.

21