Internationalof Computational
Intelligence
Systems
Journal ofJournal
Robotics,
Networking andArtificial Life
Vol.
XX(Z);
Month(2019),(Year), pp.
pp. 162–166
xx–yy
Vol.
6(3);
December
DOI: 10.1080/XXXXXXXXXXXXXX; ISSN
online
DOI: https://doi.org/10.2991/jrnal.k.191203.001;
ISSNXXXX–XXXX
2405-9021; eISSN
2352-6386
https://www.atlantis-press.com/journals/ijndc
https://www.atlantis-press.com/journals/jrnal

Research Article

Evaluation for the Synchronization of the Paradewith OpenPose
Yohei Okugawa, Masao Kubo, Hiroshi Sato, Bui Duc Viet
Department of Computer Science, National Defense Academy of Japan, Yokosuka, Kanagawa 239-8686, JapanARTICLE INFOABSTRACT

Article History

In military organization, the synchronization in the parade is important for showing their majesty. However, we do not have any
quantitative evaluation methods for the parade. In this research, as a first step of the quantitative evaluation for the parade, we
propose an evaluation method focusing on the synchronization level of “Marking Time”, in which soldiers is moving their legs
as in marching without step forward, between two persons with OpenPose. In order to measure the synchronization level, ourmethod is based on the trend of the transition of arm swing angles in the parade, and the result shows our method is appreciable
for measuring the synchronization level for two persons.

Received 09 November 2018
Accepted 30 November 2018
Keywords
OpenPose
unityparade
synchronization
multi-person pose estimation
multi-agent© 2019 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
In the military parade, the synchronization is a major factor for theperformance. Shimohagi, Ishihara, and Okugawa [1–3] conductedscientific analysis for the synchronization of the parade; however,
it is difficult to measure the synchronization level of the parade
because it requires a large number of posture data at same time.
We do not have any quantitative evaluation methods for the synchronization level of the parade. In this research, we propose theevaluation method for the synchronization level with OpenPose.
OpenPose [4] can measure many kinds of postures from images
simultaneously, therefore, it is much easier to get a large numberof posture data than conventional motion analysis with motion
capture [5,6], and it is suitable for analyzing cooperative operationof multiple people such as parade (Figure 1).
As a first step of this research, for measuring the synchronizationlevel of the whole of the parade, we focus on the arm swing of two
persons in “Marking Time”. Marking Time is a kind of parade style,in which soldiers is moving their legs as in marching without step
forward. In this paper, we analyze the synchronization level of themarking time, between two persons with OpenPose. When weevaluate the synchronization level of the two persons appropriately,
we can evaluate the synchronization level of the parade.
This paper is organized as follows. After introducing OpenPosein the next section, we describe the data processing method with
OpenPose. In Section 4, we propose the evaluation method for thesynchronization level. After explaining the two experiments with
3D CG and real behavior of the marking time by two persons, weCorresponding author. Email: em57037@nda.ac.jp*

show our results from the experiments. In addition, this paper isupdated with the results of reviews from the proceeding paper in
ICAROB [7].

2. OPENPOSE
OpenPose represents simultaneous posture recognition of multiple
people on single images. The left side of Figure 2 shows an exampleof pose estimation with OpenPose. If the movements of the two aresynchronized, we can expect that we get the same trend data.
The right side of Figure 2 shows the points of posture data.
As the three-dimensional data of [x, y, likelihood] on each point,
we can obtain 75 data for one piece of posture information. It
is difficult to measure the synchronization because the values
and the accuracy of the posture data are different depending on
the position and body size. Therefore, it is necessary to use thetrend of data which is not influenced by position or height as anindex. We can get the posture data of each person in same framewith OpenPose, but the data is not related to the posture data inthe before and the after frames. In order to obtain the time series
posture data of each person, we did cluster analysis based on thelocation in each frame.

3. PROPOSED METHOD
In this section, we propose the measurement method using indicators focusing on inflection point of movement. In the following,
we explain the proposed method in order.

Y. Okugawa et al. / Journal of Robotics, Networking and Artificial Life 6(3) 162–166

163

K-means clustering aims to partition the n observations into ksets S = {S1, S2}, where mi is the mean of points in Si and Var(Si) issquared deviation of Si.
In this section, we analyze the marking time movies to get timeseries data of each two persons, therefore, we set K = 2 for the cluster analysis using K-means method.
karg min S ∑∑ x − mi2

i =1 x ∈SiFigure 1 | Military Parade with OpenPose (https://www.youtube.com/watch?v=zaviumhF7aA).

k= arg min S ∑ Si Var(Si ) (1)i =13.3. Smoothing
In order to read the inflection point of the time series data of arm
swing angle accurately, we use Simple moving average [Equation (2)]for smoothing out short-term fluctuations of the angle data. Simple
moving average is the unweighted mean of the n data. When we
have a series of the angle data (Angt, Angt+1, …, Angt+n), then thesimple moving average Angsma is	   Ang sma =Ang t + Ang t +1 +  + Ang t + n
(2)nIn this method, we set n = 8 for reading the inflection points accurately.
Figure 4 shows an outline of the data before and after the smoothing process. It can be seen that the outline is smooth, and the inflection points are easy to read.
Figure 2 | Multi-person pose estimation with OpenPose.

3.4. Evaluation
After smoothing out, we get the time series data of arm swing angle[Equation (3)].
		 A = [Ang A1 , Ang A 2 , , Ang Ai ] (3)A is the time series data of arm swing angle of member A in animage, and AngA1 is the arm swing angle of member A at the first
frame of the movie.

Figure 3 | Focus points of OpenPose.

3.1. Focus Points

From each time series data of the arm swing angle, we use correlation coefficients for evaluation. We define the synchronization levelr as follow [Equation (4)]:n   r =∑ (Ang − Ang )(Ang − Ang )ABiB((∑ (Ang − Ang ) )(∑ (Anng − Ang ) ))ni =1

Focusing on the synchronized arm swing of the left hand, we obtain
the data of each shoulder (point 5 in Figure 3), elbow (point 6 inFigure 3) and palm (point 7 in Figure 3). Each data, which are points
5–7, are one-unit set. After we get time series data of each member,
we converted the data into four angles: j, y, l, q (Figure 3).

Ai

i =1

AiA2

ni =1

Bi

3.2. K-means Clustering
In order to get time series data of each person’s arm swing angle
in the image, we conduct the clustering using K-means clustering[Equation (1)]. We use K-means clustering to the data of shoulder(point 5 in Figure 3), Given a set of data (x5frame 1 , x5frame 2 , x5frame 3 ,¼, x5frame n ),Figure 4 | Data smoothing before and after.

B

2

12

(4)

Y. Okugawa et al. / Journal of Robotics, Networking and Artificial Life 6(3) 162–166

164

We use Pearson’s moment correlation coefficient as the synchronization level. When we havemean Ang A and Ang B , the arithmetic
which is from the data A and B, the synchronization level r isexpressed as [Equation (4)]. If r approaches 1, it is synchronized.
If r approaches −1, it is expected that the phase will be in opposite.

4. EXPERIMENTIn this chapter, we conducted two kinds of experiments. In Experi­
ment 1, we examined the measurement accuracy and the evaluation index of our method using 3D CG models. After confirming
the effectiveness of the evaluation method, in Experiment 2, weevaluate the synchronization level r in real marking time behavior
with our method.

4.1. Experiment 1: Accuracy of Our Method
For measuring the effectiveness of our method, we analyzed
the 3D CG models created by Unity. Since we can adjust thesynchronization level between two models, it can be an ideal
analysis target. 3D CG is appropriate for evaluating the synchronization level of the parade because we can completely control
human-like model.

4.2. Experiment 2: Evaluationof Marking Time
Since we can show our evaluation method is appreciable, we evaluatedthe synchronization of the real marking time. As in Experiment 1,
we analyzed the marking time of two persons. We experimented
with 6 pairs of two cadets in the National Defense Academy, whichhave parade training every day (Figure 7).
For Experiment 2, we have three types of arm swing: Phase Syn­chro­nization, Phase Difference and Phase Opposition. Figure 7 shows a snapshot of Phase Opposition. Both cadet arm swing is opposite in Figure 7.
Figure 8 shows the average of the synchronization level r of marking
time. As shown in Figure 8, we can confirm that the synchronization level r decreases as the difference of the arm swing angle shifts.
Figure 9 shows the synchronization level r in each angle with standard deviation. In Phase Difference case, we did not set the different level of the arm swing quantitatively like as Experiment 1.
Therefore, the standard deviation of Phase Difference is wider thanIn the 3D CG, the two models are only doing the marking time(Figure 5).
The arm swing angle of the 3D CG models was implemented
according to the simple vibration [Equation (5)]. In this section, thephase difference of the arm swing angle is expressed by changing
the initial condition of q0 as 0, p/8, p/4, p/2 and p.	   x = A sin(w t + q 0 )(w = 2p ) (5)Figure 6 | The synchronization level r of 3D CG.

Figure 6 shows the synchronization level r of time series angle dataof two models with four different angles. Each angle is representeddifferent figure such as j is represented as square. As the initialvalue q_0 shifts from 0 to p, the value r decreases in four angles.
Also, when the initial value q_0 is p, that is, each arm swing angle isin opposite phase, it approaches −1 in four angles. The synchronization level of four angles are almost same in Figure 6.
From the result, the synchronization level r is monotonicallydecreasing according to the value q0. Therefore, it was confirmedthat our evaluation method is appreciable for measuring the synchronization level in the 3D CG marking time movie.

Figure 5 | An example of 3D CG models with OpenPose.

Figure 7 | A capture of real marking time with OpenPose.

Figure 8 | The synchronization level r of marking time.

Y. Okugawa et al. / Journal of Robotics, Networking and Artificial Life 6(3) 162–166165

Figure 9 | The synchronization level r in each angle with standard deviation.

other cases. However, the results of Experiment 2 show the sametrend as the result of Experiment 1, and it is our evaluation method
is appreciable for measuring the synchronization level in the realParade. In addition, it was suggested that measuring the synchronization level with armpit angle (j) has good sensitivity for evaluation comparing to other angles.

[2]

5. CONCLUSION[4]

We propose the evaluation method for the synchronization level of themarking time with OpenPose. We focus on the arm swing angle andanalyze the marking time for two persons. Through the experimentusing 3D CG, it was confirmed that our method based on time seriesarm swing angle is appreciable for measuring the synchronizationlevel. Moreover, through the experiment of real marking time of six
pairs of two cadets, our evaluation method is appreciable for real environment. In addition, it was suggested that measuring the synchronization level with armpit angle has good sensitivity for evaluation.

REFERENCES[1]

T. Shimohagi, Parade analysis based on network system,
Proceedings of National Defense Academy of Japan Graduation
Thesis, 2017, pp. 61–62 (in Japanese).

[3]

[5]

[6]

[7]

T. Ishihara, Development of VR parade training system,
Proceedings of National Defense Academy of Japan Graduation
Thesis, 2018, pp. 47–48 (in Japanese).
Y. Okugawa, M. Kubo, T. Shimohagi, T. Ishihara, H. Sato, Analysis
of a parade with the Kuramoto model -for better performance
without trainings-, Proceedings of IEEE SMC, IEEE, Miyazaki,
Japan, 2018, pp. 4005–4010.
Z. Cao, T. Simon, S.E. Wei, Y. Sheikh, Realtime multi-person 2D
pose estimation using part affinity fields, 2017 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), IEEE,
Honolulu, HI, USA, 2017, pp. 1302–1310.
K. Inada, T. Ishikawa, H. Watanabe, Coincidence evaluationof multiple people’s dance using OpenPose, Proceedings of the80th National Convention of IPSJ, IPSJ, Tokyo, Japan, 2018,
pp. 257–258.
M. Nakai, Y. Tsunoda, H. Hayashi, H. Murakoshi, Predictionof basketball free throw scoring by OpenPose, Predictionof Basketball Free Throw Shooting by OpenPose, New
Frontiers in Artificial Intelligence JSAI-isAI 2018 Lecture
Notes in Computer Science, vol 11717, Springer, Cham, 2018,pp. 435–446.
Y. Okugawa, M. Kubo, H. Sato, B.D. Viet, Evaluation for the synchronization of the Parade with OpenPose, Proceedings of the2019 International Conference on Artificial Life and Robotics(ICAROB), Oita, Japan, 2019, pp. 443–446.

Y. Okugawa et al. / Journal of Robotics, Networking and Artificial Life 6(3) 162–166

166

Authors Introduction
Mr. Yohei Okugawa

Dr. Hiroshi Sato
He is the LTJG of Japan Maritime Self
Defense Force. He currently belongs tothe Department of Computer Science atthe National Defense Academy of Japanas a graduate student. He graduated fromthe College of Media Arts, Science andTechnology at the University of Tsukubain 2014.

Dr. Masao Kubo

He graduated Doctor course at Dep. of
Computational intelligence and System
Science in Tokyo Institute of Technologyin Japan. He is an Associate Professorof Department of Computer Science at
National Defense Academy in Japan. His
research interests include SICE, IEEJ,IEICE, JSAI member.

Mr. Bui Duc Viet
He is an Associate Professor of Dep. ofComputer Science at National Defense
Academy in Japan. He graduated fromthe Precision Engineering Department,
Hokkaido University, in 1991. He received
his PhD degree in Computer Science from
Hokkaido University in 1996.

He received his bachelor’s degree fromthe Department of Computer Science,
National Defense Academy, Japan in 2019.
He is currently a Master course student in
National Defense Academy, Japan.