Effect of shoe insoles on body swayin video-based person identiﬁcation
Haruki Nakayama∗ , Takuya Kamitani∗ , Masashi Nishiyama∗† , Yoshio Iwai∗† ,∗ Graduate School of Engineering, Tottori University† Cross-informatics Research Center, Tottori University

Abstract—Body sway has been attracting attention as an
informative cue for identifying people. Body sway is a minute
movement that occurs naturally when a person is standing,even when they consciously try to stop it. In the ﬁeld ofbiomechanics, it has been analytically reported that shoes affectbody sway. However, these analytical studies did not consider
the effect of shoe insoles on body sway. Here, we investigated
the identiﬁcation performance of body sway recorded by acamera when insoles were placed in the shoes. We found thatthe identiﬁcation performance signiﬁcantly decreased, the lowfrequency components of the frequency response of body sway
increased, and the maximum magnitude of body sway increased
when shoe insoles were worn.
Index Terms—Body sway, Shoe insole, Identiﬁcation, Videosequence

I. INTRODUCTION
To create a safe and secure society, techniques are needed toidentify people using security cameras [1]–[3]. To accurately
identify people, it is necessary to obtain informative cues that
represent individuals from the video sequences obtained by asecurity camera. Body sway in standing individuals has been
attracting attention in recent years as one such cue. Bodysway is a minute movement that occurs naturally when aperson is standing, even when they consciously try to stopit, as described in [4]. Kamitani et al. [5] showed that thebody sway recorded by an overhead camera is an effective
cue for person identiﬁcation. However, they did not consider
identiﬁcation performance in real environments. We need toconsider various factors to evaluate identiﬁcation performancein real environments. In this study, we focus on shoes as afactor.
In the ﬁeld of biomechanics, analytical studies have been
performed using a footplate force sensor to evaluate the effectof shoes on body sway. One analytical study [6] focused on astanding person with and without shoes on. It reported that themagnitude of body sway is greater in people who wear shoes
than in people who do not wear shoes. The analytical study [7]evaluated body sway in people wearing shoes with low heels
and shoes with high heels. It reported that the velocity ofbody sway is greater in people who wear high heels than inpeople who wear low heels. However, these studies did nottake into account the effect of insoles placed in the shoes.
Therefore, shoe insoles could degrade the performance ofperson identiﬁcation using body sway, as illustrated in Fig. 1.
In this study, we investigated the effect of shoe insoleson the performance of person identiﬁcation by extracting

CameraCameraBody sway

Body sway

CameraCameraBody sway

Body sway✓×

Identification

Identification

Without
insoleWithout
insoleWithinsoleWithout
insoleQueryTargetQuery

TargetFig. 1. Effect of shoe insoles on the performance of person identiﬁcation
using body sway. The performance depends signiﬁcantly on whether or notshoe insoles are worn.

the features of body sway from a video sequence. We usedslippers and insoles to evaluate the effect of shoe insoles.
In the following, we refer to the state of wearing slippers
only as “without insoles” and the state of wearing slippers
with insoles as “with insoles.” The evaluation results reveal
that, in the case of the without-insole target samples, theidentiﬁcation performance for the with-insole query samples
decreased signiﬁcantly when compared with the identiﬁcation
performance for the without-insole query samples. The power
spectral density computed from the time-series signals of bodysway showed that there was a signiﬁcant difference in thelow-frequency components. In addition, we found that themaximum magnitude of body sway increased when insoles
were worn.
II. EVALUATION METHOD
A. Shoe factors
Body sway changes depending on which part of the shoe
is varied. In particular, changes in the condition of the solesof the feet that support the body can have a signiﬁcant impact
on even the slight movement of body sway. We focus on theinsole, which is a variation in the condition of the soles ofthe feet. Insoles are used to protect the ankles and to increase
body height.
Assuming that the without-insole features are stored as thetarget samples, we evaluate the following hypothesis.

•When with-insole features are used as query samples,the identiﬁcation performance is worse than when thewithout-insole features are used as query samples.

Step1

Time

B. Extraction of body sway features

III. EXPERIMENTS

Record a color video
sequence

Color video sequence

Step2
Estimate the silhouette
video sequence fromthe color video sequence

TimeSilhouette video sequence

Step3
Select the reference
silhouette frame

Reference silhouette frame

Step4

Time

Create the difference-valuevideo sequence
Difference-value video sequence

Front

Step5
Left
Divide the video sequence
into local blocks

Step6
Calculate the amount ofmovement

TimeRightBack

Amount ofmovement

Person identiﬁcation is a technique that determines whether
or not a person appearing in a query video sequence is stored in
the target dictionary. It is important to extract features from thevideo sequence of body sway to obtain high identiﬁcation performance. We extract spatio-temporal features by calculating
the amount of body sway movement at each time step from thevideo sequence using an existing method [5]. Figure 2 shows
the process of feature extraction. The details are as follows.
Step 1: We use a camera to record a color video sequence ofbody sway. The camera is placed above the body to observe
the head, which is the part that moves the most in body sway.
We assume that the person is standing in an upright position
under the camera.
Step 2: We estimate a silhouette video sequence of the headregion from the color video sequence. For the head region, thepixel value is assumed to be 1, otherwise, it is 0.
Step 3: We select the reference silhouette frame for eachsilhouette video sequence. We ﬁnd a representative silhouetteframe in the silhouette video sequence that is similar to othersilhouette frames at other time steps. We use this reference
silhouette frame.
Step 4: We create a difference-value video sequence to extract
the movement of the head region at each time step. To do this,we calculate the difference in the pixel values of the silhouetteframe at each time step and the reference silhouette frame.
Step 5: The difference-value video sequence is divided intolocal blocks to observe the spatial difference of the head shape.
Speciﬁcally, the video sequence is divided into I blocks in theradial direction from the center of the head region. In Step 5of Fig. 2, the number of local blocks is I = 4 as an example.
Step 6: We compute the amount of movement at each time toextract a time-series signal in the head region. We obtain thesum of the difference values in each local block and use thissum to represent the amount of movement at each time.
Step 7: We extract a feature for person identiﬁcation. We calculate the power density (PSD) from the amount of movementin each local block. We then combine the PSDs of all blocks
to determine the feature used for identiﬁcation.

Left Back Right Front

TimeWe acquired without- and with-insole video sequences toevaluate the effect of shoe insoles on identiﬁcation performance. In this experiment, we used slippers and insoles, as
shown in Figure 3. The thickness of each insole was 3.5 cm.
Twenty-two participants (age: 22.9 ±1.0 years) participatedin the experiment. The duration of each video sequence wasset to 120 s. Each participant was videoed three times in thewithout- and with-insole states. As shown in Fig. 4(a), weasked the participants to stand with an upright posture under
the camera. The arrangement of the experimental environment

Step7

Extract the features

Log power

A. Dataset
Left Back Right Front

Frequency
Fig. 2. Process of feature extraction for person identiﬁcation using the video
sequences of body sway∗∗ ∶ 𝑝 < .01

Accuracy

100

Without insoleWith insoleFig. 3. Slippers and insoles used in the experimentCamera

240 cmColor video
sequence(a) Upright postureParticipant

Landmark(b) Environment

Fig. 4. Experimental setting

is illustrated in Figure 4(b). The camera was positioned at aheight of 240 cm. This camera has a resolution of 1920 × 1080
pixels and a frame rate of 30 frames per second. The video
sequence was cropped to 200 × 200 pixels in size around theperson.
B. Performance of person identiﬁcation
First, we describe the experimental conditions for theidentiﬁcation method presented in Section II-B. Based on
the assumptions described in Section II-A, we evaluated theidentiﬁcation performance when the without-insole data wereused as the target sample and the without- and with-insole data
were used as the query samples. In each of the trials, different
samples from the three collected samples were used for thetarget sample and query sample. The measure of accuracy
is the percentage of rank-1 answers that were correct. For
identiﬁcation, we use the nearest neighbors method based on
Euclidean distance.
Figure 5 shows the identiﬁcation performance results for thewithout- and with-insole query samples. We conﬁrmed that theidentiﬁcation performance for with-insole queries is about 18
points lower than it is for without-insole queries. We used theStudent t-test to determine if there was a signiﬁcant differencein the identiﬁcation performance.
The signiﬁcance level was set to p < .01. The results showthat there was a signiﬁcant difference in the identiﬁcation
performance results of the without- and with-insole samples.

∗∗90
80
70

60

Without-insole query

With-insole queryFig. 5. Identiﬁcation performance results for without- and with-insole query
samples. The signiﬁcance level was set to p < 0.01 and is indicated by **.

C. Comparison of features
We compared the without- and with-insole features to examine the reason for the decrease in identiﬁcation performance.
The feature extraction method is described in Section II-B.
For each participant, we calculated the difference between thewithout- and with-insole features at each frequency. As shownin step Step 5 of Fig. 2, four blocks were used. For eachof the four blocks, we calculated the absolute value of thedifference between the features. Figure 6(a) shows the averagevalues of the without-insole features, Figure 6(b) shows theaverage values of the with-insole features, and Figure 6(c)shows the absolute difference between the without- and withinsole features. We found that the low-frequencies componentsof the left and right blocks are generally different.
IV. EFFECT OF SHOE INSOLES ON BODY SWAY
A. Purpose
We also investigated how shoe insoles change the bodysway. In one analytical study [6], the effect of shoes on bodysway was analyzed in two directions: the anterior–posterior(AP) and medial–lateral (ML) directions. The observation in
the AP direction corresponds to observing the person from thefront, whereas the observation in the ML direction corresponds
to observing the person from the side. In [6], it was reported
that wearing shoes can signiﬁcantly increase the maximum
magnitude of body sway in the AP direction. However, theanalytical study used a footplate force sensor to observe themagnitude of body sway, whereas we use a camera instead.
B. Investigation of the maximum magnitude of body sway
To extract the maximum magnitude of body sway from a
video sequence, we ﬁrst perform feature extraction using steps
Step 1 to Step 6 in Fig. 2, and the amount of movement
is then calculated. The maximum value of the amount ofmovement from the start to the end of the video is selected
as the maximum magnitude of the body sway. The maximum
magnitude of body sway is analyzed in both the AP and ML
directions, in accordance with the analytical study. As shownin Fig. 7(a), the magnitude of body sway in the AP directionis the sum of the amount of movement in the front and backblocks. As shown in Fig. 7(b), the magnitude of body swayin the ML direction is the sum of the amount of movement inthe left and right blocks.

Log power

121086420-2Left

BackRightFront

TABLE IC OMPARISON OF THE AVERAGE MAXIMUM MAGNITUDE OF BODY SWAY IN
THE WITHOUT- AND WITH - INSOLE STATES . T HE SIGNIFICANCE LEVELWAS SET TO p < 0.01 AND IS INDICATED BY **.

APMLDC

15 DC
15 DC
15 DC
Frequencies of local blocks [Hz]Log power

LeftBackRight

Difference in Log powerStatisticalsigniﬁcance∗∗∗∗AP direction. We consider that this difference is due to thefact that the analytical study focused on body sway without
and with shoes, whereas we focused on body sway without
and with insoles. Furthermore, this difference in the maximum
magnitude of the body sway appeared in the low-frequency
components, as described in Section III-C.

Front

V. CONCLUSIONS

DC

15 DC
15 DC
15 DC
Frequencies of local blocks [Hz]15(b) With-insole feature
Left0.8

BackRight

Front

0.6

0.4
0.2
0.0

With insoles(pixels)
2225.1 ± 637.7
1858.5 ± 517.2

15(a) Without-insole feature
121086420-2

Without insoles(pixels)
1988.2 ± 568.0
1439.0 ± 400.2

DC

15 DC
15 DC
15 DC
Frequencies of local blocks [Hz]15(c) Difference in the without-insole and with-insole features
Fig. 6. Comparison of the average without- and with-insole features

Front
LeftBack(a) AP (Front, Back)We investigated the effect of shoe insoles on the performance of person identiﬁcation by extracting the featureof body sway from video sequences. The evaluation results
revealed that the identiﬁcation performance decreased significantly when with-insole query samples were used insteadof without-insole query samples. The power spectral density
computed from the time-series signals of body sway showedthat there was a signiﬁcant difference in the components of
the low-frequency components. In addition, we found that themaximum magnitude of body sway is increased when insoles
are worn.
In future work, we would like to develop a method thatis robust to the presence of shoe insoles and increase identiﬁcation performance. We also plan to expand the scope ofevaluation to include more factors in addition to shoe insoles.

Right(b) ML (Left, Right)

Fig. 7. Local blocks for the AP and ML directions

C. Changes in the maximum magnitude of body sway
Table I compares the maximum magnitude of the body swayin the AP and ML directions in the without- and with-insolestates. To analyze the results, we performed the Wilcoxon
signed-rank test with a signiﬁcance level of p < .01. The
results show that there was a signiﬁcant difference in themaximum magnitude of body sway in both directions. Bycontrast, in [6], a signiﬁcant difference was found only in theR EFERENCES[1] A. Bedagkar-Gala and S. Shah, “A survey of approaches and trends inperson re-identiﬁcation,” Image and vision computing, vol. 32, no. 4, pp.
270–286, 2014.
[2] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable
person re-identiﬁcation: A benchmark,” In Proceedings of the IEEE
international conference on computer vision, pp. 1116–1124, 2015.
[3] L. Zheng, H. Zhang, S. Sun, M. Chandraker, Y. Yang, and Q. Tian, “Person re-identiﬁcation in the wild,” In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pp. 1367–1376, 2017.
[4] A. Bottaro, M. Casadio, P. G. Morasso, and V. Sanguineti, “Body sway
during quiet standing: Is it the residual chattering of an intermittentstabilization process?” Human Movement Science, vol. 24, no. 4, pp. 588–615, 2005.
[5] T. Kamitani, H. Yoshimura, M. Nishiyama, and Y. Iwai, “Temporal andspatial analysis of local body sway movements for the identiﬁcation ofpeople,” IEICE Transactions on Information and Systems, vol. 102, no. 1,pp. 165–174, 2019.
[6] B. G. Susana, V. C. Rafael, L. A. Collange, N. F. M. Hugo, Pasini, andS. O. Claudia, “Interference of high-heeled shoes in static balance among
young women,” Human Movement Science, vol. 31, no. 5, pp. 1247–1252,2012.
[7] A. Mika, L. Oleksy, R. Kielnar, and M. Swierczek, “The inﬂuenceof high- and low-heeled shoes on balance in young women,” Acta ofBioengineering and Biomechanics, vol. 18, no. 3, pp. 97–103, 2016.