2014 International Conference on Emerging Security Technologies

Gait identiﬁcation using invisible shadows: robustness to appearance changes
Yumi Iwashita †, Ryo Kurazume †, Adrian Stoica ‡†Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan‡Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA
Email: yumi@ieee.org

As one of options to deal with these problems, Iwashita
et al. proposed a person identiﬁcation system [9] using
shadow biometrics [7] and a shadow database. In thissystem, they placed multiple lights obliquely to project
shadows of a subject on the ground, and placed a cameraon the ceiling to capture the shadows. Shadow areas,
which are projections of the subject’s body on the groundby multiple lights, can be considered as body areas captured by multiple cameras from different viewpoints, so
the system enabled to capture multiple body areas fromonly one camera. Furthermore, in the system they usedinfrared lights, with an advantage of less stress on subjects
compared with normal visible lights; in addition it is anobservation that is transparent to the subjects, and offers
no reason for falsifying the gait.
The work described in [9] relied on extracted afﬁnemoment invariants, calculated from an average image,as gait features. The average image was created fromsequential silhouette images, which were aligned with acenter position of the subject area in each silhouette image.
The performance was not high enough, due to the factthat when the subject position with respect to the lightsource changes compared with the one in the database, theshape of the subject area changes. This causes variationsof center positions of the subject’s areas in sequential
silhouette images, which make different average images
compared with the ones in the database, and the discrimination capability of the average image decreases. Besides,
in case that the appearance of the subject changes due
to variations of clothes and bags, the performance of thesystem [9] gets worse.
In this paper we propose an identiﬁcation method,which achieves higher performance compared with theprevious method [9] and is robust to appearance changes.
The proposed method is based on the following two
methods; (i) a voting-based method which achieves highperformance [1] and (ii) a person identiﬁcation method
robust to appearance changes [8]. Both the voting-based
method [1] and the person identiﬁcation method robust to
appearance changes [8] are for identifying walking peopleusing captured images of their bodies, and the methods
did not use any shadow images. In the present paper, we
propose a method which integrates both previous methods[1] [8]. In the proposed method, ﬁrstly we divide a shadow
area into several parts, and estimate the discrimination
capability for each part based on gait features between
gallery datasets and probe dataset. Next, according to
the estimated capability, the priorities of these parts for

Abstract—This paper presents a person identiﬁcation technique that uses information from person’s shadow, andis robust to appearance changes caused by variations ofclothes and carried objects. The technique uses invisible
lights and resulting shadows and has advantages fromundetected sensing. The shadows on the ground obtained
through illumination by multiple lights can be consideredas silhouettes captured by multiple virtual cameras placed
at light positions. Thus, a single camera, e.g. in the ceiling,
is able to obtain multiple silhouettes, equivalent to a multicamera system. If the person’s appearance changes comparedto the training cases in the database, e.g. by wearing different
clothes or carrying a/another bag, then the identiﬁcation
performance gets worse. To deal with this problem, we introduce a new shadow-based identiﬁcation technique robust to
appearance changes. Firstly, we divide each shadow area into
several parts, and estimate the discrimination capability foreach part based on gait features between gallery datasets andprobe dataset. Next, according to the estimated capability,
we adaptively control the priorities of these parts in the
person identiﬁcation method. We constructed a new shadow
database with a variety of clothes and bags, and carriedout successful experiments to verify the effectiveness of the
proposed technique.

I. I NTRODUCTION
Gait is a powerful remote biometric, offering the advantages of identiﬁcation from a distance, and of being
unobtrusive, as body-invasive sensing is not needed tocapture gait information [10] [3]. In applications related
to security in controlled spaces, such as at the airport
corridors leading to an airport security passport control,
etc there is an interest to enhance the capability of person
identiﬁcation, for advanced warning (hence the need for aremote biometric) as well as for the purpose of achieving a
multi-modal system for increased correct classiﬁcation rate(i.e. working in addition to face biometrics, for example).
Gait identiﬁcation methods based on appearance information use measurements of gait features from silhouettes,
obtained by feature extraction methods, such as gait energy
image(GEI) [5], active energy images (AEI), [13], Fouriertransforms [11] , and afﬁne moment invariants [6]. Thesemethods have been used with good results on humanidentiﬁcation. Correct classiﬁcation rates in person identiﬁcation are generally better when multiple cameras fromdifferent viewpoints are used, yet most of conventional
methods have used one camera, because of the reduced
cost and fewer technical difﬁculties including (i) easierinstallation in real environments, less cameras and no needto synchronize cameras, (ii) a reduction of computation
costs.
978-1-4799-7007-0/14 $31.00 © 2014 IEEE
DOI 10.1109/EST.2014.18

34

people), (ii) hand bag (18 people), (iii) traveling bag (18people), as shown in Fig. 4 (a) ∼ (c). The sequences in
the IRSD-CL are categorized into 3 categories, (i) down
jacket (18 people), (ii) laboratory coat (18 people), (iii)
coat (18 people), as shown in Fig. 4 (d) ∼ (f).

the person identiﬁcation are controlled adaptively. We
show the effectiveness of the proposed method throughexperiments with a new shadow database including 54
people.
This paper is organized as follows. Section 2 describes
a new shadow database with a variety of clothes and bags,
and section 3 describes the details of the proposed person
identiﬁcation method. Section 4 explains the experiments
using the database. Conclusions are presented in section
5.

IR light

CameraInfrared lights

II. I NVISIBLE SHADOW DATABASE
Figure 1 shows examples of captured images in which
a subject’s shadow was projected by an infrared light(Bosch, IR Illuminator 850 nm, UFLED30-8BD). Figure
1 (a) shows an example of captured images with a visible
light transmitting ﬁlter, which are equivalent to images
people see with their eyes, and Fig. 1 (b) shows anexample of images with an infrared transmitting ﬁlter.
These captured images show that a shadow by the infrared
right is not visible to human.
IR light

Camera with an
infrared transmitting filter

IR light

Walking
direction

Walking direction

Shadow invisible tothe subject(a)Figure 2.

(b)(a) Experimental setting, (b) actual scene.

IR light

Shadow1

Body
Shadow2

(a)
(a)

(b)

Figure 3.

Figure 1.
Examples of captured shadows projected by an infrared
light, (a) the image with a visible light transmitting ﬁlter (this image isequivalent to how people see), (b) the image with a infrared transmitting
ﬁlter.

(b)(a) Standard cloth, (b) an example of captured images.

III. P ERSON IDENTIFICATION USING INVISIBLESHADOW

In this section, we describe the details of the proposed
method. At ﬁrst, we describe a method of feature extraction from gait images, and we introduce the person
identiﬁcation method which achieves higher performancecompared with the previous method [9] and is robust to
appearance changes.

To collect a shadow database, we used two infrared
lights and a camera (PointGrey Research Inc., Grasshopper2 M/C). The infrared lights were placed obliquely andthe camera was placed on the ceiling perpendicular to theground as shown in Fig. 2. Previous papers showed thatfront-view gait images had the best performance comparedwith the rest of viewpoints [12], thus we placed oneof the cameras to capture front-view images of walking
people. We placed the other camera to get lateral-view
images which produce a big difference of their appearance compared with the frontal images. The velocity ofthe subject in the observation area is constant because
the initial acceleration of walking occurred outside theobservation area. The image resolution and the frame rateare 1600 × 1200 and 30 Hz, respectively. The number ofsubjects is 54, 6 gait sequences for each, which consistsof 4 standard gait sequences (IRSD-ST), 1 carrying-bag
sequence (IRSD-BG), and 1 changing-cloths sequence(IRSD-CL). Figure 3 (a) shows the standard cloth and an
example of captured images. The sequences in the IRSDBG are categorized into 3 categories, (i) backpack (18A. Extraction of a subject’s area and gait features
First, a background subtraction method is applied tocaptured images to a subject’s area including his bodyand shadow areas. Figure 5 shows results of backgroundsubtraction of Fig. 3 (b) and one of Fig. 4 (a2) (backpack).
There is noise and deﬁcit in the silhouette images.
Next, afﬁne moment invariants are extracted as gaitfeatures [6]. Afﬁne moment invariants are moment-based
descriptors, which are invariant under a general afﬁnetransform. The moments describe shape properties of an
object as it appears. For an image the centralized momentof order (p + q) of an object O is given by(x − xg )p (y − yg )q I(x, y). (1)μpq =(x,y)∈O

35

(a1)

(d1)

(a) Backpack(d) Down jackaet

(a2)

(b1)

(d2)

(e1)

Figure 4.

(b) Hand bag

(b2)

(c1)

(e2)

(f1)

(e) Laboratory coat(c2)(c) Traveling bag(f2)(f) Coat(a1) ∼ (f1) Variations of clothes and bags, (a2) ∼ (f2) examples of captured images.

features in the training sequences is calculated as follows.
djn,s,f = w ASU B j − w ADB n,s,f (a)Here, w ASU B j and w ADB n,s,f show whitened afﬁnemoment invariants of the subject and a person in the
database, respectively. j, n, s, and f are 1 ≤ j ≤ J (J isthe number of images of the one gait cycle of the subject),1 ≤ n ≤ N (N is the number of people in the database),1 ≤ s ≤ S (S is the number of sequences of each personin the database), and 1 ≤ f ≤ F (F is the number ofimages of the one gait cycle in the database), respectively.
 ·  means euclidean norm of ·. The person, who has thesmallest distance, in the training dataset, is voted at eachframe of the subject’s sequence. Figure 6 shows examplesof the voting. In the identiﬁcation phase, the person who
get the most votes is chosen as the subject.

(b)

Figure 5. (a) Extracted subject area of Fig. 3 (b), (b) extracted subject
area of Fig. 4 (a2) (backpack).

Here, xg and yg are the center of the object and I(x, y)
represents the intensity of the pixel (x, y). The number of
afﬁne moment invariants (A = (A1 , A2 , . . . . , AM )T ) is
M ; we show two of them in the following [4].

A1

=

A2

=

1
(μ20 μ02 − μ211 )μ400
1(μ230 μ203 − 6μ30 μ21 μ12 μ03 +μ10
00
4μ30 μ312 + 4μ03 μ321 − 3μ221 μ212 )(3)t......

(2)

VotingDatabase A

C

A

A

B. Method robust to appearance changes
Figure 6.

The proposed method is based on the voting-based
method [1] and the person identiﬁcation method robust to
appearance changes [8]. We brieﬂy explain both methods
in this section, and introduce the proposed method.
1) Voting-based method [1]: The voting-based method[1] is proposed to achieve high performance even withvariations of center positions of a subject’s areas insequential silhouette images. In the training phase, gaitfeatures of afﬁne moment invariants are extracted fromeach image in the training sequences. Then, gait features
are extracted from each image of a subject’s sequence inthe test dataset. At each frame of the subject’s sequence,the distance between the feature of the subject and all

An example of the voting-based method.

2) Person identiﬁcation method robust to appearancechanges [8]: In the method robust to appearance changes,
gait features are extracted from an average image createdfrom silhouette images of one gait cycle. At ﬁrst the
subject’s area is divided equally into K areas according to
the height of the subject’s area. Figure 7 shows examples
of a human body area divided into 5 areas. At each area
afﬁne moment invariants are calculated. Database is built
from a set of afﬁne moment invariants of multiple people
who wear standard clothes without belongings.
Next a matching weight at each area is estimatedaccording to the similarity between the features of the36

Fig. 9 (b) ∼ (d). In terms of the option (i), gait informationof a left shadow area in a silhouette image cannot be extracted efﬁciently, since to extract gait features efﬁciently
from the left shadow area, it is desirable to divide theleft area with vertical lines. Compared with the option(i), the option (ii) can efﬁciently extract gait informationfrom each shadow area. In terms of the option (iii), if asubject’s appearance changes, there is a chance that in all
three areas the subject’s appearance changes. This resultsin decreasing the performance of people identiﬁcation. If
we divide a silhouette area into more areas than the areasof the option (iii), we have bigger chance to identify the
subject with higher performance. Thus in the proposed
method, we use the option (ii), to divide the subject area
equally according to the height and width, and we extract
gait information from each area separately. Figure 8 shows
examples (the number of divided areas K = 1, 4, and
6). Comparisons of the options (ii) and (iii) are shownin experiments.
The performance of the previous method using theinfrared-based shadows [9] gets worse in both cases ofappearance changes of a subject and variations of center
positions of the subject’s areas. On the other hand, theproposed method achieves better performance, which isshown in experiments. Moreover, an advantage of the proposed method is to identify the subject whose appearancechanges, even if the proper number of divided areas is notknown, since the priorities of divided areas for the person
identiﬁcation are controlled adaptively.

Matching weight

Area
1
2

High
High

3

Low

4

Slightly-high
High

5
(a)

(b)

Figure 7. (a) An example of average images in the database, (b) an
example of average images of subjects with a shoulder bag.

subject and those in the database. In case that the subject’s
appearance is different from that in the database as shownin Fig. 7, matching weights of areas with appearancechanges are set to low. On the other hand, matchingweights of areas with less appearance changes are set tohigh. This method doesn’t utilize gait features extracted
from areas with low matching weights, which are due
to changes of clothes / belongings, but utilizes featuresfrom areas with high matching weights. Therefore themethod enables person identiﬁcation robust to changes inappearance.
The distance between the feature of the subject and thatin the database is used to determine a matching weight ateach area, and small and large distances mean high andlow matching weights, respectively. The procedure to estimate the matching weight is brieﬂy explained as follows.
At each area k, a threshold th(k) is set adaptively, and weselect sequences from the database if dn,s (k) < th(k).
These selected sequences are considered to have highsimilarities. Non-selected sequences can be considered tohave low similarities, and the distance dn,s (k) is redeﬁned
as a value dmax (k), which is the largest distance among
all sequences at area k. The above process is done allareas, followed by people identiﬁcation. For more details,
please refer to [8].
3) Proposed method: In the paper [8], Iwashita et
al. applied the method, which is robust to appearancechanges, to the CASIA gait database, and obtained thehigher performance compared with conventional methods[2] [13]. In the present paper, we apply the procedure
explained in section 3.2.2 to the voting-based method to
calculate the matching weight at each frame, and replace
the distance of Eq. 3 with the matching weight. We
also divide a subject’s area into multiple areas, and thevoting-based method with the matching weight explained
in section 3.2.2 is applied to each area, followed by person
identiﬁcation.
In images in the shadow database, there are two shadow
areas and one body area, as shown in Fig. 3 (b). There areseveral options to divide a silhouette area, which includes
two shadow areas and one body area, into multiple areas
such as (i) areas divided according to the height of thesilhouette area, using the same process with [8], and (ii)areas divided according to the height and width as shownin Fig. 8, and (iii) each shadow and body area as shown in

(a)

(b)

(c)

Figure 8. Examples of divided areas (the number of divided areas K =1, 4, and 6).

IV. E XPERIMENTS
This section shows the results of person identiﬁcation
experiments using the shadow database in Section 2.
The database contains 54 people with 6 sequences forevery subject. Six sequences consist of 4 standard gait
sequences (IRSD-ST), 1 carrying-bag sequence (IRSDBG), and 1 changing-cloth sequence (IRSD-CL). We carried out three experiments: (i) person identiﬁcation withstandard sequences (IRSD-ST), (ii) person identiﬁcation
with appearance-changed sequences (IRSD-BG and IRSDCL), and (iii) person identiﬁcation to validate discrimination capabilities of shadow and body areas. The correctclassiﬁcation rate was estimated based on the idea of 4fold cross validation. The concrete procedure is as follows.
Three sequences in the IRSD-ST are used as training

37

Figure 9.

(a) Whole area(b) Shadow 1(c) Shadow 2(d) Bodychanges [8]. We varied the parameter of K (the numberof divided areas) from 1 to 20 and M (the total number
of afﬁne moment invariants) from 1 to 80. We tested
all combinations of K and M . Figures 11 and 12 show
examples of CCRs (K=4, 8, 14, and 20) of IRSD-BG and
IRSD-CL. From these results, it is clear that the proposed
method outperformed the voting-based method and CCR
increased with the parameters K and M . However, CCRsof IRSD-CL are relatively lower than those of IRSD-BG,
thus we checked the CCRs for each category of IRSD-CL,
as shown in Table I. From these results, the CCR of down
jacket is the worst among all categories. One of the reasonsis that the discrimination capability of the subject’s areaof down jacket decreased due to the thickness of the down
jacket.

Examples of divided areas into each shadow and body area.

100
90
80
70
60
50
40
30
20
10
0

datasets, thus there are 4 kinds of combinations of training
datasets. Regarding a test dataset, in case of the IRSDST, the rest of the sequence which is not used in the
training dataset is used as a test dataset, and in case ofboth IRSD-BG and IRSD-CL each sequence is used. A
correct classiﬁcation rate is calculated by averaging resultsof 4-fold cross validation.

20

40

60

80

CCR [%]

M
0

20

40

60

80

(b)

Figure 11. Comparison of correct classiﬁcation rates of (a) the proposed
method and (b) the voting-based method (IRSD-BG).

In this experiment, to show the effectiveness of thevoting-based method [1], we compared the voting-based
method [1], which is not robust to appearance changes,
and a conventional method which used average images
to extract gait features [6] [9]. We did not divide thesubject area (K, the number of divided areas, is 1) and
we varied the parameter of M (the total number of afﬁnemoment invariants) from 1 to 80. Figure 10 shows theresults of the experiments with respect to the change ofM . From these results, it is clear that the voting-based
method outperformed the conventional method.

100
90
80
70
60
50
40
30
20
10
0

CCR [%]

M
0

20

40

60

100
90
80
70
K=4
60
K=8
50
K=14 40
K=20 30
20
10
0

80

CCR [%]

M
0

20

(a)

40

60

80

(b)

Figure 12. Comparison of correct classiﬁcation rates of (a) the proposed
method and (b) the voting-based method (IRSD-CL).

CCR [%]

Table IC ORRECT CLASSIFICATION RATE OF EACH CATEGORY OF
CASIA-B-BG BY THE PROPOSED METHOD [%].
The voting-based method
A conventional method

CCR [%]

20

40

60

Down jacket
26.4

Laboratory coat
77.8

Coat
91.7

C. Person identiﬁcation to validate discrimination capabilities of shadow and body areas
As we explained in Section 3.2.3, there are anotheroptions to divide a subject’s area, and one of them is todivide the subject’s area into each body and shadow area[7]. In this section we separate the body and shadow areas
as shown in Fig. 9, by detecting the lowest pixel of shadow
1 and the leftest pixel of shadow 2, and extract featuresfrom each area followed by person identiﬁcation. We carried out ﬁve experiments with standard sequences (IRSDST), carrying-bag sequence (IRSD-BG), and changingcloth sequence (IRSD-CL):

M
0

M
0

100
90
80
70
K=4
60
K=8
50
K=14 40
K=20 30
20
10
0

(a)

A. Person identiﬁcation with standard sequences (IRSDST)

100
90
80
70
60
50
40
30
20
10
0

CCR [%]

80

Figure 10. Correct classiﬁcation rates by the voting-based method [1]and a conventional method [6] (IRSD-ST).

B. Person identiﬁcation with appearance-changed sequences (IRSD-BG and IRSD-CL)
In next experiments, we applied the proposed method to
IRSD-BG and IRSD-CL. In addition to that, we applied
the voting-based method [1], which does not utilize themethod robust to appearance changes, to see the effectiveness of the use of the method robust to appearance

38

(1)

Person identiﬁcation with a whole area (i.e. thearea was not divided, Fig. 9 (a))(2)
Person identiﬁcation with shadow1 (Fig. 9 (b))(3)
Person identiﬁcation with shadow2 (Fig. 9 (c))(4)
Person identiﬁcation with body (Fig. 9 (d))(5)
Person identiﬁcation with combination of
shadow1, shadow2, and bodyHere, the voting-based method [1], which is not robust to
appearance changes, is applied for evaluation. Figures 13∼ 15 show results of the above ﬁve experiments with standard sequences (IRSD-ST), carrying-bag sequence (IRSDBG), and changing-cloth sequence (IRSD-CL). We varied
the parameter of M from 1 to 80. Figures 13 ∼ 15 also
show results by the proposed method with K = 20. The
results show that the proposed method achieved the highest
performance. Interestingly, the results of shadow 1 showed
better results compared with other areas (i.e. body andshadow 2). This is because the area of shadow 1 tends toreceive smaller inﬂuence of appearance changes comparedwith other areas.
100
90
80
70
60
50
40
30
20
10
0

100
90
80
70
60
50
40
30
20
10
0

60

80

Whole area
Shadow 1
Shadow 2
Body
Combination of 3 areas
The proposed method

M
20

40

60

60

80

[1] Authors. Identiﬁcation of people walking along curved
trajectories. Pattern Recognition Letters, Supplied as additional material prl.pdf.
[2] K. Bashir, T. Xiang, and S. Gong. Gait recognition
without subject cooperation. Pattern Recognition Letters,
31(13):2052–2060, 2010.
[3] I. Bouchrika, M. Goffredo, J. Carter, and M. Nixon. On
using gait in forensic biometrics. Journal of Forensic
Sciences, 56(4):882–889, 2011.
[4] J. Flusser, T. Suk, and B. Zitova. Moments and moment
invariants in pattern recognition. Wiley & Sons Ltd., 2009.
[5] J. Han and B. Bhanu. Individual recognition using gaitenergy image. IEEE Trans. PAMI, 28(2):316–322, 2006.
[6] Y. Iwashita and R. Kurazume. Person identiﬁcation fromhuman walking sequences using afﬁne moment invariants.
Proc. IEEE Int. Conf. Robotics and Automation, pages 436–
441, 2009.
[7] Y. Iwashita, A. Stoica, and R. Kurazume. Gait identiﬁcation
using shadow biometrics. Pattern Recognition Letters,
33:2148–2155, 2012.
[8] Y. Iwashita, K. Uchino, and R. Kurazume. Gait-based person identiﬁcation robust to changes in appearance. Sensors,
13(6):7884–7901, 2013.
[9] Y. Iwashita, K. Uchino, R. Kurazume, and A. Stoica. Gait
identiﬁcation from invisible shadows. SPIE BiometricTechnology for Human Identiﬁcation IX, 2012.
[10] M. Nixon, T. Tan, and R. Chellappa. Human identiﬁcation
based on gait. Springer, 2005.
[11] K. Sugiura, Y. Makihara, and Y. Yagi. Gait identiﬁcation
based on multi-view observations using omnidirectional
camera. Asian Conf. on Computer Vision, 1:452–461, 2007.
[12] S. Yu, D. Tan, and T. Tan. A framework for evaluating theeffect of view angle, clothing and carrying condition on gait
recognition. Int. Conf. on Pattern Recognition, 4:441–444,
2006.
[13] E. Zhang, Y. Zhao, and W. Xiong. Active energy image plus
2dlpp for gait recognition. Signal Processing, 90(7):2295–
2302, 2010.

CCR [%]

040

R EFERENCES

Figure 13. Comparison of correct classiﬁcation rates of (1) whole area,(2) shadow 1, (3) shadow 2, (4) body, (5) combination of 3 areas, and(6) the proposed method (K = 20) (IRSD-ST).

100
90
80
70
60
50
40
30
20
10
0

20

ACKNOWLEDGMENT
This work is supported by Strategic Information andCommunications R&D Promotion Program (SCOPE)
No.121810005.

M
40

M
0

which are projections of one’s body on the ground by
multiple lights, can be considered as body areas captured
from different viewpoints. We carried out experimentswith the new database, and showed the robustness of the
proposed method compared with conventional methods
against appearance changes in all, except one clothing
condition (down jacket).

Whole area
Shadow 1
Shadow 2
Body
Combination of 3 areas
The proposed method

20

Whole area
Shadow 1
Shadow 2
Body
Combination of 3 areas
The proposed method

Figure 15. Comparison of correct classiﬁcation rates of (1) whole area,(2) shadow 1, (3) shadow 2, (4) body, (5) combination of 3 areas, and(6) the proposed method (K = 20) (IRSD-CL).

CCR [%]0

CCR [%]

80

Figure 14. Comparison of correct classiﬁcation rates of (1) whole area,(2) shadow 1, (3) shadow 2, (4) body, (5) combination of 3 areas, and(6) the proposed method (K = 20) (IRSD-BG).

V. C ONCLUSION
We proposed a shadow-based person identiﬁcation technique robust to appearance changes caused by variations ofclothes and carried bags conditions. In addition, we built a
new shadow database with a variety of clothes and bags.
The system to collect the database consists of multiple
infrared lights and a camera, and it has the advantages ofan invisible and undetected sensing system. Shadow areas,39