Grenoble INP  Ensimag
École Nationale Supérieure d'Informatique et de Mathématiques Appliquées

Final year project report
Performed at Okayama University

Human behavior analysis

Analysis of social relation and interaction in pedestrian groups

Adrien Gregorj

3rd year  Option MMIS Bio
19th February, 2018  20th July, 2018

Supervisors

1-1, Tsushima-Naka, 1-Chome

Akito Monden
Zeynep Yücel

Okayama 700-8530

School Tutor

Okayama University

Japan

James Crowley

2

CONTENTS

Contents
1

Context

42

Motivation and ob jectives

43

Background and related work

5

3.1
3.2
3.3

5
6
6

4

5

Crowd modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Gait analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Action recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Recognition of social relations

7

4.1
4.2
4.3
4.4
4.5
4.6

7
8
10
11
12
14

Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Observables and empirical distributions . . . . . . . . . . . . . . . . . . . .
Bayesian model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Alternative methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Detection of gestures during interactions

14

5.1
5.2
5.3
5.4
5.5
5.6

16
18
18
19
20
21

Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Pose estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Angles extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Pitch detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

Estimation of intensity of interaction

22

7Conclusion

24

8Work organization

25

9Societal and environmental impact

25

9.1
9.2

26
26

Societal impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Environmental impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10 Personal feedback

10.1 Work environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2 Benets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2.1 Working experience . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2.2 Personal experience . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.3 Diculties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.4 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

27
27
27
28
28
29

CONTENTS

3

A Detailed performance of alternative methods for recognition of social
relation

35

B

Contact information

35

C

French abstract

37

4

1

2

MOTIVATION AND OBJECTIVES

Context

This internship takes place in an academic context. The home structure is the Graduate
School of Natural Science and Technology of Okayama University. In particular, the hostlaboratory is part of the Division of Industrial Innovation Sciences, in the Department ofComputer Science.
Okayama University is a well-ranked university in Japan located in Okayama Prefecture,in the Chugoku region of the main island of Honshu. The school was founded in 1870 andwelcomes around 14000 students (10000 undergraduates, 3000 postgraduates and 1000
doctoral students). It's motto is  Creating and fostering higher knowledge and wisdom .
Okayama University and Université Grenoble Alpes have a long history of exchange thathas been reactivated for the past three years with around a dozen of Grenoble students
performing research oriented internship in Okayama every year. As a University Research
Administrator, Dr Bernard Chenevier's mission is to develop research collaborations at
the international level and, for this purpose, he works to strengthen the bond between
these two universities. It is in this context that internship topics are proposed to French
students.
The Graduate School of Natural Science and Technology was originally established in
April 1987. It focuses on research in fundamental sciences such as global climate change,
plant photosynthesis or supernova neutrinos. The Division of Industrial Innovation Sciences
works especially on applied engineering in the eld of computer science, robotics, materialsciences among others.
In the Department of Computer Science, the topics of research are the basic theory
and application of information technology, articial intelligence and computer technology.
Examples of research projects include the development of a visualization tool for a processor
or human tracking algorithm by means of attention control. Dr Akito Monden is theprofessor of the Theory of Programming and Articial Intelligence laboratory, where thisinternship takes place and Dr Zeynep Yücel is the assistant professor in the same laboratory.

2

Motivation and objectives

Dr Zeynep Yücel has been working for a few years on modeling crowd movements [1][2][3].
This study is an extension to their work in the last couple of years.
The motivations behind constructing crowd or pedestrian motion models are numerous:
testing architectural designs for an evacuation process, designing accurate crowd models
for movies or video games, etc., and it is a very active eld of research as it is requisite inrecent domains such as autonomous driving.
One of the main challenges behind pedestrian motion modelling is to take into account the social relation and interactions between social pedestrian groups. Indeed, [3] hasdemonstrated that the motion patterns of individuals performing gestures (i.e. physicalcontact, mutual gaze, conversation and arm gestures) are signicantly dierent from theones of not-interacting individuals. In addition, [4] proved that pedestrian pairs, which are5in dierent social relations, have dierent motion characteristics. Both [3] and [4] rely onhuman annotations to model the dierent motion patterns and evaluate their performance.
Therefore, this project aims to develop methods to automatize the recognition andclassication process of pedestrians'. Namely, our goals are twofold: (i) recognition ofsocial relations and (ii) detection of gestures. We study each of these problems in Section 4
and Section 5. For recognition of social relations we devise a probabilistic method with a
Bayesian approach, whereas for detecting the gestures of interaction we propose a method
inspired from audio signal processing.
Thanks to the methods proposed in Sec 4 and 5, instead of relying on human referees
to decide whether or not a set of pedestrians are presumably in a given relationship andinteracting (performing gestures) in a given video, this decision can be made automatically,
which enables real time processing and reaction of social interaction robots.

3

Background and related work

This section will detail the previous works that have been done in the dierent domainsthat this project involves.

3.1 Crowd modeling
When modeling the crowd, two kinds of models are generally in use:
This kind of models [5, 6, 7] is used to describe often high density
crowds. They are generally based on dynamic uid representation of the people
and study the density and ow of pedestrians in simulated environments (see Figure1a). Their applications include planing evacuation processes or large scale events foravoiding stampedes by identifying high density areas.

Macroscopic models:These models, where the most prominent one is Social Force Model [8],describe pedestrians as particles with a certain position and velocity and subject toa variety of forces: attractive forces from other particles of the same group, repulsive forces from obstacles like walls or other pedestrians, etc (see Figure 1b). They
are used in modeling the interaction at a smaller scale and can be used to develop
autonomous agents such as companion robots or smart wheelchairs.

Microscopic models:This study focuses on motion of pedestrians in small groups. The most popular modelfor this topic is the Social Force model (SFM). Many studies based their models on theoriginal SFM and extended it to enhance the simulated behaviors of pedestrians to prevent
sudden changes in velocity or direction [11, 12].
Most recently, data driven techniques such as neural networks have been used to compute socially acceptable trajectories [13, 14]. Mainly, networks are trained on real trajectorydata to generate missing portions considering the beginning of a trajectory.

6

3

(a)

BACKGROUND AND RELATED WORK(b)

Figure 1: (1a) Macroscopic [9] and (1b) microscopic [10] crowd models illustrations.

3.2 Gait analysis
Motion patterns of pedestrians are signicantly aected by gait. Length and frequency ofsteps of walking individuals have been particularly studied to inventory empirical values
for panels of people in various environments.
In [15], Hui et al. manually measured the gait parameters of pedestrians to study theircorrelation with gender and age. A particularly vast number of studies have investigated
the impact of age on gait parameters [16, 17, 18, 19, 20]. The recurring nding is thatolder people exhibit slower walking speed and shorter step length than younger ones. Sun
et al. analysed the impact of surface slope on human gait and discovered that pedestrians'
step length decrease during ramp descent [21].
The uniqueness of human gait rythme has been established in various studies [22,23]. This specicity was found to be a discriminating feature to build pedestrian detectors [24][25] or dierentiate pedestrians from other moving vehicles [23]. Moreover, gait
analysis has also been used as a way to identify people's gender or age range [26][27].
Few studies present automatized methods to measure the gait parameters of pedestrians. In [28], Niyogi et al. detect the characteristic braided pattern of walking in thespatiotemporal volume by computing an autocorrelation sequence of trajectory values (theinverse Fourier transform of the magnitude of the Fourier transform) and nding the peakin this sequence. Saunier et al. [29] and Hediyeh et al. [30] compute power spectra of speed
proles to extract gait frequency of pedestrians.

3.3 Action recognition
The problem of action detection and recognition has been addressed in many ways as it is at
the center of various other topics such as video surveillance or human-computer interaction.
Initial methods used dense trajectories, where feature points are tracked across frames anddescriptors such as Fisher Vectors are computed to capture the motion information of thesubjects in the video [31].
More recently, machine learning algorithms, such as neural networks, have been widely
applied to this task. Skeleton-based action recognition using Recurrent Neural Networks

7

Figure 2: Density map of the environment.
(RNNs) [32, 33] or their extension using Long Short-Term Memory (LSTMs) units [34] has
shown signicant improvements over the previous methods.
Recently introduced Graph Convolutional Networks (GCNs) [35] generalize the convolution operation on 1D and 2D arrays to graph data structure. In [36], Yan et al. proposea Spatio-Temporal Graph Convolutional Network (STGCN) to classify actions in videos.
Their study relies on building a graph composed of the skeletons of a given person at
multiple successive frames, each joint being linked to its detection on the previous andfollowing frame. Their classier takes this graph as input to perform action classication.

4

Recognition of social relations

This section focuses on the impact of social relations on pedestrian motion. The denitionof the relationships is based on [37], where Bugental proposes a domain-based approachand divides social life into ve non-overlapping domains as attachment (e.g. family),hierarchical power, mating (e.g. couple), reciprocity (e.g. friends) and coalitional (e.g.
colleagues).
In this study, hierarchical relation is eliminated, since it does not apply to the full
extent to pedestrians in a public space. For this simplied case, we contain ourselves tothe two most distinct social relations as pointed by [4]: mating and coalitional.

4.1 Dataset
The dataset used for studying the impact of social relations on walking patterns wasintroduced in [38]. This dataset was obtained by setting up a tracking environment in theATC shopping center in Osaka. The system was composed of multiple 3D range sensors,
covering an area of about 900 m2. It is composed of a set of trajectory data les containing
the coordinates of the pedestrians, their height as well as their velocities.
For annotation purposes, the tracking area was recorded with 16 cameras. A subset ofthe footage was annotated by human coders to identify the pedestrian social groups andrelation (apparent purpose of the visit, apparent gender, apparent relation, apparent age)for 988 dyads. The relation between dyads is distributed as follows: 358 coalitional, 9684RECOGNITION OF SOCIAL RELATIONS

Figure 3: Schema of the observables depicted on a single dyad.
mating, 216 attachment and 318 reciprocal.

4.2 Observables and empirical distributions
The raw trajectory les are exposed to a preprocessing operation. Namely, the data points,
which are not uniform over time, are averaged over intervals of 0.5 s to obtain uniform
sampling, and to decrease the eect of measurement noise and pedestrian gait. Figure 2
shows the cumulative density map of the environment.
From the preprocessed data, a certain number of observables are computed for eachdyad at each sampling instant according to the group reference frame. Namely, we adopt a
dynamic reference frame such that its x-axis is always aligned with the direction of motion.
This enables eliminating the eect of maneuvering for taking curves or avoiding obstacles.
The observables, which are illustrated on Figure 3, are dened explicitly as follows:
1. Interpersonal distance, δij , is dened as the distance between the peers.
2. Group velocity, vg , is the magnitude of the average instantaneous velocities of thepeers,~vi + ~vj.
vg(i,j) =2
3. Absolute dierence of velocity vectors, ωij , is dened as the magnitude of the dierence vector,
ωij = |~
vi − v~j | .
4. Height dierence, is∆η = |ηi − ηj |,where ηi and ηj stand for the height of the pedestrians pi and pj , respectively.

4.2

9

Observables and empirical distributions

0.005

2

C
M

0.004

C
M

1.5

p(δ)

p(vg)

0.003

1

0.002
0.5
0.001

0

0
0

500

1000

1500

2000

0

0.5

1

1.5

2

2.5

vg (m/s)

δ(mm)

(a)

(b)

8

0.01

C
M

C
M

0.008
6

p(Δη)

p(ω)

0.006
4

0.004

2

0

0.002

0
0

0.1

0.2

0.3
ω(m/s)

(c)

0.4

0.5

0

100

200

300

400

500

600

Δη(mm)(d)Figure 4: Empirical distributions of (4a) δ , (4b) vg , (4c) ω and (4d) ∆η , of dyads incoalitional (C, blue) and mating (M, red) relation.

10

4

RECOGNITION OF SOCIAL RELATIONS

After computing the observables, their probability density distributions are approximated with cumulative histograms (see Figure 4). It is clear that the distributions are
signicantly dierent. Moreover, an ANOVA analysis conrmed that the observables δ ,vg , ω and ∆η have a p-value smaller than 10−4 . Considering that in the literature ([7]), a
value lower than 0.05 is generally admitted to demonstrate statistical signicance, it can beconcluded that coalitional relation and mating relation are considerably dierent in termsof the observables.

4.3 Bayesian model
Following the ndings of [4], the idea is to build a predictive model based on the a priori knowledge of the observable distributions to classify dyads into their apparent social
relations as coalitional or mating.
Let Σ(t) = {δ, vg , ω, ∆η } be the set of observables at time t. The probability that Σ(t)comes from a group in social relation of r, where r ∈ {C, M } can be computed as

Pt (r|Σ) =Pt (Σ|r)Pt (r).
Pt (Σ)(1)

The term Pt (Σ|r) can be decomposed as the product of probabilities of each single
observable, provided that their independence can be assumed. To verify this hypothesis,
the Jaccard distance is computed.
Let Θ and ∆ be two random variables. Their Jaccard distance is dened asD(Θ, ∆) =H(Θ, ∆) − I(Θ, ∆),H(Θ, ∆)(2)

where H(Θ, ∆) and I(Θ, ∆) are the joint entropy and mutual information of the variables,
respectively, and are described as

H(Θ, ∆) = −X(3)p(θi , δj ) log2 (p(θi , δj )),i,jI(Θ, ∆) =Xi,jp(θi , δj ) log2p(θi , δj )p(θi )p(δj )


.

(4)

The distance D(Θ, ∆) should be 1 for uncorrelated variables and closer to 0 for correlated ones.
Table 1 shows Jaccard distance values between pairs of observables for two sample
subsets of dyads coming from coalitional and mating relations. The results being always
higher than 0.94, the independence hypothesis can be considered as reasonable.
Therefore, the term Pt (Σ|r) can be decomposed as follows,

Pt (Σ|r) = Pt (δ|r)Pt (vg |r)Pt (ω|r)Pt (∆η |r).

(5)

4.4

11

Results

Table 1: Jaccard distance between observables for subsets of (1a) coalitional and (1b)mating relations.
(b)

(a)

vg
ω
δ
∆η

vg

ω

δ

∆η

0
-

0.98
0
-

0.99
0.98
0
-

0.98
0.97
0.98
0

vg
ω
δ
∆η

vg

ω

δ

∆η

0
-

0.96
0
-

0.97
0.95
0
-

0.96
0.94
0.96
0

Without any prior belief, the probability of being in a C or M relation are initializedwith unbiased equal probabilities,P0 (r) = 0.5 0.5 .
(6)As time elapses, we propose updating the prior as in Equation 7, where the parameter αdenes the rate of update,

Pt (r) = αP0 (r) + (1 − α)Pt−1 (r).

(7)

In this manner, Pt (C) and Pt (M ) are computed at every time instant t and the relation
associated with the larger probability is considered as the estimated relation at that instant.
For quantifying estimation performance, the mean rate of correct detections over all time
instants for all dyads are reported.

4.4 Results
Table 2: Detection performance for varying α (in %).

α

C

M

Total

0
0.5
1

88.5 ± 2.1
88.1 ± 2.1
87.1 ± 2.3

73.4 ± 5.0
79.1 ± 3.8
81.3 ± 4.1

85.3
87.2
86.5

In practice, 30% of the pairs are randomly chosen and their trajectories are used tobuild the probability density functions in Eq. 5. The remaining 70% are used to test theestimation method. Moreover, repeating this validation procedure 20 times, the mean andstandard deviations of performance values are computed to investigate the sensitivity (i.e.
dependence) of the observables on training set. By randomly picking 30% of the entire

124RECOGNITION OF SOCIAL RELATIONS

samples and repeating this procedure 20 times, the probability that a particular sample is
not used for training falls below 10−3 .
From the recognition rates presented in Table 2, it is observed that coalitional relationis recognized with a somewhat higher rate for all values of α, which could be due to theimbalance of samples in the dataset as given in Section 4.1. Moreover, although the overall
performance rate for α = 1 seems sligtly lower than that of α = 0, taking a xed andunbiased prior is regarded to perform better in the sense that it yields a more balanced
estimation performance for the social relations. In addition, due to the very low standard
deviation values, the eect of random shuing is regarded to be minute, which suggeststhat the observables are stable across samples and the method is resilient to changes intraining set.
It is important to note that the algorithm relies on previous annotations made bycoders for the training step (construction of the distributions). Yet it proves to be able togeneralize the classication process to new data.

4.5 Alternative methods
The proposed method detailed in Section 4 considers each set of observations Σ(t) ={δ, vg , ω, ∆η } collected at every sampling instant t and yields a probability of being in a Cor M relation. In that sense, it enables a local decision, which may gradually evolve.
In this section, we describe a few alternative methods, which work on the entire set ofobservations for all time instants providing global decisions. In other words, for a particular
dyad of interest, we obtain distributions of each observable at the end of its trajectory andcompare them to the corresponding representative distributions of C and M relations. At
this point, we need to measure the distance between each pair of probability distribution
on a statistical manifold. Therefore, it is important to dene proper distance metrics.
In what follows, we dene the metrics that we considered. Let two discrete probability
distributions be denoted with P and Q,
1. Kullback-Leibler (KL) divergence from P to Q is dened as,

DKL (P ||Q) =Xi

P (i) log

P (i)
.
Q(i)

(8)

Equation 8 can be interpreted as the amount of information lost when Q is used toapproximate P . Obviously, DKL (P ||Q) 6= DKL (Q||P ). Therefore, in order to havea fair comparison we compute the divergence from the observed distribution to therepresentative distribution of C and M . In addition, the logarithmic term is likely
to suer from the zero probabilities, if the observation is not suciently long. We
tackle this issue simply by ignoring the terms with P (i) = 0 and Q(i) = 0.
2. Jensen-Shanonn (JS) divergence, is derived from KL divergence as follows,

1
1
DJS (P ||Q) = DKL (P ||M ) + DKL (Q||M ),
2
2

(9)

4.5

13

Alternative methods

where M = 21 (P + Q), and it has the advantage to be symmetric. Therefore, it does
not matter whether we compute DJS (P ||Q) or DJS (Q||P ), but the issue relating thezeros in the logarithmic term pertain.
3. Earth mover's distance (EMD) is dened as,
Pm Pn

EM D(P, Q) =fi,j di,j,
j=1 fi,j

i=1
j=1
P
m Pn
i=1

(10)

where fi,j is the ow between pi and qj and di,j is the ground distance between pi
and qj . If the distributions are pictured as pile of dirt, the EM D represents theminimum cost to turn one pile into the other (the cost corresponding to the amountof dirt moved times the distance by which it is moved).
4. Log likelihood follows,

LL(P ∼ Q) = −

X

(11)

pi log(qi ).

i

It represents the plausibility that the histogram of P was sampled from a distribution
Q. If Q has any zero values we skip those observations, since they make the entire
term undened.
To create a decision algorithm, we compute the previous metrics for each observables.
For the Kullback-Leibler divergence, the Jensen-Shanonn divergence and the EMD, weselect the relation that minimize these distances. Conversely, for the Log Likelihood, weselect the relation that maximize it.
Table 3: Detection performance for the other proposed metrics.
Metricvg

ωδProposed (α = 1)
KL
JS
EMD
LL

76.6
77.8
77.2
76.0
76.8

68.1
37.1
63.4
63.6
66.3

77.8 76.4
67.7 81.2
75.8 77.4
67.6 76.2
76.3 74.8∆ηThe performance obtained using these alternative methods are detailed in Table 3 (total
accuracy for both classes), together with a comparison to the proposed method for sameobservables. For the sake of brevity, we skip reporting detailed detection performance foreach class but we refer the reader to the appendix for these results.
Table 3 shows that the proposed metric has comparable or better performance to thealternative methods. A closer look into the individual detection rates reveal that the14

5

DETECTION OF GESTURES DURING INTERACTIONS

alternative methods may sometimes yield a high detection rate due to a bias to sort mostobservations as C , which has many more samples than M , and thus misleadingly increases
the total performance.
In addition, one important inference drawn from Table 3 is that blending all observables
into a single decision strategy improves the performance of the proposed method. It may be
worthwhile, to test whether this integration may help the alternative methods. However,
our expectation is that DKL and DJS suer more from the zeros in the logarithmic term,since such cases may arise more frequently (i.e from any one of the observables), reducing
the available amount of data used in the decision process. In addition, LL may haveproblems concerning machine precision due to the growing number of decimal terms in the
product.

4.6 Discussion
Using only trajectory and height information of pairs of pedestrians, we are able to estimate
their social relation with over 80% accuracy. Considering the challenge of the problem, theproposed method is regarded to achieve signicant accuracy.
To the best of our knowledge, this is the rst study on detection of social relationof interacting pedestrians. Therefore, it is not possible to compare our performance toalternative methods. We can propose several improvements or extensions to the current
study.
First of all, in this work we considered only the coalitional and mating relations but inthe future we could extend this study to discriminate more challenging and less distinct
relations such as attachment and reciprocity.
Moreover, other methods to classify the trajectories can be imagined. One promisingidea could be to develop a neural network that takes the observables as input and returns
the probability of each relation. One basic implementation could be done with a fully
connected Multi-Layer Perceptron (a basic fully connected neural network) that takes a xnumber of observables (from portions of the trajectory) to classify. However, LSTM neuralnetworks would be more suited for this task, as they take in account the chronologicalcomponent of the data.

5

Detection of gestures during interactions

In this section, we study detection of gestures of interaction. We rst choose a public
dataset with an abundant number of pedestrian dyads and annotate it for a set of gestures,
as conversation, gaze exchange, arm gestures and physical contact. Arm gestures turn out
to have the highest rate of inter-rater agreement, and we decided to develop a method for
detecting them.
The idea is that human body is subject to somewhat regular oscillations due to thewalking rhythm and any disruption of this regularity (around the wrist, elbow, shoulder

15

0.005

2

C
M
A
R

0.004

C
M
A
R

1.5

p(δ)

p(vg)

0.003

1

0.002
0.5
0.001

0

0
0

500

1000

1500

2000

0

0.5

1

1.5

2

2.5

vg (m/s)

δ(mm)

(a)

(b)

8

0.01

C
M
A
R

6

C
M
A
R

0.008

p(Δη)

p(ω)

0.006
4

0.004

2

0

0.002

0
0

0.1

0.2

0.3
ω(m/s)

(c)

0.4

0.5

0

100

200

300

400

500

600

Δη(mm)(d)Figure 5: Empirical distribution of (5a) δ , (5b) vg , (5c) ω and (5d) ∆η , of peers for dyads
in coalitional (C, blue), mating (M, red), attachment (A, green) and reciprocity (R,orange)
relation.

16

5

DETECTION OF GESTURES DURING INTERACTIONS

Figure 6: Top view of the Duke University campus with the elds of view of the cameras
used to record the DukeMTMC dataset in color.
area) can be considered to arise from a gesture within a social dyad. We propose using amethod inspired from pitch detection to pinpoint these disruptions.

5.1 Dataset
We started by searching for a dataset that could be exploited for the gesture detection task.
Many video datasets involving pedestrians are publicly available but most of them are notsuitable for studying gestures of pedestrian interaction. Some of them are based on bird's
eye view camera footage, in which pedestrian interactions can not easily be observed (BIWI
dataset [39]), whereas some datasets involve too sparse pedestrian trac, too short footage
or acted and unnatural behaviors (CAVIAR dataset [40], ETHZ [41], PETS 2009 [42]).
The DukeMTMC dataset, introduced in [43], is found to be the most appropriate set
for studying gestures of interacting pedestrians. It presents pedestrian trac lmed at
multiple locations in the campus of the Duke University. Figure 6 shows a top view of theelds of view of each camera of the datasets. In addition to a very large quantity of data
(85 minutes of 1080p and 60 fps video for 8 cameras, with more than 2,000 identities), it
involves several ground truth values.
Trajectories on image coordinate frame and real-world coordinates, as well as the pose
estimations of each pedestrian, are provided as ground truth. Trajectories on image plane
are obtained by manual annotation. Namely, coders were asked to mark the feet positionof pedestrians on certain key frames and the trajectory points between those were linearlyinterpolated. This gives piece-wise linear trajectories for each individual (see Figure 7). The
real-world trajectories are computed using the homography matrices of each corresponding
camera (that are also available).
In addition, a subset of this dataset (20 minutes of videos for the camera 1, 2, 4 and
5) was also annotated for groups: the DukeMTMC-Groups dataset, introduced in [44]. In

5.1

17

DatasetFigure 7: Example of annotations on the DukeMTMC dataset. Yellow dots were manuallyplaced by coders between the feet of the pedestrians and black lines are linear interpolation
between those keyframe annotations.
Table 4: Inter-rater agreement analysis
GestureSpeaking
Touching
Arms gesture
GazingIntensity

Cohen's κKrippendor's α

0.45
0.49

0.44
0.48

0.77

0.77

0.49

0.48

-

0.64

total, 64 groups are identied and tracked for an average of 400 frames per group.
In order to get the ground truth for gesture and intensity of interaction concerning
DukeMTMC-Groups dataset, we carried out an additional annotation task. Two coders
are asked to watch clips for each group on each camera and annotate the four kinds ofgestures: conversation, gaze exchange, arm gestures, physical contact. In addition, they
were asked to rank the intensity I of the interaction on a scale from 0 to 3, I = 0 being nointeraction and I = 3 the maximum level of interaction.
We performed inter-rater agreement analysis using the Cohen's kappa coecient (κ) andKrippendor's alpha coecient (α) [45]. The results are displayed in Table 4. Concerning
Cohen's κ, values greater than 0.67 are often considered to be statistically reliable, andvalues at the level of 0.80 are regarded to indicate to a substantial agreement. Since theagreement is best for the arm gestures, we decided to focus on this particular interactionin our analysis. Regarding Krippendor's α, there does not exist a benchmark value forsignicance but values around 0.66 are regarded as satisfactory [45].

18

5

DETECTION OF GESTURES DURING INTERACTIONS

Figure 8: Schema of the OpenPose joints numbering with the elbow and shoulder angles.

5.2 Pose estimation
As mentioned in Section 5.1, pose estimation had already been performed by the researchteam of the DukeMTMC dataset and is publicly available.
The pose estimations are obtained using the publicly available OpenPose library, whichis state of the art for this problem [46]. To identify joints in every frame, Cao et al. employ
body parts inference, which we briey explain in what follows.
The model is mainly composed of a multi-stage Convolutional Neural Network that
evaluates the condence maps for body parts (18 dierent joints: right and left elbow,
right and left feet, etc.) but also for what Cao et al. call  part anity elds  (PAF),
which correspond to vectors that encode the direction between the two joints for each pixel
on a given limb. Multiple iterations enable gradual renement of estimations so as to avoid
false detections (e.g. right wrist instead of the left).
A graph can be modeled using the detected body parts as vertices and by scoring theedges using the part anity elds with the outputs provided by the neural network. The
integral of the PAF over the line that joins the two parts gives a cost that encodes thecondence of the joints belonging to the same limb. In order to retrieve actual skeletonsfrom this graph, which is an NP Hard problem, the authors introduced a greedy algorithm
that performs well and rapidly. They actually subdivided the problem to work on smallerbi-partite graphs (considering successively all the possible association of joints: right wrist- right elbow, right foot - right knee, etc.) instead of considering all the joints at once.

5.3 Angles extraction
We dened and computed two kinds of angles: elbow and shoulder angles (see Figure 8).
Due to the occlusions or misdetections, there is the possibility of having some instability onthese values. For eliminating this sort of inconsistencies, a median ltering (with a windowof size 9) is carried on the elbow and shoulder angles. In addition, only those pedestrians

19

Pitch detection

200

200

160

160

120

120

Angle (°)

Angle (°)

5.4

80

40

0

80

40
extracted angles
processed angles
0

1

2

3

4

5

6

0

extracted angles
processed angles
0

1

t (s)

2

3

4

5

6

t (s)

(a)

(b)

Figure 9: (9a) Elbow and (9a) shoulder angles before and after preprocessing.
with less than 20% of missing values along their trajectory are considered in the analysis.
Figure 9 shows elbow and shoulder angles for a pedestrian in a non-interacting dyad. It
appears that the walking motion (swinging arms or just the overall oscillations due to thebody movements) causes a certain periodicity in the signal, which we intend to identify.
After examining the empirical observations and the videos, we decided to focus onelbow angles in our analysis mainly for the two following reasons. Firstly, the region of theshoulder joint is inherently broader than the region of the elbow, which makes estimationsof OpenPose vary more from one frame to the next. This noisy behavior of shoulder angle
estimations and how it compares to those of elbow angles can be seen on Figure 9. In
addition, most gestures encountered in pedestrian interaction (pointing, waving, etc.) are
observed to involve more pronounced motions of the elbow rather than the shoulder, whichputs greater importance on elbow angles.

5.4 Pitch detection
The problem of retrieving the oscillation of the walking motion from the angle data isvery similar to pitch detection problems, where the goal is also to identify a low frequency
periodicity from a noisy signal. Dierent methods exist to perform such tasks. In our
analysis, we decided to use the average magnitude dierence function (AMDF) introduced
in [47] as,
NX
−τ −1
1
|x(n) − x(n + τ )|,D(τ ) =N − τ − 1 n=0

where τ is the lag number.
For mere walking action, AMDF should resemble a sine wave and have minimas at lagscorresponding to the period of the walking oscillations and its multiples (see Figure 10).
Provided that the input signal contains a prominent periodic component, its AMDF
can be approximated with a sinusoidal waveform, D(τ ) ∼ y(τ ), such that,y(τ ) = A sin(ωτ + φ) + c.

(12)

20

5

DETECTION OF GESTURES DURING INTERACTIONS

200

3

160

Angle (°)

Angle (°)

2
120

80

1
40

0

extracted angles
processed angles
0

1

2

3

4

5

0

6

amdf
ﬁtted sine
0

0.5

t (s)

1

1.5

2

t (s)

(a)

(b)

Figure 10: (10a) Elbow angles and (10b) AMDF.
200

3

160

Angle (°)

Angle (°)

2
120

80

1
40

0

extracted angles
processed angles
0

1

2

3

4

5

0

6

amdf
ﬁtted sine
0

0.5

t (s)

1

1.5

2

t (s)

(a)

(b)

Figure 11: (11a) Elbow angles and (11b) AMDF and tted sinus for a no-gesture group.
In Equation 12, A stands for the amplitude of the sinusoidal waveform, ω is its frequency,
φ is the phase and c is the oset. If periodicity is not pronounced enough, deriving suchan approximation would not be possible. In other words, for our specic case, solving for{A, ω, φ, c} would be possible, only if the limbs are subject to predominant oscillations(possibly due to walking).

5.5 Results
Running our decision algorithm on all the groups, we obtain the confusion matrix of Table 5.
We get a precision of 0.47 and a recall of 0.90, which suggests a model biased toward positive
200

40

120

Angle (°)

Angle (°)

160

80

20

40

0

extracted angles
processed angles
0

1

2

3
t (s)

(a)

4

5

6

0

amdf
0

0.5

1

1.5

2

t (s)

(b)

Figure 12: (12a) Elbow angles and (12b) AMDF for a strong gesture group.

5.6

21

Discussion

200

20

15

120

Angle (°)

Angle (°)

160

80

5

40

0

10

extracted angles
processed angles
0

0.5

1

1.5

2

2.5

3

3.5

4

t (s)

0

amdf
0

0.5

1

1.5

2

t (s)

(a)

(b)

Figure 13: (13a) Elbow angles and (13b) AMDF for a weak gesture group.
Table 5: Estimation performance in identication of arm gestures.

Ground truth

No gest.
Gest.

Estimation
No gest. Gest.
41
31
3
27

gesture detection. Overall, the accuracy of the model is 0.68.
In Figures 12 and 13, we present two examples, where the algorithm detects the gesturestate successfully. In Figure12, the angle waveform is subject to abrupt changes, and the
AMDF curve ascertains this. Thus, the dyad is correctly identied as no gesture. In
Figure13, although the waveform involves such disruptions only for a very short duration,
namely less than 2 seconds, the shape of the AMDF reects the existence of a gestureclearly.

5.6 Discussion
Various sources of error that can have an impact on the performance of the model wereidentied:• Pose estimation errors or gaps may arise from occlusions (by object or other peers)and changing view angle due to taking a curve (see Figure 14). Indeed, OpenPose
detections relies on the identication of body parts in the picture. If occlusions
occurs, the algorithm returns either guessed improbable detections (like for the hips
on Figure 14) or no detections (the coordinates are set to (0, 0)). While computing
the angles, if on joints is missing, the angle is ignored for this frame, but in case ofa misplaced joint, a false value will be computed.
• Actions other than gestures may be performed (e.g. switching cup from one handto the other). As a matter of fact, when analyzing the clips that were annotated, itturns out that some pedestrians are engaged in arm gestures that are unrelated totheir interaction (i.e. they are not pointing at something for instance). The coders

22

6

ESTIMATION OF INTENSITY OF INTERACTION

Figure 14: Misdetection of pose due to occlusion.
therefore did not considered these gestures, but the classication algorithm detects
them, which increase the number of false-positives.

• Coder's incorrect labels. It is possible that the annotators coded a group as non
performing arm gestures, while actually some are being performed. As coders lookfor gestures during the annotations process and put a positive value only when they
notice one, all positive labels can be assumed to be correct while negative ones may
be due to failure in noticing the gesture. Consequently, such errors only impact thefalse-negative side of the confusion matrix.
Future work for this study would rst require to improve the results of the skeletons
estimations to correct missing or false detections. This could be done by performing someinterpolation computation to generate skeletons by assuming regular motions of the pedestrians. Another more advanced idea would be to use the recent studies in the eld of neuralnetwork with graph input data to build a model which would, given a set of detections,generate detections for the missing time steps.

6

Estimation of intensity of interaction

We explored the possibility of estimating intensity of interaction from the observables as
described in Section 4. People's height being unavailable in the DukeMTMC dataset, weonly computed vg , ω and δ .
The resulting empirical distributions of the observables for pedestrians interacting at agiven intensity are as given in Figure 15.
We expected to see a clearer pattern in the peak location or dispersion of the distributions. For instance, δ for I = 0 (Figure 15a), is concentrated around a smaller expectedvalue whereas for I = 3 the expected value as well as the tail grow. We would expect
the distributions for I = 2 and I = 1 to follow a gradual pattern between these. In other

23

2.5

2.5

0
1
2
3

2

2

1.5

p(δ)

p(vg)

1.5

1

1

0.5

0.5

0

0
1
2
3

0
0

0.5

1

1.5

2

0

0.5

1

1.5

2

2.5

vg (m/s)

δ(m)

(a)

(b)
40

0
1
2
3

p(ω)

30

20

10

0

0

0.1

0.2

0.3

0.4

0.5

ω (m/s)(c)

Figure 15: Empirical distribution of (15a) δ , (15b) vg and (15c) ω of peers for dyads
interacting at dierent intensity levels.

24

7

CONCLUSION

words, we would expect δ for I = 2 to have a smaller expected value and lighter tail thanfor I = 3. But we observe that this hypothesis is not veried as the order is reversed.
We believe that one reason that can explain these unexpected results comes from theway that the trajectory ground truth is computed in the DukeMTMC dataset. Indeed, as
explained in Section 5.1 and illustrated on Figure 7, pedestrians feet positions are annotatedat certain frames and then the missing values are linearly interpolated. This leads to threepossible sources of error in the trajectories: (i) precision of the annotations limited whenclicking between people's feet on an image, (ii) approximation errors when projecting fromthe image coordinates to the real-world coordinates, and (iii) non-realistic values due tothe interpolation. The latest is, in our opinion, the most problematic issue, as it leads topiece-wise constant velocities.

7

Conclusion

To sum up the work exposed in this report, we proposed solutions to the tasks introducedin Section 2, namely (i) recognition of social relations and (ii) detection of gestures.
For (i), we introduced a Bayesian model based on empirical distributions of a set ofobservables computed from pedestrian trajectories. By computing the probabilities ofobservables to have been sampled from a given empirical distributions, we are able toclassify the social relation of a pair of pedestrians (coalitional or mating) with an accuracy
larger than 85%.
We also compared our results for this task with global methods (see Section 4.5), basedon distributions comparison and showed that the Bayesian model gives the best performance.
For (ii), we proposed a method based on the detection of walking motion pattern frompedestrian arm movements. The idea is that for non-interacting pedestrians, the AMDF
of elbow angles movements should resemble a sinusoidal waveform. In case of interaction
arm gestures being performed, trying to t such a sinusoidal function to the computed
AMDF should fail. Using this decision algorithm we obtained promising results for what
we believe to be a novel approach to pedestrians gesture detection.
We tried to apply the idea used for social relation classication to dierentiate theintensity of interactions between pedestrians. Namely, as detailed in Section 6, we computed empirical distributions of observables for trajectories of pedestrians interacting withdierent level of intensity. The results being inconclusive, we left it out for the time being.
Japan being very involved in the robotic industries, one possible application of thisproject could be to analyse behaviors pairs of pedestrians to develop robots, that canpotentially walk along side with people in the most natural manner.

25
Weeks
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

Bibliography research
Report redaction
Paper redaction
Preprocessing
Pose estimation
Action recognition
Test and review

Figure 16: Estimated schedule showing progress of the project.
Weeks
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

Bibliography research
Report redaction
Presentation preparation
Bayesian model
Paper redaction
Dataset research
Preprocessing
Pose estimation
Arm gestures detection
Test and review

Figure 17: Final schedule of the project.

8Work organization

As detailed in Section 1, this internship took place in an academic context. It follows that,
although the goal to accomplish was well dened, the way to reach it was only drafted. A
substantial part of the project consisted in bibliography research and documentation on
the various domains that the subject covered.
Figure 16 shows the estimated schedule established six weeks after the beginning of theinternship as shown in the pre-report written at that time. Figure 17, on the other hand,shows the actual planning that has been followed during the project.
The major dierence is the disappearance of the Action classication, which has
been replaced by Arms gesture detections. Indeed, for various reasons (limited quantityof data, specicity of the action to detect), the rst idea that we had (namely, using a
STGCN-like neural network to perform action classication of the pedestrians gesture)
turned out to be impractical.

9Societal and environmental impact

This section will give some ideas of the impact of this project on an societal and environmental level.

26

9

SOCIETAL AND ENVIRONMENTAL IMPACT

9.1 Societal impact
For this project, the possible applications are in the eld of robotics. As detailed in
Section 2, this could allow to created embodied agents able to follow socially acceptabletrajectories. This could be used in a variety of ways, such as guidance or assistantship fordisabled and elderly.
Relating to the privacy of the data, for the dataset used in the social relation detection(see Section 4.1), it does not contain video of the pedestrians thus insuring their complete
anonymity. For the work on gesture detections, the DukeMTMC dataset (see Section 5.1)
is publicly available for research purpose.

9.2 Environmental impact
Regarding the environment, the rst part of the project was fullled on my personal laptop
computer. It was mainly used in battery mode, being charged around two hours perday. According to its technical documentation, this represents a power consumption of
approximately 300Wh/day. For the 21 weeks of the internship, this corresponds to a total
consumption of around 30kWh.
The project only took place in Okayama University. Except from the round trip from
France to Japan, no further travels have been performed, in the context of this internship.
Yet, this represents a substantial carbon footprint of 4t of CO2, almost twice the maximum
amount that individuals are encouraged not to exceed to prevent global warming. In termsof power consumption, this journey corresponds to more than 8000kWh, which is roughly
equivalent to using 250 laptops during the duration of the internship.
Moreover, Okayama being a very at city, many people (and especially students) use
bikes to move around. Likewise, I used a bike to go to work therefore limiting my environmental impact, the international dormitory being very close from the university.
Japan is very involved in environmental procedure such as sorting of waste. In thelaboratory oce, we had two thrash cans, one for the plastic bottles and another one forthe rest of the trash. In the main buildings of the university, there were around 7 dierent
trashes for various kind of waste: burnable, non burnable, cans, etc.

27

10

Personal feedback

10.1 Work environment
On a typical day of work, I arrive at the laboratory at 9:30AM and leave around 17:30PM(working 7.5 hours/day). I am supposed to share the laboratory with another French
student, a Chinese student and 10 Japanese students. The six rst weeks of my internshipcoincided with the end of the school year in Japan, so the laboratory was never actually
full and most of the times we were only 4 or 5 working. Dr Akito Monden's oce is locatedin the same oor as the students' oce and Dr Yücel's oce is located two oors lower.
The overall ambience in the laboratory was great and the integration process went
very smoothly, probably mainly because we were all students and because of the Japanese
helpfulness and kindness. The furniture in the oce also reect this friendly environments,
with bookshelves full of mangas, coding books and retro console and games. A microwave
and a fridge are also available directly in the oce room. Once a week we all met with theprofessors to eat together.

10.2 Benets
10.2.1

Working experience

Overall, this project was a great working experience.
During my previous internships (in 1rst and 2nd year), the work that I performed wasnever really connected to my orientations ideas. I mainly worked on Web development forindustrial companies, and although it was two great work experiences, it did not correspondto the kind of task I pictured myself doing in the future.
On the other hand, during this project, I was able to concretely apply notion that I haveadressed during classes at Ensimag. In 3rd year, in the Pattern recognition and machinelearning lectures, we studied a bayesian model applied to face detection and during thenal exam, we worked on developing another bayesian model, applied to digit recognition
this time. My work on social relations detection was greatly facilitated by these previous
studies.
Moreover, I discovered new domains that I had not studied or only overown during
my studies at Ensimag. For instance, in 2nd year for the 3D Graphic lecture we quickly
introduced microscopic Social Force Model applied to video games or 3D crowd simulation.
During this internship, I had the opportunity to deeply improve my knowledge of this kindof models.
On a very technical level, most of the code implemented during this internship was
Python code. I was already familiar with this langage that I have been using for a few years,
but I took some time to assimilate some good practices that I was not following before.
For instance, I used work environment, which are a great way to handle dependencies in
Python projects, especially when alternating between dierent version in dierent projects.
Finally, I was comforted in my desire to pursue a PhD, probably in the computer visionand machine learning elds of study. The technical freedom associated with research really

28

10

PERSONAL FEEDBACK

suits my expectations. A big portion of this internship, as an academic research work,
consisted in looking for documentation regarding the state of the art of the diverse stepsof the project (pose estimation, action recognition, and pedestrian models) and previous
work related to the studied topics. I nd this part of the work extremely rewarding on thescientic level as it allows to dive into a wide variety of interesting methods and solutions.
Even if most of the papers describe algorithms that could not be applied on this specicproject, I enjoyed reading them.

10.2.2

Personal experience

I had the chance to do my internship in Okayama, Japan. This allowed me to discover
the Japanese culture, which is very dierent from what I was used to. Japanese people
are extremely helping and always willing to assist in case of dicult situations. Nevertheless, administration is very rigorous and respecting deadlines and regulations is especially
important.
Moreover, I had the opportunity to meet a large number of people during this project,
not only students from my laboratory but also from the dormitory and other laboratories
accros the university. This turned out to be a great human experience and a chance forme to get to know new people from all over the world.
Okayama is a very well situated city in Japan. As a matter of fact, it is close to big
cities such as Osaka or Kyoto which are great to visit. The weather is also particularly nicein Okayama prefecture which is rightfully known as the Land of Sunshine . I had thechance to be in Japan during the cherry blossom and participate to Hanami, the traditionalcustom during which Japanese people admire the owering of the cherry trees.

10.3 Diculties
In relation to the project, the main diculties that I experienced were linked with environment issues. I started working on my personal Mac Book Pro but many algorithms that I
needed to use to test feasibility of considered solutions required an NVidia GPU. Indeed,most machine learning algorithms require heavy computation that is generally accelerated
on GPU, very often using the CUDA toolkit that integrates well with traditional machinelearning libraries such as Torch or Tensorow. I started preprocessing the video data until
I could use the wanted algorithms on a new computer.
On advantage of the research domain is the freedom of choice when designing a solutions. However a drawback is that there is generally no guarantee that the project willcomplete. Moreover, the solution might drastically change during if it turns out not to be
viable. As detailed in Section 8, I started working on action recognition algorithms andI was planning to use methods based on STGCN [36] to classify the gestures. This idea
turned out to be unpracticable due to the specicity of the action that had to be detected
as well as the relatively small quantity of data.

10.4

Acknowledgment29

10.4 Acknowledgment
I would like to thank all the people that have made this internship possible: Dr. Akito
Monden and Dr. Zeynep Yücel for proposing the subject as well as Dr. Bernard Chenevier
for giving me the opportunity to postulate. Dr Yücel, in particular, was overseeing mywork and was extremely available during the entirety of the project. Dr James L. Crowley
was also supervising my work, as an Ensimag representative, and gave me very interesting
feedback and help regarding the project.
I would also like to thank all the student from Monden's laboratory for their invaluablekindness and helpfulness.

30

REFERENCES

References[1] Francesco Zanlungo, Zeynep Yucel, and Takayuki Kanda. The eect of social roles
on group behaviour. Pedestrian and Evacuation Dynamics Conference 2016 Heifei
China, (25):110, 2017.
[2] Zeynep Yücel, Francesco Zanlungo, Tetsushi Ikeda, Takahiro Miyashita, and Norihiro Hagita. Deciphering the crowd: Modeling and identication of pedestrian groupmotion. Sensors (Switzerland), 13(1):875897, 2013.
[3] Zeynep Yücel, Francesco Zanlungo, and Masahiro Shiomi. Walk the Talk: Gesturesin Mobile Interaction. In Abderrahmane Kheddar, Eiichi Yoshida, Shuzhi Sam Ge,
Kenji Suzuki, John-John Cabibihan, Friederike Eyssel, and Hongsheng He, editors,
Social Robotics, pages 220230, Cham, 2017. Springer International Publishing.
[4] Francesco Zanlungo, Zeynep Yücel, Draºen Br²£i¢, Takayuki Kanda, and Norihiro
Hagita. Intrinsic group behaviour: Dependence of pedestrian dyad dynamics on principal social and personal features. PLoS ONE, 2017.
[5] Bertrand Maury, Aude Roudne-Chupin, and Filippo Santambrogio. A macroscopic
crowd motion model of gradient ow type. Arxiv preprint arXiv, 2010.
[6] Yan Qun Jiang, Peng Zhang, S. C. Wong, and Ru Xun Liu. A higher-order macroscopic
model for pedestrian ows. Physica A: Statistical Mechanics and its Applications,
2010.
[7] M. Twarogowska, P. Goatin, and R. Duvigneau. Macroscopic modeling and simulations of room evacuation. Applied Mathematical Modelling, 2014.
[8] Dirk Helbing and Péter Molnár. Social force model for pedestrian dynamics. PhysicalReview E, 1995.
[9] Systematica. http://www.systematica.net.
[10] Winnie Daamen. Modelling Passenger Flows in Public Transport Facilities. Delft
University Press Science, 2004.
[11] Yuan Gao, Peter B. Luh, Hui Zhang, and Tao Chen. A modied social force model
considering relative velocity of pedestrians. 2013 IEEE International Conference on
Automation Science and Engineering (CASE), 2013.
[12] Yan Qun Jiang, Bo Kui Chen, Bing Hong Wang, Weng Fai Wong, and Bing Yang
Cao. Extended social force model with a dynamic navigation eld for bidirectional
pedestrian ow. Frontiers of Physics, 2017.

REFERENCES

31[13] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li FeiFei, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded
Spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2016.
[14] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social
GAN: socially acceptable trajectories with generative adversarial networks. In CVPR,
volume abs/1803.10892, 2018.
[15] Xiong Hui, Lv Jian, Jiang Xiaobei, and Li Zhenshan. Pedestrian Walking Speed, Step
Size, and Step Frequency from the Perspective of Gender and Age: Case Study in
Beijing, China. In Transportation Research Board 86th Annual Meeting, 2007.
[16] P. A. Hageman and D. J. Blanke. Comparison of gait of young women and elderly
women. Physical Therapy, 1986.
[17] R. J. Elble, S. Sienko Thomas, C. Higgins, and J. Colliver. Stride-dependent changes
in gait of older people. Journal of Neurology, 1991.
[18] M. P. Murray, R. C. Kory, and B. H. Clarkson. Walking patterns in healthy old men.
Journal of gerontology, 1969.
[19] Alan Crowe, Monique M. Samson, Marja J. Hoitsma, and Alexandra A. Van Ginkel.
The inuence of walking speed on parameters of gait symmetry determined fromground reaction forces. Human Movement Science, 1996.
[20] John H. Hollman, Francine M. Kovash, Jared J. Kubik, and Rachel A. Linbo. Agerelated dierences in spatiotemporal markers of gait stability during dual task walking.
Gait and Posture, 2007.
[21] Jie Sun, Megan Walters, Noel Svensson, and David Lloyd. The inuence of surface
slope on human gait characteristics: A study of urban pedestrians walking on an
inclined surface. Ergonomics, 1996.
[22] Hideo Mori, N. Moghadam Charkari, and Takeshi Matsushita. On-Line Vehicle andPedestrian Detections Based on Sign Pattern. IEEE Transactions on Industrial Electronics, 1994.
[23] S. Yasutomi, H. Mori, and S. Kotani. Finding pedestrians by estimating temporalfrequency and spatial-period of the moving objects. Robotics and Autonomous Systems, 1996.
[24] Chiraz BenAbdelkader, Ross Cutler, and Larry Davis. Stride and cadence as a biometric in automatic person identication and verication. In Proceedings - 5th IEEE
International Conference on Automatic Face Gesture Recognition, FGR 2002, 2002.

32

REFERENCES[25] Mark S. Nixon, John N. Carter, Michael G. Grant, Layla Gordon, and James B.
Hayfron-Acquah. Automatic recognition by gait: Progress and prospects. Sensor
Review, 2003.
[26] J W Davis. Visual Categorization of Children and Adult Walking Styles. Proceedingsof the Third International Conference on Audio- and Video-Based Biometric Person
Authentication, 2001.
[27] Yasushi Makihara, Hidetoshi Mannami, and Yasushi Yagi. Gait analysis of genderand age using a large-scale multi-view gait database. Computer VisionACCV 2010,2011.
[28] Sourabh A Niyogi and Edward H Adelson. Analyzing and Recognizing Walking Figures in XYT. In Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition CVPR-94, 1994.
[29] Nicolas Saunier, Ali El Husseini, Karim Ismail, Catherine Morency, Jean-Michel
Auberlet, and Tarek Sayed. Estimation of Frequency and Length of Pedestrian Stride
in Urban Environments with Video Sensors. Transportation Research Record: Journalof the Transportation Research Board, 2011.
[30] Houman Hediyeh, Tarek Sayed, Mohamed H. Zaki, and Greg Mori. Pedestrian gait
analysis using automated computer vision techniques. Transportmetrica A: Transport
Science, 2014.
[31] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In
Proceedings of the IEEE International Conference on Computer Vision, 2013.
[32] Vivek Veeriah, Naifan Zhuang, and Guo-Jun Qi. Dierential Recurrent Neural Networks for Action Recognition. International Conference on Computer Vision (ICCV),2015.
[33] Yong Du, Wei Wang, and Liang Wang. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE Computer Society Conference
on Computer Vision and Pattern Recognition, 2015.
[34] Jun Liu, Amir Shahroudy, Dong Xu, Alex Kot Chichung, and Gang Wang. SkeletonBased Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[35] Tomas Kipf. Graph Convolutional Networks. Blog, 2017.
[36] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial Temporal Graph Convolutional
Networks for Skeleton-Based Action Recognition. CoRR, 2018.
[37] Daphne Blunt Bugental. Acquisition of the Algorithms of Social Life: A Domain-BasedApproach. Psychological Bulletin, 2000.

33

REFERENCES[38] Drazen Brscic, Takayuki Kanda, Tetsushi Ikeda, and Takahiro Miyashita. Person
Tracking in Large Public Spaces Using 3-D Range Sensors. IEEE Transactions on
Human-Machine Systems, 2013.
[39] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool. You'll never walk alone: Modelingsocial behavior for multi-target tracking. In Proceedings of the IEEE International
Conference on Computer Vision, 2009.
[40] Caviar test case scenarios.
CAVIARDATA1/.

http://groups.inf.ed.ac.uk/vision/CAVIAR/[41] Andreas Ess, Bastian Leibe, and Luc Van Gool. Depth and appearance for mobile
scene analysis. In Proceedings of the IEEE International Conference on Computer
Vision, 2007.
[42] J Ferryman and A Shahrokni. PETS2009: Dataset and challenge. Pets, 2009.
[43] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In Lecture
Notes in Computer Science (including subseries Lecture Notes in Articial Intelligenceand Lecture Notes in Bioinformatics), 2016.
[44] Francesco Solera, Simone Calderara, Ergys Ristani, Carlo Tomasi, and Rita Cucchiara.
Tracking social groups within and across cameras. IEEE Transactions on Circuits andSystems for Video Technology, 2017.
[45] Klaus Krippendor. Content Analysis: An Introduction to Its Methodology. Sage
Publications, 2004.
[46] Zhe Cao, Tomas Simon, Shih En Wei, and Yaser Sheikh. Realtime multi-person 2D
pose estimation using part anity elds. In Proceedings - 30th IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2017, 2017.
[47] Andrew Cohen, Richard Freudberg, Andrew Cohen, Richard Freudberg, Myron J.
Ross, Harry L. Shaer, and Harold J. Manley. Average Magnitude Dierence Function
Pitch Extractor. IEEE Transactions on Acoustics, Speech, and Signal Processing,
1974.

34

LIST OF TABLES

List of Figures
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

(1a) Macroscopic [9] and (1b) microscopic [10] crowd models illustrations.
6
Density map of the environment. . . . . . . . . . . . . . . . . . . . . . . .
7
Schema of the observables depicted on a single dyad. . . . . . . . . . . . .
8Empirical distributions of (4a) δ , (4b) vg , (4c) ω and (4d) ∆η , of dyads incoalitional (C, blue) and mating (M, red) relation. . . . . . . . . . . . . . .
9Empirical distribution of (5a) δ , (5b) vg , (5c) ω and (5d) ∆η , of peers fordyads in coalitional (C, blue), mating (M, red), attachment (A, green) andreciprocity (R,orange) relation. . . . . . . . . . . . . . . . . . . . . . . . . 15
Top view of the Duke University campus with the elds of view of thecameras used to record the DukeMTMC dataset in color. . . . . . . . . . . 16
Example of annotations on the DukeMTMC dataset. Yellow dots were manually placed by coders between the feet of the pedestrians and black lines
are linear interpolation between those keyframe annotations. . . . . . . . . 17
Schema of the OpenPose joints numbering with the elbow and shoulder angles. 18(9a) Elbow and (9a) shoulder angles before and after preprocessing. . . . 19(10a) Elbow angles and (10b) AMDF. . . . . . . . . . . . . . . . . . . . . . 20(11a) Elbow angles and (11b) AMDF and tted sinus for a no-gesture group. 20(12a) Elbow angles and (12b) AMDF for a strong gesture group. . . . . . . 20(13a) Elbow angles and (13b) AMDF for a weak gesture group. . . . . . . 21
Misdetection of pose due to occlusion. . . . . . . . . . . . . . . . . . . . . . 22
Empirical distribution of (15a) δ , (15b) vg and (15c) ω of peers for dyads
interacting at dierent intensity levels. . . . . . . . . . . . . . . . . . . . . 23
Estimated schedule showing progress of the project. . . . . . . . . . . . . . 25
Final schedule of the project. . . . . . . . . . . . . . . . . . . . . . . . . . 25

List of Tables
1
2
3
4
5
6
7
8
9

Jaccard distance between observables for subsets of (1a) coalitional and (1b)mating relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Detection performance for varying α (in %). . . . . . . . . . . . . . . . . .
Detection performance for the other proposed metrics. . . . . . . . . . . .
Inter-rater agreement analysis . . . . . . . . . . . . . . . . . . . . . . . . .
Estimation performance in identication of arm gestures. . . . . . . . . . .
Detailed performance rates for recognition of social relations using KL divergence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Detailed performance rates for recognition of social relations using JS divergence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Detailed performance rates for recognition of social relations using EMD. .
Detailed performance rates for recognition of social relations using Loglikelihood. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11
11
13
17
21
35
35
35
36

35

A

Detailed performance of alternative methods for recognition of social relation

Table 6: Detailed performance rates for recognition of social relations using KL divergence.

M
C
Total

vg

ω

δ

∆η

71.15 ± 7.30
81.09 ± 4.50
78.90

90.58 ± 12.40
24.73 ± 15.40
39.27

78.37 ± 7.83
69.40 ± 10.75
71.38

44.81 ± 7.38
92.60 ± 2.04
82.04

Table 7: Detailed performance rates for recognition of social relations using JS divergence.

vg
M
C
Total

ω

δ

∆η

72.60 ± 7.27 68.46 ± 5.10 72.12 ± 4.65 55.96 ± 4.55
79.13 ± 2.88 64.07 ± 7.98 78.77 ± 5.79 85.00 ± 3.57
77.68
65.04
77.30
78.59

Table 8: Detailed performance rates for recognition of social relations using EMD.

vg
M
C
Total

B

ω

δ

∆η

76.06 ± 6.03 63.37 ± 5.11 74.23 ± 4.19 59.33 ± 5.78
76.45 ± 3.19 64.02 ± 4.37 67.51 ± 6.06 82.62 ± 3.37
63.87
76.36
69.00
77.48

Contact information

Name

Role

Email

Phone number

Adrien Gregorj
Dr Akito Monden
Dr Zeynep Yücel
Dr James L. Crowley

InternSupervisor
Project supervisor
Ensimag supervisoradrien.gregorj@gmail.com
monden@okayama-u.ac.jp
zeynep@okayama-u.ac.jp
james.crowley@inria.fr+81 80 4099 5120
+81 86 251 8180
+81 86 251 8245
+33 4 76 61 53 96

36

B

CONTACT INFORMATION

Table 9: Detailed performance rates for recognition of social relations using Log-likelihood.

vg
M
C
Total

ω

δ

∆η

73.94 ± 6.19 63.46 ± 7.30 72.12 ± 4.19 60.38 ± 6.12
78.20 ± 2.78 68.11 ± 8.50 78.93 ± 5.03 81.31 ± 4.10
77.26
67.09
77.43
76.69

37

C

French abstract

Ce rapport présente le travail réalisé dans le cadre d'un PFE se déroulant en milieu
académique, à l'université d'Okayama, au Japon. Le Dr Zeynep Yücel, superviseure de ce
stage, travaille depuis plusieurs années sur la problématique de la modélisation de mouvement de foule et cette étude étend ces travaux. Des résultats dans ce domaine sontparticulièrement utiles dans les domaines de la robotique, de la video surveillance ou de laconduite autonome.
Les objectifs du projet sont doubles : (i) la reconnaissance de relations sociales et (ii)la detection de gestes. Pour (i) une méthode probabiliste avec une approche Bayesienne aété développé et pour (ii) une méthode inspirée du traitement de signal audio est proposée.
Pour étudier la reconnaissance de relations sociales, le jeu de données est composé destrajectoires (coordonnées et vitesses) et des tailles des piétons, obtenues préalablement
à l'aide d'un environnement de traçage. Un ensemble de données sont calculées pour les
trajectoires de paires de piétons : la vitesse du groupe, la diérence de vitesse des membres
du groupe, la diérence de tailles et la distance entre les membres. Le modèle Bayesiendéveloppé calcule la probabilité qu'un groupe de piétons soit engagé dans une relation
donnée connaissant les variables précédemment introduites.
Ce modèle est également comparé avec des méthodes de comparaison globale de distributions, à savoir la divergence de Kullback-Leibler, la divergence de Jensen-Shanonn,la distance du cantonnier et la log-likelihood. Les meilleurs résultats sont obtenus avec lemodèle Bayesien et sont de l'ordre de 80% de précision.
Pour la détection des gestes, le jeu de données DukeMTMC-Groups est utilisé. En
plus des annotions de groupes déjà présentes, il a également du être annoté pour relever laprésence d'un ensemble de gestes : conversation, échange de regards, contact physique etmouvements des bras. Cependant, seul ce dernier geste est considéré dans cette étude.
L'idée développée ici est que le corps humain est sujet à des oscillations relativement
régulières causées par le rythme de marche et que des irrégularités dans ces oscillations(autour des poignets, coudes et épaules) peuvent être considérées comme provenant d'ungeste, au sein d'une paire de piétons. Pour detecter et analyser ces irrégularités, uneméthode inspirée de la detection de tonalité est utilisée.
Des poses générées par la librairie OpenPose sont utilisées pour calculer l'angle del'articulation du coude des piétons à chaque image. L'AMDF (average magnitude dierencefunction) du signal est calculée et une fonction sinusoïdale est paramétrée pour approcher
la courbe au plus près. Dans le cas où la fonction ne peut pas susamment être assimilée
à une fonction sinusoïdale, on peut considérer qu'un geste est eectué. En utilisant cet
algorithme décisionnel, la précision obtenue est de 68%, et le modèle semble biaisé en faveurde résultats positifs (i.e. geste eectué).
An d'améliorer ce modèle, les principales sources d'erreurs sont analysées et des pistes
de solutions sont proposées (amelioration de la cohérence des poses détectées par exemple).