Crowd Scene Understanding with Coherent Recurrent
Neural Networks
Hang Su, Yinpeng Dong, Jun Zhu
Department of Computer Science and Technology, Tsinghua University

July 12, 2016

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 20161

Outline

1Introduction

2

LSTM Recap

3

Coherent LSTM

4Experimental Results

5Conclusion

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 20162

Outline

1Introduction

2

LSTM Recap

3

Coherent LSTM

4Experimental Results

5Conclusion

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

3

Background
Understanding Collective behaviors has a wide range applicationsin video surveillance and crowd management.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

4

Background
Understanding Collective behaviors has a wide range applications
in video surveillance and crowd management.
In the real scenes, pedestrians tend to form groups and theirtrajectories are influenced by others and obstacles.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

4

Background
Understanding Collective behaviors has a wide range applications
in video surveillance and crowd management.
In the real scenes, pedestrians tend to form groups and theirtrajectories are influenced by others and obstacles.
The main challenges of crowd motion analysis are nonlinear
dynamics and coherent motion.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

4

Problem Formulation

Obtain reliable tracklets from each scene using KLT trackers. At
any time-instant t, the ith person is represented by his/her
coordinate (xi (t), yi (t)).

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

5

Problem Formulation

Obtain reliable tracklets from each scene using KLT trackers. At
any time-instant t, the ith person is represented by his/hercoordinate (xi (t), yi (t)).
Predict future trajectories of pedestrians and use extracted hiddenfeatures to recognize crowd motions.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 20165

Previous WorkSocial Force model
Optimize energy function
Hand-crafted functions
Hard to generalize

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 20166

Previous WorkSocial Force model
Optimize energy function
Hand-crafted functions
Hard to generalize

Probabilistic Forecasting
Gaussian Process

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 20166

Previous WorkSocial Force modelOptimize energy function
Hand-crafted functions
Hard to generalize

Probabilistic Forecasting
Gaussian Process

Recurrent Neural Networks
N-LSTM [Alahi et al., 2016]

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

6Outline

1Introduction

2

LSTM Recap

3

Coherent LSTM

4Experimental Results

5Conclusion

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

7

LSTM

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

8

LSTM

Structure
Input / Output / Forget
gate
Memory state ct

Advantage
Prevent vanishing gradientproblem
Nonlinear characteristicGeneralizationct = f tct−1 + it

Hang Su, Yinpeng Dong, Jun Zhu

tanh(Wxc xt + Whc ht−1 + bc )(1)

July 12, 20168

Outline

1Introduction

2

LSTM Recap

3

Coherent LSTM

4Experimental Results

5Conclusion

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 20169

Why Coherent LSTM?
LSTM can model individual behaviors but can’t capture theinteraction in a group.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201610

Why Coherent LSTM?
LSTM can model individual behaviors but can’t capture theinteraction in a group.
When the neighboring relationship of individuals remain invariant
over time and correlation of their velocities remain high, they tend
to have similar hidden state.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

10Why Coherent LSTM?
LSTM can model individual behaviors but can’t capture theinteraction in a group.
When the neighboring relationship of individuals remain invariant
over time and correlation of their velocities remain high, they tend
to have similar hidden state.
The trajectories of pedestrians not only follow the old trend, butalso are influenced by current environment.
LSTM
Coherentregularization

LSTM

Motion Prediction

Coherentregularization
LSTM

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201610

cLSTM Unit
ct = ftct−1 + ittanh(Wxc xt + Whc ht−1 + bc ) +X

λj (t)ftj

cjt−1j∈N(2)
Forget Gate

σσ

InputGateϕ

CoherentRegularization

Cellxtht −1

Output Gate

σht

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201611

Coherent Motion Modeling
Use coherent filtering [Zhou et al., 2012] [Shao et al., 2014] to discover
the coherent group.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201612

Coherent Motion Modeling
Use coherent filtering [Zhou et al., 2012] [Shao et al., 2014] to discover
the coherent group.

The dependency relationship between two tracklets within the samegroup is measured as:
τj (t) =Hang Su, Yinpeng Dong, Jun Zhu

vi (t) · vj (t)kvi (t)kkvj (t)k(3)July 12, 201612

Dependency Coefficient
The dependency coefficient between the ith and jth tracklets in Eq. (2)is defined asτj (t) − 11
λj (t) =exp
∈ (0, 1]
(4)
Zi
2σ 2

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201613

Dependency Coefficient
The dependency coefficient between the ith and jth tracklets in Eq. (2)is defined asτj (t) − 1
1
λj (t) =exp
∈ (0, 1]
(4)
Zi
2σ 2
Zi : normalization constant corresponding to the ith tracklet.
λj (t) ' Z−1
if vi (t) ' vj (t) which implies that tracklets i and jiare similar.
Coherent regularization encourages the tracklets to learn similarfeature distributions by sharing information across tracklets withina coherent group.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201613Framework
Unsupervised encoder-decoder cLSTM framework:hT = cLSTMe (xT , hT −1 ),(5)x̂t = cLSTMdr (ht , x̂t+1 ), where t ∈ [1, T ],(6)x̂t = cLSTMdp (ht , x̂t−1 ). where t > T,

(7)x̂3x̂2

Wrd
x1x2

Wex̂1

Wrd

Reconstruction
Decoderx3

We

hTx̂4

Encoder
Learnt HiddenFeaturesx̂6x̂5Wpd

WpdPrediction
Decoder

Coherent Regularization

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

14

Crowd Scene Profiling

Solve critical tasks in crowd scene analysis:Group state estimation
Crowd video classification

Softmax classification using the feature learnt from theunsupervised cLSTM.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201615

Outline

1

Introduction

2

LSTM Recap

3

Coherent LSTM

4Experimental Results

5Conclusion

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201616

Datasets and Settings

CUHK Crowd Dataset
http://www.ee.cuhk.edu.hk/~xgwang/CUHKcrowd.html
Scene: streets, shopping malls, airports and parksMore than 400 sequences and more then 200,000 traklets

Settings
128 hidden units in cLSTM
2/3 of tracklets as the input and 1/3 as the predicted tracklets toevaluate the performance.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201617

Future Path Forecasting

Table 1: Error of Path Prediction(pixels)

Kalman Filter
9.32 ± 1.99

Un-coherent LSTM
6.64 ± 1.76

Hang Su, Yinpeng Dong, Jun Zhu

Coherent LSTM
4.37±0.93
July 12, 201618

Group State Estimation(a) Gas(b) Solid(a) Collective Transition(d) Un-coherent LSTM(c) Pure Fluid(b) Prediction LSTM(d) Impure Fluid(c) Reconstruction LSTM(e) Coherent LSTM

Confusion matrices of estimating group states using different methods:(a) collective transition [Shao et al., 2014]; (b) prediction LSTM; (c)reconstruction LSTM; (d) un-coherent LSTM; and (e) coherent LSTM.
Hang Su, Yinpeng Dong, Jun Zhu

July 12, 2016

19

Crowd Video Classification
All video clips are annotated into 8 classes as 1) Highly mixed
pedestrian walking; 2) Crowd walking following a mainstream and wellorganized; 3) Crowd walking following a mainstream but poorlyorganized; 4) Crowd merge; 5) Crowd split; 6) Crowd crossing in
opposite directions; 7) Intervened escalator traffic; and 8) Smoothescalator traffic.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201620

Outline

1

Introduction

2

LSTM Recap

3

Coherent LSTM

4Experimental Results

5Conclusion

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201621

Conclusion

A novel recurrent neural network with coherent long shortterm memory unit;
Introduce a coherent regularization to consider the collectiveproperties;
Outperform other methods in group state estimation andcrowd video classification.

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201622

Thanks for your time!
Questions?

Hang Su, Yinpeng Dong, Jun Zhu

July 12, 201623