x̂3x̂2

Wrd
x1x2

Wex̂1

Wrd

Reconstruction
Decoderx3WehTx̂4

Encoder
Learnt HiddenFeaturesx̂6x̂5Wpd

WpdPrediction
DecoderCoherent Regularization

Crowd Scene Understanding with Coherent Recurrent
Neural Networks
Hang Su, Yinpeng Dong, Jun Zhu, Haibin Ling, and Bo Zhang{suhangss, dcsjz, dcszb}@mail.tsinghua.edu.cn, donyp13@mails.tsinghua.edu.cn, hbling@temple.edu

I NTRODUCTIONM ODEL C ROWD M OTIONSR ESULTS

Understanding collective behaviors in crowd scenes has
a wide range of applications in video surveillance andcrowd management [3]. It has the following challenges:

We use LSTM to model the crowd dynamic. Each LSTM
unit has a cell state ct which preserves the information.
Forget Gateσ• Crowd spatio-temporal behavior patterns behave
abundantly nonlinear dynamics, such as limit cycles,
quasi-period and even chaos.

σInputGate

CoherentRegularization

ϕCell• Collective effect (or coherent motion), e.g. pedestrians in crowds tend to form coherent groups by aligning with other neighbors.

Output Gatext

σht −1ht

Update the cell state by incorporating with its neighboring agents by a coherent regularization asX
jj
ct = ft ct−1 +λj (t)ft ct−1Experiments are conducted on CUHK dataset. Sample
results of path forecasting are demonstrated in the figure
above. In Table 1, we report the prediction error measured
by average pixel distance.

j∈N

C ONTRIBUTIONS
We propose to explore the crowd dynamics with a coherent Long Short Term Memory (LSTM) architecture [4]. Our
main contributions are:
• We propose to investigate the crowd dynamics with astacked LSTM model, such that the complex and nonlinear crowd motion patterns are well captured;• To consider the collective properties in crowd motion
patterns, we propose to improve LSTM by introducing
a coherent regularization which encourages a consistent spatio-temporal hidden feature;• We adopt the hidden features learnt from the coherent LSTM to critical tasks in crowd scene analysis, including future path prediction, group state estimation, and crowd behavior classification.

+ ittanh(Wxc xt + Whc ht−1 + bc )(1)

C OHERENT M OTION
We use the coherent filtering [2] to detect coherent
groups.

We train a softmax classifier using the hidden features
learnt by our cLSTM, and then implement the group state
estimation and crowd video classification.

(a) Collective Transition

The dependency coefficient between the ith and jth tracklets in Eq. (1) is defined as1
τj (t) − 1
λj (t) =exp∈(0,1],(2)Zi
2σ 2(d) Un-coherent LSTM(b) Prediction LSTM(c) Reconstruction LSTM(e) Coherent LSTM

and τj (t) is:

R EFERENCES[1] Nitish Srivastava, Elman Mansimov, and Ruslan
Salakhutdinov: Unsupervised learning of video representations using lstms, In arXiv preprint arXiv:1502.04681, 2015[2] Bolei Zhou, Xiaoou Tang, and Xiaogang Wang: Scalable
Deep Poisson Factor Analysis for Topic Modeling, In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1-8, 2015[3] Noah Sulman, Thomas Sanocki, Dmitry Goldgof, andRangachar Kasturi: How effective is human video surveillance performance? In Pattern Recognition, 19th International Conference on (ICPR), pages 1-8, 2008[4] Sepp Hochreiter and Jürgen Schmidhuber : Long shortterm memory, In Neural Computation, pages 1735–1780,
1997

vi (t) · vj (t)τj (t) =kvi (t)kkvj (t)k(3)C LSTM F RAMEWORKTo learn an informative representation, we take the"encoder-decoder" approach [1].
x̂3x̂2

Wrd
x1x2

We

C ONCLUSION

x̂1

Wrd

Reconstruction
Decoder

x3

We

hTx̂4

Encoder
Learnt Hidden
Featuresx̂6x̂5

Wpd

Coherent Regularization

Wpd

Prediction
Decoder• A novel recurrent neural network with coherent longshort term memory unit;• Introduce a coherent regularization to consider thecollective properties;• Outperform other methods in group state estimationand crowd video classification.