2021 The 21st International Conference on Control, Automation and Systems (ICCAS 2021)
Ramada Plaza Hotel, Jeju, Korea, Oct. 12~15, 2021

Prediction of pedestrian trajectory based on long short-term memory of data
Tomoya Ono1 and Takashi Kanamaru2*
1

Major of Mechanical Engineering, Kogakuin University,
Tokyo 192-0015, Japan (am20015@ns.kogakuin.ac.jp)
2
Department of Mechanical Science and Engineering, Kogakuin University,
Tokyo 192-0015, Japan (kanamaru@cc.kogakuin.ac.jp) * Corresponding author
Abstract: Recently, development of navigation robots and autonomous cars are rapidly progressing. When such robots
become popular in our daily life, their collisions with humans should be avoided for safety. For that purpose, we predictpedestrian trajectories with LSTM (long short-term memory) networks and conventional neural networks, and wecompare their results. In order to predict sequential data, we use the following two methods: (I) predicting 𝑛 steps ofdata with 𝑛 models, and (II) predicting 𝑛 steps of data with a model by applying one-step prediction several times. By
examining these two methods, it was found that the performances of the method I with the LSTM network and theconventional neural network are comparable, and the performance of the method II with the LSTM network is
significantly better than that with the conventional neural network.
(Proceedings of 2021 21st International Conference on Control, Automation and Systems (ICCAS2021), pp.1676-1679.)
Keywords: machine learning, neural networks, long short-term memory, LSTM, pedestrian trajectory

1. INTRODUCTION
Recently, development of navigation robots andautonomous cars are rapidly progressing. When realizing
such robots, safety for humans is highly important. For
that purpose, robots are often operated with slow speed.
If robots can avoid collisions with humans appropriately,such limitations for robots would be removed.
Therefore, in this study, we model the pedestrian
trajectory to avoid collisions of robots with humans.

2. RELATED STUDIESIn order to predict the motion of pedestrians, Helbing
et al. used an equation of motion in which a pedestrian ismodeled as a point mass and other pedestrians and effectsof obstacles are described as forces [1]:𝑚=𝑚+∑ ()𝑓+ ∑ 𝑓 . (1)Although this model can incorporate various effects
for pedestrians, it would be difficult to determine various
parameters of the model.
On the other hand, we can also obtain a model ofpedestrian motion based on the machine learning using
only data of pedestrian motions. For example, Sakata etal. trained a Seq2Seq model that predicts the future,𝑝from the past
sequential data 𝑝 , 𝑝 ⋯ 𝑝sequential data 𝑥,𝑥⋯ 𝑥 , 𝑥 [2]. Alahi et al.
proposed Social LSTM that is an LSTM model with asocial pooling layer that can model human-humaninteractions, and they predicted the pedestrian

trajectories [3].
Models can be trained with the machine learning whensufficient amount of data are available. Recently, it has
become easy to treat large amount of data because of theimprovement of computer performances and thedevelopment of software libraries. Therefore, we use amodel with the machine learning in this study.
In order to predict the future sequential data,𝑝from the past sequential data
𝑝 ,𝑝 ⋯𝑝𝑥,𝑥⋯ 𝑥 , 𝑥 , Sakata et al. [2] obtained 𝑛steps of future data at once using the Seq2Seq, and Alahiet al. [3] repeated one-step predictions by adding 𝑝to the past data. In this study, we examine the effective
method for predicting 𝑛 steps of future data.

3. METHODS
When applying the prediction of pedestrian trajectory
to realistic mobile robots, predicting multiple steps ofdata would be useful. Therefore, we predict the future
sequential data 𝑝 , 𝑝 ⋯ 𝑝,𝑝with 𝑛 steps
from the past sequential data 𝑥,𝑥⋯ 𝑥 ,𝑥with 𝑚 steps where 𝑚 = 15 and 𝑛 = 10.
For that purpose, we investigated the following twomethods.
I.
II.

Predicting 𝑛 steps of data with 𝑛 models.
Predicting 𝑛 steps of data with a model byapplying one-step prediction several times.

As shown in Fig. 1, with the method I, 𝑛 models aretrained by the machine learning, and the 𝑖 th model(1 ≤ 𝑖 ≤ 𝑛).
predicts the 𝑖 steps-ahead data 𝑝
On the other hand, with the method II, only one modelis trained by the machine learning that predict one stepahead data 𝑝
as shown in Fig. 2. By repeating one
step-ahead prediction 𝑛 times, we can obtain 𝑝 .
In the following sections, we compare these twomethods.

Fig. 1. Method I. Predicting 𝑛 steps with 𝑛 modelsmulti-layer neural network (NN). The LSTM network isa kind of recurrent neural network (RNN) and it is known
that the LSTM network is suitable for learning sequential
data. For the NN, we set the number of hidden layers as
1, which is composed of 300 neurons.
In both cases, the networks are trained to minimize
mean squared error (MSE).

4.2 Dataset
As a dataset, we use the ATC pedestrian tracking
dataset [4] of pedestrian trajectories of general shoppers
in the ATC shopping center in Osaka, which includes
3,758,348 trajectories obtained from various sensors at
10 40 [Hz].
In this study, we chose only the trajectories confinedin an area with 10 [m]
10 [m], and we use 200
trajectories for training, and 20 trajectories for test.

4.3 Method I
MSE of each prediction step with the method I for theLSTM network and the NN are shown in Fig.3. The mean
and the standard deviation of MSE of 20 test data areshown.

Fig. 2. Method II. Predicting 𝑛 steps with a model.

4. EXPERIMENTS
4.1 Network
In this section, we compare the two methods definedin section 3.
We treat pedestrian trajectories represented by twodimensional coordinates (𝑥 , 𝑦 ). We define the input
data as(𝑥,𝑦,𝑥,𝑦⋯𝑥 ,𝑦 )where 𝑚 = 15, and we predict the future sequential data
written as (𝑝𝑥 , 𝑝𝑦 , 𝑝𝑥 , 𝑝𝑦 ⋯ 𝑝𝑥 , 𝑝𝑦 )where 𝑛 = 10.
As models of the machine learning, we use a longshort-term memory (LSTM) network and a conventional

Fig. 3. MSE of each prediction step with the method I
for the LSTM network and the NN.
In Fig. 3, the significant difference ( 𝑝 0.05 )between the results of the LSTM network and the NNwas not observed.
Some examples of the predicted trajectories are shownin Fig. 4. The data with blue line show the original data,
and the data with orange line show the predicted data.
𝑝in Fig. 6. The data with blue line show original data, and.
the data with orange line show the predicted data 𝑝

Fig. 4. Method I. Predicting 𝑛 steps with 𝑛 models.
4.4 Method II
MSE of each prediction step with the method II for theLSTM network and the NN are shown in Fig.5. The mean
and the standard deviation of MSE of 20 test data areshown.

Fig. 5. MSE of each prediction step with the method II
for the LSTM network and the NN.
In Fig. 5, the significant difference ( 𝑝 0.05 )between the results of the LSTM network and the NN
was observed.
Some examples of the predicted trajectories are shown

Fig. 6. Method I. Predicting 𝑛 steps with 𝑛 models.
4.5 Comparison
By comparing the results of the method I and II, it isobserved that the predictions with the method I give
better results than those with the method II as shown inFigs. 3 and 5. This result would not be surprising because
the method I can use multiple steps-ahead data 𝑥(1 ≤ 𝑖 ≤ 𝑛) for training.
By comparing the results of the LSTM network and theNN, it is observed that there was no significant difference
between the results of the LSTM network and the NNwith the method I as shown in Fig. 3.
On the other hand, with the method II, the LSTM
network shows better prediction than that of the NN asshown in Fig. 5. It would be because that the LSTM
network is suitable for learning sequential data.
With the method I, the trained data were not sequential,(1 ≤ 𝑖 ≤ 𝑛)i.e., the multiple steps-ahead data 𝑥
were trained with the 𝑖th LSTM network; therefore, each

LSTM network could not show better results than that ofthe NN.

5. CONCLUSION
In this study, we investigated two methods forpredicting pedestrian trajectories. In order to predict thesequential data, we used the following two methods: (I)predicting 𝑛 steps of data with 𝑛 models, and (II)predicting 𝑛 steps of data with a model by applying onestep prediction several times.
It was found that the method I gives better predictions
than those with the method II. However, in order to use
the method I, we have to train 𝑛 models using multiple
steps-ahead data 𝑥(1 ≤ 𝑖 ≤ 𝑛).
When solving realistic problems, it would not be
possible to train arbitrarily large number of models, andpredictions exceeding the range of training would often
be required. In such cases, the method II would beimportant.
When using the method II, it was found that the LSTM
networks give better predictions than those of theconventional neural networks. By improving the method
II with the LSTM network, this method would become
applicable to predictions for general purposes.
In this study, we did not use interactions amongpedestrians and effects from obstacles. By incorporatingsuch effects into our model, further improvement would
be possible.

REFERENCES
[1]
[2]

[3]

[4]

D. Helbing, and P. Molnar, “Social Force Model for
Pedestrian Dynamics”, Physical Review E, Vol. 51,
No. 5, pp. 4282-4286, 1995.
N. Sakata, Y. Kinoshita and Y. Kato, “Predicting aPedestrian Trajectory Using Seq2Seq for Mobile Robot Navigation”, The 44nd Annual Confer
ence of the IEEE Industrial Electronics Society,pp. 4300-4305, 2008.
A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, LiFei-Fei, S. Savarese, “Social LSTM: Human
Trajectory Prediction in Crowded Spaces”,
Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 961971, 2016.
D. Brscic, Y. Kanda, T. Ikeda and T. Miyashita,“Person Position and Body Direction Tracking inLarge Public Spaces Using 3D Range Sensors,”
IEEE Transactions on Human-Machine Systems,
Vol. 43, No. 6, pp. 522-534, 2013.