Sep. 17, 2021
The 20 th Behavior Modeling in Transportation Networks
Lecture series #2-3

LSTM & RNN for day-to-day panel data
On the use of ML (particularly NN) for representing
temporal dependencies in transport studies

Hiroshima University
Makoto Chikaraishi

INTRODUCTION

2

Applications of deep learning method intransport studies

Varghese, V. Chikaraishi, M., Urata, J. (2020) Deep Learning in Transport Studies: A Meta‐Analysis
on the Prediction Accuracy, Journal of Big Data Analytics in Transportation, Vol. 2, 199–220.

3

Tension between theory-driven methods(classical choice models) and data-driven
methods (machine learning)•

Winner of the 2018 Eric Pas Best Dissertation Award– Timothy Brathwaite• The Holy Trinity: Blending Statistics, Machine Learning and Discrete
Choice with Applications to Strategic Bicycle Planning•

ICMC2019 keynote– Joan Walker• Choice modelling in an age of machine learning•

Honorable Mention of the 2019 Eric Pas BestDissertation Award– Shenhao Wang
• Deep neural networks for choice analysis
And many others...

4

A conventional derivation of logit model(for behavior modelers)

Pi1  Pr U i1  U i 2 f  i1 ,  i 2  Pr Vi1   i1  Vi 2   i 2  i2 Pr  i 2   i1  Vi1  Vi 2 
i 1   i1

 i 1 Vi 1 Vi 2

 f  i1 ,  i 2 d i1d i 2

http://www.biwako.shiga‐u.ac.jp/sensei/mnaka/ut/sozai/prob.html

 i 2  

 i2 VUi2

 i 2   i1  Vi1  Vi 2i2

Vi1
 i1

U i15

A conventional derivation of logit model(for behavior modelers)

Assuming εi1 and εi2 are independent,Pi1 


i 1   i 1 Vi 1 Vi 2f  ,  d d


i1i2

i1i2i 2   i1 Vi1 Vi 2    f  i 2 d i 2  f  i1 d i1
 i 1   
  i 2  

Figure. Probability Density Function for
Gumbel and Normal Distributions

 i1 Vi1 Vi 2
    exp  i 2  exp exp i 2 d i 2  f  i1 d i1
 i 1   
  i 2   ↑ Assuming Gumbel distribution exp exp  V  V  f  d
i1i1i2i1i1i 1  

exp exp  V  V exp   exp exp d
i1i 1  i1i2i1i1↑ Assuming Gumbel distributioni16

Problem setting• Standard logit model:• The conventional form of:– Linear approximation (rooted to the Taylor'stheorem)– Also known as a linear-in-parameter model• Problem at hand:– Is there any better way to determine thefunctional form?
• Obviously, taking into account the non-linearity of 𝑉
would improve the goodness-of-fit.
• What is the cost of doing that?
7Problem setting• Can we understand the non-linear
logically?
transformation ofExample: contribution of travel time to mode/route choice model𝑉Highly non-linear transformation

Linear approximation
Logarithm transformation
Travel time of mode j8

Non-linearity through neural network (NN):Itʼs about how to construct network architecture
Network architecture

Logit 𝐕𝐢𝛽 𝑥

exp 𝑉⋯𝛽𝛽𝑉Mode 2…

Travelcost ofmode 𝑗𝛽𝛽𝑉𝑃

Mode 1𝑉𝑃

Mode 2𝑉

Mode JNetwork architecture(Deep) NN(Fully connected)

exp 𝑉𝑃Σ𝑉

exp 𝑉𝑓𝑃…

Traveltime ofmode 𝑗𝑃𝑓𝑓𝐱𝐢

An example of 𝑓: Rectified linear unit (ReLU)

Mode J𝑓𝑓𝑓

𝑉Softmax Logit 𝐕𝐢𝛽𝑥𝑉…

𝑉ΣMode 1𝛽𝛽…

𝑃exp 𝑉Logit 𝐕𝐢A classical MNL

𝑃9𝑃

Universal approximation theorem• Universal approximation theorem by Hornik et al.
(1989), Cybenko (1989)– This theorem says that neural networks canapproximate any function.

• This theorem also said that “shallow” network
structure can approximate any function, while it isalso known that more efficient learning can beachieved with “deep” network structure.
• Another important issue is the explainability of thefully connected NN.
10Fully connected DNN often does not work well(and produce less explainable results)

Seeking a better
network structure
in between.

11Efforts to keep both explainability
and accuracy• Wang (2020)– From fully connected deep neural network (F-DNN) toDNN with alternative-specific utility functions (ASU-DNN)
A classical MNL

Proposed network architecture

Mode 1…
Mode 2…
…

Mode 1𝑉

Mode 2𝑉

Mode J𝑉𝑃

Mode JLogit 𝐕𝐢…

(Deep) NN𝑃Mode 1

Mode 2𝑃

Mode J12

Efforts to keep both explainability
and accuracy• Sifringer et al. (2020)– Traditional linear-in-parameters are assumed forimportant policy variables, while DNN is used for the restof variables (TB-ResNets proposed by Wang (2020) also follows a similar idea,
but use a different method to implement it)

A classical MNL

Proposed network architecture

Mode 1…
Mode 2…
…

Mode 1𝛽𝛽𝑉

Mode 2𝛽𝛽𝑉

𝑃…

…

(Deep) NN

Mode J𝛽𝛽𝑉Logit 𝐕𝐢Mode J𝑃

Mode 1Mode 2

Otherfactors𝑃Mode J13Chikaraishi, M., Garg, P., Varghese, V., Yoshizoe, K., Urata, J., Shiomi, Y.
and Watanabe, R.: On the possibility of short-term traffic prediction
during disaster with machine learning approaches: An exploratory
analysis. Transport Policy 98, 91-104, 2020.

WHAT WOULD HAPPEN IF
WE NAIVELY APPLY ML?
14

An example of less explainable results

1/3

1. Predict loop detector 9ʼs
traffic flow (Q) and time
occupancy (K) using
different ML methods.
2. Check the consistency ofthe results with theory
Traffic flow theory said: traffic
state should be dependent on
traffic volume on the upstream
and/or time occupancy on thedownstream in congested
situation (not the other wayaround).

15

An example of less explainable results

2/3

Prediction
accuracy
XGB is thebest in terms
of prediction
accuracy.
DNN also
performs well.

16

An example of less explainable results

3/3

Particularly XGB does
NOT really mimic themechanisms ofcongestion occurrence.
What we have learned:
The model which produces
the best prediction
accuracy is not always thebest for practical use.

17

Goodfellow I, Bengio Y, Courville A. Deep learning, MITPress; 2016.

NETWORK ARCHITECTURE FOR
REPRESENTING TEMPORAL
DEPENDENCIES
18

Recurrent neural network• Recurrent neural network– A neural network that is specialized forprocessing a sequence of values (e.g., timeseries data).
– Parameter sharing• A recurrent neural network typically shares the same
parameters across time steps.
• This is needed to generalize and make it possible topredict future.
• An example:– Recurrent structure:» Tomorrow will come after today.
– Non-recurrent structure:» Sep. 18, 2021 will come after Sep. 17, 2021.

– There are a wide variety of recurrent neuralnetworks (next slide).
19

Examples of RNN structures

Recurrent networks that produce an output
at each time step and have recurrent
connections between hidden nodes.

Recurrent networks that produce an output at
each time step and have recurrent
connections only from the output at the next
step to the hidden units at the next time step.

Adding connection from the output at time
t to the hidden unit at time t+1
Bidirectional recurrent networks

Make network deeper

Various structures exist (similar with time series models with lagged variables)

20

Hierarchical structure of network:
use the concept of “cell”
Having a cell (a set of nodes with a particular network structure),
instead of simply having a node.

21

LSTM cell (Long short-term memory)
Hochreiter and Schmidhuber (1997)
The weight on the self‐loop conditioned on the context(rather than fixed), resulting in the dynamic control of
the time scale and forgetting behavior of different units.
(Conceptually, it is similar with the introduction of moderators.
Please refer to Yamamoto senseiʼs lecture note as well)LSTM cell

22

LSTM cell (Long short-term memory)
Hochreiter and Schmidhuber (1997)
Forget gate

Input gate

State

OutputOther variants...
 GRU: Gated recurrent unit (Cho etal., 2014)
 Simpler than LSTM, but theperformance is similar to that of
LSTM for some applications.

Output gate

23

Remaining concerns in applications
to transport issues• Temporal dependencies are notindependent from spatial dependencies.
– LSTM is designed for handling temporaldependencies, not spatial dependencies.

24Cui, Z., Henrickson, K., Ke, R.and Wang, Y.: Traffic Graph
Convolutional Recurrent Neural Network: A Deep Learning
Framework for Network-Scale Traffic Learning andForecasting. IEEE Transactions on Intelligent Transportation
Systems 21, 4883-4894, 2020.

TGC-LSTM(TRAFFIC GRAPH CONVOLUTIONAL RNN)

25

Network architecture employedAn free-flow reachable matrix
1,
𝑖𝑓 𝑆 , 𝑚∆𝑡 𝐷𝑖𝑠𝑡 ,𝐹𝐹𝑅 ,0,𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒0,∀ 𝑣 ,𝑣

OutputForget
gate

Input
gate
State

Outputgate

An adjacency matrix 𝐴∈ℝ𝑁×𝑁, in which eachelement 𝐴𝑖,𝑗=1 if there is an edge connecting node𝑖 and node 𝑗 and 𝐴𝑖,𝑗=0 otherwise.
min 𝐴 𝐼 , , 1𝐴,(called a 𝑘 –hop neighborhood matrix)
26

Empirical resultsa) LOOP dataset covering the freeway network in Seattle
area; (b) INRIX dataset covering the downtown Seattlearea, where traffic segments are plotted with colors.

Histogram of performance comparison for theinfluence of orders (hops) of graph convolution
in the TGC LSTM on INRIX and LOOP datasets.

Validation loss versus training epoch (batch size
= 40 and early stopping patience = 10 epochs)

27

Limitations
1. Undirected graph is used, not directed one.
– Limited applications of directed graph convolution.
– Recently, some researchers have proposed approximation
methods such as Tong et al. (2020)

2. Travel time and congestions are not endogenously modeled.
– choice [C] = f(travel time [TT]), but also TT = g(C) = g(f(TT))

28

Conclusions• Take-away messages:– Shifting from “choosing theory-driven OR data-driven” to“integrating theory-driven AND data-driven”.
– Temporal dimension can be well modeled using LSTM etc.,
while temporal dimension and spatial dimension cannot be
simply separated in transport studies.
– Some researchers have been actively working on thedevelopment of the methods for handling both temporaland spatial dependencies in a consistent way with theories
in the transportation field. Yet, still a number of challenges(e.g., endogenous representation of travel time withdirected graph) remain.
29

References
••

•

•

•••
•
•

Brathwaite, T.: The Holy Trinity: Blending Statistics, Machine Learning and Discrete
Choice, with Applications to Strategic Bicycle Planning, PhD Dissertation at UC Berkeley,2018, https://escholarship.org/uc/item/1pk9p2ct
Chikaraishi, M., Garg, P., Varghese, V., Yoshizoe, K., Urata, J., Shiomi, Y. and Watanabe,R.: On the possibility of short-term traffic prediction during disaster with machinelearning approaches: An exploratory analysis. Transport Policy 98, 91-104, 2020.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H.andBengio, Y.: Learning phrase representations using RNN encoder-decoder for statisticalmachine translation. arXiv preprint arXiv:1406.1078, 2014.
Cui, Z., Henrickson, K., Ke, R.and Wang, Y.: Traffic Graph Convolutional Recurrent
Neural Network: A Deep Learning Framework for Network-Scale Traffic Learning andForecasting. IEEE Transactions on Intelligent Transportation Systems 21, 4883-4894,2020.
Goodfellow I, Bengio Y, Courville A. Deep learning, MIT Press; 2016.
Hochreiter, S.and Schmidhuber, J.: Long short-term memory. Neural computation 9,1735-1780, 1997.
Sifringer, B., Lurkin, V.and Alahi, A.: Enhancing discrete choice models withrepresentation learning. Transportation Research Part B 140, 236-261, 2020.
Wang, S.: Deep neural networks for choice analysis, PhD Dissertation at MIT,2020
https://dspace.mit.edu/handle/1721.1/129894Tong, Z., Liang, Y., Sun, C., Rosenblum, D.S.and Lim, A.: Directed graph convolutional
network. arXiv preprint arXiv:2004.13970, 2020. https://arxiv.org/pdf/2004.13970.pdf

30