Sep. 23, 2022
The 21th Behavior Modeling in Transportation Networks

Fundamentals of neural networkmodeling and its applications totravel behavior analysis
Hiroshima University
Makoto Chikaraishi

Contents

1. Fundamentals of neural network modeling

2. Applications to travel behavior analysis

2

Main reference: Goodfellow, I., Bengio, Y., Courville, A.
(2016) Deep Learning, MIT Press.
Recommend to read this textbook to understand the details!

1. FUNDAMENTALS OF NEURALNETWORK MODELING
3

Deep feedforward networks (FFN)• Also called as multilayer perceptrons• Goal of using (deep) FFN– Approximate some function f that maps an input
x to output y:– Specifically, estimate parameters θ that result inthe best function approximation f• Itʼs called feedforward, since no feedback
connections:↑↑↑
Output Second First
layer layer
layer

4

Why (Deep) NN?
• Limitations of linear regression– Cannot consider 1) non-linearity of input variables & 2)
interactions among input variables• Strategies: Non-linear transformation of x:
1. Use very generic(e.g., Kernel)
2. Manually tune
（using expert knowledge）
3. Use neural network to tune

5

An example of FFN

Different
notation

W: mapping from x to h (parameters)
w: mapping from h to y (parameters)
An example:

Rectified linear unit (ReLU)

Affine transformation

6

Designing and training a NN
1. Define a cost function
e.g., using maximum likelihood (i.e., the crossentropy between the training data and the model
distribution)

Note: there are several other cost functions, including meansquared error, mean absolute error, etc.

7Designing and training a NN
2. Output units
Ø Linear units for Gaussian output distributions
Ø

Only use an affine transformation

Ø Sigmoid units for Bernoulli output distributionsØAffine transportation + logit transformation

Ø Softmax units for Multinoulli output distibutionsØAffine transformation + MNL transformation

8

Designing and training a NN
3. Hidden units
Ø Activation functions are used on top of anaffine transformation:
Ø A widely used activation function:Ø

ReLU: Rectified Linear Unit

Ø Generalizations of ReLUØ

Using nonzero slope αi when z<0
1.
2.
3.

Absolute value rectification: !" = −1
Leaky ReLU: Give a small fixed value to !" such as 0.01
Parametric ReLU: treat !" as a learnable parameterØ There are many other activation functions

9Architecture design• Architecture refers to the overall structure of the
network. Most NNs are organized into groups ofunits called layers.

・・• Main architectural considerations– The depth of the network and the width of each layer– Whether to have skip connections or not– How to connect a pair of layers to each other (W)EX) Convolutional NN, Recurrent NN• The ideal network architecture for a task must be
found via experimentation guided by monitoring
the validation set error.
10Universal approximation theorem• Universal approximation theorem by Hornik et al.
(1989), Cybenko (1989)– This theorem says that neural networks canapproximate any function.

• This theorem also said that “shallow” network
structure can approximate any function, while it isalso known that more efficient learning can beachieved with “deep” network structure.
• Another important issue is the interpretability of
the fully connected NN.
11Learning algorithm• Back-propagation algorithm (Rumelhart et al., 1986)– Computing the gradient• Stochastic gradient descent (SGD)– Perform learning using the gradient
Error gradient (Prediction – Output)

Prediction results

Backpropagation
algorithm

Update parameters

SGD

Gradient for the network

12

Stochastic gradient descent (SGD)• Conventional gradient descent• Stochastic gradient descent– Use mini-batch !′ (small subset of training data)to compute the gradient

Learning rate

13

Back-propagation algorithm
Computationalgraph of NN

Add additional
nodes of gradients
Recall that
Scholar:!

Vector:

!
!′

!

!
!′

!

×

!
!′

×
14

Regularization• What is regularization– “any modification we make to a learningalgorithm that is intended to reduce its
generalization error but not its training error”• Some methods
1. Put extra constraints on a ML model
2. Add extra terms in the objective function
3. Ensemble methods, combining multiple
hypotheses that explain the training data.

• Fundamental properties– Generally increasing bias for reducing variance.

15

Parameter norm penalities• Regularized objective function
The norm penalty term

• L2 parameter regularization (ridge regression)• L1 parameter regularization (LASSO)• Elastic net

16

2. APPLICATIONS TO TRAVELBEHAVIOR ANALYSIS
17

Applications of deep learning
method in transport studies

Varghese, V. Chikaraishi, M., Urata, J. (2020) Deep Learning in Transport Studies: A Meta-Analysis
on the Prediction Accuracy, Journal of Big Data Analytics in Transportation, Vol. 2, 199–220.

18

Tension between theory-driven methods(classical choice models) and data-driven
methods (machine learning)•

Winner of the 2018 Eric Pas Best Dissertation Award– Timothy Brathwaite• The Holy Trinity: Blending Statistics, Machine Learning and Discrete
Choice with Applications to Strategic Bicycle Planning•

ICMC2019 keynote– Joan Walker• Choice modelling in an age of machine learning•

Honorable Mention of the 2019 Eric Pas Best
Dissertation Award– Shenhao Wang
• Deep neural networks for choice analysis
And many others...

19

Problem setting• Standard logit model: $"# =exp !"#
Σ#*+,
exp !"#*• The conventional form of !"# :– Linear approximation (rooted to the Taylor's
theorem)– Also known as a linear-in-parameter model• Problem at hand:– Is there any better way to determine thefunctional form?
• Obviously, taking into account the non-linearity of !"#would improve the goodness-of-fit.
• What is the cost of doing that?
20

Problem setting• Can we understand the non-linear
transformation of !"# logically?
Example: contribution of travel time to mode/route choice model!"#

Highly non-linear transformation

Linear approximation
Logarithm transformation
Travel time of mode j21Non-linearity through neural network (NN):Itʼs about how to construct network architecture
Network architecture
Mode 1

6

Σ#345 exp 1"#3

1"5

75
7:

Mode 2

1":…

1"# = 75 8"#5 + 7: 8"#: + ⋯

75
7:

…

Travelcost ofmode <

75
7:

1"6

!"6

Mode 11"5!"5Mode 21":Mode J(Fully connected)

exp 1"#
6

Σ#345 exp 1"#3

1"# = = >=:= 5 ?,An example of =: Rectified linear unit (ReLU)

Mode J

=5

=:

=>

1"6

Softmax Logit(+, )Network architecture(Deep) NN

!"# =

!":

…

Traveltime ofmode <!"5…

!"# = Logit(+, ) =exp 1"#

Logit(+, )A classical MNL

!":

22

!"6

Fully connected DNN often does not work well(and produce less explainable results)
Non-linearity through neural network (NN):Itʼs all about network architecture
Network architecture
Mode 1

6
Σ#345 exp 1"#3

1"5

Mode 2

75
7:

1":

…

1"# = 75 8"#5 + 7: 8"#: + ⋯

75
7:

…

Travelcost ofmode <

75
7:

1"6

!"6

Mode 11"5!"5Mode 21":Mode J(Fully connected)

exp 1"#

6

Σ#345 exp 1"#3

1"# = = > = := 5 ?,An example of =: Rectified linear unit (ReLU)

Mode J

=5

=:

=>

1"6

Softmax Logit(+, )Network architecture(Deep) NN

!"# =

!":

…

Traveltime ofmode <!"5…

Seeking a better
network structure
in between.

!"# = Logit(+, ) =exp 1"#

Logit(+, )A classical MNL

!":

9

!"6

23

ü Sifringer, B., Lurkin, V.and Alahi, A.: Enhancing discrete choice models withrepresentation learning. Transportation Research Part B 140, 236-261, 2020.
ü Wang, S.: Deep neural networks for choice analysis, PhD Dissertation at MIT,2020
https://dspace.mit.edu/handle/1721.1/129894
ü Han Y, Pereira FC, Ben-Akiva M, Zegras C. A neural-embedded discrete choicemodel: Learning taste representation with strengthened interpretability.
Transportation Research Part B: Methodological. 2022;163:166-86.

SOME EFFORTS TO IMPROVE
THE INTERPRETABILITY
24

Interpretability (Lipton, 2018)• Motivations for asking interpretability– trust, causality, transferability, informativeness,and fair and ethical decision making• Two broader categories of interpretability– Transparency• i.e., how does the model work.
• The opposite of “black-boxness”– Post hoc explanations• i.e., what else can the model tell me.
• Extracting information from learned models.

25

Efforts to keep both interpretability
and accuracy• Wang (2020)– From fully connected deep neural network (F-DNN) to
DNN with alternative-specific utility functions (ASU-DNN)
A classical MNL

Proposed network architecture

Mode 1…
Mode 2…
…

Mode 1!"#Mode 2!"$Mode J!"%

Mode J/"#
Logit(,- )…

(Deep) NN/"$

Mode 1Mode 2/"%Mode J

26

Efforts to keep both interpretability
and accuracy• Sifringer et al. (2020)– Traditional linear-in-parameters are assumed forimportant policy variables, while DNN is used for the restof variables (TB-ResNets proposed by Wang (2020) also follows a similar idea,
but use a different method to implement it)

A classical MNL

Proposed network architecture

Mode 1…
Mode 2…
…

Mode 1$#$%!"#Mode 2

$#
$%

!"%

0"#

…

…

(Deep) NN

Mode J$#$%!"&

Logit(-. )Mode J

0"%Mode 1Mode 2Other
factors0"&Mode J

27

Model of Sifringer et al. (2020)
A classical MNL!"# = Logit(+, ) =Sifringer et al. (2020)

exp 1"#
6
Σ#345 exp 1"#3

789
789
1"#789 = :5 ;"#5
+ := ;"#=+⋯!"# = Logit(+, ) =exp 1"#
6

Σ#345 exp 1"#3

1"# = 1"#789 + 1"#88
Note: 1"#88 should not include variables
used in 1"#789(Deep) NN(Fully connected)!"# =exp 1"#
6

Σ#345 exp 1"#3

1"#88 = ? @?=? 5 A ,BB

An example of ?: Rectified linear unit (ReLU)

28

Han et al. (2022): Further extensionof Sifringer et al. (2020)
Sifringer et al. (2020)!"# = Logit(+, ) =1"# = 1"#789 + 1"#88

exp 1"#
6
Σ#345 exp 1"#3

Han et al. (2022)!"# = Logit(+, ) =exp 1"#
6

Σ#345 exp 1"#3

<8
789
1"# = ;"<8 ="#+ ; 789 ="#;"<8 = >?@ABCBA D" ; F
Individual characteristics

Note: Han et al. (2022) and Sifringer
et al. (2020) are essentially the same<8
when ="#
only contain the constant.

Taste heterogeneities are considered in Han et al. (2022),
which represents through individual characteristics D"
29

Relationship between Han et al.
(2022) and latent class model
Mixed logit
1"# = 2 !"# % 3 % 4%

where!"# (%) =Choice Mixing
probability distribution
conditionalon %exp %+"#
0

Σ#-./ exp %+"#-

Latent class(i.e., the discrete mixing distribution is assumed in the mixed logit)

1"# = 5 !"# %6 1"6 7"
6

Probabilityof belonging
to class 8

where!"# (%6 ) =exp %6 +"#0Σ#-./ exp %6 +"#30

From a perspective of moderation effects
Mixed logit

Varying due to unobserved factors!"#
Latent class$"#

Varying depending on individual characteristics '"!"#Han et al. (2022)

%

%&

$"#

Varying depending on individual characteristics '"

!"#

%&

$"#

Latent class model is comparable to feed-forward neuralnetwork under a certain condition (Vermunt and Magidson. 2003)

31

From a perspective of moderation effects• The drawback of the straightforward
application of neural network is in its
interpretability.
• From the latent class model perspective,
this issue essentially comes from the factthat all input variables have double roles,
i.e., (1) variables directly affecting the
choice decisions, and (2) variables affecting
the class belonging probability which playsa moderator role.

32Han Y, Pereira FC, Ben-Akiva M, Zegras C. A neural-embedded discrete
choice model: Learning taste representation with strengthened
interpretability. Transportation Research Part B: Methodological.
2022;163:166-86.

REVISITING THE ROLE OF
REGULARIZATION
33

Interesting finding in Han et al.
Generate synthetic
datasets using “true”
utility function shownon the right

True utility function

When !"#$%& and '("!%&are correlated () = 0.6),
The performance ofTasteNet-MNL could
outperform that of
TRUE model !!

34

Why it can be better than the truemodel? Han et al.ʼs interpretation• According to Han et al. (2022)– With a wide hidden layer, there is enough flexibility tolearn two taste parameters that do not interfere with eachother and together optimize the objective function.
– The introduced L2 regularization constrains the taste
parameters, making estimated taste variations relatively
stable.

• What we can learn from this?
– The importance of regularization !
– Mixed logit model (with the continuous mixture
distribution) naturally introduces a L2-like regularization
term into the model estimation (see next page).
– The similar regularization may need to made when themodel is fairly complex by introducing interaction terms.
35Evgeniou et al. (2007)••

Compare the proposed L2-like regularization
method and hierarchical Bayes (HB) for
discrete choice model (which is in principle
equivalent to the mixed logit (Train, 2009)).
Confirm that the performance of proposed
L2-like regularization method outperforms
that of HB through two case studies.

Proposed convex optimization approach

HB

L2-like regularization method

Shrinks toward the mean of thefirst-stage prior

Shrink toward the population
mean

Samples from posterior
distribution

Minimize a convex loss function

Posterior distribution is a functionof parameters of the second-stagepriorsLoss function is a function of theregularization parameter !

The parameters of the secondstage priors are set exogenously! is determined using crossvalidation

Key would be in the flexibility of the regularization parameter

36

Take-away messages– Shifting from “choosing theory-driven OR data-driven” to“integrating theory-driven AND data-driven”.
– To utilize NN in travel behavior analysis, we should design
the network structure with paying attention to:• How to handle nonlinearity in an interpretable manner• How to handle moderation effects in an interpretable manner– Han et al.ʼs (2022) work paying attention to both, while it
opens up further possible ways to tune the model.
• Ishii et al. (2022): Tuning moderation effects using a Tucker
decomposition technique.
• Han et al. avoid positive travel impedance parameters by taking
exponential etc., but this does not mean the first derivative is always
negative.

37

References
•
•

•

•

•
•
•

•
•
•
•
•
•

Brathwaite, T.: The Holy Trinity: Blending Statistics, Machine Learning and Discrete Choice, with Applications toStrategic Bicycle Planning, PhD Dissertation at UC Berkeley, 2018, https://escholarship.org/uc/item/1pk9p2ct
Chikaraishi, M., Garg, P., Varghese, V., Yoshizoe, K., Urata, J., Shiomi, Y. and Watanabe, R.: On the possibilityof short-term traffic prediction during disaster with machine learning approaches: An exploratory analysis.
Transport Policy 98, 91-104, 2020.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H.and Bengio, Y.: Learningphrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint
arXiv:1406.1078, 2014.
Cui, Z., Henrickson, K., Ke, R.and Wang, Y.: Traffic Graph Convolutional Recurrent Neural Network: A Deep
Learning Framework for Network-Scale Traffic Learning and Forecasting. IEEE Transactions on Intelligent
Transportation Systems 21, 4883-4894, 2020.
Evgeniou, T., Pontil, M., Toubia, O. (2007) A Convex Optimization Approach to Modeling Consumer
Heterogeneity in Conjoint Estimation. Marketing Science 26, 805-818.
Goodfellow I, Bengio Y, Courville A. Deep learning, MIT Press; 2016.
Han Y, Pereira FC, Ben-Akiva M, Zegras C. A neural-embedded discrete choice model: Learning taste
representation with strengthened interpretability. Transportation Research Part B: Methodological.
2022;163:166-86.
Hochreiter, S.and Schmidhuber, J.: Long short-term memory. Neural computation 9, 1735-1780, 1997.
Ishii, Y., Hayakawa, K., Koide, S., Chikaraishi, M. (2022) Data-driven appraisal of public transport fare policies
using an entropy Tucker model with heterogeneous demand (submitted to…).
Lipton ZC. The mythos of model interpretability: In machine learning, the concept of interpretability is both
important and slippery. Queue. 2018;16(3):31-57.
Sifringer, B., Lurkin, V.and Alahi, A.: Enhancing discrete choice models with representation learning.
Transportation Research Part B 140, 236-261, 2020.
Wang, S.: Deep neural networks for choice analysis, PhD Dissertation at MIT,2020
https://dspace.mit.edu/handle/1721.1/129894
Tong, Z., Liang, Y., Sun, C., Rosenblum, D.S.and Lim, A.: Directed graph convolutional network. arXiv preprint
arXiv:2004.13970, 2020. https://arxiv.org/pdf/2004.13970.pdf

38

39

Appendix 1

RECURRENT NEURALNETWORK40

Recurrent neural network• Recurrent neural network– A neural network that is specialized forprocessing a sequence of values (e.g., timeseries data).
– Parameter sharing• A recurrent neural network typically shares the same
parameters across time steps.
• This is needed to generalize and make it possible topredict future.
• An example:– Recurrent structure:» Tomorrow will come after today.
– Non-recurrent structure:» Sep. 18, 2021 will come after Sep. 17, 2021.

– There are a wide variety of recurrent neuralnetworks (next slide).
41

Examples of RNN structures

Recurrent networks that produce an output
at each time step and have recurrent
connections between hidden nodes.

Recurrent networks that produce an output at
each time step and have recurrent
connections only from the output at the next
step to the hidden units at the next time step.

Adding connection from the output at time
t to the hidden unit at time t+1
Bidirectional recurrent networks

Make network deeper

Various structures exist (similar with time series models with lagged variables)

42

Hierarchical structure of network:
use the concept of “cell”
Having a cell (a set of nodes with a particular network structure),
instead of simply having a node.

43

LSTM cell (Long short-term memory)
Hochreiter and Schmidhuber (1997)
The weight on the self-loop conditioned on the context(rather than fixed), resulting in the dynamic control of
the time scale and forgetting behavior of different units.
(Conceptually, it is similar with the introduction of moderators.
Please refer to Yamamoto senseiʼs lecture note as well)

LSTM cell44

LSTM cell (Long short-term memory)
Hochreiter and Schmidhuber (1997)
Forget gate

Input gate

State

OutputOther variants...
ü GRU: Gated recurrent unit (Cho etal., 2014)
ü Simpler than LSTM, but theperformance is similar to that of
LSTM for some applications.

Output gate45