Sep. 23, 2022 The 21th Behavior Modeling in Transportation Networks Fundamentals of neural network modeling and its applications to travel behavior analysis Hiroshima University Makoto Chikaraishi
1. Fundamentals of neural network modeling 2. Applications to travel behavior analysis
2
Main reference: Goodfellow, I., Bengio, Y., Courville, A. (2016) Deep Learning, MIT Press. Recommend to read this textbook to understand the details! 1. FUNDAMENTALS OF NEURAL NETWORK MODELING
• Also called as multilayer perceptrons • Goal of using (deep) FFN – Approximate some function f that maps an input x to output y: y = f*(x) – Specifically, estimate parameters θ that result in the best function approximation f y = f(x;θ) • It's called feedforward, since no feedback connections:
Output   Second layer layer layer
4

•   Limitations of linear regression – Cannot consider 1) non-linearity of input variables & 2) interactions among input variables Strategies: Non-linear transformation of x: φ(x) • 1. 2. Use very generic φ( x ) (e.g., Kernel) Manually tune φ( x ) (using expert knowledge) 3. Use neural network to tune  φ(x)
5

An example: W: mapping from x to h (parameters) w: mapping from h to y (parameters) Rectified linear unit (ReLU) f (x; W , c, w, b) = w T max{0, W T x + c} +
Affine transformation

1. Define a cost function e.g., using maximum likelihood (i.e., the cross- entropy between the training data and the model distribution) J(θ) = − Ex,y~îdata  log Pmodel(Y|x) Note: there are several other cost functions, including mean squared error, mean absolute error, etc.
7

2. Output units
▷ Linear units for Gaussian output distributions ➤ Only use an affine transformation ŷ = wTh + b ▷ Sigmoid units for Bernoulli output distributions ▷ Affine transportation + logit transformation ŷ = σ(wTh + b)   where   σ(x) = 1+exp(−x) ▷ Softmax units for Multinoulli output distibutionsんAffine transformation + MNL transformation  
8

3. Hidden units
➤ affine transformation: h = g(W T x + b) ➤ A widely used activation function: A ReLU: Rectified Linear Unit g(z) = max{0, z} > Generalizations of ReLUムUsing nonzero slope α i when z<0 h i = g(z, α) i = max{0, z i } + α i min{0, z i } Absolute value rectification: α i = −1 1. Leaky ReLU: Give a small fixed value to α i such as 0.01 2. 3. Parametric ReLU: treat α i as a learnable parameter ▷ There are many other activation functions
9

•  Architecture refers to the overall structure of the network. Most NNs are organized into groups of units called layers. h(¹) = g(¹)(W(¹)T x + b(¹)) h(²) = g(²)(W(²)T x + b(²)) •  Main architectural considerations – The depth of the network and the width of each layer – Whether to have skip connections or not –  How to connect a pair of layers to each other (W) EX) Convolutional NN, Recurrent NN • The ideal network architecture for a task must be found via experimentation guided by monitoring the validation set error.
10

•  Universal approximation theorem by Hornik et al. (1989), Cybenko (1989) – This theorem says that neural networks can approximate any function. •  This theorem also said that “shallow” network structure can approximate any function, while it is also known that more efficient learning can be achieved with “deep” network structure. •  Another important issue is the interpretability of the fully connected NN.
11

• Back-propagation algorithm (Rumelhart et al., 1986) – Computing the gradient •  Stochastic gradient descent (SGD) – Perform learning using the gradient Prediction results Error gradient (Prediction – Output) propagation
• Conventional gradient descent J(θ) = ↑ ∑ i=1 L(x(i), y(i), θ) ∇θJ(θ) = ↑ ∑ i=1 ∇θL(x(i), y(i), θ) • Stochastic gradient descent – Use mini-batch m' (small subset of training data) to compute the gradient g = m ⁄ ∑ i=1 ∇ θ L(x(i), y(i), θ ) θ ← θ — eg Learning rate
13

Computational graph of NN Add additional nodes of gradients

• What is regularization
– “any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error”
•  Some methods
1.  Put extra constraints on a ML model 2.  Add extra terms in the objective function 3.  Ensemble methods, combining multiple hypotheses that explain the training data. •  Fundamental properties – Generally increasing bias for reducing variance.
15

• Regularized objective function J(θ; X, y) = J(θ; X, y) + αΩ(θ)  The norm penalty term •  L² parameter regularization (ridge regression) J(w;X,y) = J(w;X,y) + αwTw •  L¹ parameter regularization (LASSO) J(w; X, y) = J(w; X, y) + α ∑ i |w i | • Elastic net
16
2. APPLICATIONS TO TRAVEL BEHAVIOR ANALYSIS
Applications of deep learning

Abbreviations for area of application: DB, driver behaviour; M–AC, mode and activity choice; TD, travel demand; TTP, travel time prediction; Dist., travel distance; Occ, occupancy Varghese, V. Chikaraishi, M., Urata, J. (2020) Deep Learning in Transport Studies: A Meta-Analysis on the Prediction Accuracy, Journal of Big Data Analytics in Transportation, Vol. 2, 199–220. 18
• Winner of the 2018 Eric Pas Best Dissertation Award – Timothy Brathwaite •  The Holy Trinity: Blending Statistics, Machine Learning and Discrete Choice with Applications to Strategic Bicycle Planning •
ICMC2019 keynote
– Joan Walker • Choice modelling in an age of machine learning •
Honorable Mention of the 2019 Eric Pas Best
Dissertation Award – Shenhao Wang • Deep neural networks for choice analysis
•  The conventional form of Vij : – Linear approximation (rooted to the Taylor's theorem) – Also known as a linear-in-parameter model •  Problem at hand: – Is there any better way to determine the functional form? • Obviously, taking into account the non-linearity of would improve the goodness-of-fit. • What is the cost of doing that?
20

• Can we understand the non-linear transformation of V i  logically? Example: contribution of travel time to mode/route choice model Travel time of mode j
21

A classical MNL Network architecture Network architecture exp( Vii
Travel Travel time of cost of mode j mode j (Deep) NN (Fully connected)
An example of f : Rectified linear unit (ReLU) f (æ; W , c, w, b) = w T max{0, W T æ + c} + b
Seeking a better network structure in between.
23
> Sifringer, B., Lurkin, V.and Alahi, A.: Enhancing discrete choice models with representation learning. Transportation Research Part B 140, 236-261, 2020. ✓ Wang, S.: Deep neural networks for choice analysis, PhD Dissertation at MIT,2020 https://dspace.mit.edu/handle/1721.1/129894 > Han Y, Pereira FC, Ben-Akiva M, Zegras C. A neural-embedded discrete choice model: Learning taste representation with strengthened interpretability. Transportation Research Part B: Methodological. 2022;163:166-86.
SOME EFFORTS TO IMPROVE

24

•  Motivations for asking interpretability – trust, causality, transferability, informativeness, and fair and ethical decision making
• Two broader categories of interpretability
– Transparency
• i.e., how does the model work. • The opposite of “black-boxness”
– Post hoc explanations
• i.e., what else can the model tell me. • Extracting information from learned models.
25

•
Wang (2020)
–  From fully connected deep neural network (F-DNN) to DNN with alternative-specific utility functions (ASU-DNN) A classical MNL Proposed network architecture
26

Sifringer et al. (2020)
.
– Traditional linear-in-parameters are assumed for important policy variables, while DNN is used for the rest of variables (TB-ResNets proposed by Wang (2020) also follows a similar idea, but use a different method to implement it) Proposed network architecture
27

A classical MNL
Sifringer et al. (2020)
Vij =  VMNL + VNN Note: V N N should not include variables used in VMNL (Deep) NN
An example of f : Rectified linear unit (ReLU) f ( x ; W , c, w, b ) = w T max { 0, W T x + c} + b
Sifringer et al. (2020) Pij = Logit(Vi) = Vij =  VMNL + VNN Han et al. (2022) exp( Vii
Vij = βinxTN + βmnlxMnl ββTN = TasteNet(zi; W) Individual characteristics Note: Han et al. (2022) and Sifringer et al. (2020) are essentially the same when x i N only contain the constant. Taste heterogeneities are considered in Han et al. (2022), which represents through individual characteristics z i
29

Mixed logit
Latent class (i.e., the discrete mixing distribution is assumed in the mixed logit)
Mixed logit Varying due to unobserved factors B Pij Latent class Varying depending on individual characteristics z ¡ βκ Pij Han et al. (2022) Varying depending on individual characteristics z ¡ Xij βκ Pij Latent class model is comparable to feed-forward neural network under a certain condition (Vermunt and Magidson. 2003) 31
• The drawback of the straightforward application of neural network is in its interpretability. • From the latent class model perspective, this issue essentially comes from the fact that all input variables have double roles, i.e., (1) variables directly affecting the choice decisions, and (2) variables affecting the class belonging probability which plays a moderator role.
32
Han Y, Pereira FC, Ben-Akiva M, Zegras C. A neural-embedded discrete choice model: Learning taste representation with strengthened interpretability. Transportation Research Part B: Methodological. 2022;163:166-86.

33
Interesting finding in Han et al.
Generate synthetic datasets using “true” utility function shown on the right True utility function Vin = ASCi — cost in + ( − 0.1 − 0.5inc n − 0.1 full n + 0.05 flex n –0.2inc n * full n + 0.05inc n * flex n + 0.1full n * flex n ) * timein + ( − 0.2 − 0.8inc n − 0.3 full n + 0.1 flex n −0.3inc n * full n +0.08inc n * flex n +0.3full n * flex n ) * waitin When timein and waitin are correlated (ρ = 0.6), The performance of TasteNet-MNL could outperform that of TRUE model !!
•  According to Han et al. (2022) –  With a wide hidden layer, there is enough flexibility to learn two taste parameters that do not interfere with each other and together optimize the objective function. – The introduced L2 regularization constrains the taste parameters, making estimated taste variations relatively stable.
•  What we can learn from this?
– The importance of regularization ! –  Mixed logit model (with the continuous mixture distribution) naturally introduces a L2-like regularization term into the model estimation (see next page). – The similar regularization may need to made when the model is fairly complex by introducing interaction terms.
35

• Compare the proposed L2-like regularization method and hierarchical Bayes (HB) for discrete choice model (which is in principle equivalent to the mixed logit (Train, 2009)). • Confirm that the performance of proposed L2-like regularization method outperforms that of HB through two case studies. Proposed convex optimization approach
Key would be in the flexibility of the regularization parameter
36

–  Shifting from “choosing theory-driven OR data-driven” to “integrating theory-driven AND data-driven”. – To utilize NN in travel behavior analysis, we should design the network structure with paying attention to: •  How to handle nonlinearity in an interpretable manner •  How to handle moderation effects in an interpretable manner –  Han et al.’s (2022) work paying attention to both, while it opens up further possible ways to tune the model. •  Ishii et al. (2022): Tuning moderation effects using a Tucker decomposition technique. •  Han et al. avoid positive travel impedance parameters by taking exponential etc., but this does not mean the first derivative is always negative.
37
. . . Brathwaite, T.: The Holy Trinity: Blending Statistics, Machine Learning and Discrete Choice, with Applications to Strategic Bicycle Planning, PhD Dissertation at UC Berkeley, 2018, https://escholarship.org/uc/item/1pk9p2ct Chikaraishi, M., Garg, P., Varghese, V., Yoshizoe, K., Urata, J., Shiomi, Y. and Watanabe, R.: On the possibility of short-term traffic prediction during disaster with machine learning approaches: An exploratory analysis. Transport Policy 98, 91-104, 2020. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H.and Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. . Cui, Z., Henrickson, K., Ke, R.and Wang, Y.: Traffic Graph Convolutional Recurrent Neural Network: A Deep Learning Framework for Network-Scale Traffic Learning and Forecasting. IEEE Transactions on Intelligent Transportation Systems 21, 4883-4894, 2020. . . . Evgeniou, T., Pontil, M., Toubia, O. (2007) A Convex Optimization Approach to Modeling Consumer Heterogeneity in Conjoint Estimation. Marketing Science 26, 805-818. Goodfellow I, Bengio Y, Courville A. Deep learning, MIT Press; 2016. Han Y, Pereira FC, Ben-Akiva M, Zegras C. A neural-embedded discrete choice model: Learning taste representation with strengthened interpretability. Transportation Research Part B: Methodological. 2022;163:166-86. . . Hochreiter, S.and Schmidhuber, J.: Long short-term memory. Neural computation 9, 1735-1780, 1997. Ishii, Y., Hayakawa, K., Koide, S., Chikaraishi, M. (2022) Data-driven appraisal of public transport fare policies using an entropy Tucker model with heterogeneous demand (submitted to…). . . Lipton ZC. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue. 2018;16(3):31-57. Sifringer, B., Lurkin, V.and Alahi, A.: Enhancing discrete choice models with representation learning. Transportation Research Part B 140, 236-261, 2020. . Wang, S.: Deep neural networks for choice analysis, PhD Dissertation at MIT,2020 https://dspace.mit.edu/handle/1721.1/129894 . Tong, Z., Liang, Y., Sun, C., Rosenblum, D.S.and Lim, A.: Directed graph convolutional network. arXiv preprint arXiv:2004.13970, 2020. https://arxiv.org/pdf/2004.13970.pdf Appendix 1 RECURRENT NEURAL NETWORK
– A neural network that is specialized for processing a sequence of values (e.g., time series data). – Parameter sharing • A recurrent neural network typically shares the same parameters across time steps. • This is needed to generalize and make it possible to predict future. • An example: – Recurrent structure: » Tomorrow will come after today. – Non-recurrent structure: » Sep. 18, 2021 will come after Sep. 17, 2021. – There are a wide variety of recurrent neural networks (next slide).
Recurrent networks that produce an output at each time step and have recurrent connections between hidden nodes. Adding connection from the output at time t to the hidden unit at time t+1 Recurrent networks that produce an output at each time step and have recurrent connections only from the output at the next step to the hidden units at the next time step. Bidirectional recurrent networks Various structures exist (similar with time series models with lagged variables)
Having a cell (a set of nodes with a particular network structure), instead of simply having a node.
Hochreiter and Schmidhuber (1997)
Hochreiter and Schmidhuber (1997) Other variants… > GRU: Gated recurrent unit (Cho et al., 2014) > Simpler than LSTM, but the performance is similar to that of LSTM for some applications. Forget gate Input gate Lj UyjXjt) State Output (t) Output gate