The 18th Behavior Modeling Summer School
Sep. 21 – 23 , 2019 @ The University of Tokyo

Basic inference and validation in discrete choice modeling
Giancarlos Troncoso Parady – The University of Tokyo

The 18th Behavior Modeling Summer School
Sep. 21 – 23 , 2019 @ The University of Tokyo

Basic inference discrete choice modelingWhy is inference important ?
Variable name

CoefficientS.E.

t statistic

1.45

0.393

3.70

In-vehicle time (min)-0.00890.0063-1.42Out-of-vehicle time (min)-0.03080.0106-2.90

Auto out-of-pocket cost (c)-0.0115

0.0026-4.39

Transit fare

-0.00700.0038-1.87

Auto ownership (specific to auto mode)-0.770

0.213

3.16

Downtown workplace (specific to auto mode)-0.561

0.306

-1.84

Number of observations

1476

Number of cases

1476

LL(0)

-1023

LL(β)

-347.4-2[LL(0)-LL(β)]

1371

𝜌2

0.660

𝜌ҧ 2

0.654

Auto constant

Table adapted from Ben-Akiva and Lerman (1985)

Magnitudes are not directly interpretable.
We can only interpret the effect direction,or use them to calculate utilities,
and choice probabilities

To make some sense of theseparameters we must calculate
elasticities or marginal effects

Basic Inference in discrete choice models
MNL: Logit Elasticities (Point elasticities)•

Direct elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in an attribute of that same
alternative.

𝜕𝑃𝑛 (𝑖) 𝑥𝑖𝑛𝑘
𝑃 𝑖
𝐸𝑥𝑖𝑛𝑘 =∙=
𝜕𝑥𝑖𝑛𝑘 𝑃𝑛 (𝑖)•1 − 𝑃𝑛 𝑖 𝑥𝑖𝑛𝑘 𝛽𝑘

Cross-elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in a competing alternative.

𝜕𝑃𝑛 (𝑖) 𝑥𝑗𝑛𝑘
𝑃 𝑖
𝐸𝑥𝑗𝑛𝑘 =∙= −𝑃𝑛 𝑗 𝑥𝑗𝑛𝑘 𝛽𝑘𝜕𝑥𝑗𝑛𝑘 𝑃𝑛 (𝑖)Hensher, David A., John M. Rose, and William H. Greene (2015)

Because of IIA, crosselasticities are uniform
across all alternatives

Basic Inference in discrete choice models
MNL: Logit Elasticities (Point elasticities)•

The elasticities shown before are individual elasticities (Disaggregate)•

To calculate sample (aggregate) elasticities we use the probability weighted
sample enumeration method:
𝑃 𝑖
𝑁
෠
σ
𝑃
𝑖
𝐸
𝑛=1 𝑛
𝑥𝑖𝑛𝑘
𝑃(𝑖)
𝐸𝑥𝑖𝑛𝑘 =෠σ𝑁
𝑛=1 𝑃𝑛 𝑖𝑃(𝑖)𝐸𝑥𝑗𝑛𝑘 =

Sample direct elasticity

𝑃 𝑖
෠
σ𝑁
𝑃
𝑖
𝐸
𝑛=1 𝑛
𝑥𝑗𝑛𝑘

෠
σ𝑁
𝑛=1 𝑃𝑛 𝑖

Sample cross-elasticity

Where 𝑃(𝑖) is the aggregate choice probability of alternative I, and 𝑃෠𝑖𝑛 𝑖 is an estimated choice probability•

Uniform cross-elasticities do not necessarily hold at the aggregate level•

Also note that elasticities for dummy variables are meaningless!

Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
Relation between elasticity of demand, change in price and revenue

Px

Px

Perfectly
inelastic

X

Px

Relatively
inelastic

X

Px

Unit
elastic

X

Px

Relatively
elastic

X

Perfectly
elastic

X

Direct elasticity:
1% increase in X results in a
0% decrease in P(i)1% increase in X results in aless than 1% decrease in P(i)1% increase in X results in a
1% decrease in P(i)1% increase in X results in amore than 1% decrease in P(i)1% increase in X results
in a ∞ decrease in P(i)1% increase in X results in aless than 1% increase in P(j)1% increase in X results inno percent change in P(j)1% increase in X results in amore than 1% increase in P(j)1% increase in X results
in a ∞ increase in P(j)Cross elasticity:
1% increase in X results in a
0% increase in P(j)Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects•

Direct marginal effects: measures the change in the probability (absolute change) of choosing
a particular alternative in the choice set with respect to a unit change in an attribute of that same
alternative.

𝜕𝑃𝑛 (𝑖)
𝑃 𝑖
𝑀𝑥𝑖𝑛𝑘 == 𝑃𝑛 𝑖𝜕𝑥𝑖𝑛𝑘•

1 − 𝑃𝑛 𝑖 𝛽𝑘

Cross-marginal effects: measures the change in the probability (absolute change) of choosing
a particular alternative in the choice set with respect to a unit change in a competing alternative.

𝜕𝑃𝑛 (𝑖)𝑃 𝑖
𝑀𝑥𝑗𝑛𝑘 == 𝑃𝑛 𝑖𝜕𝑥𝑗𝑛𝑘

Hensher, Rose, and Greene (2015)−𝑃𝑛 𝑗 𝛽𝑘

Basic Inference in discrete choice models
MNL: Marginal Effects•

We can also calculate sample (aggregate) marginal effects we using e the probability weighted
sample enumeration method:
𝑃 𝑖
𝑁
෠
σ
𝑃
𝑖
𝑀
𝑛=1 𝑛
𝑥𝑖𝑛𝑘
𝑃(𝑖)
𝑀𝑥𝑖𝑛𝑘 =෠σ𝑁
𝑛=1 𝑃𝑛 𝑖
Sample direct marginal effect𝑃(𝑖)𝑀𝑥𝑗𝑛𝑘 =

𝑃 𝑖
෠
σ𝑁
𝑃
𝑖
𝑀
𝑛=1 𝑛
𝑥𝑗𝑛𝑘

෠
σ𝑁
𝑛=1 𝑃𝑛 𝑖

Sample cross-marginal effect

Where 𝑃(𝑖) is the aggregate choice probability of alternative I, and 𝑃෠𝑖𝑛 𝑖 is an estimated choice probability•

Marginal effects for dummy variables do make sense as we are talking about unit changes!

Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects
P(i)𝜕𝑃𝑛 (𝑖)𝜕𝑥𝑖𝑛𝑘

xi
Marginal effects as the slopes of the Tangent lines to the cumulative probability curve
Hensher, David A., John M. Rose, and William H. Greene (2015)

Basic Inference in discrete choice models
Incremental Logit for prediction•

An alternative approach to using elasticities or marginal effects for prediction•

Prediction of changes in behavior based on existing choice probabilities

𝑃′ 𝑖

=

exp 𝑉𝑖𝑛 + ∆𝑉𝑖𝑛
σ𝑗∈𝐶 exp 𝑉𝑗𝑛 + ∆𝑉𝑗𝑛

𝐾;

𝑤ℎ𝑒𝑟𝑒 ∆𝑉𝑖𝑛 = ෍ 𝛽𝑘 ∆𝑥𝑖𝑛𝑘
𝑘=1∆𝑥𝑖𝑛𝑘 is a marginal change in the kth independent variable for alternative i and individual n•

In fact, for linear-in-parameter models we need not calculate the utilities again

𝑃′ 𝑖

=

Ben-Akiva and Lerman (1985)

exp 𝑉𝑖𝑛 + ∆𝑉𝑖𝑛

σ𝑗∈𝐶 exp 𝑉𝑗𝑛 + ∆𝑉𝑗𝑛=

P(𝑖)exp ∆𝑉𝑖𝑛

σ𝑗∈𝐶 𝑃(𝑗) exp ∆𝑉𝑗𝑛

The 18th Behavior Modeling Summer School
Sep. 21 – 23 , 2019 @ The University of Tokyo

Validation practices in discrete choicemodeling

A credibility crisis in science and engineering?

Most published research findings are likely to be false due to factors such as lack of
power of the study, small effect sizes, and great flexibility in research design, definitions,outcomes and methods.
(Ioannidis, 2005)

A credibility crisis in science and engineering?
What about the transportation field?
In practice:
Demand forecasting is the “Achilles’ heel” of the transport planning model (Banister, 2002)
Demand overestimation:

30% Highway trips

35% Transit trips→(UK, 1962 – 1972)

Forecasts have not become more accurate (between 1969-1998). (Flyvbjerg, kamris Holm, & Buhl, 2005)

Unlike the natural sciences◼ Dependence on cross-section observational studies.
◼ Classic scientific hypothesis testing is more difficult.
◼ Underscores the need for proper validation practices.

“There is little tradition of confronting and confirming
predictions of cross-sectional models with outcomesin either back-casting or detailed before-and-afterstudies”(Boyce & William, 2015)→ While in practice, a feedback loop exists between forecast outputs and implementation results inthe form of measurable forecasting errors, in academia such feedback loop rarely exists.

Term definitions and research scope◼ Estimation: “the use of statistical analysis techniques and observed data to develop model parameters or
coefficients”◼ Calibration: “the adjustment of constants and other model parameters in estimated or asserted models in aneffort to make the models replicate observed data for a base (calibration) year or otherwise produce morereasonable results”◼ Validation: “the application of the calibrated models and comparison of the results against observed data”. This
comparison is done in terms of predictive ability.

Systemwide validation

Model level validation

More common in practiceMore common in research◼ Sensitivity analysis: At the individual model level, refers to the analysis of changes in outcomes given changes
in input variables such as elasticities, marginal effects, etc. At the system level, it refers to the application of amodel system using alternative input data or assumptions.

→

Scope is limited to discrete choice models in the peer-reviewed transportation literatureCambridge Systematics (2010)

A general overview of model validation methods
Estimation and calibration

𝒙𝒂 , 𝒚𝒂Model𝑦ො𝑎 = 𝑓(𝒙𝒂 )A general overview of model validation methods
Estimation and calibration

𝒙𝒂 , 𝒚𝒂Model𝑦ො𝑎 = 𝑓(𝒙𝒂 )A general overview of model validation methods
Validation

𝒙𝒃 , 𝒚𝒃Model𝑦ො𝑏1 = 𝑓(𝒙𝒃 )?
𝐿1 (𝑦𝑏 , 𝑦ො𝑏1 )?

?

𝐿2 (𝑦𝑏 , 𝑦ො𝑏2 )

𝐿3 (𝑦𝑏 , 𝑦ො𝑏3 )A general overview of model validation methods
Checking a model predictive ability◼ Validation with:◼ An Independent sample from the same population←

Ideal, but limited by practical considerations◼ A subset of the same sample (holdout, cross-validation)←

Reduces overfitting risks, but still tied to the same data

◼ Within sample predictive check (Information criteria, etc.)• Over-optimistic since it uses the same data
• Risk of overfitting• Asymptotic equivalence with cross validation relies onstronger distributional assumptions (Arlot & Celisse, 2009)

A general overview of model validation methods
Performance measures
Direct prediction accuracy
measures◼ Directly interpretable◼ Objective indicator of theprediction accuracy of a model

Error-based measures
Likelihood-based measures◼ Scores not directly interpretable◼ Only meaningful in relative terms◼ Useful for model selection

The best model among a set of models
can still be a very bad model

MeasurePredicted vs observed market outcomes
Percentage of correct predictions% clearly right (t)% clearly wrong (t)% unclear (t)Fitting factor
Maximum market share deviation
Correlation
Absolute percentage error
Sum of square error
Root sum of square error
Mean absolute error
Mean absolute percentage error
Mean squared error
Root mean square error
Brier Scoreχ2 testLog-likelihood
Mean log-likelihood loss
ρ2 , likelihood ratio test (LR), AIC

Abbrv.
PVO
FPR
％CR
％CW
%U
FF
MSD
Corr
APE
SSE
RSSE
MAE
MAPE
MSE
RMSE
BS
CHISQ
LL
MLLL
f(LL)

See Parady, Ory and Walker (2019) for specific details

Validation and reporting practices in the transportation literature
Using the Web of Science Core Collection maintained by Clarivate Analytics we reviewed validation
and reporting practices in the transportation literature from the last 5 years (2014 to 2018). Articles
were selected based on the following criteria:◼ Peer-reviewed journal articles published between 2014 and 2018
◼ Analysis uses discrete choice models◼ Target choice dimensions are:
Destination choice, Model choice and Route choice◼ Web of Science Database fields are:Transportation; transportation science and technology; economics; civil engineering◼ Research scope is limited to land transport and daily travel behavior (tourism, evacuation
behavior, etc. were excluded)◼ Articles use empirical data (Studies using numerical simulations only were excluded)◼ Methodological papers only included if the use empirical data

Validation and reporting practices in the transportation academic literature

282 articles reviewed
91% reported a goodness of fit statistics
66% reported a policy-related inference
Marginal effects, elasticities, odds ratios, value of time estimates,
marginal rates of substitution, and policy scenario simulations

17% reported a validation measure
Validation Method
Holdout validation
Repeated learning testing cross validation
Validation with independent sample from same population
Validation with post-intervention data
K-fold cross validation
Other**All indicators computed on calibration sample only

Abbrv.
HOV
RLT
ISV
PIDV
K-CV
O

Frequency
25
11
7
3
1
1

Percentage
52.1%
22.9%
14.6%
6.3%
2.1%
2.1%

Validation and reporting practices in the transportation academic literature
Evaluation measureLog-likelihood
Percentage of correct predictions or First Preference Recovery
Mean absolute error
Mean log-likelihood loss
Predicted vs observed market outcomes
Other functions of LL.ρ2 , AIC, likelihood ratio test (LR)% clearly right (t)Mean absolute percentage error (MAPE)
Root mean square error
Absolute percentage error
Chi-square
Sum of square error / Brier Score% clearly wrong (t)Mean squared error
Maximum market share deviation
Fitting factor
Correlation
Brier Score

Abbrv.
LL
FPR
MAE
MLLL
PVO
f(LL)
%CR
MAPE
RMSE
APE
CHISQ
SSE
%CW
MSE
MSD
FF
Corr
BS

Frequency
16
14
6
6
5
4
3
3
3
2
2
1
1
1
1
1
1
1

% Studies
33.3%
29.2%
12.5%
12.5%
10.4%
8.3%
6.3%
6.3%
6.3%
4.2%
4.2%
2.1%
2.1%
2.1%
2.1%
2.1%
2.1%
2.1%

*Note that some studies reported more than one measure

73% of studies reported at least one likelihood-based measure
25% of studies reported at least one of both
46% of studies reported at least one prediction accuracy measureValidation and reporting practices in the transportation academic literature

Is a randomized
controlled
trial possible?

Yes

Conduct arandomized
controlled trial

Report:1. Predicted vs observed
market shares (for route
choice models, a correlation
measure)2. An error-based performancemeasureNo
No
Is there an
independent
dataset available?

Yes

Conduct validation
with an independent
datasetIs validation data
in disaggregate
form?
Yes

No

Is the mode too
computationallyintensive?

No

Conduct crossvalidation

Yes

Conduct a holdout
validation

Report:1. Predicted vs observed
market shares (for route
choice models, a correlation
measure)2. Percentage of correct
predictions
3. A clearness of prediction
measure4. An error-based or
likelihood-based measure

Recommended validation practices
given available resourcesTowards better validation practices in the field◼ Make model validation mandatory:• Non-negotiable part of model reporting and peer-review in academic journals for any
study that provides policy recommendations.
• Cross-validation is the norm in machine learning studies.
◼ Share benchmark datasets:• A fundamental limitation in the field is the lack of benchmark datasets and a generalculture of sharing code and data.
◼ Incentivize validation studies:• Lot of emphasis on theoretically innovative models.
• Encourage submissions that focus on proper validation of existing models and theories.

◼ Draw and enforce clear reporting guidelines:• In addition to detailed information of survey characteristics such as sampling method,
discussion on representativeness of the data, validation reporting is required.
• Efforts to improve reporting are well documented in other fields(i.e. STROBE statement (von Elm et al., 2007))

Wait a minute…

“I’m not validating my model because I’m not trying to build a predictive framework. I’mtrying to learn about travel behavior”

The more orthodox the type of analysis conducted (such as the dimensions of travel
behavior covered in this study), the stronger the onus of validation.

Wait a minute…

“Does every study that uses a discrete choice model
should be conducting validation?”

In short, yes. At the very least, any article that makes policy recommendations should be
subject to proper validation given the dependence of the field on cross-section
observational studies, and the lack of a feedback loop in academia.

Wait a minute…

“Is what we learn about travel behavior fromcoefficient estimation less valuable if not conducted?”

There is a myriad of reasons why some skepticism is warranted against any particularmodel outcome. the most obvious one being model overfitting.

Finally

Better validation practices will not solve the credibility crisis in the field, but it’s a step inthe right direction.
Model validation is no solution to the causality problem in the field, but we want to underscore thatthe reliance on observational studies inherent to the field demands more stringent controls toimprove external validity of results.

The 18th Behavior Modeling Summer School
Sep. 21 – 23 , 2019 @ The University of Tokyo

References:1.

Ben-Akiva, M. E., Lerman, S. R. (1985). Discrete choice analysis: theory andapplication to travel demand. MIT press.

2.

Hensher, D. A., Rose, J. M., & Greene, W. H. (2015). Applied choice analysis: aprimer. Cambridge University Press. 2nd Edition.

3.

Parady G., Ory, D., Walker, J. (2019) “The overreliance on statistical goodness of fit
and under-reliance on empirical validation in discrete choice models: A review ofvalidation practices in the transportation academic literature” Presented at the 6thInternational choice modelling conference, Kobe, Japan, August 19-21, 2019.

Appendix: Definition of model validation performance measures reported in the literature
Type
Directpredictive
accuracymeasureMeasurePredicted vsobserved
outcomes

Percentage ofcorrect predictions
or First Preference
RecoveryAbbrv.

Equation

Notes

PVO

-

𝑁

FPR

𝑦𝑛𝑐 is the observed choice made by individual n, and 𝑦ො𝑛𝑐 is the choice with thehighest predicted probability.

100෍ 𝑦ො𝑛𝑐 = 𝑦𝑛𝑐
𝑁
𝑛=1

𝑁

% clearly right (t)Simple comparison of predicted and observed outcomes (i.e. market shares, trips
by mode, etc.). Usually in the form of a table or plot. No prediction accuracy
statistics are calculated.

％CR

100
෍ 𝑃෠ 𝑦𝑛𝑐 > 𝑡N
𝑛=1

𝑃෠ 𝑦𝑛𝑐 is the estimated choice probability of the chosen alternative.
𝑃෠ 𝑦𝑛!𝑐 is the estimated choice probability of an alternative other than thechosen one.

𝑁

% clearly wrong (t)

％CW

100
෍ 𝑃෠ 𝑦𝑛!𝑐 > 𝑡N𝑛=1% unclear (t)%U

100 – % clearly right (t) +% clearly wrong (t)𝑁

Fitting factorFF𝑃෠ 𝑦𝑛𝑐 is the estimated choice probability of the chosen alternative.

1
෍ 𝑃෠ 𝑦𝑛𝑐
N
𝑛=1

CorrelationCorr

corr(s, 𝑠)Ƹ

Correlation between predicted and observed outcomes. s is a continuous
aggregate outcome measure (i.e. train ridership, etc.)

Appendix: Definition of model validation performance measures reported in the literature
Type
Relative
predictive
accuracymeasures

MeasureAbsolute
percentage error
Sum of squareerror

Abbrv.

Equation

Notes

APE

100 ∙

𝑠𝑚
Ƹ − 𝑠𝑚𝑠𝑚

𝑀SSE෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2
𝑚=1

Root sum ofsquare error

𝑀

RSSE

෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2

M is the number of alternatives in the choice set.
𝑠𝑚 is an aggregate outcome measure, such as the market share of alternative
m (i.e. modal market share), choice frequency, etc.
𝑃෠ 𝑦𝑛𝑚 is the predicted probability that individual n chooses alternative m andynm is the actual outcome variable valued 0 or 1. In the particular case of abinary choice, the second summation sign disappears.

𝑚=1

Mean absolute
error

𝑀

1
෍ 𝑠Ƹ𝑚 − 𝑠𝑚
𝑀

MAE

𝑚=1

Mean absolute
percentage error

𝑀

MAPE

100
𝑠Ƹ𝑚 − 𝑠𝑚
෍
𝑀
𝑠𝑚
𝑚=1

Mean squared
error

𝑀

1
෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2
𝑀

MSE

𝑚=1

Root mean square
error

𝑀

1
෍ 𝑠𝑚
Ƹ − 𝑠𝑚 2
𝑀

RMSE

𝑚=1

𝑁

Brier Score

BS

𝑀

1
2
෍ ෍ 𝑃෠ 𝑦𝑛𝑚 − 𝑦𝑛𝑚N
𝑛=1 𝑚=1𝑀χ2 test

CHISQ

𝑓𝑚 − 𝐸(𝑓𝑚 ) 2
෍𝐸(𝑓𝑚 )

𝑚=1

𝑓𝑚 is the observed choice frequency of alternative m, and 𝐸(𝑓𝑚 ) is the expectedchoice frequency

Appendix: Definition of model validation performance measures reported in the literature
Type
Relative
predictive
accuracymeasures

MeasureAbbrv.

Maximum market
share deviation

MSD

Log-likelihood

LL

Equation
max 𝑆
𝑁

𝑀

෍ ෍ 𝑐𝑛𝑚 𝑙𝑜𝑔𝑃෠ 𝑦𝑛

Notes
S is the set of all market share deviations

𝑐𝑛𝑚 is a variable that takes value 1 if alternative m was chosen by individual n,
and 0 otherwise.

𝑛=1 𝑚=1

Mean log-likelihoodloss

ρ2 , likelihood ratio
test (LR), AIC

MLLL

1
1
෍−
𝐿𝐿
R
𝑉𝑆𝑟

Where LL is the log-likelihood, VS is the size of the validation (holdout) sample r,
and R is number of validation samples generated.

෡ −𝐾
𝐴𝐼𝐶 = 𝐿𝐿 𝜷෡
𝐿𝐿 𝜷
𝜌2 = 1 −𝐿𝐿 0
𝐴𝐼𝐶𝜌2 = 1 −𝐿𝐿 0෡
𝐿𝑅 = −2 𝐿𝐿 0 − 𝐿𝐿 𝜷

LL(0) is the log-likelihood when all parameters are zero. LL(β) is the maximized
likelihood.
K is the number of freely estimated parameters in the model.

𝑟f(LL)