The 19th Behavior Modeling Summer School
Sep. 12 – 15 , 2020 @ The University of Tokyo

Basic inference and validation in discrete choice modeling
Giancarlos Troncoso Parady – The University of Tokyo

Basic inference discrete choice modelingWhy is inference important ?
Variable name

CoefficientS.E.

t statistic

1.45

0.393

3.70

In-vehicle time (min)-0.00890.0063-1.42Out-of-vehicle time (min)-0.03080.0106-2.90

Auto out-of-pocket cost (c)-0.0115

0.0026-4.39

Transit fare

-0.00700.0038-1.87

Auto ownership (specific to auto mode)-0.770

0.213

3.16

Downtown workplace (specific to auto mode)-0.561

0.306

-1.84

Number of observations

1476

Number of cases

1476

LL(0)

-1023

LL(β)

-347.4-2[LL(0)-LL(β)]

1371

𝜌2

0.660

𝜌ҧ 2

0.654

Auto constant

Table adapted from Ben-Akiva and Lerman (1985)

Magnitudes are not directly interpretable.
We can only interpret the effect direction,or use them to calculate utilities,
and choice probabilities

To make some sense of theseparameters we must calculate
elasticities or marginal effects

Basic Inference in discrete choice models
MNL: Logit Elasticities (Point elasticities)•

Direct elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in an attribute of that same
alternative.

𝜕𝑃𝑛 (𝑖) 𝑥𝑖𝑛𝑘
𝑃 𝑖
𝐸𝑥𝑖𝑛𝑘 =∙=
𝜕𝑥𝑖𝑛𝑘 𝑃𝑛 (𝑖)•1 − 𝑃𝑛 𝑖 𝑥𝑖𝑛𝑘 𝛽𝑘

Cross-elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in a competing alternative.

𝜕𝑃𝑛 (𝑖) 𝑥𝑗𝑛𝑘
𝑃 𝑖
𝐸𝑥𝑗𝑛𝑘 =∙= −𝑃𝑛 𝑗 𝑥𝑗𝑛𝑘 𝛽𝑘𝜕𝑥𝑗𝑛𝑘 𝑃𝑛 (𝑖)Hensher, David A., John M. Rose, and William H. Greene (2015)

Because of IIA, crosselasticities are uniform
across all alternatives

Basic Inference in discrete choice models
MNL: Logit Elasticities (Point elasticities)•

The elasticities shown before are individual elasticities (Disaggregate)•

To calculate sample (aggregate) elasticities we use the probability weighted
sample enumeration method:
𝑃 𝑖
𝑁
෠
σ
𝑃
𝑖
𝐸
𝑛=1 𝑛
𝑥𝑖𝑛𝑘
𝑃(𝑖)
𝐸𝑥𝑖𝑛𝑘 =෠σ𝑁
𝑛=1 𝑃𝑛 𝑖𝑃(𝑖)𝐸𝑥𝑗𝑛𝑘 =

Sample direct elasticity

𝑃 𝑖
෠
σ𝑁
𝑃
𝑖
𝐸
𝑛=1 𝑛
𝑥𝑗𝑛𝑘

෠
σ𝑁
𝑛=1 𝑃𝑛 𝑖

Sample cross-elasticity

Where 𝑃(𝑖) is the aggregate choice probability of alternative I, and 𝑃෠𝑖𝑛 𝑖 is an estimated choice probability•

Uniform cross-elasticities do not necessarily hold at the aggregate level•

Also note that elasticities for dummy variables are meaningless!

Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
Relation between elasticity of demand, change in price and revenue

Px

Px

Perfectly
inelastic

X

Px

Relatively
inelastic

X

Px

Unit
elastic

X

Px

Relatively
elastic

X

Perfectly
elastic

X

Direct elasticity:
1% increase in X results in a
0% decrease in P(i)1% increase in X results in aless than 1% decrease in P(i)1% increase in X results in a
1% decrease in P(i)1% increase in X results in amore than 1% decrease in P(i)1% increase in X results
in a ∞ decrease in P(i)1% increase in X results in aless than 1% increase in P(j)1% increase in X results inno percent change in P(j)1% increase in X results in amore than 1% increase in P(j)1% increase in X results
in a ∞ increase in P(j)Cross elasticity:
1% increase in X results in a
0% increase in P(j)Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects•

Direct marginal effects: measures the change in the probability (absolute change) of choosing
a particular alternative in the choice set with respect to a unit change in an attribute of that same
alternative.

𝜕𝑃𝑛 (𝑖)
𝑃 𝑖
𝑀𝑥𝑖𝑛𝑘 == 𝑃𝑛 𝑖𝜕𝑥𝑖𝑛𝑘•

1 − 𝑃𝑛 𝑖 𝛽𝑘

Cross-marginal effects: measures the change in the probability (absolute change) of choosing
a particular alternative in the choice set with respect to a unit change in a competing alternative.

𝜕𝑃𝑛 (𝑖)𝑃 𝑖
𝑀𝑥𝑗𝑛𝑘 == 𝑃𝑛 𝑖𝜕𝑥𝑗𝑛𝑘

Hensher, Rose, and Greene (2015)−𝑃𝑛 𝑗 𝛽𝑘

Basic Inference in discrete choice models
MNL: Marginal Effects•

We can also calculate sample (aggregate) marginal effects we using e the probability weighted
sample enumeration method:
𝑃 𝑖
𝑁
෠
σ
𝑃
𝑖
𝑀
𝑛=1 𝑛
𝑥𝑖𝑛𝑘
𝑃(𝑖)
𝑀𝑥𝑖𝑛𝑘 =෠σ𝑁
𝑛=1 𝑃𝑛 𝑖
Sample direct marginal effect𝑃(𝑖)𝑀𝑥𝑗𝑛𝑘 =

𝑃 𝑖
෠
σ𝑁
𝑃
𝑖
𝑀
𝑛=1 𝑛
𝑥𝑗𝑛𝑘

෠
σ𝑁
𝑛=1 𝑃𝑛 𝑖

Sample cross-marginal effect

Where 𝑃(𝑖) is the aggregate choice probability of alternative I, and 𝑃෠𝑖𝑛 𝑖 is an estimated choice probability•

Marginal effects for dummy variables do make sense as we are talking about unit changes. But
you still need to frame the problem correctly.

Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects
P(i)𝜕𝑃𝑛 (𝑖)𝜕𝑥𝑖𝑛𝑘

xi
Marginal effects as the slopes of the Tangent lines to the cumulative probability curve
Hensher, David A., John M. Rose, and William H. Greene (2015)

Basic Inference in discrete choice models
Incremental Logit for prediction•

An alternative approach to using elasticities or marginal effects for prediction•

Prediction of changes in behavior based on existing choice probabilities

𝑃′ 𝑖

=

exp 𝑉𝑖𝑛 + ∆𝑉𝑖𝑛
σ𝑗∈𝐶 exp 𝑉𝑗𝑛 + ∆𝑉𝑗𝑛

𝐾;

𝑤ℎ𝑒𝑟𝑒 ∆𝑉𝑖𝑛 = ෍ 𝛽𝑘 ∆𝑥𝑖𝑛𝑘
𝑘=1∆𝑥𝑖𝑛𝑘 is a marginal change in the kth independent variable for alternative i and individual n•

In fact, for linear-in-parameter models we need not calculate the utilities again

𝑃′ 𝑖

=

Ben-Akiva and Lerman (1985)

exp 𝑉𝑖𝑛 + ∆𝑉𝑖𝑛

σ𝑗∈𝐶 exp 𝑉𝑗𝑛 + ∆𝑉𝑗𝑛=

P(𝑖)exp ∆𝑉𝑖𝑛

σ𝑗∈𝐶 𝑃(𝑗) exp ∆𝑉𝑗𝑛

Validation practices in discrete choicemodeling

A credibility crisis in science and engineering?

Most published research findings are likely to be false due to factors such as lack of
power of the study, small effect sizes, and great flexibility in research design, definitions,outcomes and methods.
(Ioannidis, 2005)

A credibility crisis in science and engineering?
What about the transportation field?
In practice:
Demand forecasting is the “Achilles’ heel” of the transport planning model (Banister, 2002)
Demand overestimation:

30% Highway trips

35% Transit trips→(UK, 1962 – 1972)

Forecasts have not become more accurate (between 1969-1998). (Flyvbjerg, kamris Holm, & Buhl, 2005)

Unlike the natural sciences◼ Dependence on cross-section observational studies.
◼ Classic scientific hypothesis testing is more difficult.
◼ Underscores the need for proper validation practices.

“There is little tradition of confronting and confirming
predictions of cross-sectional models with outcomesin either back-casting or detailed before-and-afterstudies”(Boyce & William, 2015)→ While in practice, a feedback loop exists between forecast outputs and implementation results inthe form of measurable forecasting errors, in academia such feedback loop rarely exists.

Term definitions and research scope▪

Predictive accuracy: the degree to which predicted outcomes match observed outcomes.
oCalibration*: the agreement between observed outcomes and predictionsodiscrimination: the ability of model predictions to discriminate between those with and those without anobserved outcome▪ Generalizability: the ability of a model or system of models to maintain its predictive accuracy in a different
sample.

o

Reproducibility: the ability of a model or system of models to maintain its predictive accuracy in different
samples from the same population.

oTransportability (transferability) the ability of a model or system of models to maintain its predictive

accuracy in samples from different but plausibly related populations or in samples collected with different
methodologies.

Term definitions and research scope•

Validation: the evaluation of the generalizability of a statistical model.
•Internal validation evaluates reproducibility•

External validation evaluates transferability

A general overview of model validation methods
Estimation

𝒙𝒂 , 𝒚𝒂Model𝑦ො𝑎 = 𝑓(𝒙𝒂 )A general overview of model validation methods
Validation

𝒙𝒃 , 𝒚𝒃Model𝑦ො𝑏1 = 𝑓(𝒙𝒃 )?
𝐿1 (𝑦𝑏 , 𝑦ො𝑏1 )?

?

𝐿2 (𝑦𝑏 , 𝑦ො𝑏2 )

𝐿3 (𝑦𝑏 , 𝑦ො𝑏3 )Model Validation
Internal
In-sample testingOut-of-sample testing: the data used for validation is not used in estimation

Data exclusion (holdout,
cross-validation) or
resampling (Bootstrapping)

Apparent predictive
accuracy:
Predictive accuracy of amodel estimated on thesame data used toestimate the model

External

Different
sample fromsame
population

Reproducibility:

Data fromdifferent time
periodData fromdifferentcity/region

Data gatheredwith different
methods

Temporaltransferability

Spatialtransferability

Methodologicaltransferability

The extent to which a model maintains its
predictive accuracy in different samples from thesame population.

Transferability:
The extent to which a model maintains its predictive
accuracy in a different but plausibly related populations orin data gathered with different methodologies.

Generalizability

A general overview of model validation methods(Some) Performance measures
Direct prediction accuracy
measures◼ Directly interpretable◼ Allows for comparison acrossmodels (to some extent)

Relative prediction accuracy
measures◼ Scores not directly interpretable◼ Only meaningful in relative terms◼ Useful for model selection

The best model among a set of models
can still be a very bad model

MeasurePredicted vs observed market outcomes
Percentage of correct predictions% clearly right (t)% clearly wrong (t)% unclear (t)Fitting factor
Correlation
ρ2
Brier Score
Absolute percentage error
Sum of square error
Root sum of square error
Mean absolute error
Mean absolute percentage error
Mean squared error
Root mean square error
χ2 test*Log-likelihood
Mean log-likelihood loss
likelihood ratio test (LR), AIC

Abbrv.
PVO
FPR
％CR
％CW
%U
FF
Corr
RHOSQ
BS
APE
SSE
RSSE
MAE
MAPE
MSE
RMSE
CHISQ
LL
MLLL
f(LL)

See Parady, Ory and Walker (2019) for specific details

Validation and reporting practices in the transportation literature
Using the Web of Science Core Collection maintained by Clarivate Analytics we reviewed validation
and reporting practices in the transportation literature from the last 5 years (2014 to 2018). Articles
were selected based on the following criteria:◼ Peer-reviewed journal articles published between 2014 and 2018
◼ Analysis uses discrete choice models◼ Target choice dimensions are:
Destination choice, Model choice and Route choice◼ Web of Science Database fields are:Transportation; transportation science and technology; economics; civil engineering◼ Research scope is limited to land transport and daily travel behavior (tourism, evacuation
behavior, etc. were excluded)◼ Articles use empirical data (Studies using numerical simulations only were excluded)◼ Methodological papers only included if the use empirical data
◼ SP only papers are excluded

Validation and reporting practices in the transportation academic literature

228 articles reviewed
92% reported a goodness of fit statistics
64% reported a policy-related inference
Marginal effects, elasticities, odds ratios, value of time estimates,
marginal rates of substitution, and policy scenario simulations

19% reported a validation measure
MethodAbbvr.

Percentage

Holdout validation

HOV

58.8%

Repeated learning-testing cross validation

RLT

26.5%

Validation against an independent sample

IS

8.8%

K-fold cross validation

K-CV

2.9%

Other sample splitting methods

SS-O

2.9%

Validation and reporting practices in the transportation academic literature
Performance measure

Abbrv.

Log-likelihood

LL

32.6%

Percentage of correct predictions or First Preference Recovery

FPR

23.3%

Predicted vs observed market outcomes

PVO

16.3%

Mean absolute error

MAE

14.0%

Mean log-likelihood loss

MLLL

14.0%

Root mean square error

RMSE

9.3%

Absolute percentage error

APE

4.7%

Mean absolute percentage error

MAPE

4.7%

Chi-square

CHISQ

4.7%

Rho-Square

RHOSQ

4.7%

Transfer index

TI

4.7%

% clearly right (t)

%CR

2.3%

Sum of square error

SSE

2.3%

Brier Score

BS

2.3%

Concordance index

C

2.3%

Correlation

Corr

2.3%

Transferability test statistic

TTS

2.3%

All others

Percentage

0.0%Validation and reporting practices in the transportation academic literature
Randomized
controlled
trial possible?

Yes

Conduct arandomized
controlled trial

Report:1. Predicted vs observed
market shares (for route
choice models,a correlation
performancemeasuremeasure)2. A relative performancemeasureNo
No
Independent
dataset fromplausibly related
population
available?

Yes

Conduct external
validation

Is validation data
in disaggregate
form?
Yes

No

Independent
dataset fromsame population
available?
No

Conduct internal
validation via
data/splitting or
bootstrapping

Yes

Conduct internal
validation withindependent sample

Report:1. Predicted vs observed
market shares (for route
choice models, a correlation
measure)2. Percentage of correct
predictions
3. A discrimination measure
4. Relative accuracy measure
5. Variability of performance
estimates.

Recommended validation practices
given available resourcesTowards better validation practices in the field◼ Make model validation mandatory:• Non-negotiable part of model reporting and peer-review in academic journals for any
study that provides policy recommendations.
• Cross-validation is the norm in machine learning studies.
◼ Share benchmark datasets:• A fundamental limitation in the field is the lack of benchmark datasets and a generalculture of sharing code and data.
◼ Incentivize validation studies:• Lot of emphasis on theoretically innovative models.
• Encourage submissions that focus on proper validation of existing models and theories.

◼ Draw and enforce clear reporting guidelines:• In addition to detailed information of survey characteristics such as sampling method,
discussion on representativeness of the data, validation reporting is required.
• Efforts to improve reporting are well documented in other fields(i.e. STROBE statement (von Elm et al., 2007))

Wait a minute…

“I’m not validating my model because I’m not trying to build a predictive
framework. I’m trying to learn about travel behavior”

The more orthodox the type of analysis conducted (such as the dimensions of travel
behavior covered in this study), the stronger the onus of validation.

Wait a minute…

“Does every study that uses a discrete choice model
should be conducting validation?”

In short, yes. At the very least, any article that makes policy recommendations should be
subject to proper validation given the dependence of the field on cross-section
observational studies, and the lack of a feedback loop in academia.

Wait a minute…

“Is what we learn about travel behavior fromcoefficient estimation less valuable if not conducted?”

There is a myriad of reasons why some skepticism is warranted against any particularmodel outcome. the most obvious one being model overfitting.

Finally

Better validation practices will not solve the credibility crisis in the field, but it’s a step inthe right direction.
Model validation is no solution to the causality problem in the field, but we want to underscore thatthe reliance on observational studies inherent to the field demands more stringent controls toimprove external validity of results.

References:1.

Ben-Akiva, M. E., Lerman, S. R. (1985). Discrete choice analysis: theory andapplication to travel demand. MIT press.

2.

Hensher, D. A., Rose, J. M., & Greene, W. H. (2015). Applied choice analysis: aprimer. Cambridge University Press. 2nd Edition.

3.

Parady G., Ory, D., Walker, J. (2019) “The overreliance on statistical goodness of fit
and under-reliance on empirical validation in discrete choice models: A review ofvalidation practices in the transportation academic literature” Presented at the 6thInternational choice modelling conference, Kobe, Japan, August 19-21, 2019.

Appendix: Definition of model validation performance measures reported in the literature
Type
Directpredictive
accuracymeasureMeasurePredicted vsobserved
outcomes
Percentage ofcorrect predictions
or First Preference
Recovery% clearly right (t)Abbrv.

Equation

Notes

PVO

Simple comparison of predicted and observed outcomes (i.e. market shares, trips
by mode, etc.). Usually in the form of a table or plot. No prediction accuracy
statistics are calculated.

-

𝑦𝑛𝑐 is the observed choice made by individual n, and 𝑦ො𝑛𝑐 is the choice with thehighest predicted probability.

𝑁100
෍ 𝑦ො𝑛𝑐 = 𝑦𝑛𝑐
𝑁

FPR

𝑛=1
𝑁

𝑃෠ 𝑦𝑛𝑐 is the estimated choice probability of the chosen alternative.

𝑛=1

𝑃෠ 𝑦𝑛!𝑐 is the estimated choice probability of an alternative other than thechosen one.

100
෍ 𝑃෠ 𝑦𝑛𝑐 > 𝑡
N

％CR

𝑁

% clearly wrong (t)

％CW

100
෍ 𝑃෠ 𝑦𝑛!𝑐 > 𝑡N𝑛=1% unclear (t)%U

100 – (% clearly right (t) +% clearly wrong (t))
𝑃෠ 𝑦𝑛𝑐 is the estimated choice probability of the chosen alternative.

𝑁

Fitting factor1
෍ 𝑃෠ 𝑦𝑛𝑐
N

FF

𝑛=1

CorrelationCorr

corr(s, 𝑠)Ƹ

BS12෍ ෍ 𝑃෠ 𝑦𝑛𝑚 − 𝑦𝑛𝑚
N𝑁

Brier Score

𝑀

Correlation between predicted and observed outcomes. s is a continuous
aggregate outcome measure (i.e. train ridership, etc.)𝑃෠ 𝑦𝑛𝑚 is the predicted probability that individual n chooses alternative m and ynm
is the actual outcome variable valued 0 or 1. In the particular case of a binary
choice, the second summation sign disappears.

𝑛=1 𝑚=1

ρ2 , 𝜌ҧ 2

RHOSQ

෡
𝐿𝐿 𝜷

𝐴𝐼𝐶

𝜌2 = 1 − 𝐿𝐿 0 ; 𝜌ҧ 2 = 1 − 𝐿𝐿 0

LL(0) is the log-likelihood when all parameters are zero. LL(β) is the maximized
likelihood.
K is the number of freely estimated parameters in the model.

Appendix: Definition of model validation performance measures reported in the literature
Type
Relative
predictive
accuracymeasures

MeasureAbsolute
percentage error
Sum of squareerror

Abbrv.

Equation

Notes

APE

100 ∙

𝑠𝑚
Ƹ − 𝑠𝑚𝑠𝑚

𝑀SSE෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2
𝑚=1

Root sum ofsquare error

𝑀

RSSE

෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2

M is the number of alternatives in the choice set.
𝑠𝑚 is an aggregate outcome measure, such as the market share of alternative
m (i.e. modal market share), choice frequency, etc.
𝑃෠ 𝑦𝑛𝑚 is the predicted probability that individual n chooses alternative m andynm is the actual outcome variable valued 0 or 1. In the particular case of abinary choice, the second summation sign disappears.

𝑚=1

Mean absolute
error

𝑀

1
෍ 𝑠Ƹ𝑚 − 𝑠𝑚
𝑀

MAE

𝑚=1

Mean absolute
percentage error

𝑀

MAPE

100
𝑠Ƹ𝑚 − 𝑠𝑚
෍
𝑀
𝑠𝑚
𝑚=1

Mean squared
error

𝑀

MSE

1
෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2𝑀
𝑚=1

Root mean square
error

𝑀

1
෍ 𝑠Ƹ𝑚 − 𝑠𝑚 2
𝑀

RMSE

𝑚=1

𝑀

2

χ testCHISQ*

𝑓𝑚 − 𝐸(𝑓𝑚 ) 2
෍𝐸(𝑓𝑚 )

𝑚=1

𝑓𝑚 is the observed choice frequency of alternative m, and 𝐸(𝑓𝑚 ) is the expectedchoice frequency

Appendix: Definition of model validation performance measures reported in the literature
Type
Relative
predictive
accuracymeasures

MeasureMaximum market
share deviation

Abbrv.

Equation
max 𝑆

MSD
𝑁

Log-likelihood

LL𝑀෍ ෍ 𝑐𝑛𝑚 𝑙𝑜𝑔𝑃෠ 𝑦𝑛

Notes
S is the set of all market share deviations

𝑐𝑛𝑚 is a variable that takes value 1 if alternative m was chosen by individual n,
and 0 otherwise.

𝑛=1 𝑚=1

Mean log-likelihood
loss

MLLL

likelihood ratio test
(LR), AIC

f(LL)

1
1
෍−
𝐿𝐿
R
𝑉𝑆𝑟

Where LL is the log-likelihood, VS is the size of the validation (holdout) sample r,
and R is number of validation samples generated.

෡ −𝐾
𝐴𝐼𝐶 = 𝐿𝐿 𝜷

LL(0) is the log-likelihood when all parameters are zero. LL(β) is the maximized
likelihood.
K is the number of freely estimated parameters in the model.

𝑟෡
𝐿𝑅 = −2 𝐿𝐿 0 − 𝐿𝐿 𝜷