The 21st Behavior Modeling Summer School

Sep. 23 – 25 , 2022 @ The University of Tokyo

Size Matters:

The Use and Misuse of Statistical Significance in Discrete
Choice Models
Giancarlos Parady – The University of Tokyo

Size Matters:

The Use and Misuse of Statistical Significance in Discrete
Choice Models

Basic inference with discrete choicemodelsWhy is inference important ?
Variable name

CoefficientS.E.

t statistic

1.45

0.393

3.70

In-vehicle time (min)-0.00890.0063-1.42Out-of-vehicle time (min)-0.03080.0106-2.90

Auto out-of-pocket cost (c)-0.0115

0.0026-4.39

Transit fare

-0.00700.0038-1.87

Auto ownership (specific to auto mode)-0.770

0.213

3.16

Downtown workplace (specific to auto mode)-0.561

0.306

-1.84

Number of observations

1476

Number of cases

1476

LL(0)

-1023

LL(β)

-347.4-2[LL(0)-LL(β)]

1371

𝜌𝜌2

0.660

Auto constant𝜌𝜌̅ 2

Table adapted from Ben-Akiva and Lerman (1985)

0.654

Coefficients are not directly interpretable.
We can only interpret the effect direction,or use them to calculate utilities,
and choice probabilities

To make some sense of theseparameters we must calculate
elasticities, marginal effects orother quantity of interest suchas marginal rates of substitution(i.e. VoTT)Basic Inference with discrete choice models
MNL: Logit Elasticities (Point elasticities)• Direct elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in an attribute of that same
alternative.

𝜕𝜕𝑃𝑃𝑛𝑛 (𝑖𝑖) 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖
𝑃𝑃 𝑖𝑖=
𝐸𝐸𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖 =�𝜕𝜕𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖 𝑃𝑃𝑛𝑛 (𝑖𝑖)

1 − 𝑃𝑃𝑛𝑛 𝑖𝑖 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖 𝛽𝛽𝑘𝑘

• Cross elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in a competing alternative.

𝜕𝜕𝑃𝑃𝑛𝑛 (𝑖𝑖) 𝑥𝑥𝑗𝑗𝑛𝑛𝑛𝑛
𝑃𝑃 𝑖𝑖
𝐸𝐸𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 =�= −𝑃𝑃𝑛𝑛 𝑗𝑗 𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 𝛽𝛽𝑘𝑘
𝜕𝜕𝑥𝑥𝑗𝑗𝑛𝑛𝑛𝑛 𝑃𝑃𝑛𝑛 (𝑖𝑖)

Because of IIA, crosselasticities are uniform
across all alternatives

Basic Inference with discrete choice models
MNL: Logit Elasticities (Point elasticities)•

The elasticities shown before are individual elasticities (Disaggregate)•

To calculate sample (aggregate) elasticities we use the probability weighted
sample enumeration method:
𝑃𝑃 𝑖𝑖
𝑁𝑁
�
∑
𝑃𝑃
𝑖𝑖
𝐸𝐸
𝑛𝑛=1 𝑛𝑛𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖
𝑃𝑃(𝑖𝑖)
𝐸𝐸𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖 =�∑𝑁𝑁
𝑛𝑛=1 𝑃𝑃𝑛𝑛 𝑖𝑖
Sample direct elasticity

𝑃𝑃(𝑖𝑖)

𝐸𝐸𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 =

𝑃𝑃 𝑖𝑖
�
∑𝑁𝑁
𝑃𝑃
𝑖𝑖
𝐸𝐸
𝑛𝑛=1 𝑛𝑛
𝑥𝑥𝑗𝑗𝑛𝑛𝑛𝑛

�∑𝑁𝑁
𝑛𝑛=1 𝑃𝑃𝑛𝑛 𝑖𝑖

Sample cross-elasticity

Where 𝑃𝑃(𝑖𝑖) is the aggregate choice probability of alternative I, and 𝑃𝑃�𝑖𝑖𝑖𝑖 𝑖𝑖 is an estimated choice probability•

Uniform cross-elasticities do not necessarily hold at the aggregate level•

Also note that elasticities for dummy variables are meaningless!

Basic Inference with discrete choice models
NL: Logit Elasticities (Point elasticities)
𝑒𝑒 𝑉𝑉𝑗𝑗 /𝜏𝜏𝑒𝑒 𝜏𝜏𝐼𝐼𝐼𝐼(𝑖𝑖)𝑃𝑃 𝑗𝑗 = 𝐼𝐼𝐼𝐼(𝑖𝑖) � 𝐼𝐼
∑𝑖𝑖=1 𝑒𝑒 𝜏𝜏𝐼𝐼𝐼𝐼(𝑖𝑖)𝑒𝑒

Direct Elasticity← NL RUM2 specification

When alternative 𝑗𝑗 does not belong to any
nest

When alternative 𝑗𝑗 belongs to nest 𝑖𝑖

𝑃𝑃 𝑗𝑗
𝐸𝐸𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 =𝑃𝑃 𝑗𝑗
𝐸𝐸𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 =1 − 𝑃𝑃𝑛𝑛 𝑗𝑗 𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 𝛽𝛽𝑘𝑘

Cross Elasticity

When alternatives 𝑗𝑗 and 𝑗𝑗𝑗 belong todifferent nests

𝑃𝑃 𝑗𝑗
𝐸𝐸𝑥𝑥 ′ = −𝑃𝑃𝑛𝑛 𝑗𝑗 ′ 𝑥𝑥𝑗𝑗 ′𝑛𝑛𝑛𝑛 𝛽𝛽𝑘𝑘
𝑗𝑗 𝑛𝑛𝑛𝑛

1 − 𝑃𝑃𝑛𝑛 𝑗𝑗

1
+
−1
𝜏𝜏

1 − 𝑃𝑃 𝑗𝑗 𝑖𝑖

𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 𝛽𝛽𝑘𝑘

When alternatives 𝑗𝑗 and 𝑗𝑗𝑗 belong to thesame nest

𝑃𝑃 𝑗𝑗
𝐸𝐸𝑥𝑥 ′ = −𝑗𝑗 𝑛𝑛𝑛𝑛

𝑃𝑃𝑛𝑛

𝑗𝑗 ′

1
+
− 1 𝑃𝑃 𝑗𝑗 ′ 𝑖𝑖𝜏𝜏

𝑥𝑥𝑗𝑗 ′𝑛𝑛𝑛𝑛 𝛽𝛽𝑘𝑘

Basic Inference with discrete choice models
Graphical illustration of elasticities
Let 𝑥𝑥𝑖𝑖 be the price of alternative 𝑖𝑖
𝑥𝑥𝑖𝑖

𝑥𝑥𝑖𝑖

+1%

𝑥𝑥𝑖𝑖

+1%

𝑥𝑥𝑖𝑖

+1%

𝑥𝑥𝑖𝑖

+1%

+1%

-1%

Perfectly
inelastic

𝑃𝑃(𝑖𝑖)

Direct elasticity:

1% increase in 𝑥𝑥𝑖𝑖 results
in a 0% change in 𝑃𝑃(𝑖𝑖)Cross elasticity:

1% increase in 𝑥𝑥𝑗𝑗 results
in a 0% change in 𝑃𝑃(𝑖𝑖)Relatively
inelastic

𝑃𝑃(𝑖𝑖)

Unit
elastic

𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑖𝑖 results in 1% increase in 𝑥𝑥𝑖𝑖 results ina less than 1% decrease in a 1% decrease in 𝑃𝑃(𝑖𝑖)
𝑃𝑃(𝑖𝑖)
1% increase in 𝑥𝑥𝑗𝑗 results ina less than 1% increase in𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑗𝑗 results inno percent change in 𝑃𝑃(𝑖𝑖)Relatively
elastic

𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑖𝑖 results ina more than 1% decrease in𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑗𝑗 results in amore than 1% increase in𝑃𝑃(𝑖𝑖)

Perfectly
elastic

𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑖𝑖 results
in a ∞ percent decreasein 𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑗𝑗 results
in a ∞ percent increase in𝑃𝑃(𝑖𝑖)

Basic Inference with discrete choice models
Graphical illustration of elasticities
Let 𝑥𝑥𝑖𝑖 be the price of alternative 𝑖𝑖
𝑥𝑥𝑗𝑗

𝑥𝑥𝑗𝑗

+1%

𝑥𝑥𝑗𝑗

+1%

𝑥𝑥𝑗𝑗

+1%

𝑥𝑥𝑗𝑗

+1%

+1%

+1%

Perfectly
inelastic

𝑃𝑃(𝑖𝑖)

Direct elasticity:

1% increase in 𝑥𝑥𝑖𝑖 results
in a 0% change in 𝑃𝑃(𝑖𝑖)Cross elasticity:

1% increase in 𝑥𝑥𝑗𝑗 results
in a 0% change in 𝑃𝑃(𝑖𝑖)Relatively
inelastic

𝑃𝑃(𝑖𝑖)

Unit
elastic

𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑖𝑖 results in 1% increase in 𝑥𝑥𝑖𝑖 results ina less than 1% decrease in a 1% decrease in 𝑃𝑃(𝑖𝑖)
𝑃𝑃(𝑖𝑖)
1% increase in 𝑥𝑥𝑗𝑗 results ina less than 1% increase in𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑗𝑗 results inno percent change in 𝑃𝑃(𝑖𝑖)Relatively
elastic

𝑃𝑃(i)

1% increase in 𝑥𝑥𝑖𝑖 results ina more than 1% decrease in𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑗𝑗 results in amore than 1% increase in𝑃𝑃(𝑖𝑖)

Perfectly
elastic

𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑖𝑖 results
in a ∞ percent decreasein 𝑃𝑃(𝑖𝑖)

1% increase in 𝑥𝑥𝑗𝑗 results
in a ∞ percent increase in𝑃𝑃(𝑖𝑖)

Basic Inference with discrete choice models
MNL: Marginal Effects••

Direct marginal effect: measures the change in the probability (absolute change) of choosing
a particular alternative in the choice set with respect to a unit change in an attribute of that same
alternative.

𝜕𝜕𝑃𝑃𝑛𝑛 (𝑖𝑖)
𝑃𝑃 𝑖𝑖
𝑀𝑀𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖 == 𝑃𝑃𝑛𝑛 𝑖𝑖
𝜕𝜕𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖

1 − 𝑃𝑃𝑛𝑛 𝑖𝑖 𝛽𝛽𝑘𝑘

𝜕𝜕𝑃𝑃𝑛𝑛 (𝑖𝑖)𝑃𝑃 𝑖𝑖
𝑀𝑀𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 == 𝑃𝑃𝑛𝑛 𝑖𝑖
𝜕𝜕𝑥𝑥𝑗𝑗𝑛𝑛𝑛𝑛−𝑃𝑃𝑛𝑛 𝑗𝑗 𝛽𝛽𝑘𝑘

Cross marginal effect: measures the change in the probability (absolute change) of choosing
a particular alternative in the choice set with respect to a unit change in a competing alternative.

Basic Inference with discrete choice models
MNL: Marginal Effects•

We can also calculate sample (aggregate) marginal effects using the probability weighted
sample enumeration method:
𝑃𝑃 𝑖𝑖
𝑁𝑁
�
∑
𝑃𝑃
𝑖𝑖
𝑀𝑀
𝑛𝑛=1 𝑛𝑛𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖
𝑃𝑃(𝑖𝑖)
𝑀𝑀𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖 =�∑𝑁𝑁
𝑛𝑛=1 𝑃𝑃𝑛𝑛 𝑖𝑖
Sample direct marginal effect𝑃𝑃(𝑖𝑖)
𝑀𝑀𝑥𝑥𝑗𝑗𝑗𝑗𝑗𝑗 =

𝑃𝑃 𝑖𝑖
�
∑𝑁𝑁
𝑖𝑖
𝑀𝑀
𝑃𝑃
𝑛𝑛=1 𝑛𝑛
𝑥𝑥𝑗𝑗𝑛𝑛𝑛𝑛

�∑𝑁𝑁
𝑛𝑛=1 𝑃𝑃𝑛𝑛 𝑖𝑖

Sample cross-marginal effect

Where 𝑃𝑃(𝑖𝑖) is the aggregate choice probability of alternative I, and 𝑃𝑃�𝑖𝑖𝑖𝑖 𝑖𝑖 is an estimated choice probability•

Marginal effects for dummy variables do make sense as we are talking about unit changes, but adifferent procedure is necessary to estimate marginal effects.

Basic Inference with discrete choice models
MNL: Marginal Effects
𝑃𝑃(𝑖𝑖)
𝜕𝜕𝑃𝑃𝑛𝑛 (𝑖𝑖)𝜕𝜕𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖

𝑥𝑥𝑖𝑖

Marginal effects as the slopes of the Tangent lines to the cumulative probability curveBasic Inference with discrete choice models
MNL: Marginal Effects
Calculating marginal effects for dummy variables
Calculated via simulation:
1.

Set the values of the variable of interest to 0

2.

Estimate base predictions (at the individual level)3.

Set the values of the variable of interest to 1

4.

Estimate new predictions (at the individual level)5.

Calculate marginal effects by taking the mean of the difference in individual predictions

Basic Inference with discrete choice models

Rather obvious…
Is it though?

Size Matters:

The Use and Misuse of Statistical Significance in Discrete
Choice Models

The problem: Quantitative researchers areobsessed with statistical significance.

Where does the field stand regarding the use andmisuse of statistical significance in empirical analysis?

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature• Follow-up study to the work of Parady et al., (2021) that showed that while
92% of studies reported goodness-of-fit statistics, only 18.1% reported
validation.
• Based on the seminal work of McCloskey and Ziliak (1996) in economics.
• We adapt McCloskey and Ziliak (1996)’s 19 questions to the academictransportation literature to evaluate where the field stands regarding the use
and misuse of statistical significance in empirical analyses

Does the article...

(Think about the last article you wrote)

Out of which:% Yes Comprehe
Limitedly
nsively

0.00
-

-

-

Q15: Report effect confidence intervals, using them to interpret economic significance not merely as areplacement for pointwise statistical significance?

7.37

0

7.37

Q10: Discuss the scientific conversation within which an effect or other quantity of interest would be judged large
or small?

13.68

-

-

Q12: Do a simulation to determine whether the estimated effects or other quantities of interest are reasonable
and/or to better illustrate the magnitude of estimated effects?

29.47

-

-

Q13: In the conclusions and implications sections, keep statistical significance separate from economic policy
and scientific significance?

32.63

-

-

Q9: Make a judgement on effect magnitudes?
Q14: In the estimation, conclusions, and implication sections, avoid using the word "significance" inambiguous ways?
Q7: In the model results section, eschew "sign econometrics"?

36.84

13.68

23.16

37.63

-

-

60.64

27.66

32.98

Q8: Discuss the magnitude of estimated effects or other quantities of interest?

64.21

33.68

30.53

Q2: Use coefficients to calculate elasticities, or some other quantity that addresses the question of "how large
is large"?

65.26

45.26

20.00

Q11: Avoid choosing variables for inclusion solely on the basis of statistical significance?

75.53

-

-

Q3: Report all traditionally reported statistics?
Q1: Report descriptive statistics for model variables?
Q6: Eschew "asterisk econometrics"?

76.84
78.95
100.00

65.26
-

13.68
-

Q4: Consider the power of the test?
Q5: Examine the power function?

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature
67% of reviewed studies did not distinguish statistical significance from economic, policy orscientific significance.

86% of studies did not discuss the scientific conversation within which the magnitude of acoefficient can be judged to be “large” or “small.”

62% of studies ambiguously used the word “significant” to mean statistically different from the null

sometimes and to mean practically important at other times.

39% explained model results exclusively based on the sign of the coefficient.
24% explicitly stated to have used statistical significance as an exclusive criterion to drop variablesfrom a model.

0% of the reviewed studies considered the statistical power of the tests.

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature
DO NOT!

Discuss your findings only in terms of significance and sign (sign econometrics)

Statistical significance is just a measureof ONE kind of error, but it does not tell
you anything about whether that effect ispractically important

Plus or minus, who cares? Without ameasure of size, this information isuseless!

“Variable X has a significant and positive effect on…”

With a large enough sample,
EVERYTHING is significant
Statistical significance asks whether an
effect exists, not how big is it.

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature
Good Practices: Discussing effect magnitude (not significance!) and making judgement about
how large an effect isKhan, Kockelman and Xiong (2014) make clear judgements of magnitude when they state that “network
connectivity (measured as 4-way intersections within 0.5 mile) plays a major role: a single standard
deviation change in this variable is estimated to increase walking probability by 34%” and go on to statethat “parking prices and free-parking availability variables were not found to have much of an effect.”de Luca and Di Pace (2015) also make clear judgments of magnitude when they discuss the magnitude
of value of travel time estimates and state that “aside from being similar to those estimated in different
Italian case studies, [the magnitude] indicates the extreme importance of parking location. Assumingthat the average one-way travel monetary cost is equal to 3 €, 10 min walking time (about 700 m at 4km/h) is more than half of the whole travel monetary cost.”

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature
Other common mistakes: Confusing statistical significance with practical importanceKamargianni et al. (2014) state of a latent construct of walking preference that “this component is themost statistically significant variable…indicating the strong influence that parents have on thedevelopment of their children’s attitudes towards walking”

Qin et al. (2017) argue in a study of mode-shifting behavior that “bus service level has the most significant
positive t-value, which indicates that improving the bus service level can increase the shifting proportionof car travelers to bus significantly.”

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature
Recommendations:
Always report of effect magnitudes and their confidence intervals (it should be mandatory).
Statistical significance should not be more than one of many criteria of evaluation, but it should certainly
not be the most important one. The discussion of statistical models should focus on effect magnitude andother policy relevant quantities. Is it large enough to matter for policy?
Provide to the extent possible judgements of magnitude that convey what the authors consider
are “small,” “medium,” or “large” effects (or other quantities of interest) and the basis for suchjudgement.
This is certainly not an easy task, there is a discussion to be had regarding what effects or quantities arepolicy relevant and how to assess such relevance. Furthermore, such discussions should ideally be
accompanied by a discussion on the cost implications of changing the policy variables in question.

The Use and Misuse of Statistical Significance in Discrete Choice Models
A review of reporting practices in the Transportation Academic Literature
Recommendations:Compare, whenever possible, effect magnitudes or other quantities of interest to existing studies.
For the most regularly reported values, such as value of travel time, there is a myriad of studies reporting
such values for many contexts, so there are no reasons why such comparisons cannot be made. For lessoften reported values, there will be certainly times when such a task will be difficult, but if we all do it intime, proper discussion of scientific context should be widespread.

For new studies, take statistical power into consideration when defining sample size to guarantee
the effects the research wants to detect can in fact be detected with enough power. For studies
using secondary data (i.e., national household survey data, etc.) report post-hoc power levels of
tests reported in the study.

The 21st Behavior Modeling Summer School

Sep. 23 – 25 , 2022 @ The University of Tokyo

Size Matters:

The Use and Misuse of Statistical Significance in Discrete
Choice Models
But it is
So what!?
significant!

To summarize…

Read the full paper here: Parady, G., Axhausen K.W.: Size Matters: The Useand Misuse of Statistical Significance in Discrete Choice Models in theTransportation Academic Literature

For self-study:

Validation practices in discrete choicemodeling

A credibility crisis in science and engineering?

Baker and Penny (2016)

A credibility crisis in science and engineering?
Most published research findings are likely to be false due to factors such as lack of
power of the study, small effect sizes, and great flexibility in research design, definitions,outcomes and methods.
Focused on experimental studies(Ioannidis, 2005)

In the transportation field

Unlike the natural sciences Dependence on cross-section observational studies
 Classic scientific hypothesis testing is more difficult
 Impact evaluation of policies drawn based on model-based academic research is rarely conducted No feedback in terms of how right or how wrong are these models and the policy recommendations derived from them These issues underscore the need for proper validation practices

Term definitions
Predictive accuracy: The degree to which predicted outcomes match observed outcomes.
Predictive accuracy is a function of：•

Calibration: The degree to which predicted probabilities match the relative frequency of observed outcomes.

•

Discrimination ability: The ability of a model or system of models to discriminate between those instances withand without a particular outcome.

Generalizability: The ability of a model, or system of models to maintain its predictive accuracy
in a different sample.
Generalizability of a model is a function of：•Reproducibility: The extent to which a model or system of models maintains its predictive ability in different
samples from the same population.

•

Transferability: The extent to which a model or system of models maintains its predictive ability in samples fromdifferent but plausibly related populations or in samples collected with different methodologies (sometimes called
transportability)

Parady, Ory & Walker (2021)

Term definitions
Model validation: The evaluation of the generalizability of a statistical model.
Types of model validation：••

Internal validation: The evaluation of the reproducibility of a model.
•

Data splitting (i.e., cross-validation), resampling methods (i.e., bootstrapping)•Different sample from the same population

External validation: The evaluation of the transferability of a model.
•

Temporal transferability•

Spatial transferability•

Methodological transferability

Parady, Ory & Walker (2021)

Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Holdout validation: Dataset is randomly split into an estimation dataset and a validation dataset.
Estimation data

Validation data

For illustration purposes, let us define Q 𝑦𝑦𝑛𝑛 , 𝑦𝑦�n as a measure of prediction correctness forthe 𝑛𝑛th instance, for the binary choice case as:
0 𝑖𝑖𝑖𝑖 𝑦𝑦𝑛𝑛 = 𝑦𝑦�n
Q 𝑦𝑦𝑛𝑛 , 𝑦𝑦�n = �
1 𝑖𝑖𝑖𝑖 𝑦𝑦𝑛𝑛 ≠ 𝑦𝑦�nwhere is 𝑦𝑦𝑛𝑛 the observed outcome, and 𝑦𝑦�n is the predicted outcome for instance n.

The holdout estimator is

𝑁𝑁𝑣𝑣

1
𝐻𝐻𝑂𝑂𝑂𝑂 =� 𝑄𝑄[𝑦𝑦𝑛𝑛𝑣𝑣 , 𝑦𝑦�𝑛𝑛𝑒𝑒𝑣𝑣 ]𝑁𝑁𝑣𝑣
𝑛𝑛𝑣𝑣 =1

where 𝑦𝑦�𝑛𝑛𝑒𝑒𝑣𝑣 is the predicted outcome for instance 𝑛𝑛 in sample v, using the model estimated with sample 𝑒𝑒, and 𝑁𝑁𝑣𝑣 is thevalidation sample size.
Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Cross-validation: When the holdout process is repeated multiple times, thus generating a set of
randomly split estimation-validation data pairs, we refer to the validation procedure as crossvalidation (CV).
𝐶𝐶𝐶𝐶 =

1
� 𝐻𝐻𝐻𝐻𝑉𝑉𝑏𝑏
B 𝑏𝑏

where B is the number of estimation-validation data pairs generated and is the holdout estimator for set b.

𝑏𝑏 = 1𝑏𝑏 = 2

Validation data
Estimation data

𝑏𝑏 = 3𝑏𝑏 = 4𝑏𝑏 = 5
Parady, Ory & Walker (2021)

Estimation data
Validation data

Estimation data

Estimation data
推定データ
Validation
data
Estimation data

Estimation data
Validation data

Estimation data
A 5-fold cross validation illustration

Estimation data
Validation dataA brief introduction to internal validation (data splitting methods)
Cross-validation：Commonly used methods
𝐶𝐶𝐶𝐶 =

1
� 𝐻𝐻𝐻𝐻𝑉𝑉𝑏𝑏
B 𝑏𝑏• Cross-validation methods differ from one another in the way the data is split.
• When the data splitting considers all possible estimation sets of size , the splitting is
exhaustive, otherwise the splitting is partial. (Arlot and Celisse, 2009)。
Exhaustive splitting methods• Leave-one-out : estimation set size is Ne = 𝑁𝑁 − 1, and B = N. The model is fitted leaving out one

instance per iteration, and the outcome of that single instance is predicted based on the estimatedmodel.

• Leave-p-out : Ne = 𝑁𝑁 − 𝑝𝑝. The model is fitted leaving out p-instances per iteration, and theoutcome of the remaining instances is predicted based on the estimated model.

Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Cross-validation：Commonly used methods
1
𝐶𝐶𝐶𝐶 = � 𝐻𝐻𝐻𝐻𝑉𝑉𝑏𝑏
B 𝑏𝑏• Cross-validation methods differ from one another in the way the data is split.
• When the data splitting considers all possible estimation sets of size , the splitting is
exhaustive, otherwise the splitting is partial. (Arlot and Celisse, 2009)。
Partial splitting methods (lower calculation cost)• K-fold cross-validation: data is partitioned into K mutually-exclusive subsets of roughly equal
size, and B=K.
• Repeated learning-testing: a B number of randomly-split estimation-validation pairs are
generated. This method is also called repeated holdout validation.
Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Performance measures
Market share comparison• Easy to execute• Does not provide a quantitative
measure to evaluate the level ofagreement between predictions
and observations

Hasnine and Habib (2018)

A brief introduction to internal validation (data splitting methods)
Performance measures
Percentage of correct predictions: the alternative with the highest probability is defined as thepredicted choice. However,Model A：

Model B：

Model C：

• Alt. A: 0.34＊• Alt. A: 0.50＊• Alt. A: 0.90＊• Alt. B: 0.33

• Alt. B: 0.30

• Alt. B: 0.05

• Alt. C: 0.33

• Alt. C: 0.20

• Alt. C: 0.05

＊Observed choice

Cannot discriminate differences in estimated probabilities.
A measures that accounts for “clearness” of prediction is necessary.

A brief introduction to internal validation (data splitting methods)
Performance measures
Clearness of prediction:
Percentage of clearly right choices: “the percentage of users in the sample whose observed
choices are given a probability greater than threshold t by the model”
100 𝑁𝑁𝑣𝑣∑𝑛𝑛𝑣𝑣=1 𝐶𝐶𝑅𝑅𝑛𝑛𝑣𝑣
％CR =
𝑁𝑁𝑣𝑣

𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤

1 𝑖𝑖𝑖𝑖 𝑃𝑃� 𝑦𝑦𝑛𝑛𝑒𝑒𝑣𝑣 > 𝑡𝑡
C𝑅𝑅𝑛𝑛𝑣𝑣 = �0𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜

Percentage of clearly wrong choices: “the percentage of users in the sample for whom the model
gives a probability greater than threshold t to a choice alternative differing form the observed one”
𝑁𝑁𝑣𝑣

100
％CW =
� 𝐶𝐶W𝑛𝑛𝑣𝑣
𝑁𝑁𝑣𝑣
𝑛𝑛𝑣𝑣 =1

𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤

1 𝑖𝑖𝑖𝑖 𝑃𝑃̂ ! 𝑦𝑦𝑛𝑛𝑒𝑒𝑣𝑣 > 𝑡𝑡
𝐶𝐶𝑊𝑊𝑛𝑛𝑣𝑣 = �0𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜

𝑃𝑃� ! 𝑦𝑦𝑛𝑛𝑒𝑒𝑣𝑣 is the estimated choice probability of an alternative other than the chosen one.

de Luca and Cantarella (2009)A brief introduction to internal validation (data splitting methods)
Performance measures
Clearness of prediction: defining threshold t•

To be meaningful, the threshold t must be “considerably larger”
than c−1, where c is the choice set size.

•

Values used in the literature: Binary model： 𝑡𝑡 = 0.9 (de Luca and Di Pace, 2015)

 Trinary model： 𝑡𝑡 = 0.5 (Glerum, Atasoy and Bierlaire , 2014)de Luca and Di Pace (2015)

See appendix for a list of commonly used indicators

Validation and reporting practices in the transportation academic literature

226 articles reviewed by Parady, Ory and Walker (2021)
92% reported a goodness of fit statistics
64.6% reported a policy-related inference

Marginal effects, elasticities, odds ratios, value of time estimates,
marginal rates of substitution, and policy scenario simulations

18.1% reported a validation measureTowards better validation practices in the field Make model validation mandatory:• Non-negotiable part of model reporting and peer-review in academic journals for any
study that provides policy recommendations.
• Cross-validation is the norm in machine learning studies.
 Share benchmark datasets:• A fundamental limitation in the field is the lack of benchmark datasets and a generalculture of sharing code and data.
 Incentivize validation studies:• Lot of emphasis on theoretically innovative models.
• Encourage submissions that focus on proper validation of existing models and theories.
 Draw and enforce clear reporting guidelines:• In addition to detailed information of survey characteristics such as sampling method,
discussion on representativeness of the data, validation reporting is required.
• Efforts to improve reporting are well documented in other fields(i.e. STROBE statement (von Elm et al., 2007))

Wait a minute…

Q: “I’m not validating my model because I’m not trying to build a predictive framework. I’mtrying to learn about travel behavior”
A: The more orthodox the type of analysis, the stronger the onus of validation.

Q: “Does every study that uses a discrete choice model should be conducting validation?”
A: In short, yes. At the very least, any article that makes policy recommendations should be subject
to proper validation given the dependence of the field on cross-section observational studies, and thelack of a feedback loop in academia.

Q: “Is what we learn about travel behavior from coefficient estimation less valuable if notconducted?”
A: There is a myriad of reasons why some skepticism is warranted against any particular modeloutcome. the most obvious one being model overfitting.

Finally

Better validation practices will not solve the credibility crisis in the field, but it’s a step inthe right direction.
Model validation is no solution to the causality problem in the field, but we want to underscore thatthe reliance on observational studies inherent to the field demands more stringent controls toimprove external validity of results.

References:1.

Ben-Akiva, M. E., Lerman, S. R. (1985). Discrete choice analysis: theory and application to travel demand. MIT press.

2.

M. Baker, D. Penny (2016) Is there a reproducibility crisis? Nature, 533 (7604) pp. 452-454

3.

de Luca, S. De and Cantarella, G. E. (2009) ‘Validation and comparison of choice models’, in Saleh, W. and Sammer, G. (eds) Travel Demand Management
and Road User Pricing: Success, Failure and Feasibility. Ashgate publications, pp. 37–58. doi: 10.1017/cbo9780511619960.008.

4.

de Luca, S., and R. Di Pace (2015). Modelling Users’ Behaviour in Inter-Urban Carsharing Program: A Stated Preference Approach. Transportation ResearchPart A: Policy and Practice, Vol. 71, pp. 59–76. https://doi.org/10.1016/j.tra.2014.11.001.

5.

Glerum, A., Atasoy, B. and Bierlaire, M. (2014) ‘Using semi-open questions to integrate perceptions in choice models’, Journal of Choice Modelling. Elsevier,10(1), pp. 11–33. doi: 10.1016/j.jocm.2013.12.001.

6.

Hasnine, M. S. and Habib, K. N. (2018) ‘What about the dynamics in daily travel mode choices? A dynamic discrete choice approach for tour-based mode
choice modelling’, Transport Policy. Elsevier Ltd, 71(August), pp. 70–80. doi: 10.1016/j.tranpol.2018.07.011.

7.

Hensher, D. A., Rose, J. M., & Greene, W. H. (2015). Applied choice analysis: a primer. Cambridge University Press. 2nd Edition.

8.

Kamargianni, M., M. Ben-Akiva, and A. Polydoropoulou (2014) Incorporating Social Interaction into Hybrid Choice Models. Transportation, Vol. 41, No. 6, pp.
1263–1285.

9.

Qin, H., J. Gao, H. Guan, and H. Chi. (2017) Estimating Heterogeneity of Car Travelers on Mode Shifting Behavior Based on Discrete Choice Models.
Transportation Planning and Technology, Vol. 40, No. 8, pp. 914–927. https://doi.org/10.1080/03081060.2017.1355886.

10. Khan, M., K. M. Kockelman, and X. Xiong. (2014) Models for Anticipating Non-Motorized Travel Choices, and the Role of the Built Environment. Transport
Policy, Vol. 35, pp. 117–126. https://doi.org/10.1016/j.tranpol.2014.05.008.
11. McCloskey, D. N., and S. T. Ziliak.(1996) The Standard Error of Regressions. Journal of Economic Literature, Vol. 34, No. 1, pp. 97–114.
12. Parady, G., Ory, D., Walker, J. (2021) The overreliance on statistical goodness of fit and under-reliance on validation in discrete choice models: A review ofvalidation practices in the transportation academic literature . Journal of Choice Modelling 38, 100257 (Open Access)13. Parady, G., Axhausen K.W.: Size Matters: The Use and Misuse of Statistical Significance in Discrete Choice Models in the Transportation Academic Literature

Appendix: Definition of model validation performance measures reported in the literature
Parady, Ory & Walker (2021)

Index
Mean absolute percentage error
平均絶対誤差率
Root sum of square error
二乗平方根誤差和
Mean absolute error
平均絶対誤差

Type
MAPE

Notes

𝑀𝑀

100
𝑠𝑠̂ 𝑒𝑒 𝑣𝑣,𝑚𝑚 − 𝑠𝑠𝑣𝑣,𝑚𝑚�
𝑠𝑠𝑣𝑣,𝑚𝑚
𝑀𝑀
𝑚𝑚=1

RSSE

MAE

Mean squared error
平均二乗誤差MSE

Root mean square error
二乗平均平方根誤差

RMSE

Brier Score
ブライアスコア

Absolute

FormulaBS

𝑀𝑀

Relative

Aggregate: Relative
Disaggregate: Absolute
Aggregate: Relative
Disaggregate: Absolute
Aggregate: Relative
Disaggregate: Absolute

Absolute� 𝑠𝑠̂ 𝑒𝑒 𝑣𝑣,𝑚𝑚 − 𝑠𝑠𝑣𝑣,𝑚𝑚

𝑚𝑚=1

2

𝑀𝑀

1
� 𝑠𝑠̂ 𝑒𝑒 𝑣𝑣,𝑚𝑚 − 𝑠𝑠𝑣𝑣,𝑚𝑚
𝑀𝑀
𝑚𝑚=1
𝑀𝑀

1
2
� 𝑠𝑠̂ 𝑒𝑒 𝑣𝑣,𝑚𝑚 − 𝑠𝑠𝑣𝑣,𝑚𝑚
𝑀𝑀
𝑚𝑚=1
𝑀𝑀

1
2
� 𝑠𝑠̂ 𝑒𝑒 𝑣𝑣,𝑚𝑚 − 𝑠𝑠𝑚𝑚𝑚𝑚,
𝑀𝑀

𝑁𝑁𝑣𝑣

𝑚𝑚=1
𝑀𝑀

1
2
� � 𝑃𝑃� 𝑦𝑦𝑛𝑛𝑒𝑒𝑣𝑣,𝑚𝑚 − 𝑦𝑦𝑛𝑛𝑣𝑣 ,𝑚𝑚
𝑁𝑁𝑣𝑣
𝑛𝑛𝑣𝑣 =1 𝑚𝑚=1

M is the number of alternatives in the
choice set.
𝑠𝑠𝑣𝑣,𝑚𝑚 is an aggregate outcome measurein sample v, such as the market shareof alternative m (i.e. modal market
share), choice frequency, etc.
𝑠𝑠̂ 𝑒𝑒 𝑣𝑣,𝑚𝑚 is an aggregate outcome measurein sample v, such as the market shareof alternative m, predicted from model
estimated on sample e.
𝑃𝑃� 𝑦𝑦𝑛𝑛𝑒𝑒𝑣𝑣 ,𝑚𝑚 is the predicted probability that
individual n chooses alternative m,
predicted from model estimated on
sample e.
ynm is the actual outcome variablevalued 0 or 1.

Appendix: Definition of model validation performance measures reported in the literature
Parady, Ory & Walker (2021)

Index
Log-likelihood
対数尤度

Log-likelihood loss
対数尤度損失Rho-square𝜎𝜎 2Transfer rho-square
移転 𝜎𝜎 2
Transfer index
移転指標

Transferability test statistic
移転性検定統計量

χ2 test

Type
LL

LLL

Formula11
� 𝑒𝑒
� 𝐿𝐿𝐿𝐿𝑣𝑣,𝑟𝑟 𝜷𝜷�−R𝑁𝑁𝑣𝑣,𝑟𝑟𝑟𝑟

𝑛𝑛𝑣𝑣,𝑟𝑟∀ 1 ≤ 𝑟𝑟 ≤ 𝑅𝑅

� 𝑒𝑒
𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷
𝐿𝐿𝐿𝐿𝑣𝑣 𝟎𝟎

RHOSQ

Absolute

TRHOSQ𝜌𝜌2 = 1 −Relative2𝜌𝜌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡=1−

TI

Pass/Fail

TTS

CHISQ

Relative

Pass/Fail� 𝑒𝑒 is log-likelihood of the model
𝐿𝐿𝐿𝐿𝑣𝑣,𝑟𝑟 𝜷𝜷
estimated on data e applied to thevalidation data vr.

� 𝑒𝑒
𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷

Relative

Absolute

Notes

� 𝑒𝑒
𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷
𝐿𝐿𝐿𝐿𝑣𝑣 𝑴𝑴𝑺𝑺𝒗𝒗

� 𝑒𝑒 − 𝐿𝐿𝐿𝐿𝑣𝑣 𝑴𝑴𝑺𝑺𝒗𝒗
𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷
� 𝑣𝑣 − 𝐿𝐿𝐿𝐿𝑣𝑣 𝑴𝑴𝑺𝑺𝒗𝒗
𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷

� 𝑣𝑣 − 𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷� 𝑒𝑒−2 𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷
𝑀𝑀

�

𝑚𝑚=1

𝑒𝑒
𝑓𝑓𝑚𝑚 − 𝐸𝐸 𝑓𝑓𝑣𝑣,𝑚𝑚𝑒𝑒
𝐸𝐸 𝑓𝑓𝑣𝑣,𝑚𝑚

2

𝑁𝑁𝑣𝑣,𝑟𝑟 is the size of the validation(holdout) sample r, and R is number ofvalidation samples generated.
𝐿𝐿𝐿𝐿𝑣𝑣 𝟎𝟎 is log-likelihood of the model
when all parameters are zero for data v.
� 𝑣𝑣 is the likelihood of the model
𝐿𝐿𝐿𝐿𝑣𝑣 𝜷𝜷
estimated on the validation data v.
𝐿𝐿𝐿𝐿𝑣𝑣 𝑴𝑴𝑺𝑺𝒗𝒗 is a base model estimated on
validation data v (i.e. market sharemodel.)
2𝜌𝜌𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
is the local rho-square of themodel.

𝑓𝑓𝑚𝑚 is the observed choice frequency of𝑒𝑒alternative m in sample v, and 𝐸𝐸 𝑓𝑓𝑣𝑣,𝑚𝑚
isthe expected choice frequency
predicted from model estimated on
sample e.

Appendix: Validation and reporting practices in the transportation academic literature, Brier Score

Heuristic to select validation method given available resources and recommended performance measures to reportAppendix: Validation and reporting practices in the transportation academic literature