The 20th Behavior Modeling Summer School
Sep. 15 – 19 , 2020 @ The University of Tokyo

Basic inference and validation in discrete choice modeling
Giancarlos Parady – The University of Tokyo

Basic inference discrete choice modelingWhy is inference important ?

When talking about policy,

Size matters!

→ Ask not whether there is an effect, ask how big is it?
→ Is the effect big enough to make a policy successful?
→ Is the effect big enough to make invest millions of dollars?
→ 64.6% of studies in the transportation field report some sort ofpolicy-relevant inference analysis (Parady, Ory and Walker, 2021)

Parady G., Ory, D., Walker, J. (2021）

Why is inference important ?
Variable name

CoefficientS.E.

t statistic

1.45

0.393

3.70

In-vehicle time (min)-0.00890.0063-1.42Out-of-vehicle time (min)-0.03080.0106-2.90

Auto out-of-pocket cost (c)-0.0115

0.0026-4.39

Transit fare

-0.00700.0038-1.87

Auto ownership (specific to auto mode)-0.770

0.213

3.16

Downtown workplace (specific to auto mode)-0.561

0.306

-1.84

Number of observations

1476

Number of cases

1476

LL(0)

-1023

LL(β)

-347.4-2[LL(0)-LL(β)]

1371

𝜌2

0.660

𝜌ҧ 2

0.654

Auto constant

Table adapted from Ben-Akiva and Lerman (1985)

Magnitudes are not directly interpretable.
We can only interpret the effect direction,or use them to calculate utilities,
and choice probabilities

To make some sense of theseparameters we must calculate
elasticities or marginal effects

Basic Inference in discrete choice models
MNL: Logit Elasticities (Point elasticities)•

Direct elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in an attribute of that same
alternative.

𝜕𝑃𝑛 (𝑖) 𝑥𝑖𝑛𝑘
𝑃 𝑖
𝐸𝑥𝑖𝑛𝑘 =∙=
𝜕𝑥𝑖𝑛𝑘 𝑃𝑛 (𝑖)•1 − 𝑃𝑛 𝑖 𝑥𝑖𝑛𝑘 𝛽𝑘

Cross-elasticity: measures the percentage change in the probability of choosing a particularalternative in the choice set with respect to a given percentage change in a competing alternative.

𝜕𝑃𝑛 (𝑖) 𝑥𝑗𝑛𝑘
𝑃 𝑖
𝐸𝑥𝑗𝑛𝑘 =∙= −𝑃𝑛 𝑗 𝑥𝑗𝑛𝑘 𝛽𝑘𝜕𝑥𝑗𝑛𝑘 𝑃𝑛 (𝑖)Hensher, David A., John M. Rose, and William H. Greene (2015)

Because of IIA, crosselasticities are uniform
across all alternatives

Basic Inference in discrete choice models
MNL: Logit Elasticities (Point elasticities)•

The elasticities shown before are individual elasticities (Disaggregate)•

To calculate sample (aggregate) elasticities we use the probability weighted
sample enumeration method:
𝑃 𝑖
𝑁
෠
σ
𝑃
𝑖
𝐸
𝑛=1 𝑛
𝑥𝑖𝑛𝑘
𝑃(𝑖)
𝐸𝑥𝑖𝑛𝑘 =෠σ𝑁
𝑛=1 𝑃𝑛 𝑖𝑃(𝑖)𝐸𝑥𝑗𝑛𝑘 =

Sample direct elasticity

𝑃 𝑖
෠
σ𝑁
𝑃
𝑖
𝐸
𝑛=1 𝑛
𝑥𝑗𝑛𝑘

෠
σ𝑁
𝑛=1 𝑃𝑛 𝑖

Sample cross-elasticity

Where 𝑃(𝑖) is the aggregate choice probability of alternative I, and 𝑃෠𝑖𝑛 𝑖 is an estimated choice probability•

Uniform cross-elasticities do not necessarily hold at the aggregate level•

Also note that elasticities for dummy variables are meaningless!

Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
NL: Logit Elasticities (Point elasticities)
𝑒 𝑉𝑗 /𝜏𝑒 𝜏𝐼𝑉(𝑖)𝑃 𝑗 = 𝐼𝑉(𝑖) ∙ 𝐼
σ𝑖=1 𝑒 𝜏𝐼𝑉(𝑖)𝑒← NL RUM2 specification

Direct Elasticity
When alternative 𝑗 does not belong to any
nest

When alternative 𝑗 belongs to nest 𝑖

𝑃 𝑗
𝐸𝑥𝑗𝑛𝑘 =𝑃 𝑗
𝐸𝑥𝑗𝑛𝑘 =

1 − 𝑃𝑛 𝑗 𝑥𝑗𝑛𝑘 𝛽𝑘

1 − 𝑃𝑛 𝑗

1+−1𝜏

1−𝑃 𝑗 𝑖𝑥𝑗𝑛𝑘 𝛽𝑘

Cross Elasticity
When alternatives 𝑗 and 𝑗’ belong todifferent nests

When alternatives 𝑗 and 𝑗’ belong to thesame nest

𝑃 𝑗
𝐸𝑥 ′ = −𝑃𝑛 𝑗 ′ 𝑥𝑗 ′𝑛𝑘 𝛽𝑘
𝑗 𝑛𝑘

𝑃 𝑗
𝐸𝑥 ′ = −
𝑗 𝑛𝑘

𝑃𝑛

𝑗′

1
+
− 1 𝑃 𝑗′ 𝑖𝜏

𝑥𝑗 ′𝑛𝑘 𝛽𝑘Basic Inference in discrete choice models
Relation between elasticity of demand, change in price and revenue

𝑃

𝑃

Perfectly
inelastic𝑥𝑃

Relatively
inelastic

𝑥

𝑃

Unit
elastic𝑥

𝑃

Relatively
elastic𝑥

Perfectly
elastic𝑥

Direct elasticity:
1% increase in 𝑥𝑖 results in a
0% decrease in 𝑃(𝑖)1% increase in 𝑥𝑖 results in aless than 1% decrease in 𝑃(𝑖)1% increase in 𝑥𝑖 results in
a 1% decrease in 𝑃(𝑖)1% increase in 𝑥𝑖 results in amore than 1% decrease in 𝑃(𝑖)1% increase in 𝑥𝑖 results
in a ∞ decrease in 𝑃(𝑖)1% increase in 𝑥𝑗 results in aless than 1% increase in 𝑃(𝑖)1% increase in 𝑥𝑗 results inno percent change in 𝑃(𝑖)1% increase in 𝑥𝑗 results in amore than 1% increase in 𝑃(𝑖)1% increase in 𝑥𝑗 results
in a ∞ increase in 𝑃(𝑖)Cross elasticity:
1% increase in 𝑥𝑗 results in a
0% increase in 𝑃(𝑖)Adapted from Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects•

Direct marginal effect: measures the change in the probability (absolute change) of choosing aparticular alternative in the choice set with respect to a unit change in an attribute of that same
alternative.

𝜕𝑃𝑛 (𝑖)
𝑃 𝑖
𝑀𝑥𝑖𝑛𝑘 == 𝑃𝑛 𝑖𝜕𝑥𝑖𝑛𝑘•

1 − 𝑃𝑛 𝑖 𝛽𝑘

Cross-marginal effect: measures the change in the probability (absolute change) of choosing aparticular alternative in the choice set with respect to a unit change in a competing alternative.

𝜕𝑃𝑛 (𝑖)𝑃 𝑖
𝑀𝑥𝑗𝑛𝑘 == 𝑃𝑛 𝑖𝜕𝑥𝑗𝑛𝑘

Hensher, Rose, and Greene (2015)−𝑃𝑛 𝑗 𝛽𝑘

Basic Inference in discrete choice models
MNL: Marginal Effects•

We can also calculate sample (aggregate) marginal effects using the probability weighted
sample enumeration method:
𝑃 𝑖
𝑁
෠
σ
𝑃
𝑖
𝑀
𝑛=1 𝑛
𝑥𝑖𝑛𝑘
𝑃(𝑖)
𝑀𝑥𝑖𝑛𝑘 =෠σ𝑁
𝑛=1 𝑃𝑛 𝑖
Sample direct marginal effect𝑃(𝑖)𝑀𝑥𝑗𝑛𝑘 =

𝑃 𝑖
෠
σ𝑁
𝑃
𝑖
𝑀
𝑛=1 𝑛
𝑥𝑗𝑛𝑘

෠
σ𝑁
𝑛=1 𝑃𝑛 𝑖

Sample cross-marginal effect

Where 𝑃(𝑖) is the aggregate choice probability of alternative I, and 𝑃෠𝑖𝑛 𝑖 is an estimated choice probability•

Marginal effects for dummy variables do make sense as we are talking about unit changes, but adifferent procedure is necessary to estimate marginal effects.

Hensher, Rose, and Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects
𝑃(𝑖)𝜕𝑃𝑛 (𝑖)𝜕𝑥𝑖𝑛𝑘

𝑥𝑖
Marginal effects as the slopes of the Tangent lines to the cumulative probability curve
Hensher, David A., John M. Rose, and William H. Greene (2015)

Basic Inference in discrete choice models
MNL: Marginal Effects
Calculating marginal effects for dummy variables
Calculated via simulation:
1.

Set the values of the variable of interest to 0

2.

Estimate base predictions (at the individual level)3.

Set the values of the variable of interest to 1

4.

Estimate new predictions (at the individual level)5.

Calculate marginal effects by taking the mean of the difference in individual predictions

Validation practices in discrete choicemodeling

A credibility crisis in science and engineering?

Image removed due to copyright issues
See original article here:
M. Baker, D. Penny (2016) Is there a reproducibility crisis? Nature, 533(7604) pp. 452-454

Baker and Penny (2016)

A credibility crisis in science and engineering?
Most published research findings are likely to be false due to factors such as lack of
power of the study, small effect sizes, and great flexibility in research design, definitions,outcomes and methods.
Focused on experimental studies(Ioannidis, 2005)

In the transportation field

Unlike the natural sciences◼ Dependence on cross-section observational studies◼ Classic scientific hypothesis testing is more difficult◼ Impact evaluation of policies drawn based on model-based academic research is rarely conducted◼ No feedback in terms of how right or how wrong are these models and the policy recommendations derived from them

◼ These issues underscore the need for proper validation practices

Term definitions
Predictive accuracy: The degree to which predicted outcomes match observed outcomes.
Predictive accuracy is a function of：•

Calibration: The degree to which predicted probabilities match the relative frequency ofobserved outcomes.

•

Discrimination ability: The ability of a model or system of models to discriminate between
those instances with and without a particular outcome.

Parady, Ory & Walker (2021)

Term definitions
Generalizability: The ability of a model, or system of models to maintain its predictive accuracy
in a different sample.
Generalizability of a model is a function of：•Reproducibility: The extent to which a model or system of models maintains its predictive ability indifferent samples from the same population.

•

Transferability: The extent to which a model or system of models maintains its predictive ability insamples from different but plausibly related populations or in samples collected with different
methodologies (sometimes called transportability.)

Parady, Ory & Walker (2021)

Term definitions
Model validation: The evaluation of the generalizability of a statistical model.
Types of model validation：••

Internal validation: The evaluation of the reproducibility of a model.
•

Data splitting, resampling methods•

Different sample from the same population

External validation: The evaluation of the transferability of a model.
•

Temporal transferability•

Spatial transferability•

Methodological transferability

Parady, Ory & Walker (2021)

Parady, Ory & Walker (2021)

A brief introduction to internal validation•

Internal validation: The evaluation of the reproducibility of a model.

•

Due to the high costs of datal collection the most common approaches are•

Data splitting (Holdout validation, Cross validation)•

Resampling methods (Bootstrapping)Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Holdout validation: Dataset is randomly split into an estimation dataset and a validation dataset.
Estimation data

Validation data

For illustration purposes, let us define Q 𝑦𝑛 , 𝑦ොn as a measure of prediction correctness forthe 𝑛th instance, for the binary choice case as:
0 𝑖𝑓 𝑦𝑛 = 𝑦ොn
Q 𝑦𝑛 , 𝑦ොn = ቊ
1 𝑖𝑓 𝑦𝑛 ≠ 𝑦ොnwhere is 𝑦𝑛 the observed outcome, and 𝑦ොn is the predicted outcome for instance n.

The holdout estimator is
𝑁𝑣

𝐻𝑂𝑉 =

1
෍ 𝑄[𝑦𝑛𝑣 , 𝑦ො𝑛𝑒𝑣 ]
𝑁𝑣
𝑛𝑣 =1

where 𝑦ො𝑛𝑒𝑣 is the predicted outcome for instance 𝑛 in sample v, using the model estimated with sample 𝑒, and 𝑁𝑣 is thevalidation sample size.
Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Cross-validation: When the holdout process is repeated multiple times, thus generating a set of
randomly split estimation-validation data pairs, we refer to the validation procedure as crossvalidation (CV).

𝐶𝑉 =1
෍ 𝐻𝑂𝑉𝑏B 𝑏where B is the number of estimation-validation data pairs generated and is the holdout estimator for set b.

𝑏=1

Validation data

𝑏=2

Estimation data

𝑏=3

Estimation data
Validation data

Estimation data

Estimation data

推定データ
Validation
data

𝑏=4

Estimation data

𝑏=5

Estimation data

Estimation data

Validation data

A 5-fold cross validation illustration
Parady, Ory & Walker (2021)

Estimation data
Validation dataA brief introduction to internal validation (data splitting methods)
Cross-validation：Commonly used methods
𝐶𝑉 =1
෍ 𝐻𝑂𝑉𝑏B 𝑏• Cross-validation methods differ from one another in the way the data is split.
• When the data splitting considers all possible estimation sets of size , the splitting is
exhaustive, otherwise the splitting is partial. (Arlot and Celisse, 2009)。
Exhaustive splitting methods• Leave-one-out : estimation set size is Ne = 𝑁 − 1, and B = N. The model is fitted leaving out one
instance per iteration, and the outcome of that single instance is predicted based on the estimatedmodel.

• Leave-p-out : Ne = 𝑁 − 𝑝. The model is fitted leaving out p-instances per iteration, and theoutcome of the remaining instances is predicted based on the estimated model.
Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Cross-validation：Commonly used methods
1
𝐶𝑉 = ෍ 𝐻𝑂𝑉𝑏
B 𝑏
• Cross-validation methods differ from one another in the way the data is split.
• When the data splitting considers all possible estimation sets of size , the splitting is
exhaustive, otherwise the splitting is partial. (Arlot and Celisse, 2009)。
Partial splitting methods (lower calculation cost)• K-fold cross-validation: data is partitioned into K mutually-exclusive subsets of roughly equal

size, and B=K.
• Repeated learning-testing: a B number of randomly-split estimation-validation pairs are
generated. This method is also called repeated holdout validation.

Parady, Ory & Walker (2021)

A brief introduction to internal validation (data splitting methods)
Performance measures
Market share comparison• Easy to execute• Does not provide a quantitative
measure to evaluate the level ofagreement between predictions
and observations

Image removed due to copyright issues
See original article here:
Hasnine, M. S. and Habib, K. N. (2018) ‘What about the dynamics indaily travel mode choices? A dynamic discrete choice approach fortour-based mode choice modelling’, Transport Policy. Elsevier Ltd,
71(August), pp. 70–80. doi: 10.1016/j.tranpol.2018.07.011.

Hasnine and Habib (2018)

A brief introduction to internal validation (data splitting methods)
Performance measures

Percentage of correct predictions: the alternative with the highest probability is defined as thepredicted choice. However,Model A：

Model B：

Model C：

• Alt. A: 0.34＊• Alt. A: 0.50＊• Alt. A: 0.90＊• Alt. B: 0.33

• Alt. B: 0.30

• Alt. B: 0.05

• Alt. C: 0.33

• Alt. C: 0.20

• Alt. C: 0.05

＊Observed choice

Cannot discriminate differences in estimated probabilities.
A measures that accounts for “clearness” of prediction is necessary.

A brief introduction to internal validation (data splitting methods)
Performance measures

Clearness of prediction:
Percentage of clearly right choices: “the percentage of users in the sample whose observed
choices are given a probability greater than threshold t by the model”
100 𝑁𝑣
σ𝑛𝑣=1 𝐶𝑅𝑛𝑣
％CR =
𝑁𝑣

𝑤ℎ𝑒𝑟𝑒

1 𝑖𝑓 𝑃෠ 𝑦𝑛𝑒𝑣 > 𝑡
C𝑅𝑛𝑣 = ቊ0𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Percentage of clearly wrong choices: “the percentage of users in the sample for whom the model
gives a probability greater than threshold t to a choice alternative differing form the observed one”
𝑁𝑣

100
％CW =
෍ 𝐶W𝑛𝑣
𝑁𝑣
𝑛𝑣 =1

𝑤ℎ𝑒𝑟𝑒

1 𝑖𝑓 𝑃Ƹ ! 𝑦𝑛𝑒𝑣 > 𝑡
𝐶𝑊𝑛𝑣 = ቊ0𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒𝑃෠ ! 𝑦𝑛𝑒𝑣 is the estimated choice probability of an alternative other than the chosen one.
de Luca and Cantarella (2009)A brief introduction to internal validation (data splitting methods)
Performance measures

Clearness of prediction: defining threshold t•

To be meaningful, the threshold t must be “considerably larger”
than c−1, where c is the choice set size.

•

Values used in the literature:➢ Binary model： 𝑡 = 0.9 (de Luca and Di Pace, 2015)➢ Trinary model： 𝑡 = 0.5 (Glerum, Atasoy and Bierlaire , 2014)

See appendix for a list of commonly used indicators

Image removed due to copyright issues
See original article here:de Luca, S. De and Cantarella, G. E. (2009)‘Validation and comparison of choicemodels’, in Saleh, W. and Sammer, G. (eds)
Travel Demand Management and Road
User Pricing: Success, Failure andFeasibility. Ashgate publications, pp. 37–58.

Validation and reporting practices in the transportation academic literature

226 articles reviewed by Parady, Ory and Walker (2021)
92% reported a goodness of fit statistics

64.6% reported a policy-related inference
Marginal effects, elasticities, odds ratios, value of time estimates,
marginal rates of substitution, and policy scenario simulations

18.1% reported a validation measureTowards better validation practices in the field◼ Make model validation mandatory:• Non-negotiable part of model reporting and peer-review in academic journals for any
study that provides policy recommendations.
• Cross-validation is the norm in machine learning studies.
◼ Share benchmark datasets:• A fundamental limitation in the field is the lack of benchmark datasets and a generalculture of sharing code and data.
◼ Incentivize validation studies:• Lot of emphasis on theoretically innovative models.
• Encourage submissions that focus on proper validation of existing models and theories.
◼ Draw and enforce clear reporting guidelines:• In addition to detailed information of survey characteristics such as sampling method,
discussion on representativeness of the data, validation reporting is required.
• Efforts to improve reporting are well documented in other fields(i.e. STROBE statement (von Elm et al., 2007))

Wait a minute…

“I’m not validating my model because I’m not trying to build a predictive
framework. I’m trying to learn about travel behavior”

The more orthodox the type of analysis conducted (such as the dimensions of travel
behavior covered in this study), the stronger the onus of validation.

Wait a minute…

“Does every study that uses a discrete choice model
should be conducting validation?”

In short, yes. At the very least, any article that makes policy recommendations should be
subject to proper validation given the dependence of the field on cross-section
observational studies, and the lack of a feedback loop in academia.

Wait a minute…

“Is what we learn about travel behavior fromcoefficient estimation less valuable if not conducted?”

There is a myriad of reasons why some skepticism is warranted against any particularmodel outcome. the most obvious one being model overfitting.

Finally

Better validation practices will not solve the credibility crisis in the field, but it’s a step inthe right direction.
Model validation is no solution to the causality problem in the field, but we want to underscore thatthe reliance on observational studies inherent to the field demands more stringent controls toimprove external validity of results.

References:1.

Ben-Akiva, M. E., Lerman, S. R. (1985). Discrete choice analysis: theory andapplication to travel demand. MIT press.

2.

Hensher, D. A., Rose, J. M., & Greene, W. H. (2015). Applied choice analysis: aprimer. Cambridge University Press. 2nd Edition.

3.

M. Baker, D. Penny (2016) Is there a reproducibility crisis? Nature, 533 (7604) pp.
452-454

4.

Parady, G., Ory, D., Walker, J. (2021) The overreliance on statistical goodness of fit
and under-reliance on validation in discrete choice models: A review of validation
practices in the transportation academic literature . Journal of Choice Modelling 38,
100257 (Open Access)5.

de Luca, S. De and Cantarella, G. E. (2009) ‘Validation and comparison of choicemodels’, in Saleh, W. and Sammer, G. (eds) Travel Demand Management and Road
User Pricing: Success, Failure and Feasibility. Ashgate publications, pp. 37–58. doi:
10.1017/cbo9780511619960.008.

6.

Glerum, A., Atasoy, B. and Bierlaire, M. (2014) ‘Using semi-open questions tointegrate perceptions in choice models’, Journal of Choice Modelling. Elsevier, 10(1),pp. 11–33. doi: 10.1016/j.jocm.2013.12.001.

7.

Hasnine, M. S. and Habib, K. N. (2018) ‘What about the dynamics in daily travel
mode choices? A dynamic discrete choice approach for tour-based mode choicemodelling’, Transport Policy. Elsevier Ltd, 71(August), pp. 70–80. doi:
10.1016/j.tranpol.2018.07.011.

Appendix: Definition of model validation performance measures reported in the literature
Parady, Ory & Walker (2021)

Index
Mean absolute percentage error
平均絶対誤差率Type

FormulaNotesAbsolute

100𝑠Ƹ 𝑒 𝑣,𝑚 − 𝑠𝑣,𝑚
෍
𝑀
𝑠𝑣,𝑚

𝑀

MAPE

M is the number of alternatives in the
choice set.

𝑚=1

Root sum of square error
二乗平方根誤差和𝑀

RSSE

Relative෍ 𝑠Ƹ 𝑒 𝑣,𝑚 − 𝑠𝑣,𝑚2

𝑚=1

Mean absolute error
平均絶対誤差

MAE

Aggregate: Relative
Disaggregate: Absolute𝑀1෍ 𝑠Ƹ 𝑒 𝑣,𝑚 − 𝑠𝑣,𝑚𝑀
𝑚=1

Mean squared error
平均二乗誤差MSE

Aggregate: Relative
Disaggregate: Absolute𝑀12෍ 𝑠Ƹ 𝑒 𝑣,𝑚 − 𝑠𝑣,𝑚𝑀
𝑚=1

Root mean square error
二乗平均平方根誤差

RMSE

Aggregate: Relative
Disaggregate: Absolute𝑀12෍ 𝑠Ƹ 𝑒 𝑣,𝑚 − 𝑠𝑚𝑣,
𝑀
𝑚=1

Brier Score
ブライアスコア

𝑁𝑣

BS

Absolute𝑀12
෍ ෍ 𝑃෠ 𝑦𝑛𝑒𝑣,𝑚 − 𝑦𝑛𝑣 ,𝑚𝑁𝑣
𝑛𝑣 =1 𝑚=1

𝑠𝑣,𝑚 is an aggregate outcome measurein sample v, such as the market shareof alternative m (i.e. modal market
share), choice frequency, etc.

𝑠Ƹ 𝑒 𝑣,𝑚 is an aggregate outcome measurein sample v, such as the market shareof alternative m, predicted from model
estimated on sample e.
𝑃෠ 𝑦𝑛𝑒𝑣 ,𝑚 is the predicted probability that
individual n chooses alternative m,
predicted from model estimated on
sample e.
ynm is the actual outcome variablevalued 0 or 1.

Appendix: Definition of model validation performance measures reported in the literature
Parady, Ory & Walker (2021)

Index
Log-likelihood
対数尤度

Log-likelihood loss
対数尤度損失

Type
LL

LLL

FormulaNotes

Relative෡𝑒
𝐿𝐿𝑣 𝜷෡ 𝑒 is log-likelihood of the model
𝐿𝐿𝑣,𝑟 𝜷estimated on data e applied to thevalidation data vr.

Absolute

11෡𝑒෍−෍ 𝐿𝐿𝑣,𝑟 𝜷R𝑁𝑣,𝑟𝑁𝑣,𝑟 is the size of the validation(holdout) sample r, and R is number ofvalidation samples generated.

𝑟

𝑛𝑣,𝑟

∀1≤𝑟 ≤𝑅
Rho-square
𝜎2

RHOSQ

Transfer rho-square
移転 𝜎 2TRHOSQ

Transfer index
移転指標

Transferability test statistic
移転性検定統計量

TI

TTS

Absolute

Relative

Pass/Fail

Relative𝜌2 = 1 −CHISQ

Pass/Fail

෡𝑒
𝐿𝐿𝑣 𝜷
𝐿𝐿𝑣 𝟎

2
𝜌𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟=1−

෡ 𝑣 is the likelihood of the model
𝐿𝐿𝑣 𝜷estimated on the validation data v.

෡𝑒
𝐿𝐿𝑣 𝜷
𝐿𝐿𝑣 𝑴𝑺𝒗

෡ 𝑒 − 𝐿𝐿𝑣 𝑴𝑺𝒗
𝐿𝐿𝑣 𝜷෡ 𝑣 − 𝐿𝐿𝑣 𝑴𝑺𝒗
𝐿𝐿𝑣 𝜷෡ 𝑣 − 𝐿𝐿𝑣 𝜷෡𝑒−2 𝐿𝐿𝑣 𝜷𝑀χ2 test𝐿𝐿𝑣 𝟎 is log-likelihood of the model
when all parameters are zero for data v.

෍
𝑚=1

𝑒
𝑓𝑚 − 𝐸 𝑓𝑣,𝑚𝑒
𝐸 𝑓𝑣,𝑚2

𝐿𝐿𝑣 𝑴𝑺𝒗 is a base model estimated on
validation data v (i.e. market sharemodel.)
2
𝜌𝑙𝑜𝑐𝑎𝑙is the local rho-square of themodel.

𝑓𝑚 is the observed choice frequency of𝑒alternative m in sample v, and 𝐸 𝑓𝑣,𝑚isthe expected choice frequency
predicted from model estimated on
sample e.

Appendix: Validation and reporting practices in the transportation academic literature, Brier Score

Heuristic to select validation method given available resources and recommended performance measures to reportAppendix: Validation and reporting practices in the transportation academic literature