Model choice
By: Graeme Keith
Tags: Model Choice, Prediction, Data Analysis, Philosophy Of Science, Mathematical Modeling

Photo by Florian Schmetz on Unsplash

Model choice

Whenever a model appears to you as the only possible one, take this as a sign that you have neither understood the model nor the problem which it was intended to solve

Karl Popper, Philosopher (1902–1994)

If models are cheap, don’t choose. The more models the merrier. But we mustn’t waste our time on rubbish and there’s more to a good model than the accuracy of its predictions.

What is a good model?

At catastrophic economic cost, governments across the world have imposed various levels of lockdown on the basis of results of a handful of mathematical models. Many of these models have since received withering criticism, bringing into question the efficacy of the decisions they supported.

Leaving aside, for now, issues of how best to build and deploy mathematical models to select strategic options in corporate and public policy, we address here only the pertinent and pressing question of what constitutes a good model. What do we mean when we say a model was good, or bad?

This is necessarily a very quick and shallow overview over a deep and rich subject.

Actionability

Decisions and objectives are the poles of the axis on which good models are built. If the decision that the model is supposed to inform isn’t built into the model and connected, causally, to the objective we’re trying to achieve then we’re playing in a sandpit (or maybe a university). Nothing wrong with playing in a sandpit, as long as you’re not dictating policy from your sandpit.

Empirical performance

When I put this question out on LinkedIn for discussion, the overwhelming response was that the merit of a model is its consistency with empirical data: consistency with historical data — preferably data that weren’t used to construct the model in the first place — or a proven track record in prediction. Many contributors took it so much as given that empirical verification was the only measure of a model that they simply interpreted the question as which paradigm of verification was the most effective and suggested appropriate metrics.

No argument here. We build quantitative models exactly so that we can hold them up to scrutiny by empirical verification. This is the critical examination that keeps us honest and keeps our models bound to the world in which they help us navigate.

Empirical adequacy

Actionability opens up for a notion of empirical adequacy. A model doesn’t have to be the best empirically to be the best model. Considerations of cost allow for a richer kind of optimization. A low cost model that is fit for purpose — sufficiently accurate to resolve the decisions at hand (within an acceptable level of uncertainty) — is better than an empirically superior counterpart that is unnecessarily costly or complex.

But I would argue that — even with a notion of adequacy — empirical performance is far from the only criterion for assessing the merit of a model. In some cases, we may not have historical data on which to test, and predictions may only be assessed after the critical decisions have been taken.

And there are yet other circumstances where models that under-perform empirically may be preferred to their predictively superior counterparts.

Explanatory power and causality

An explanation of a phenomenon is an account of the causal factors that lead to it. Causal models propagate the effects of decisions into a network of intervening causal connections and through to the objectives the decisions are trying influence. Sometimes called first-principles, mechanical or structural models, causal models stand in contrast to empirical or exogenous models, which are built on simpler correlative (rather than causal) relationships.

Causal models allow for direct analysis of the effects of decisions on objectives, but also open up for a much richer conversation regarding the scope of those decisions, and possible alternative interventions. They also facilitate the rolling up or packing out of clusters and categories of cause, which is a key step on the path to simplicity.

Simplicity

A good model aligns its complexity with the granularity of the decisions and the objectives it’s trying to inform. A relatively crude model might be good enough in some situations (spelling out the horror scenario of an unchecked, rampant contagion), but not good enough in others (deciding whether cafes should open before hairdressers).

Einstein never said “Everything should be made as simple as possible, but not simpler”, but Einstein was a vigorous advocate for what Quine later called ontological parsimony (being tight-fisted about the number of elements in a model), the sense that the models that have the greatest predictive power with least information are preferred to complex models, even if they are empirically superior. David MacKay gives a gorgeous justification of this in terms of Bayes theory in his (freely available online) book.

For more on simplicity, including what Einstein actually said, read my blog on virtuous and vicious simplification.

Consistency

Good models are consistent: internally, logically consistent, consistent with data and consistent with other models. When I started modelling in the real world, I thought this would be the hoe with which weed the garden of the nonsensical mathematically meaningless models with which the world abounds. But it turns out that if someone using a model hasn’t yet realized it doesn’t make sense on its own terms, it’s incredibly difficult to persuade them otherwise.

Fruitfulness

Good models will predict phenomena they weren’t constructed to predict and motivate measurements they weren’t intended to inform.

In conclusion

Good models are aligned with the decisions they support and the objectives those decisions are designed to achieve. They are actionable, built around an axis whose poles are those decisions and objectives. They are empirically adequate, causal, (virtuously) simple, consistent, internally and with other models, and fruitful.