Machine Learning Algorithms: Surprises at Deployment?
By: Galit Shmueli
Tags: Machine Learning, Prediction, Decision Making, Human Behavior, Deep Learning

In a recent ArXiv paper by a large group of Google researchers (+ two EE/CS professors and a PhD student), called “Underspecification Presents Challenges for Credibility in Modern Machine Learning” they report discovering a new underlying cause of deployment surprise: “under-specification”, that is, that algorithms that seem equally good at time of development (i.e. they all give a similar “solution” and therefore the problem is under-specified) perform dramatically differently during deployment, in terms of performance on subgroups.

Is the above paper’s discovery a new insight? Are deployment surprises a specialty of deep-learning? is “under-specification” a deep learning problem? Is under-specification in fact a problem in prediction?

That predictive algorithms can perform dramatically differently on data subgroups is well-known. Simpson’s Paradox is an extreme example, where a correlation between an input and output changes direction when examining subgroups of the data. The larger the number of predictors, the larger the chance of a Simpson’s Paradox. Predictive models are also easily “fooled” when the training dataset includes a minority group that has a different input-output relationship than the rest of the training data. Models are fooled because metrics used to train and evaluate algorithms give equal weight to each observation (e.g. least squares or maximum likelihood for training; RMSE and accuracy metrics for evaluation).

Illustration of Simpson’s paradox: a positive x-y relationship appears for each group separately, but a negative x-y relationship appears when the groups are combined. (Source: Wikipedia, Creative Commons)

While the Google researchers paper’s abstract concludes with a vague sentence that might mislead readers to think there’s a technological solution (“Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain”), in several places in the 59-page paper the authors conclude:

“This confirms the need to tailor and test models for the clinical settings and population in which they will be deployed.”

or

“This is compatible with previous findings that inputting medical/domain relational knowledge has led to better out of domain behaviour…, performance… and interpretability… of ML models.”

The paper closes with a proposal to circumvent the need for a context-specific dialogue between the data scientist and the end user by building models that favor “predictors that approximately respect causal structure”. While using causal structure is feasible and useful in some domains, especially in low-dimensional problems, the areas where ML shines are exactly those where causal relationships are difficult to specify. Explanation and prediction both have their merits, and predictive solutions can still be sound and useful even without underlying causal modeling if developers and users collaborate and communicate during the entire loop of design, testing, deployment, and post-deployment feedback.

At their basis, deployment surprises are a failure to understand the limitations of ML, or even statistical models. They all rely on many human choices —by data scientists, data collectors, data engineers, people on whom data are collected, the end users (e.g. decision makers), and more.

In judicial decision making, there has been a growing number of studies identifying issues related to deployment disasters, triggered by the 2016 ProPublica report on glaring errors of the COMPAS system used in several judicial decision making contexts. Many of the issues are related to discrepancies between the data used to train the algorithm and those during deployment, but there are many other context-related issues that surface when we ask “how will the ML solution be used to generate an action?” We can then ask what data the judicial decision maker will use as input to the system and compare that to the input used to train the data (different populations, different definitions of “recidivism”, etc.). We can compare the action that will be triggered (e.g. parole decision) to the action used to define the output in the training data. These are examples of the critical knowledge a dialogue would uncover.