When Data Science fails us
By: Andy Spezzatti
Tags: Scenarioplanning, Data Science, Machine Learning, Bias, Statistics

2020 was a very special year, covid hits the world at the beginning of the year and changed the plans of all industries. Timelines have been changed, customer interactions transformed and internal communication dematerialized. As a result, all of our models that were based on previous years and historical wisdom have been questioned and challenged. What is their relevance in today’s environment?

“How do we reconcile the need to use the past as a guide with our recognition that the future may be different?” The signal and the noise

When can historical data fool you?

A common practice in data science is to examine available historical data in order to draw useful conclusions for better planning.

‘Studying the European market of the 90’ would certainly be of great help to a historian; but what kind of inference can we make now that the structure of the institutions and the markets have changed so much.’ Nassim Taleb

A dataset is a tabular representation of what has happened in the past. But this representation is also accompanied by uncertainties, biases, and errors. In addition, it represents only a subset of the phenomenon you are studying and may not reflect the subset that is relevant to your problem (data availability).

Several caveats should be noted when reviewing the historical data:

the distribution may not be stationary

may not be the initial conditions are different today

are different today the actors (contributors) have evolved

(contributors) have evolved the external interactions have evolved

have evolved how to take into account the randomness ?

? the unknown unknowns (things that you’re not even aware of and that are not included)

In this context, similar inputs can lead to different results.

Is your drug really effective?: the data selection step

When designing clinical trials, an important step is the selection of the patient population. Which population category should be selected? Which age group, what is their condition, their origin, their biomarkers? How can we ensure that this sample is as close as possible to the target customers to whom we want to provide the drug, without missing anyone?

Conversely, if your sample does not accurately represent the population, your drug may be less effective than what your trial showed. For example, if you develop a drug for people with breast cancer, but do not include diabetics with obesity in your sample, you may later discover that these people do not respond well to treatment and cannot use it.

Photo by National Cancer Institute on Unsplash

The same applies to the use of historical data to build inferences, but the limitations are even more subtle to grasp. If you look at all the past data on financial loans, you may think that they reflect the general trend and that the future is likely to be the same. However, these financial loans may have been made only to people with sufficient wealth and did not take into account the rest of the population since they are not even included in the dataset.

Do you really have a causality?: The covariate shift

When the pattern is primarily a reflection of variance and randomness, the causality identified will be misleading to understand the actual behavior of the phenomenon. This is what happens to some very successful traders who benefit from the variance of trading results as Nassim Taleb points out in his book “Fooled by Randomness”. [1]

“Many are so unoriginal, they study history to find mistakes to repeat” Nassim Taleb

The challenge may be even more subtle, as a recent study found that some models trained to detect pneumonia on chest x-rays had poorer results on actual data not visible in the training set. This is partly due to the difference in image acquisition and processing.[2]

The covariate shift is when your input training set differ from the real-world data (Image by author)

This is what we call the covariate shift. When the distribution of your input variables is different in your production model (real setting) compared to your training set.

Are Blacks more often targeted by the police in the United States?: The denominator bias

Models are built not only from data but more importantly from the subset of available (vs non-available) data that we decide to focus on. This decision influences the direction the model takes in learning and can lead to more discrimination.

In her article, Laura Bronner sheds light on the misleading statistics published on the violence against the black community. [3] Many of the published statistics relate to the number of deaths per encounter with the police. However, these statistics hide the discrimination that occurs before, when police had discretion over who they engaged with on the street. The denominator of the measures should not be the encounter, but the population.

Photo by Nicole Baster on Unsplash

Similarly, in his interview with HBO, Donald Trump said that the U.S. has performed well compared to other countries, using the number of deaths per infected metric.

Around the 14'-15' of the interview, Trump tries to defend the poor performance of the United States in the fight against the pandemic by using this statistical flaw. The diagram below highlights the statistical flaw in his reasoning.