Bayesian Hierarchical Modeling (or “more reasons why autoML cannot replace Data Scientists yet”) | by Alain Tanguy | Towards Data Science

In this article, we will use a Probabilistic Programming library developed for python, pymc3. A first introduction to the efficiency of Bayesian approaches in basic statistics and to pymc3 can be found here and there.
Since early 2018, Automated Machine Learning has become one of the trendiest topic in data science. Amazon’s Sagemaker or Google AutoML to mention just a few are now accessible to most Data Scientists, to such an extent that some tend to think exploring and understanding data is not necessary anymore in order to build machine learning models.
The promise of AutoML can be summed up this way:
the high degree of automation allows non-experts to make use of machine learning models and techniques without requiring to become an expert in a particular field first.1
The reasoning is straightforward ; expertise is no longer required, just give data to the AutoML algorithm, and after testing a fixed number of predefined models, it will return the best one.
But here’s the catch... AutoML algorithms ignore what best means to us, and will merely try to minimize an empirical error. Knowledge and expertise are still required to understand what this error really means, and to what extent it differs from the one we are actually hoping to minimize.
Given n data points, the empirical error is given by
for a particular function fn over all observable values of features xi and target yi where V denotes a loss function.2
It is usually computed through a cross-validation or train/test process. Missing some links between covariates and target will generate a bias often leading to under/over-fitting and eventually to a higher test/cross-validated error.
It is too often, and wrongly, the only one taken into account by data scientists (and always by autoML engines).
generalization error(also known as the out-of-sample error) is a measure of how accurately an algorithm is able to predict outcome values for previously unseen data. It is defined by:
for a particular function fn over all possible values of features x and target y where V denotes a loss function and ρ(x, y) is the unknown joint probability distribution for x and y.3
Obviously, it is usually the one we would prefer to minimize when training our model. Unfortunately, it cannot be computed directly from data, unlike the empirical one which is merely an estimate of the former.
With enough good quality data, we interchange those. Now imagine only part of the information is available at training time, or that there just isn’t enough data at all, as it is often the case in Machine Learning. Our model will then be trained on a different distribution ρ(x, y) from the one it will be running on later, and as a consequence, the empirical error will diverge from the generalized one.
Biased data leads to biased empirical error.
We can already see how much of a problem this is: the empirical error does not estimate the desired quantity anymore, but is the only we can compute from data. We need to make sure it can still be relied on to some extent. This is where human expertise comes into plays and shows its full potential.
The same way parametric approaches are preferred over non-parametric ones when the relation between variables is clear, knowledge from the field of study compensate the lack of information from the data by explicitly modeling some dependences.
The better our model can extrapolate from partial or truncated distribution, the closer empirical and generalization error will be.
“Correlation does not imply causation.”
All data scientists have heard this sentence at least once, but it turns out only a few truly realize the implication when it comes to actual modeling. The topic of causality is actually just left out most of the time, sometimes wrongly, sometimes justifiably, but rarely knowingly.
A confounder is a variable that is causally related to both the covariate and the outcome of interest. As a causal concept, it cannot be described in terms of correlations.
The error yielded by confounders cannot be fully measured by traditional statistical methods since it is not a statistical error per se.
One could argue that when the empirical error is low, or even the generalization error; then we shouldn’t care if our model exploits true causality or spurious correlations. It might be true in some cases, but unless we explicitly know why, this situation shouldn’t be overlooked.
Simpson’s paradox, described by Edward Simpson in 1951 and George Udny Yule in 1903, is a paradox in which a statistical trend appears when data are segmented into separate groups but reverses when the groups are combined.
It is usually observed in the presence of a confounding variable (represented here by the different colour groups) when causal relations are ignored.
The impact Simpson’s paradox will have in machine learning is when it comes to decision making, where we are given the choice of which data we should consider to pick an action, the aggregated or the partitioned?
Unfortunately the answer usually cannot be inferred from the data itself. Indeed with exact similar values we can have different answers, depending on the causal relationships among the variables. The real source of error lies outside of the data itself and eventually it doesn’t matter how low the test or cross-validation error is. We are not safe from getting making completely wrong decisions unless we are able to correctly model the environment first.
We will now through a concrete example show how to handle this situation in the concrete case of hierarchical dependences.
Often data scientists have to deal with geographic data, which have the disadvantage of not being easily exploitable by classic machine learning models. The location feature usually has a really high cardinality and is often unbalanced. However it is intuitive that parameters from models will vary from region to region, depending on local variables, but will still be closely related since they model the same phenomenon across different places.
We are going to see a way to handle this spatial dependence and thus to minimize all types of error listed above through the following example.
We consider the problem of estimating the use of contraceptive in Bangladesh, and to this end we use data come from the 1988 Bangladesh Fertility Survey. It consists of a subsample of 1934 women grouped in 60 districts, with variables are defined as follows:
DISTRICT: identifying code for each district.
LC: Number of living children at time of survey.
AGE: Age of woman at time of survey.
URBAN: Type of region of residence.
and we’ll consider 3 logistic Bayesian regressions with different characteristics to model different possible approaches.
Here we simply obliviate the role of the DISTRICT variable by not using it. The result is a simple logistic regression with only 4 parameters, including the intercept. The location is not seen as a confounder and each region is assumed to behave similarly.
For each district we fit a different logistic regression, leading to a total of 60 models with 4 parameters each. Even if we assume behaviour to vary from district to district, we don’t take any advantage of the similarities they could share.
Bayesian hierarchical modelling is a statistical model written in multiple levels that estimates the parameters of the posterior distribution using the Bayesian method. The sub-models combine to form the hierarchical model, and Bayes’ theorem is used to integrate them with the observed data and account for all the uncertainty that is present.4
We assume that while βs are different for each district as in the unpooled case, now the coefficients all share similarity. We can model this by assuming that each individual coefficient comes from a common group distribution:
with
Though analytically intractable, probabilistic programming allows us to compute the posterior of all our parameters using Markov chain Monte Carlo (MCMC) to sample from the posteriors distributions. Again pymc3 offers an extremely intuitive way to model our network and to compute posteriors!
We can also easily compute a graphical representation of our Bayesian network:
First of all, let’s try to understand the differences between our models.
Inspired by https://docs.pymc.io/notebooks/GLM-hierarchical.html, we can visualize the evolution of our regression parameters over the different regressions.
We display the coefficients of each district’s non-hierarchical posterior mean, hierarchical posterior mean, and pooled posterior mean. The small amount of data available at district level led unpooled posteriors to be spread far out and thus the shrinkage effect is really important, yet differences among mean in the hierarchical model are still significant, as betas vary in order of magnitude between districts.
We use the Area Under the ROC Curve (AUC) as our error measure for comparing the models. It can be seen as the probability that our models will score a randomly chosen positive class higher than a randomly chosen negative class. It is particularly interesting to us since it has the advantage of not requiring to set a threshold to assign labels.
We considered 2 test sets in order to measure performances, a stratified by district one and a non-stratified one. The use of a non-stratified test set is more representative of a case where the generalization error will differ significantly from the empirical model error, since features distributions among districts will vary significantly between train and test sets!
The measures have been averaged over multiple seeds for the test set sampling in order to be more representative of the real performance.
┌───────────────────┬────────┬──────────┬──────────────┐│                   │ POOLED │ UNPOOLED │ HIERARCHICAL │├───────────────────┼────────┼──────────┼──────────────┤│ Train             │  0.632 │    0.818 │        0.726 ││ Stratified Test   │  0.623 │    0.618 │        0.668 ││ Unstratified Test │  0.634 │    0.603 │        0.663 │└───────────────────┴────────┴──────────┴──────────────┘
The unpooled model strongly overfits as the huge gap between the train and test AUC demonstrates.
On the other hand, the pooled model is strongly biased and clearly underfits the data.
Finally, our hierarchical model performed significantly better than the others by taking advantage of parameters geographic similarities.
The shrinkage effect provided us with an improved statistical power and can also be seen as a smart way to regularize.
Some districts have extremely few individuals from which to train, and thus the unstratified test error on those gets bigger with the unpooled model. The partial pooling takes into account the similarity between parameters, and provides low density districts with information from others, while keeping their specificities.
As a result the difference between models is even more significant regarding the unstratified test set, showing us its generalization capacity is greater as performances are almost not affected by the stratification strategy.
=> Multi-level hierarchical Bayesian models outperform basic approaches when we have multiple sets of measurements we expect to have similarity.
It would be interesting to compare this Bayesian approach to other classic data preprocessing approaches (different encodings of the district variable) or algorithms (gradient boosting, random forest, etc.).
AutoML cannot replace Data Scientists yet as it is not able to distinguish empirical error measures from actual business objectives, nor to model correctly dependences between covariates and target.
Experts are still required to understand data and model properly the problems. To this end, they have access to a range of mathematics and informatic tools, including pymc3 library and Bayesian hierarchical models, allowing to easily model and compute distributions in the very common case of hierarchically structured data
[1]: Wikipedia contributors. “Automated machine learning.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 18 Feb. 2020. Web. 1 Mar. 2020.
[2,3]: Wikipedia contributors. (2020, February 22). Generalization error. In Wikipedia, The Free Encyclopedia. Retrieved 16:40, March 1, 2020, from https://en.wikipedia.org/w/index.php?title=Generalization_error&oldid=942140633
[4]: Wikipedia contributors. “Bayesian hierarchical modeling.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 12 Dec. 2019. Web. 1 Mar. 2020.
Causality: Models, Reasoning, and Inference, Cambridge University Press (2000, 2nd edition 2009). ISBN 0–521–77362–8.
https://docs.pymc.io/notebooks/GLM-hierarchical.html
Huq, N. M., and Cleland, J. 1990. Bangladesh Fertility Survey 1989 (Main Report). Dhaka: National Institute of Population Research and Training
Legendre P. Spatial autocorrelation: Trouble or new paradigm? Ecology. 1993;74:1659–1673.