Thinking Generatively
By: Anwesh Marwade
Tags: Computer Science, Bayesian Inference, Data Science, Statistical Learning, Machine Learning

Bayesian Inference is based on generative thinking

In the Machine Learning domain, we often find ourselves in the pursuit of inferring a functional relationship between the attribute variable(s) (i.e. features or simply, input data: x) and its associated response (i.e. the target variable: y). Being able to learn this relationship allows one to build a model that can predict a response, given any set of such attribute variables (i.e. the test data).

A response y being modelled using attribute x

The usual suspect

Scrambling towards the drawing board, we quickly think of defining a loss function with an intention of minimising the said loss using whatever means necessary (pun un-intended), essentially translating a learning problem into a problem of optimisation, hoping that our model is able to learn a function good enough to potentially discriminate likeable and not-so-likeable content. This is considered as a discriminative modelling approach. In this article, however, I intend to walk you through a different approach: a generative one.

It is quite difficult to generate the absolute correct recommendation for any user as there is a multitude of factors that might be at play, like the user’s time constraints at that moment, her mood? or even a gut feeling? Wait.. it’s getting way too complex!

Recently, I have come to appreciate an approach to modelling which involves actually inferring that functional relationship by thinking generatively. Goal? we wish to model the process that generated the data (likeable content in this case) in the first place. Although it’s not possible (practically) to come up with a model to generate the exact data, the ability to think generatively is quite useful as I shall now explain.

A Need for Modelling Errors

A simple Linear Regression model

Consider a simple linear regression model (see figure) with the best fit line as a result of minimising the loss. Going back to our discriminative approach, we see that even though the assumption of a linear relationship between x and y does capture the general trend, there are deviations (sometimes considerable) between our model (i.e. the red line) and observed data points. The intuition we’re trying to develop here is that if we know these deviations or errors are going to be present (remember that it is the best fit line after all), it does make a fair case for us to build them into our model, in trying to think generatively! Fret not! We can call on our dear friend, statistics to incorporate this variability into our model using a ‘random variable’.

Intending for the reader to appreciate the intuition of thinking generatively, bear with me as I skim over the mathematical details and abstract certain caveats (definitely incurring the wrath of the more mathematically inclined).

Response y modelled generatively using additive errors sampled from a normal distribution

Our generative model with (purposely) modelled errors, looks like this after incorporating the deviations through a random variable, ‘ϵ’ for each of our n input data points. Starting simple, we assume that the errors (i.e. the epsilons) are independent and are sampled from a Gaussian (or Normal) distribution with zero mean and a variance v. Owing to the random component, we now look to maximise the likelihood of our data in the following manner to obtain the optimal parameter estimate for w:

Dataset Likelihood: likelihood of the joint density over all responses in our dataset

We look to find the values for w and v that maximise the likelihood, L or for mathematically tractable reasons, maximise the Log-likelihood. Interestingly maximising the likelihood and minimising the loss (using least squares for example) in the discriminative approach, gives the same exact optimal value for our parameter w, when the error terms are sampled from a normal distribution. Being able to estimate an optimal value of the variance: v, which basically controls the distribution of these errors, allows us to predict a range of values as opposed to an exact answer as in a discriminative model; here, a smaller range (of values) would suggest a higher level of certainty in our prediction whereas this range would be large if we are less certain about the result.

Conclusively, the benefit of thinking generatively, using this likelihood approach, is that it confers us with the ability to quantify uncertainty in our estimates and hence (more importantly) in our predictions. Crucially, it allows us to steer away from exact predictions (like in the discriminative model) which will most certainly be (close enough but still) wrong, towards a ranged interval: y ± v; where ‘y’ is the predicted response and ‘v’ is a control parameter of the chosen error distribution.

It is absolutely not necessary to sample our errors from a normal distribution (it can be any distribution like the Beta or Binomial distribution), or to even have an additive error; that was intended to keep the mathematics straightforward for the sake of this article. For an in-depth understanding, please refer to the book titled, ‘A first course in Machine Learning’ by S. Roger and M. Girolami, which was the inspiration for this article!

Hi! I am Anwesh. Pursuing a Masters at the Computer Vision lab at TU Delft, I am learning to appreciate the math and the following intuition behind machine learning models. I have come across some really cool ideas, which I would love to share. Stay tuned for upcoming articles and you can learn more about my projects on my website. Appreciate your support!

References:

Simon Rogers and Mark Girolami. 2016. A First Course in Machine Learning, Second Edition (2nd. ed.). Chapman & Hall/CRC.

Image taken from: https://www.analyticsvidhya.com/blog/2016/06/bayesian-statistics-beginners-simple-english/