Generalized Linear Models — Introduction | by Deepak Khandelwal | Towards Data Science

What is Linear Model in Machine Learning?
To illustrate the topic with an example, I have taken this dataset which has one predictor/feature/independent/explanatory variable namely Brain Weight and one response/target/dependent variable namely Body Weight .
Index Brain Weight Body Weight 1          3.385     44.500 2          0.480     15.500 3          1.350      8.100 4        465.000    423.000 5         36.330    119.500 6         27.660    115.000...
The response variable Body Wight is having a continuous distribution. We can model this distribution as a linear combination of predictors — Brain Weight,which is only one in this dataset, and a intercept term. Here, model is essentially trying to learn the weights/parameter associated with each features. Mathematically, we have to model the following linear equation —
y = Body Weight [Response]X = Brain Weight [Predictor]We are interested to model the following equation y_hat     y_hat = W1*X1 + W0     y = y_hat + e
In the above equation, W1 and W0 are generally called parameters of model. In the context of linear models, W1 and W0 are referred as coefficients and intercept respectively. W0 represents the situation where all the X1 predictors will be zero, W0 is the value our model will predict.
e is called error term. This is the error which cannot be predicted from the knowledge of the predictors. It is the difference between what is observed i.e y, and what our model has predicted i.e. W1*X1 + W0.
The regression function tells us that how much the response variable y changes on average for a unit change (increase or decrease) in the value of X1.
The following plot is the hyper plane (line in this case) that fits the data.
We will take one more example to illustrate the idea of linear model more concretely.In this example, we have this dataset which contains two predictors and one continuous response variable. In this case, we have following situation —
y = Fish Length [Response]X1 = Age [Predictor]X2 = Water Temperature[Predictor]We are interested to model the following equation y_hat     y_hat = W1*X1 + W2*X2 + W0     y = y_hat + e
The following hyper plane (plane in this case) is drawn using the parameters i.e. W1, W2, W0 learned by the model. Although it is not the best fit to the data but our focus is on the underlying concept of linear model, not on the model predictive power.
We can compare and cross-check the equation y = y_hat + e by putting the learned parameters in the equation - y_hat = W1*X1 + W2*X2 + W0 . For the sake of brevity, I have taken first ten rows of the dataset.
We can clearly see that y is very closely equal to the summation of y_hat and error term e .
Following the above discussion, we can conclude that Linear Model means that a distribution of a random variable is modeled as a linear combination of the features/predictors.
What is ‘linear’ in linear models?
When we say — ‘linear combination of features’ — it means that it is the model parameters Wi that are linear, having degree 1, not the features. We estimate linear parameters, we don’t estimate something like — W12, W1*W2 in case of linear models. Moreover, Features can have different varieties of terms such as — X1, X1*X2, X22 etc.
Now we can write a general equation for linear model as follows —
y = W0 + W1*X1 + W2*X2 + W3*X3 + ... + Wn*Xn + e
But while we perform linear regression, we make certain assumptions such as: 1. Distribution of response variable is normal i.e. Y_i ~ N(μ_i, σ2)2. Function of the predictors/explanatory variables Xi is W0 + Wi*Xi 3. Connection/Link between the explanatory variables and the the distribution of Yi is the mean of the response variable.The predicted value for each data point would be distributed around that mean with variance σ2.
                      μ_i = E(Y_i) = W0 + Wi*Xi
Limitations with linear models -1. The assumption of normality does not hold all the time, we may have our categorical features such as — gender, profession type etc. or we may have time series data which are not normally distributed.2. Linear model cannot fit a data with categorical response variable such as iris flower dataset, MNIST dataset etc.
3. Even if the response variable is a continuous random variable but takes its values within a range such as probability between 0 and 1. A traditional linear model will fail to predict a value between 0 and 1 because our features can take any value in the range (-∞, ∞).
Now we have learnt that linear models are not always right choice to model different than continuous response data and may provide rather strange results. Now what we are going to do is to generalize this model to overcome such problems. This results in GLMs which provides a unified framework for modelling data originating from the exponential family of probability distributions such as Normal, Binomial, Poisson among others.
There are 3 components of GLM.
Random Component — Which defines the response variable y and its probability distribution. One important assumption is that the responses from y1 to yn are independent to each other.Systematic Component — Which defines which explanatory variables we want to include in our model. It also allows the interactions among explanatory variables such as — X1*X2, X12 etc.That is the part that we model. It is also called linear predictor of covariates i.e. X1, X2, ... , Xn and coefficients i.e W1, W2, ... , Wn.Link Component — Which connects Random and Systematic component. It is the function of expected value of response variable i.e. E(Y) which enables linearity in the parameters and allows E(Y) to be non-linearly related to the explanatory variables. It is the link function that generalizes the linear models.
Random Component — Which defines the response variable y and its probability distribution. One important assumption is that the responses from y1 to yn are independent to each other.
Systematic Component — Which defines which explanatory variables we want to include in our model. It also allows the interactions among explanatory variables such as — X1*X2, X12 etc.That is the part that we model. It is also called linear predictor of covariates i.e. X1, X2, ... , Xn and coefficients i.e W1, W2, ... , Wn.
Link Component — Which connects Random and Systematic component. It is the function of expected value of response variable i.e. E(Y) which enables linearity in the parameters and allows E(Y) to be non-linearly related to the explanatory variables. It is the link function that generalizes the linear models.
Note that choice of link function is separate from the choice of random component.
Now we can say that the simple linear regression model is a special case of GLM, wherein the random component has Normal distribution and it takes the continuous values in the range (-∞, ∞). The systematic component is X.The link function is an identity function.
                        Y ~ N(μ, σ2)                        E(Y) = μ                        g(E(Y)) = μ = W1*X1 + e
What do we generalize in GLMs?
We generalize the distribution of response variable, that is y , can take.We generalize the link function between explanatory variables Xi and response variable Yi.
We generalize the distribution of response variable, that is y , can take.
We generalize the link function between explanatory variables Xi and response variable Yi.
Below table contains the commonly used link functions.
Let’s summarize what we have discussed so far — 1. Linear models are good to model continuous response variable but they have certain limitations.2. Generalized linear models unify many different types of response variable distributions that belong to exponential family of density.3. Link function is the key component in the GLM which enable linearity in the parameters and it is the one that generalizes the linear model.
References:[1] B. Caffo, 03 01 Part 1 of 1 Generalized Linear Models (2015)[2] R. Cooksey, Logic of the General Linear Model (GLM) (2013)[3] Actuarial Education, CT6 Introduction to generalized linear models (GLMs) (2012)