Supervised learning — the What, When, Why, Good and Bad (Part 1) | by Sheenal Srivastava | Towards Data Science

Often times in the workplace, business stakeholders and managers associate machine learning and big data with deep learning. They will often think that the best solution to all data problems is deep learning, AI or neural networks or some combination of the prior (insert your favourite buzzword). As someone with a statistics background, I find this view quite frustrating as the type of machine learning algorithm chosen should be based upon five key considerations:
Type of data problem — i.e. supervised vs unsupervisedThe variable types within the dataset — i.e. categorical or numericalThe validation of underlying assumptions of the statistical modelThe resulting model accuracyThe balance between precision vs. recall and sensitivity vs. specificity (for classification)
Type of data problem — i.e. supervised vs unsupervised
The variable types within the dataset — i.e. categorical or numerical
The validation of underlying assumptions of the statistical model
The resulting model accuracy
The balance between precision vs. recall and sensitivity vs. specificity (for classification)
In the next few sections and over a couple of blog posts, I am going to go through the different types of models for supervised learning and apply the above considerations to each one.
Most of the time, data problems require the application of supervised learning. This is when you know exactly what you want to predict — the target or dependent variable, and have a set of independent or predictor variables that you want to better understand in terms of their influence on the target variable. Then, the selection of a model is based on mapping the underlying pattern in the data to a function which is dependent upon the data’s distribution and variable types.
The name “supervised learning” is used to describe these types of models because the model learns the underlying pattern on a training set. The number of iterations/rounds determines the number of times the model has a chance to learn from its past. In each subsequent round, the model tries to improve on model accuracy (accuracy metric selected by user) based on what it has learnt in previous runs. The model stops running either after its maximum number of runs has been achieved or when it can no longer improve the model accuracy (specified by early stopping in some models).
There are two types of supervised learning algorithms as shown in the figure below where the type of outcome variable determines whether you select regression or classification.
Let’s delve into regression first.
It is common for people to think everything is a linear regression problem, but this is not the case. Regression algorithms have a very strict set of assumptions that have to be met in order for the final results to be statistically reliable (*Note: I did not write statistically significant).
Regression analysis is a predictive algorithm that is often used in time series analysis and forecasting. This is because it models the underlying relationship between predictor variables and the dependent variable and determines how the variation in the values of the predictor variables explains the variation in the outcome variable.
There are several types of regression algorithms, but I will focus on the ones below.
Before I explore the above regression techniques further. Let’s gain an understanding of the assumptions associated with regression techniques.
Residuals of the regression line are normally distributed — To ensure that the result from the regression model are valid, the residuals (difference between observed and predicted values) should follow a normal distribution (with mean zero and constant standard deviation). The residuals are also called error terms and their distribution can be observed on a normal probability plot (Q-Q). If the majority of the residuals are located on the diagonal normality line, then we can assume their normality.
Residuals of the regression line are normally distributed — To ensure that the result from the regression model are valid, the residuals (difference between observed and predicted values) should follow a normal distribution (with mean zero and constant standard deviation). The residuals are also called error terms and their distribution can be observed on a normal probability plot (Q-Q). If the majority of the residuals are located on the diagonal normality line, then we can assume their normality.
2. Variables in the dataset come from normally distributed populations — all variables in the data should also arise from normally distributed populations. Typically, in statistics if the sample size is large enough, then normality can be assumed according to the Central Limit Theorem. In some textbooks it is also low as n = 30.
3. Residuals are independent — This assumption is checked to avoid auto-correlation amongst the residuals. When auto-correlation is present within the dataset, it can lead to underestimating the standard error which in turn can make predictors look statistically significant even when they aren’t.
The Durbin-Watson Test can be utilised to check for auto-correlation. The Null Hypothesis is that the residuals are not linearly auto-correlated. Durbin-Watson’s d can take on any value between 2 and 4, but rule of thumb values of d that are between 1.5 and 2.5 are are used to indicate lack of auto-correlation in the residuals.
Note: The Durbin-Watson test can only vouch for linear auto-correlation between direct neighbors (first-order effects).
4. Data shows homoscedasticity — Homoscedasticity in the data occurs when the residuals are equally distributed along the line of best fit rather than show a pattern (residuals have constant variance). When heteroscedasticity is present, the shape of the residuals may show fanning (cone-like shape) or even a linear trend.
Homoscedasticity can be assessed by plotting the residuals (y-axis) against the predicted values (x-axis) onto a scatterplot and looking for a random scatter of the points rather than any clusters or patterns. A statistical test for heteroscedascticity is the Goldfeld-Quandt Test. It involves dividing the dataset into two groups and checking to see if the variances of the residuals between the two groups are similar.
5. Relationship is linear — When carrying out linear regression (simple or multiple), the relationship between the independent variable(s) and the dependent variable must be linear. Visually, this can be ascertained by creating a scatterplot between each independent variable and dependent variable pair and checking to see if there is a straight line that can be drawn through the points.
5. There are no significant/influential outliers — Outliers are typically points within the dataset that are +/- 3 standard deviations from the mean or are beyond the whiskers of a boxplot. As outliers are extreme values they can have a significant effect on the slope of the best-fit line.
Outliers can be visually identified in boxplots (points that lie outside the whiskers) or from skewness in histograms.
To determine if the outliers are influential, there are two statistical metrics that can be used: 1) Cook’s Distance and 2) Mahanalobis Distance
Cook’s Distance checks to see how the regression coefficients vary as each observation in the dataset is removed thereby checking their influence. Each value in the dataset is assigned a Cook’s Distance value. The larger the Cook’s distance, the more influential the observation. A typical heuristic used to determine if a value is an outlier is by checking to see if it’s Cook’s D value is greater than 4/n (where n is the size of the dataset).
Mahanalobis Distance measures how far (how many standard deviations) is each observation from the majority of the data points (or mean) and is often used for multivariate datasets. This distance can be approximated using a chi-squared distribution with n degrees of freedom. Compute the p-value for each observation in the dataset. Check to see if the p-value is less than the level of significance (i.e. 0.01) to determine if they are extreme.
6. There is no multicollinearity* amongst the independent variables — Multicollinearity occurs when the independent variables are related to each other. If this is the case, typically you want to include only one of the variables within the pair into the regression model. This assumption can be checked using three methods.
*Does not apply to simple linear regression
Variance Inflation Factor (VIF): Variables are multicollinear if their VIF value is greater than 10.
Correlation Coefficient (r): Create a correlation matrix between all pairs of variables. If the correlation coefficient is greater than or equal to -/+ 0.7, the the variables are highly correlated.
Tolerance — This metric measures the how each independent variable influences all the other independent variables and is defined by the equation T = 1 — R-squared. R-squared (Coefficient of Determination) can be calculated for each independent variable by regressing it against the remaining independent variables. A tolerance factor of 0.1 is commonly used, where when T < 0.1, this indicates the presence of multicollinearity in the dataset.
Let’s test some of the concepts we have learnt by running regression diagnostics on the Boston Housing data.
Residuals vs Fitted — This plot is used to check for linearity. Given that the line is horizontal without any distinct patterns, we can assume a linear relationship.Normal Q-Q. This plot is used to check for normality of residuals. As most of the residuals are on the diagonal line, this assumption is satisfied.Scale-Location (or Spread-Location) — this plot is used to examine whether the variance of the residuals is homogeneous (homoscedastic). We are looking for a horizontal line with equally distributed residuals below and above this line. In this case, the line has some curvature which means we have some heteroscedascticity.Residuals vs Leverage. This plot is used to determine influential observations or those that have leverage. The points that are labelled in the plot appear to be influential. By removing these, we may also resolve the heteroscedasctic problem as the same observations are labelled in the previous plot.
Residuals vs Fitted — This plot is used to check for linearity. Given that the line is horizontal without any distinct patterns, we can assume a linear relationship.
Normal Q-Q. This plot is used to check for normality of residuals. As most of the residuals are on the diagonal line, this assumption is satisfied.
Scale-Location (or Spread-Location) — this plot is used to examine whether the variance of the residuals is homogeneous (homoscedastic). We are looking for a horizontal line with equally distributed residuals below and above this line. In this case, the line has some curvature which means we have some heteroscedascticity.
Residuals vs Leverage. This plot is used to determine influential observations or those that have leverage. The points that are labelled in the plot appear to be influential. By removing these, we may also resolve the heteroscedasctic problem as the same observations are labelled in the previous plot.
Now let’s look at statistics that are commonly used to evaluate model performance.
There are a couple of metrics that can be used to measure the accuracy of a regression model. I have described the ones I have commonly used when building models.
R-squared: The coefficient of determination measures how much variation in the outcome variable can be explained by the variation in the independent variables. Higher R-squared values are seen as being reflective of higher model accuracy. However, R-squared is not to be trusted when there are multiple predictors in the model. This is because each predictor influences the R-squared value. In these instances, it is more reliable to use the adjusted R-squared. When deciding whether or not to trust the R-squared value, subject matter expertise in the area is required. For instance, R-squared can be quite high (close to 90%) in controlled environments such as chemical plants; however, when modelling human behaviour, a high R-squared value can be representative of overfitting. In these instances, it is better to use any of the below metrics to validate your model.
RMSE (Root-Mean Squared Error): It is calculated as the squared root of the variance between the predicted and observed values (residuals). the lower the value, the better the model fit. RMSE is a better metric in some cases because it penalizes large errors (residuals).
MAE (Mean Absolute Error):This metric is simpler than the RMSE and just sums the absolute value of the residuals. It is less influenced by outliers. Each residual has a proportionate contribution to the total error in the equation. Once again, lower values are better.
We can finally delve into understanding the various types of regression algorithms.
Linear regression, also called, least-squares regression, uses independent variables to approximate the best fit line for the outcome variable. It is a weighted equation that tries to minimize the sum of squared differences and is denoted by the following equation, where y is the predicted dependent variable, each β is a coefficient corresponding to each independent variable or x, β0 is the y-intercept, and e is the error term.
In simple linear regression, there is only one coefficient and one independent variable. A coefficient term (β) is added for each independent variable (x) included in multiple linear regression.
Let’s start with the y-intercept or constant, β0. It is the value predicted for y if all the independent variables (X) were zero. It is also typically the average value of the predicted y.
For coefficients of continuous predictor variables, they are interpreted as follows. Keeping all other variables constant, for every 1 unit of increase in X1, y will increase (if β1 is positive) or decrease (if β1 is negative) by β1 units on average.
Categorical independent variables are coded as 0 or 1 (dummy variables) for linear regression. So when interpreting their coefficient, which is the difference in the predicted value of y, a one unit difference is a representation of switching from one category to the other while keeping all other variables constant.
When the relationship between the independent variable and dependent variable is non-linear in nature and includes curvature, polynomial regression can be considered an option.
In the graph on the left, we can see that polynomial regression (to the order of 2) fits slightly better than straight line.
A polynomial regression equation for each independent variable is expressed below. For each order that the variable is raised to, an additional term is added for the variable into the equation.
It can be tempting to continue to raise the equation to higher degree polynomials to get the best fit; however, this can lead to over-fitting. Overfitting occurs when an attempt is made to try and explain as many points as possible, including the random errors, in the dataset (i.e. training set) making it difficult to generalise for other datasets (i.e. test sets).
Stepwise regression likes its name involves adding (forward selection starting with no independent variables in the model) or removing (backward selection starting with a full model or all independent variables in the model) independent variables in the linear model to find the best subset of predictors for the dependent variable based on the model evaluation metric.
Forward selection stops when there is no longer an improvement in the model evaluation metric (i.e. R-squared stops increasing), whereas backward selection halts after the least influential (i.e. those that contribute to R-squared) independent variables have been removed.
Sticking to the Boston housing dataset, I have run linear regression (“lm”) of all predictor variables against the median house price. Please note, that I did not “deal” with the multicollinearity, normality, and outlier issues prior to running the model.
> linearMod <- lm(medv ~ ., data=Boston) > summary(linearMod)Call:lm(formula = medv ~ ., data = Boston)Residuals:    Min      1Q  Median      3Q     Max -15.595  -2.730  -0.518   1.777  26.199Coefficients:              Estimate Std. Error t value Pr(>|t|)    (Intercept)  3.646e+01  5.103e+00   7.144 3.28e-12 ***crim        -1.080e-01  3.286e-02  -3.287 0.001087 ** zn           4.642e-02  1.373e-02   3.382 0.000778 ***indus        2.056e-02  6.150e-02   0.334 0.738288    chas         2.687e+00  8.616e-01   3.118 0.001925 ** nox         -1.777e+01  3.820e+00  -4.651 4.25e-06 ***rm           3.810e+00  4.179e-01   9.116  < 2e-16 ***age          6.922e-04  1.321e-02   0.052 0.958229    dis         -1.476e+00  1.995e-01  -7.398 6.01e-13 ***rad          3.060e-01  6.635e-02   4.613 5.07e-06 ***tax         -1.233e-02  3.760e-03  -3.280 0.001112 ** ptratio     -9.527e-01  1.308e-01  -7.283 1.31e-12 ***black        9.312e-03  2.686e-03   3.467 0.000573 ***lstat       -5.248e-01  5.072e-02 -10.347  < 2e-16 ***---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 4.745 on 492 degrees of freedomMultiple R-squared:  0.7406, Adjusted R-squared:  0.7338 F-statistic: 108.1 on 13 and 492 DF,  p-value: < 2.2e-16
Above, we can see that the adjusted R-squared value is 73.38%, which means that this amount of variation in the median house price can be explained by the predictor variables in the model. This is a pretty good R-squared value and the p-value < 0.05 indicating statistical significance.
Next, I would determine which variables contribute to this R-squared statistic. I can see that all except indus and age are non-significant contributors (p-values > 0.05).
Let’s see what happens if I run stepwise regression.
Backward
Let’s try and interpret the call to the the model first.
My dependent variable (medv) has been regressed against all independent variables (.)
I have used backward regression
The model is going to be tuned to try models consisting of 1 to 13 variables.
# Set seed for reproducibility> set.seed(123)> # Set up repeated k-fold cross-validation (10 fold)> train.control <- trainControl(method = "cv", number = 10)> # Train the model> step.model <- train(medv ~ ., data=Boston,+                     method = "leapBackward", +                     tuneGrid = data.frame(nvmax = 1:13),+                     trControl = train.control+ )> step.model$results   nvmax     RMSE  Rsquared      MAE    RMSESD RsquaredSD     MAESD1      1 6.190699 0.5542196 4.512277 0.6480792 0.03585104 0.51434972      2 5.537331 0.6393649 3.975099 0.6194193 0.08470812 0.41843833      3 5.216943 0.6757786 3.675974 0.7844579 0.09606216 0.32822874      4 5.144946 0.6843838 3.621869 0.6838151 0.08435306 0.30330785      5 4.998163 0.7010740 3.543459 0.7416779 0.08756225 0.29994316      6 5.048162 0.6953426 3.520865 0.7245789 0.08673190 0.27173747      7 5.049469 0.6952020 3.500608 0.6987592 0.08276566 0.25975088      8 5.008368 0.7004709 3.497253 0.6740135 0.08021108 0.27566749      9 5.012961 0.7018661 3.490065 0.6963818 0.08564270 0.268346810    10 4.947244 0.7098667 3.447318 0.6896883 0.08705085 0.222294311    11 4.809318 0.7249622 3.359429 0.6861143 0.08341394 0.240993712    12 4.825967 0.7231964 3.365846 0.6808653 0.08309689 0.244236013    13 4.829796 0.7227732 3.368084 0.6796633 0.08336179 0.2455718> step.model$bestTune   nvmax11    11> summary(step.model$finalModel)Subset selection object13 Variables  (and intercept)        Forced in Forced outcrim        FALSE      FALSEzn          FALSE      FALSEindus       FALSE      FALSEchas        FALSE      FALSEnox         FALSE      FALSErm          FALSE      FALSEage         FALSE      FALSEdis         FALSE      FALSErad         FALSE      FALSEtax         FALSE      FALSEptratio     FALSE      FALSEblack       FALSE      FALSElstat       FALSE      FALSE1 subsets of each size up to 11Selection Algorithm: backward          crim zn  indus chas nox rm  age dis rad tax ptratio black lstat1  ( 1 )  " "  " " " "   " "  " " " " " " " " " " " " " "     " "   "*"  2  ( 1 )  " "  " " " "   " "  " " "*" " " " " " " " " " "     " "   "*"  3  ( 1 )  " "  " " " "   " "  " " "*" " " " " " " " " "*"     " "   "*"  4  ( 1 )  " "  " " " "   " "  " " "*" " " "*" " " " " "*"     " "   "*"  5  ( 1 )  " "  " " " "   " "  "*" "*" " " "*" " " " " "*"     " "   "*"  6  ( 1 )  " "  " " " "   " "  "*" "*" " " "*" " " " " "*"     "*"   "*"  7  ( 1 )  " "  " " " "   " "  "*" "*" " " "*" "*" " " "*"     "*"   "*"  8  ( 1 )  "*"  " " " "   " "  "*" "*" " " "*" "*" " " "*"     "*"   "*"  9  ( 1 )  "*"  " " " "   " "  "*" "*" " " "*" "*" "*" "*"     "*"   "*"  10  ( 1 ) "*"  "*" " "   " "  "*" "*" " " "*" "*" "*" "*"     "*"   "*"  11  ( 1 ) "*"  "*" " "   "*"  "*" "*" " " "*" "*" "*" "*"     "*"   "*"
In the example above, I have run backwards stepwise regression with 10-fold cross-validation and 13 maximum variables. I didn’t use all of them because we know two were not significant from running a full-linear model previously. We can see that the best model consists of 11 variables (lowest RMSE, lowest MAE, and highest R-squared).
The asterisk (*) indicates that a variable was included in the corresponding model. The best 11-variables model includes all variables except indus and age, just like the full linear model.
Forward stepwise regression gives us similar results to the full linear model and backward regression.
> set.seed(123)> train.control <- trainControl(method = "cv", number = 10)> step.model <- train(medv ~ ., data=Boston,+                     method = "leapForward", +                     tuneGrid = data.frame(nvmax = 1:13),+                     trControl = train.control+ )> step.model$results   nvmax     RMSE  Rsquared      MAE    RMSESD RsquaredSD     MAESD1      1 6.190699 0.5542196 4.512277 0.6480792 0.03585104 0.51434972      2 5.537331 0.6393649 3.975099 0.6194193 0.08470812 0.41843833      3 5.216943 0.6757786 3.675974 0.7844579 0.09606216 0.32822874      4 5.243481 0.6736566 3.658219 0.7510340 0.08224804 0.33047445      5 5.098698 0.6904107 3.568604 0.7354806 0.08392928 0.27260786      6 5.045755 0.6953140 3.512300 0.7223225 0.08606320 0.27873737      7 4.969144 0.7036333 3.448631 0.7542104 0.08842114 0.29449218      8 4.985899 0.7027153 3.456932 0.7099500 0.08432911 0.28951849      9 4.985690 0.7030156 3.466382 0.6879376 0.08295944 0.278216610    10 4.912877 0.7137519 3.421565 0.6865803 0.07813425 0.271699411    11 4.809318 0.7249622 3.359429 0.6861143 0.08341394 0.240993712    12 4.825967 0.7231964 3.365846 0.6808653 0.08309689 0.244236013    13 4.829796 0.7227732 3.368084 0.6796633 0.08336179 0.2455718> step.model$bestTune   nvmax11    11
Regularization is a technique that is often used to solve the over-fitting and multicollinearity issues that may occur with regression. It does this via the addition of a penalty term to the objective function which aims to reduce standard error.
In ridge regression, a penalty term (shrinkage parameter), λ, is added to reduce the large variation in the β coefficient terms. L2 regularization is used in ride regression, where the objective is to minimize the sum of squares of coefficients by trying to bring the coefficients closer to zero.
Ridge regression is good for small datasets with a large number of variables with small to medium effect.
Lasso (Least Absolute Shrinkage and Selection Operator) regression uses L1 regularization where λ is added to the sum of absolute squares of the coefficients.
Lasso regression is good for datasets with fewer variables with medium to large effect on the dependent variable.
ElasticNet combines both Lasso and Ridge Regression techniques where both L1 and L2 regularization are applied.
When deciding on which regularization parameter to use, we can base this on two methods — 1) where we choose λ where the information criterion is minimised (such as AIC or BIC) focusing on model fit or 2) where we run the model with cross-validation and select the value of λ that minimize the cross-validated residuals focusing on prediction accuracy.
Pros
Easy to understand and present to stakeholders
Can be used for explainability — i.e relative influence of each predictor on outcome variable
Cons
Important to standardize (scale and center) all independent variables to avoid multicollinearity
Requires checking of strict model assumptions
That was all I had on regression. My next blog post will be on classification algorithms.