Machine Learning 102: Logistic Regression With Polynomial Features | by Leihua Ye, PhD | Towards Data Science

Data Scientists are rock stars!
Rock and Roll!
In my previous ML 101 article, I explained how we could apply logistic regression to classify linear questions. In this post, I want to complicate things a little bit by including nonlinear features. Just like the real world, things are intertwined and messy.
Let’s delve into the R.
# Load the dataset library(tidyverse)data5 = read_csv("nonlinear.csv")require(ggplot2)qplot(X1,X2,colour = Y,data=data5)
As we can see, black dots are surrounded by blue dots. Our job is to find a ML technique to neatly separate these two types of dots.
# build a regular logistic regressionglm_5b = glm(Y~X1+X2,data=data5)summary(glm_5b)Call:glm(formula = Y ~ X1 + X2, data = data5)Deviance Residuals:     Min       1Q   Median       3Q      Max  -0.6944  -0.5504   0.1937   0.3584   0.6213  Coefficients:            Estimate Std. Error t value Pr(>|t|)    (Intercept)  0.71038    0.05672  12.524   <2e-16 ***X1          -0.05446    0.02496  -2.182   0.0325 *  X2           0.04278    0.02708   1.580   0.1187    ---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1(Dispersion parameter for gaussian family taken to be 0.2109993)    Null deviance: 16.000  on 71  degrees of freedomResidual deviance: 14.559  on 69  degrees of freedomAIC: 97.238Number of Fisher Scoring iterations: 2
As can be seen, the regular logistic regression fails to consider the nonlinear feature and performs poorly.
Next, let’s project class labels over to finely sampled grid points and plot predictions at each point on the grid colored by class labels.
# grid of points over sample spacegr <- expand.grid(X1=seq(-5, 5, by=0.1), # sample points in X1 X2=seq(-5, 5, by=0.1)) # sample points in X2#predict class labelprobability_pred = predict(glm_5b,gr,type=”response”)# set the cutoff point at 0.5class_pred = as.factor(ifelse(probability_pred<=0.5, “0”, “1”))color_array <- c(“red”, “blue”)[as.numeric(class_pred)] plot(gr,col=color_array,pch=20,cex=0.25)
From the above plot, a regular logistic regression model does not work that well.
Let’s include 2nd degree polynomial terms of x1 and x2.
glm_5c =glm(Y~poly(X1,deg=2)*poly(X2,deg=2),data=data5)summary(glm_5c)
probability_pred_5c = predict(glm_5c,gr,type=”response”)class_pred_5c = as.factor(ifelse(probability_pred_5c<=0.5, “0”, “1”))color_array_5c <- c(“red”, “blue”)[as.numeric(class_pred_5c)] plot(gr,col=color_array_5c,pch=20,cex=0.25)
As can be seen from the ANOVA outputs, the following variables are statistically significant: X1, X1 2, X2, X2 2, X1*(X2)2, and (X1)2*(X2)2. Since the second-order terms are significant, we can’t fit a simple linear classification.
Besides, the logistic model with higher terms performs better than the simple model, as can be seen from the plot.
To play a safe card, let’s try a logistic model with 5-th degree polynomials without any interaction terms.
glm_5d =glm(Y~poly(data5$X1,deg=5)+poly(data5$X2,deg=5),data=data5)summary(glm_5d)Call:glm(formula = Y ~ poly(data5$X1, deg = 5) + poly(data5$X2, deg = 5),     data = data5)Deviance Residuals:      Min        1Q    Median        3Q       Max  -0.51652  -0.15930  -0.06256   0.17439   0.73943  Coefficients:                         Estimate Std. Error t value Pr(>|t|)    (Intercept)               0.66667    0.03309  20.145  < 2e-16 ***poly(data5$X1, deg = 5)1 -0.70547    0.30163  -2.339  0.02264 *  poly(data5$X1, deg = 5)2  0.94681    0.28791   3.289  0.00167 ** poly(data5$X1, deg = 5)3  0.82225    0.28386   2.897  0.00523 ** poly(data5$X1, deg = 5)4 -0.24777    0.29833  -0.831  0.40948    poly(data5$X1, deg = 5)5 -0.00171    0.29624  -0.006  0.99541    poly(data5$X2, deg = 5)1  0.62673    0.28989   2.162  0.03456 *  poly(data5$X2, deg = 5)2  1.70311    0.30479   5.588 5.69e-07 ***poly(data5$X2, deg = 5)3 -1.60001    0.29074  -5.503 7.84e-07 ***poly(data5$X2, deg = 5)4 -0.83690    0.28945  -2.891  0.00531 ** poly(data5$X2, deg = 5)5  0.88545    0.29321   3.020  0.00369 ** ---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1(Dispersion parameter for gaussian family taken to be 0.07885262)    Null deviance: 16.00  on 71  degrees of freedomResidual deviance:  4.81  on 61  degrees of freedomAIC: 33.498Number of Fisher Scoring iterations: 2probability_pred_5d = predict(glm_5d,gr,type=”response”)class_pred_5d = as.factor(ifelse(probability_pred_5d<=0.5, “0”, “1”))color_array_5d <- c(“red”, “blue”)[as.numeric(class_pred_5d)]plot(gr,col=color_array_5d,pch=20,cex=0.25)
The 5th degree polynomials do not improve the performance.
In summary, let’s compare the models compared in terms of bias and variance tradeoff.
The general logistic model without interaction and higher-order terms has the lowest variance but the highest bias.
The model with the 5th order polynomial term has the highest variance and lowest bias.
The model with the 2nd order polynomial and interaction terms performs the best in terms of bias-variance tradeoff.
# let's create three additional bootstrap replicates of the original dataset and fit regression models to the replicates.Boot_sample_5f <- lapply(1:3,function(i)data5[sample(1:nrow(data5),replace = TRUE),])for (i in 1:3) { glm_5b = glm(Y~X1+X2,data=Boot_sample_5f[[i]]) probability_pred_5f = predict(glm_5b,gr,type=”response”) class_pred_5f = as.factor(ifelse(probability_pred_5f<=0.5, “0”, “1”))#plot class predictions on the grid of values for each of both linear and 5th order modelscolor_array_5f <- c(“red”, “blue”)[as.numeric(class_pred_5f)]  plot(gr,col=color_array_5f,pch=20,cex=0.25)}# the 5th order polynomial term.Boot_sample_5f_2 <- lapply(1:3,function(i)data5[sample(1:nrow(data5),replace = TRUE),])for (i in 1:3) { glm_5order = glm(Y~poly(data5$X1,deg=5)+poly(data5$X2,deg=5),data=Boot_sample_5f_2[[i]]) probability_pred_5order = predict(glm_5order,gr,type=”response”) class_pred_5order = as.factor(ifelse(probability_pred_5order<=0.5, “0”, “1”)) color_array_5order <- c(“red”, “blue”)[as.numeric(class_pred_5order)]  plot(gr,col=color_array_5order,pch=20,cex=0.25)}
From Plots 1–3, the grid was divided into two parts by a straight line, referring to a large bias and a small variance.
From Plots 4–6, we observe a reversed pattern: it has a small bias as there are more variables to minimize the distance between predictions and true values. It has a large variance because the dots distributed all over the diagram.
Medium recently evolved its Writer Partner Program, which supports ordinary writers like myself. If you are not a subscriber yet and sign up via the following link, I’ll receive a portion of the membership fees.
leihua-ye.medium.com
Please find me on LinkedIn and Youtube.
Also, check my other posts on Artificial Intelligence and Machine Learning.