Selecting the Best Predictors for Linear Regression in R | by Atinakarim | Towards Data Science

To get the best fit for a multiple regression model, it is important to include the most significant subset of predictors from the dataset. However, it can be quite challenging to understand which predictors, among a large set of predictors, have a significant influence on our target variable. This can be particularly cumbersome given that the p-value for each variable is adjusted for the other terms in the model.
In this post, I will demonstrate how to use R’s leaps package to get the best possible regression model.
Leaps is a regression subset selection tool that performs an exhaustive search to determine the most influential predictors for our model(Lumley, 2020). The best predictors are selected by evaluating the combination that leads to the best adjusted r2 and Mallow’s CP.
We will use the regsubsets() function on Cortez and Morais’ 2007 forest fire dataset, to predict the size of the burned area(ha) in Montesinho Natural Park in Portugal.
The forest fire dataset contains information on 517 forest fires in Montesinho Natural Park in Portugal. There are 12 attributes in the dataset.
During EDA, I noticed multicollinearity among several dependent variables, hence this dataset is a good candidate for feature selection.
The original dataset was also transformed to fulfill the assumptions of linear regression prior to modeling. Additional dummy variables were also added because we were interested in looking at temporal interactions.
Now, that we have our dataset ready let’s start modeling.
1.Perform Linear Regression with All Predictors
Before selecting the best subset of predictors for our regression, let’s run a simple linear regression on our dataset with all predictors to set the base adjusted r2 for comparison.
lm1 <- lm(fires.clean1,formula=area ~.)summary(lm1)
We can see that with all of our variables included in the model, the base adjusted r2 is 0.02 and the Residual Standard Error is 0.6454. However, this relationship is not significant at the P<.05 level.
2. Install and load the leaps library
After installing the leaps package load, the library using the following command:
library(leaps)
3. Regsubsets()
Run the regsubsets() function on all variables.
Best_Subset <-    regsubsets(area~.,               data =fires.clean1,               nbest = 1,      # 1 best model for each number of predictors               nvmax = NULL,    # NULL for no limit on number of variables               force.in = NULL, force.out = NULL,               method = "exhaustive")summary_best_subset <- summary(regsubsets.out)as.data.frame(summary_best_subset$outmat)
4. Number of Predictors
Now, that we have run leaps through our dataset, let’s see what the package recommends in terms of the number of predictors to use for our dataset.
which.max(summary_best_subset$adjr2)
Seems like we have to use 13 predictors to get the best model.
5. What are the best predictors?
Summary table below provides details on which predictors to use for the model. The best predictors are indicated by ‘TRUE’.
summary_best_subset$which[13,]
It looks like including only the following predictors will give us the best model fit for our linear regression model : day.thu, month.aug, month.dec, month.jan, month.jul, month.jun, month.mar, month.oct, month.sep, X, DMC, temp and RH
6. Run the regression model with the best predictors
best.model <- lm(area ~ day.thu + month.aug + month.dec + month.jan + month.jul+ month.jun+month.mar+month.oct+month.sep+X+Y+DMC+temp+RH, data = fires.clean1)summary(best.model)```
We can see that compared to our model in Step 1, our adjusted r2 has improved significantly (from 0.02 to 0.05) and is significant.
The next steps would be to further evaluate the model by looking at its diagnostic plots. If the data set was split into a train and test set, we could also use the predict() function to run this model on our test dataset and subsequently, evaluate those results to fully understand our model performance.
One thing to note would be that — our adjusted r2 is still very low. The best model only explains 5% variability in the data. This may imply that we need to transform our dataset further — or try different methods of transformation. It can also imply that maybe our dataset isn’t the best candidate for linear regression.
The purpose of this post was to demonstrate how to perform variable selection for linear regression models using the leaps package. Comments and suggestions on the method or alternative (superior) methods for variable selection are welcome. Please check out the resources below to learn more about variable selection using leaps.
(2018, April 27). Retrieved May 23, 2021, from https://www.youtube.com/watch?v=3HKMjEK02Cs
Lumley, Thomas (2020, Jan 16). Package Leaps. https://cran.r-project.org/web/packages/leaps/leaps.pdf
Kassambara (2018, November 3). Best Subsets Regression Essentials in R. Statistical tools for high-throughput data analysis. http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/155-best-subsets-regression-essentials-in-r/
All subset regression with leaps, bestglm, glmulti, and meifly. Retrieved May 23, 2021 from https://rstudio-pubs-static.s3.amazonaws.com/2897_9220b21cfc0c43a396ff9abf122bb351.html
Minitab (2017, June 29). Cp and Cpk: Two Process Perspectives, One Process Reality. Minitab. https://blog.minitab.com/en/statistics-and-quality-data-analysis/cp-and-cpk-two-process-perspectives-one-process-reality#:~:text=Cp%20is%20a%20ratio%20of,indicate%20a%20more%20capable%20process.&text=When%20the%20specification%20spread%20is%20less,process%20spread,%20Cp%20is%20low.
Glen, Stephanie (2021). Adjusted R2 / Adjusted R-Squared: What is it used for. StatisticsHowTo.com: Elementary Statistics for the rest of us! https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/adjusted-r2/
P. Cortez and A. Morais (2007, December 1). A Data Mining Approach to Predict Forest Fires using Meteorological Data. In Proceedings of the 13th EPIA 2007 — Portuguese Conference on Artificial Intelligence, December, 2007. (http://www.dsi.uminho.pt/~pcortez/fires.pdf)
Zach (2020, January 08). The Four Assumptions of Linear Regression. Statology. https://www.statology.org/linear-regression-assumptions/https://www.statology.org/linear-regression-assumptions/