How to Identify Unusual Observations on Your Regression Model with R? | by Michaelino Mervisiano | Towards Data Science

In this article, I will discuss a few methods on how to detect unusual observations in regression analysis. As an analyst, you might to consider screening for unusual observations, it will help you to get a comprehensive regression model. The unusual observations require further investigation, either because they are different compared to other observations, or because they bring a disproportionate amount of influence to the outcomes of the model.
There are three types of unusual observations that I will review:
· Outliers
· High-leverage observations
· Influential observations
For illustration, I will use R sample data (state.x77). This data set give us information such as population size, average income, illiteracy rate, temperature, and murder rate for 50 states in USA.
Using code below, I would like to build a regression model that can predict the murder rate in certain states based on their population, income, illiteracy rate, and temperature. Also, we will apply car package for our data wrangling.
# How to Identify Unusual Observations in Regression?# Created by Michaelino Mervisianoinstall.packages(“car”)library(“car”)states <- as.data.frame(state.x77[,c(“Murder”, “Population”,”Illiteracy”, “Income”, “Frost”)])View(states) fit <- lm(Murder ~ Population + Illiteracy + Income + Frost, data=states)summary(fit)
From summary function, we find that our model is indeed significant with p-value of F-statistic <0.05. From t-test, we also spot that both illiteracy rate and population size are the significant on predict murder rate in a state. Next, we will investigate if we have unusual observations in our model
> summary(fit)Call:lm(formula = Murder ~ Population + Illiteracy + Income + Frost,  data = states)Residuals: Min 1Q Median 3Q Max -4.7960 -1.6495 -0.0811 1.4815 7.6210Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1.235e+00 3.866e+00 0.319 0.7510 Population 2.237e-04 9.052e-05 2.471 0.0173 * Illiteracy 4.143e+00 8.744e-01 4.738 2.19e-05 ***Income 6.442e-05 6.837e-04 0.094 0.9253 Frost 5.813e-04 1.005e-02 0.058 0.9541  — -Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 2.535 on 45 degrees of freedomMultiple R-squared: 0.567, Adjusted R-squared: 0.5285 F-statistic: 14.73 on 4 and 45 DF, p-value: 9.133e-08
Outliers
Outliers observations aren’t predicted well by regression model. They either have extremely large positive or negative residuals. If the model is underestimating the response value, then it will be indicated by positive residual. On the other hand, if the model is overestimating the response value, then it will be indicated by negative residual.
From our regression model example, we can start investigating outliers observation by using Q-Q plot. Code below helps you to plot and find potential outliers. Nevada (28th observation) and Rhode Island (39th observation) are states that detected as potential outliers.
> qqPlot(fit,labels=row.names(states), id.method=”identify”,         simulate=TRUE, main=”Q-Q Plot”) Nevada Rhode Island  28          39
Applying outlierTest function is helping us to confirm if potential outliers are indeed outliers. The statistical test is showing that Nevada undeniably detected as an outlier with p-value = 0.048.
> outlierTest(fit)       rstudent unadjusted p-value Bonferroni pNevada 3.542929         0.00095088     0.047544
Q-Q plot also confirms that Nevada has a large positive residual. It means our model underestimates the murder rate in this state. You can spot that the murder rate in Nevada is 11.5% while the model only predicts at 3.9%. We clearly understand that Nevada is indeed different compared to other states.
> states[“Nevada”,]        Murder Population Illiteracy Income FrostNevada   11.5     590        0.5      5149   188> fitted(fit)[“Nevada”] Nevada 3.878958
Therefore, you can remove this observation and re-run the test to see if other outliers are present. As an analyst, you might want to know why Nevada has a high murder rate. Perhaps you can ask post this question to Quora and let me know if you have the answer 😉
High-leverage points
Observations will be considered as high-leverage points if they resemble outliers when we compared it to other predictors. Strictly speaking, they have uncommon combination of predictor values while the response value has minor impact on determining leverage.
You can compute the high leverage observation by looking at the ratio of number of parameters estimated in model and sample size. If an observation has a ratio greater than 2 -3 times the average ratio, then the observation considers as high-leverage points. I personally like to use this simple function to identify high-leverage observations.
highleverage <- function(fit) { p <- length(coefficients(fit)) n <- length(fitted(fit)) ratio <-p/n plot(hatvalues(fit), main=”Index Plot of Ratio”) abline(h=c(2,3)*ratio, col=”red”, lty=2) identify(1:n, hatvalues(fit), names(hatvalues(fit)))}highleverage(fit)
The graph above indicates that Alaska and California have the highest ratio. It means these two states are quite unusual if we compared their predictor values compared to other states. Here is why:
· For Alaska, this state has a much higher income compared to other states, while having smaller population and lower temperature.
· For California, this state has much a bigger population than other states, while having higher income and higher temperature.
Both of these states are indeed quite particular compared to the other 48 states. One thing to remember that not all high-leverage observations are influential observations. It will depend if they are outliers or not.
Influential Observations
Observations with a disproportionate impact on the values of the model parameters can be considered as Influential observations. For instance, if you drop a single observation, then it change your model dramatically. That single observation must be influential observation. Thus, it is important for us to inspect if our data contains influential points or not.
In my experience, there are two effective methods to identify influential observations:
1. Cook’s distance
Code below provide a way to calculate the cut-off and plot Cook’s distance for each of our observation. The plot shows Alaska, Hawaii, and Nevada as influential observations. Removing these 3 states will have a significant impact on the values of the intercept and slope in our regression model.
cutoff <- 4/(nrow(states)-length(fit$coefficients)-2)plot(fit, which=4, cook.levels=cutoff)abline(h=cutoff, lty=2, col=”red”)
I need to highlight that the plot above is indeed helpful to identify influential observations, but it doesn’t give extra information about how these observations affect the model. That’s why Added Variable plots can be an option for us to dig more information.
2. Added Variable plots
Added variable plots works by plotting each predictor variable against the response variable and other predictors. It can be easily created using the avPlots() function in car package.
library(car) avPlots(fit, ask=FALSE, id.method=”identify”)
From the graph below, the straight line in each plot is the actual regression coefficient for that predictor variable. We might view the impact of influential observations by imagining how the line would alter if the point representing that observation was removed.
Now, we will incorporate all information from outlier, high-leverage, and influential observations into a single informative plot. I personally find influencePlot() is a very handy function to represent these unusual observation issues.
library(car) influencePlot(fit, id.method=”identify”, main=”Influence Plot”, sub=”Circle size is proportional to Cook’s distance”)
The graph above summarises our final verdict conveniently. Here it is:
· Nevada and Rhode island are Outliers.
· California, Washington, New York, and Hawaii are High-Leverage Observations
· Alaska is the Influential Observation
I hope you find this article is useful and kindly share it with others