One of the most common mistakes when running an ANOVA in R | by Matthieu Renard | Towards Data Science

Imagine this scenario: You are running an ANOVA in R while controlling for other variables (technically, an ANCOVA). However, the results seem weird: The order of the independent variables matter and influence the results — how can that be? And if you try to enter the same model as a linear regression, the results are also totally different. Again, how can that be?
This article aims to shed light on this common issue. First, let’s start by recreating this issue. If you are looking for a more general introduction to ANOVAs, check out my article on that topic.
We will recreate this issue using the salaries dataset included in base R. It contains data on professors' salaries and other variables that would intuitively influence professors’ salaries, such as discipline and seniority. You can access it using the following command.
df=Salaries
Now, let’s say we’re interested in whether a professor’s sex influences their salary when controlling for other intuitively important independent variables. One could analyze these differences using an ANOVA. Therefore, let’s analyze the differences between the two groups in the dataset with an ANOVA.
aov1.1=aov(salary~sex+rank+yrs.since.phd+yrs.service+discipline,df)summary(aov1.1)
This ANOVA seems to imply that sex has a significant impact on a professor’s salary. However, if you run the same analysis as a linear regression model, it leads to a different conclusion.
lm1.1=lm(salary~sex+rank+yrs.since.phd+yrs.service+discipline,df)summary(lm1.1)
Here, the impact of sex on a professor’s salary seems insignificant. So what is it? Does sex influence a professor’s salary or not? Another strange thing appears: If you rerun the ANOVA 1.1 from before but change the independent variables' order, it also shows different results. In the below example, we changed the position of the independent variables of sex and rank.
aov1.2=aov(salary~rank+sex+yrs.since.phd+yrs.service+discipline,df)summary(aov1.2)
Again, different results. Now we have three sets of results and ultimate confusion. Sex is insignificant as for the regression, but, e.g., yrs.since.phd is also insignificant, which is different from the regression results. What is going on? Which independent variable influences a professor’s salary? Why are all results different?
The reason is Types of Sum of Squares. It describes how R incorporates the error term into the regression model, and different ways of incorporating errors lead to differently fitted models. By default, R uses Type I (“one”) Sum of Squares for ANOVAs — Type 1 is perfectly fine for an ANOVA with only one independent variable. However, when you add more independent variables (technically an ANCOVA), Type I will produce incorrect results. For regressions (when using the lm() function), R uses Type III (“three”) Sum of Squares by default.
In the above example, all the ANOVAs were wrong, and only the lm() function produced correct results. While I will not explain the differences between different Types of Sums of Squares, it will suffice to say that you should almost always be using Type III Sum of Squares. If you wish to find out more about the different Types of Sums of Squares, check out this page in Andy Field’s book “Discovering Statistics Using R.”
So, if we are running an ANOVA with more than one independent variable (technically an ANCOVA), we need to override R's default settings and tell it that it has to use Type III Sum of Squares.
As the aov() function doesn’t allow us to modify the Type of Sum of Squares, we need to use another ANOVA function from a different package to specify this. We will be using the car package. Run the below command to install the car package if you haven’t already installed it.
install.packages("car")
Then load the package.
library(car)
Then, run the car Anova command on our original ANOVA object (aov1.1), specifying that we wish to use Type III errors.
Anova(aov1.1, type=”III”)
Now looking at the results, you can see that they are consistent with the regression output.