Running head : PLANNED CONTRASTS AND RESIDUAL TESTS 1 Residual tests in the analysis of planned contrasts : Problems and solutions

It is current practice that researchers testing specific, theory-driven predictions do not only use a planned contrast to model and test their hypotheses but also test the residual variance (the C+R approach). This analysis strategy relies on work by Abelson and Prentice (1997) who suggested that the result of a planned contrast needs to be interpreted in the light of the variance that is left after the variance explained by the contrast has been subtracted from the variance explained by the factors of the statistical model. Unfortunately, the C+R approach leads to six fundamental problems. In particular, the C+R approach (1) relies on the interpretation of a non-significant result as evidence for no effect, (2) neglects the impact of sample size, (3) creates problems for a priori power analyses, (4) may lead to significant effects that lack a meaningful interpretation, (5) may give rise to misinterpretations, and (6) is inconsistent with the interpretation of other statistical analyses. Given these flaws, researchers should refrain from testing the residual variance when conducting planned contrasts. Single contrasts, Bayes factors, and likelihood ratios provide reasonable alternatives that are less problematic. PLANNED CONTRASTS AND RESIDUAL TESTS 3 Residual tests in the analysis of planned contrasts: Problems and solutions Planned contrasts enable researchers to conduct tailored statistical tests of their hypotheses (e.g., Abelson, 1995; Hays, 1988; Rosenthal & Rosnow, 1985; Winer, Brown, & Michels, 1991). In contrast to the default ANOVA or GLM tests offered by many statistical software packages as standard analysis tool, planned contrasts have often a higher statistical power (e.g., Myers & Well, 1995) and allow researchers to address their hypothesis with a single test. Proponents of contrast analysis agree on the utility of the method. They disagree, however, regarding the specific analysis strategy. Some authors suggested that only a single planned contrast is needed to demonstrate whether the data support the hypothesis or not (single contrast approach; Furr, 2004; Furr & Rosenthal, 2003; Rosenthal & Rosnow, 1985). Other authors argued that a single contrast is not conclusive and that an additional, non-significant test of the residual variance is needed before one may conclude that the data convincingly support the hypothesis (C+R approach; Abelson & Prentice, 1997; Brauer & McClelland, 2005; Niedenthal, Brauer, Robin, & Innes-Ker, 2002). The disagreement on the best contrast analysis strategy is also reflected in the empirical literature. Six percent of all empirical papers that were published in 2013 in the European Journal of Social Psychology, the Journal of Experimental Psychology: General, Motivation and Emotion, the Personality and Social Psychology Bulletin, and Psychological Science used planned contrasts to model and test the predictions. Half of these papers followed the single contrast approach and conducted only the contrast test. The other half adopted the C+R approach and tested additionally the residual sum of squares. The decision between the two approaches is not trivial given that it has important implications for data interpretation. Imagine that a researcher, Dr. Mustermann, has developed a theory about the time that students spend on preparing an exam. Her theory claims that the time students invest is a function of the expected difficulty of the exam: The higher the expected difficulty, the higher the number of hours that students spend on preparing the exam. To test her theory, Dr. Mustermann runs a study in which she assesses the number of hours that students spend on preparing an easy, a PLANNED CONTRASTS AND RESIDUAL TESTS 4 moderately difficult, a difficult, and a very difficult exam. She then models her hypothesis with a planned contrast and conducts the corresponding statistical test. The test is significant. According to the single contrast approach, she would have convincingly supported her prediction. However, according to the C+R approach, she would have to conduct an additional test. She would have to subtract the sum of squares associated with her contrast from the total between-group sum of squares and test the resulting residual sum of squares. Only if this residual sum of squares–the variation explained by exam difficulty that is not captured by her contrast–does not attain statistical significance, she could claim that her data unambiguously support her hypothesis. In this paper, I will compare both contrast analysis strategies, discuss problems associated with the C+R approach, and present alternatives.