A Discussion of Low R-Squares: Concerns and Uses (2000)

A Discussion of Low R-Squares:Concerns and Uses* focus ofthis discussion is to emphasize that small R^sup 2^s have value and should not be dismissed without consideration. This is not to say that I disagree with Dr. McNeil's position that a large RZ is preferable, however, when only small R^sup 2^s are obtainable, they should not be summarily dismissed. To quote McNeil (1999), The recent debate attempting to replace statistical hypothesis testing (probability) with the amount of variance accounted for seems mis-specified. Both are needed and their relative importance depends on the stage of the research (p. 3). This statement is equally valid when small R^sup 2^s are appropriate, such as in the early stages of research. McNeil (1999), Knapp (1999), and Knapp and Tam (1997) make the point that, depending upon your question, significance testing may be meaningful regardless of the effect size. Smilarly, Rosenthal and Rosnow (1991) discuss predictive efficiency, which is defined as the ability to reduce the error in prediction. This is also very similar to the index of forecasting (explained variance, R^sup 2^). However, predictive efficiency has been shown to have relatively kttle practical value in the social sciences, since many of the predictor variables tend to have small effects. Also, in the social sciences we do not expect to be able to obtain a complete explanation of all the variance due to a specific outcome. In the soft sciences we talk about hypotheses in the if.. form, as a causal explanation. Assuming a true experimental design, one rarely if ever sees an R^sup 2^ of 1. When we get statistical significance, since the correlation is not 1, we cannot say, A... then B. What should be said is, A... then a certain proportion of is accounted for. If this data is replicable and if the proportion of B is consistently reliable, most researchers would be extremely happy, even if the R^sup 2^ is not one. That is, low predictor efficiency can yield huge effects if replicated over time. Rosenthal and Rosnow (1991) discuss an example of a major breakthrough in recent medicine, a drug that decreases the likelihood of transplant rejections. effect size is approximately 1%. By all accounts this would be considered a small effect, however the lives of approximately 10,000 people a year can be saved because of it. Personnel psychologists have known the effectiveness and usefulness of small R^sup 2^s for a long time. When developing cutoff guidelines, if they are able to select a cut score that predicts better than chance, they can improve the odds of hiring a successful employee, even if the effect size is small. This can save a company a fortune. For example, if 60% of individuals hired for a job are likely to be successful (60:40 ratio), a test with predictive validity of .3 (which is not considered high) will increase that ratio to 69:31. If the test had a validity of .45, which is still considered a small effect by many, the odds ratio would change to 74:26. That is, an R^sup 2^ of .20, which is relatively modest and not considered a large effect, increases the accuracy of hiring a successful employee by 35%. Another way of saying this is that by having a test with a validity coefficient of .45 we double the odds ratio from 60:40 to 74:26) of success. Think of what that means in terms of cost effectiveness to a company. (See a discussion of utility analysis in personnel selection.) As Rosenthal and Rosnow (1991) point out, a correlation as small as .2 will increase predictive efficiency 20% over no information. An increase of a validity coefficient from .3 to .5 will increase the predictive efficiency 20% over the .3. Table 1 (which is a summary of Table 14.7 in Rosenthal & Rosnow) presents the binomial effect size display (BESD). This table illustrates how even very small increases in treatment effectiveness, which are reflected by small is, can significantly impact the results. …