The Critical Reader—Magnitude and Strength of Findings

In Major League Baseball, left-handed batters on average have hit better than right-handed batters. In an analysis of the 2000 to 2007 seasons, the batting average difference was about 0.007 (7 points)—0.270 for lefties and 0.263 for righties. These batting averages were calculated from over 1.3 million at bats, and the difference is statistically significant. In this example, the batting average differences between left-handed and right-handed batters likely represent true differences and not random variation. There are several reasons for this advantage, including more offhanded match-ups (with a better view of the ball and a better chance at hitting breaking pitches) and larger gaps for ground balls on the pull side with a runner on first base. Based on the results from this extensive baseball study, with a statistically significant difference and a plausible explanation, should a right-handed batter switch to batting left-handed midcareer? To answer this question, I recommend that the reader consider the magnitude and strength of the findings. After all, 7 points on a 1000-point scale is less than 1%. Daily and weekly batting averages often fluctuate more than this amount. During the preseason, expert baseball analysts usually cannot predict a player’s final batting average within 7 points. In contrast, if the absolute difference was 15%, or 150 points, then I would consider switching my son this weekend. That is the difference between marginal talent and hall of fame potential. Similarly, when discovering a finding from a high-quality study with a statistically significant difference and biologically plausible reasoning, should an orthopaedic surgeon change a surgical technique, such as anterior cruciate ligament (ACL) reconstruction? How can the critical reader apply principles of epidemiology and statistics to help guide this decision? The next 2 paragraphs briefly address 2 criteria—validity and statistical significance—to assess prior to consideration of clinical significance. The subsequent paragraphs provide practical guidance for interpreting clinical significance, including some definitions, examples, and rules of thumb related to the magnitude of effect and the strength of a positive relationship. Before the relevance of a clinical finding is considered, the reader must first assess the validity of the study. Internal validity is the extent to which a study was properly conducted without major methodological problems and without bias (systematic error). Confounding, due to extra variables correlated to both predictors and outcomes, must be identified and taken into account while interpreting results. Consideration of age as a confounder is particularly important in orthopaedic surgery research. External validity is the extent to which results of a study provide a correct basis for generalization to other circumstances. Specifically, the population that was studied should be similar to the population that the reader is treating. Bias, confounding, and generalizability will be addressed in greater detail in a subsequent dedicated Critical Reader editorial. Next, the reader should evaluate the statistical significance of the findings. Studies cannot include entire populations and continue indefinitely in time. Some chance factor may result in study outcomes not representing the ultimate true values, even if bias and confounding are nonexistent. ‘‘P \ . 05’’ means that the probability that a finding could have arisen by chance alone is less than . 05 (or 1 in 20), assuming the null hypothesis is true. Fisher popularized this standard in the 1920s ‘‘to eliminate from further discussion the greater part of the fluctuations which chance causes. ’’ This cutoff has been sanctified by many years of use. Finally, if the study is considered to be valid and the findings are statistically significant, then the reader can and should assess the clinical relevance of the findings. Small differences may be statistically significant yet clinically unimportant. The type of data and the statistical comparisons dictate how the magnitude of the outcomes should be interpreted. Comparisons involving continuous outcomes are interpreted differently than comparisons involving dichotomous outcomes. For comparisons that involve continuous outcomes, many patient-oriented outcome measures have a validated value of a minimal clinically meaningful difference, which is a start. For example, with respect to the International Knee Documentation Committee (IKDC) Subjective Knee Form score, a change of 11.5 points is considered to be necessary to distinguish between those who have improved and those who have not improved. However, in many cases, these threshold values are unknown, and the reader must use some appreciation of statistical terms and perhaps a rule of thumb to complement clinical judgment and experience. For all continuous outcomes, an estimate for the size of the intervention effect is the difference in means or medians between the 2 groups. If there is no effect, then the value of this difference is 0. Cohen developed a method for calculating effect size by taking this difference between group scores and dividing it by the average of the standard deviation for both groups. Cohen classified effect size as follows: 0.2 is a small effect size, 0.5 is a moderate effect size, and 0.8 is a large effect size.