Not the last word: Harvard beats Yale and other fallacies.

One of the great academic rivalries in the United States is between the nation’s oldest institution of higher learning, Harvard University, and its New England neighbor to the south, Yale University. Both of these Ivy League schools typically stand near the top of national rankings, and jostling for the number one spot is a favored pastime among many of their alumni. When the 1968 Harvard versus Yale football game ended in a draw, for example, the Harvard student newspaper published the headline, “Harvard beats Yale 29-29.” Both Harvard and Yale boast top notch orthopaedic surgery departments, yet the orthopaedic residency at Yale is more likely to come up short on one Residency Review Committee (RRC) guideline for program evaluation, namely, that “a board certification rate [of its graduates] greater than 75% is required to maintain Residency Review Committee Accreditation” [4]. The assertion that Yale is more likely to violate this 75% RRC standard says nothing about the superior didactic programs or clinical experiences at Harvard. Rather, it relies on a long-known, but frequently forgotten mathematical phenomenon: variance increases with decreasing sample size. Therefore, a smaller program is more likely to be found at the extremes of performance, both good and bad. The larger a program is, the more it is protected against low-probability events (such has having too many recent graduates fail to pass their boards). The residency program at Harvard graduates 12 residents per year, and the program at Yale graduates five. Advantage: Harvard. Here’s why. Assuming that every graduate has a p chance of passing the boards examination, one can employ the binomial distribution to determine the chance that there will be at least m students out of n who pass in a given year [1]. For the Harvard program, if each of the 12 residents has an 80% chance of passing, there is 79.4% chance that in a given year at least nine will pass the exam. For the Yale program, with five residents with that same individual success rate of 80%, there is only a 73.7% chance that it will avoid sanction in a given year. For a 4-year average (48 versus 20 residents), the difference is even greater. With an 80% individual success rate, there is less than 1% chance that fewer than 32 of 48 Harvard graduates will pass. The probability that fewer than 15 out of 20 from Yale will pass is 20%. Under the current rules, we could easily imagine a report from the Residency Review Committee headlined “Harvard Beats Yale 80% to 80%.” This example illustrates the ease with which one can be misled when drawing inferences from extreme performance without adequately accounting for sample size differences. Wainer and Zwerling [13] provide a few additional examples. For instance, they found that the lowest rates of kidney cancer are found in rural areas, but caution against the inference that small town living is somehow healthier. That is because they also found that other small towns had the highest rates of cancer. Indeed, small areas are likely to be found at both extremes: a village with 100 inhabitants and no cancer will have the lowest rate, whereas a village of 100 inhabitants and but a single case of cancer will have a rate among the highest. One additional (or one fewer) case in a large city, by contrast, imposes a trivial effect on incidence. To appreciate the powerful influence sample size has on variance, and in turn, the probability of witnessing a rare event, imagine that you are administering Part II of the American Board of Orthopaedic Surgery exam. A candidate has presented his experience of 30 total hip replacements. You come to learn that three of these arthroplasties, 10%, have dislocated. You also recall that the literature reports a dislocation rate of 3.2%. Should you condemn the work of a surgeon whose hips dislocate at more than triple the expected number? At first glance, the 10% rate does seem to be shoddy work, yet the binomial distribution preaches caution. If the examinee’s true rate of a dislocation is 3.2% (ie, the rate that would be seen after a very large number of cases), then in the small sample of thirty cases there is a 37% chance that there will be no dislocations, a 37% chance that there will be exactly one, a 17% chance that there will be two, and 6% chance that there will be three. Further, assuming that an event with a 6% probability should be discounted as noise (a fair interpretation of the “p > 0.05” criterion for statistical significance), this candidate should not be censured. What is seen may just represent a statistical anomaly, owing to a small sample size. Consideration of sample size is of even greater importance as clinicians are increasingly critiqued and graded on every aspect of their performance. If the orthopaedic surgery community does not want to be held accountable by the public for random events — a 10% dislocation rate after 30 hip replacements comes to mind — then it behooves us to not hold each other accountable for random events. We can start by dropping this Residency Review Committee standard for residency performance, or at least modifying it to account for sample size. (For example, one could calculate a rolling average of its most recent 100 graduates). If we account for sample size, the department at Harvard may still one day be able to claim superiority, but not by hiding behind a statistical quirk.