Statistics Commentary Series: Commentary No. 34: The Jackknife, Bootstrapping, and Monte Carlo Methods.

S tatisticians are ecstatic when the data that they’re analyzing can be assumed to be normally distributed. They are also happy (albeit possibly a tad less so) when the data follow other known distributions, such as the Poisson, negative binomial, hypergeometric, or some other one. The reason is that the formulae for these distributions are known and so we can model what the data should look like and thus use tests of significance. But nature isn’t always kind to us, and there are times when the data cannot easily be described by some formula, or when the assumptions of parametric tests (ie, those where we know the parameters of the distribution) aren’t fully met. What are statisticians to do in these circumstances? Not wanting to be foiled by such piddling constraints as assumptions, they have devised a series of techniques that do not require that we know the distribution, or that the distribution meets all of the assumptions. As with most things in statistics, the path was first blazed by Sir Ronald Fisher. Although the chisquared (χ) test is often described as non-parametric, this isn’t quite accurate. Because the data in the cells are counts, they are assumed to follow a Poisson distribution, which they tend to do as long as the expected frequency in each cell is at least 5. If there aren’t at least 5, then we use Fisher’s exact test, which is a permutation test. This works out every possible layout of the data in a 2x2 table (ie, starting with the actual number of observations in the smallest cell, then decreasing it by 1, and then 2, and so on down to 0), and calculates the probability of each; the final value is the sum of these probabilities. In fact, we can calculate the exact test for larger tables and when the expected frequency is greater than 5, but the calculations become mind-boggling, especially in the days prior to computers and calculators with builtin factorial functions, which is why Fisher abandoned this approach. This approach is called a “resampling” method, because the same data are resampled for each calculation. The next step in the evolution of resampling statistics, the jackknife, was developed by Quenouille, who developed it to determine howmuchmultiple regression and other procedures were affected by outliers. It was given its sobriquet by John Tukey, because he saw it as an all-purpose tool that can be used for many things, much like a real jackknife. How it works is explained by its other name, leave-one-out. The data are analyzed multiple times, each time omitting 1 data point. Thus, the first time, the data from subjects 2 through n are analyzed, omitting subject 1; then subjects 1 and 3 through n are used, leaving out subject 2; and so forth. We can then determine not only the means of the estimates of, say, the coefficients of a multiple regression, but also how much the estimates vary due to outlying values. The major drawback of this approach is that given n subjects, we’re restricted to n samples from the data. This may be sufficient when the sample size is very large, but can be a problem if there are only 50 or so subjects, which can limit the accuracy of our estimates. Efron and Tibshirani avoided this limitation with a technique called bootstrapping, because we use the data to lift ourselves up by our bootstraps (not something you should try in real life). With this technique, we draw repeated samples of the same size from the datawith replacement. That is, if say subject 25 is included in the sample, her data point remains in the dataset, so it can be used again (and possibly yet again) in the same sample. That means that we can draw an unlimited number of samples from the dataset, and each wouldmost likely include different subgroups of people. Indeed, it’s not unusual to draw 1,000 or even 10,000 samples from a dataset with far fewer subjects. For example, let’s assume we do a small pilot project to determine the proportion of patients who improve after treatment with a certain drug, and we find that 12 out of 18 do so. Is it worth proceeding with a large randomized controlled study? A success rate of 66.7% looks fairly impressive, but with only 18 subjects, how confident arewe that this is an accurate estimate of the drug’s effect? Using bootstrapping, we can draw 1,000 samples of 18 people each. Because this is done with replacement, each of the people will be used many times, but the composition of each sample will change from one draw to the next. For example, in 1 sample, subject 7 may appear 130 times, but subject 11 may be included only 50 times. The next samplewould consist of a different combination of some people appearing verymany times and others being included less often. If we did this 4 times, we may find that the answers are 67.2%, 76.7%, 60.0%,