Lady tasting tea

**Lady tasting tea** is the name of a famous randomized experiment designed by Ronald Fisher in 1935. The experiment is the original exposition of Fisher's notion of a null hypothesis. Fisher's description is less than ten pages long and is notable for its simplicity and completeness regarding terminology, calculations and design of the experiment. The example is loosely based on an event in Fisher's life. The lady in question was Muriel Bristol, and the test used was Fisher's exact test.

## The experiment

The lady in question claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups, four of each variety, in random order. One could then ask what the probability was for her getting the number she got correct, but just by chance.

- The experiment provided the Lady with eight randomly ordered cups of tea – four prepared by first adding milk, the others prepared by first adding the tea.  She was to select the  four cups prepared by one method.
    - The Lady could compare the taste of the cups
    - The Lady was fully informed of the experimental method.
- The null hypothesis was that the Lady had no such ability.
    - Note that in Fisher's approach, there is no alternative hypothesis; this is instead a feature of the Neyman–Pearson approach.
- The test statistic was a simple count of the number of successes in selecting the four cups.
- The null hypothesis distribution was computed by the number of permutations.  The number of selected permutations and the number of unselected permutations were equal.

|scope="col"|Success count|scope="col"|Permutations of selection|scope="col"|Number of permutations|
| --- | --- | --- | --- | --- | --- |
|0|oooo|1 × 1 = 1||||
|1|ooox, ooxo, oxoo, xooo|4 × 4 = 16||||
|2|ooxx, oxox, oxxo, xoxo, xxoo, xoox|6 × 6 = 36||||
|3|oxxx, xoxx, xxox, xxxo|4 × 4 = 16||||
|4|xxxx|1 × 1 = 1||||
||Total|70||||
- The critical region was the single case of four successes of four possible based on a conventional probability criterion (< 5%; 1 of 70 ≈ 1.4%).

If and only if the Lady properly categorized all eight cups was Fisher willing to reject the null hypothesis – effectively acknowledging the Lady's ability at a 1.4% significance level (but without quantifying her ability).  Fisher later discussed the benefits of more trials and repeated tests.

David Salsburg reports that a colleague of Fisher, H. Fairfield Smith, revealed that in the test, the woman got all eight cups correct. The chance of someone who just guesses getting all correct, assuming she guesses that four had the tea put in first and four the milk, would be only 1 in 70 (the combinations of 8 taken 4 at a time).

In popular science, Salsburg published a book entitled _The Lady Tasting Tea_, which describes Fisher's experiment and ideas on randomization. Deb Basu wrote that "the famous case of the 'lady tasting tea'" was "one of the two supporting pillars . . . of the randomization analysis of experimental data".

## References

- Basu D. (1980b). "The Fisher Randomization Test", reprinted with a new preface in _Statistical Information and Likelihood : A Collection of Critical Essays by Dr. D. Basu_ ; J.K. Ghosh, editor. Springer 1988.
- Salsburg D. (2002) _The Lady Tasting Tea: how statistics revolutionized science in the Twentieth Century_ W.H. Freeman / Owl Book.