The Critical Reader—Reliability

The tomato is the edible, round fleshy product of the plant Solanum lycopersicum. Botanically, it is classified as a berry-type fruit, consisting of the ovary, together with the seeds, of a flowering plant. However, culinarily, it is classified as a vegetable, with a much lower sugar content than other edible fruits. The Tariff of 1883 required a tax to be paid on imported vegetables, but not fruit. So this classification controversy had important tax consequences. Ultimately, the United States Supreme Court declared that the tomato is a vegetable, based on the popular definition that classifies vegetables by use: Nix v. Hedden (149 US 304); that is, tomatoes are served with dinner and not dessert. Consider one seemingly less controversial feature of the tomato: color. Different pigments in tomatoes produce different balances of sugars and acids. I propose 2 tomato hypotheses: red tomatoes taste balanced (sweet, sour, and savory); green tomatoes are sour to the point of astringency. To test these hypotheses, I first need to consistently classify tomatoes by color into a red group and a green group. The classification of most tomatoes into red or green groups works out well in normal light. However, I note that I am unable to classify tomatoes that are yellow. I similarly note that some heirloom varieties are orange and even purple! Should I add more colors to the classification system? Further, some tomatoes are partly red and partly green. Should they be classified based on the predominant color, or should they be considered green until 90% red? It is easy to see how the simple classification system could be weak because of classes that are not always mutually exclusive, a high frequency of borderline cases without guidance on how to classify, or the presence of cases that cannot be classified. Further, there could be challenges in the assessment. In this example, classifying color in dim light could be challenging. Alternatively, some raters could be red-green color blind. In sum, can we reliably classify tomatoes by color? Please imagine what happens to the research on the taste of tomatoes if the tomatoes are poorly classified. Specifically, consider the effects on the research if the tomatoes that were classified as ‘‘red’’ were, in fact, actually equally red and green. Similarly, the tomatoes that were classified as ‘‘green’’ were actually also red and green. Even if there was a true underlying correlation between color and flavor, could an association be detected? No. If some other feature (such as tomato size or foliage, which are both known to be associated with flavor) systematically influenced misclassification, could a false association between color and flavor be detected? Yes. Similarly, clinical research is not useful unless you can properly characterize the patients who are studied. Suppose you read an article describing impressive results from a certain technique of articular cartilage repair. Should an orthopaedic surgeon use that surgical technique for a specific patient in the clinic with an articular cartilage defect seen on magnetic resonance imaging? Are the clinical features of this patient similar to the features of the patients who had a good outcome in the study? Can the classification of these features be considered reliable? How can the critical reader apply the principles of epidemiology and statistics to help guide this decision? The next 4 paragraphs briefly address what reliability is and what reliability is not. The subsequent paragraphs provide practical guidance for interpreting reliability including some statistical definitions, examples, and rules of thumb related to the magnitude of the reliability coefficient. Reliability in research is the consistency or repeatability of measures. It is a fundamental way to reflect the amount of error, both random and systematic, inherent in any measurement. ‘‘Our comfort resides in the knowledge that the error of measurement is a relatively small fraction of the range of observations. ’’ Consider a scenario in which the operating room team—the orthopaedic surgeon, physician assistant, resident, anesthesiologist, nurse anesthetist, circulating nurse, and scrub technician—all used local solar time as interpreted on a sundial rather than official clock time on a watch, clock, or cell phone display. How many minutes of uncertainty can be tolerated? On a cloudy day, imagine the disagreement about the start of a day in the operating room as everyone arrives at different times, but every individual person thinks that he or she arrives at 6 am. How would you measure time elapsed during a case with unreliable start and finish times? Is that not what happens when we measure patient improvement with certain untested scales? In medicine, there seem to be a lot of sundials read on cloudy days (and even at night time). It is inherent to the field. In fact, Sir William Osler stated, ‘‘Medicine is a science of uncertainty and an art of probability. ’’ To appreciate statistics related to reliability, a few terms should be reviewed. Variance measures how much a set of numbers is spread out. Variance is the average of the squared deviations from the mean. There is never negative variance. If variance is zero, then all of the values are identical. If variance is small, then data values are close to the mean. If variance is large, then data values are spread out around the mean. The standard deviation, represented by the Greek letter sigma (s), is the square root of variance. Reliability is the ratio of between-participant variability to total variability. The reliability coefficient is defined as s(patients) divided by [s(patients) 1 s(error)]. If the