On the hazards of seeing the world through intervention-colored glasses

Producing unbiased interpretations of truth can be a difficult challenge in behavioral intervention studies because the amount of behavioral change expected is often small. Having been repeatedly told what behavior change is expected, subjects may display the natural human tendency to perceive and report their behavior biased toward the expected change. Having put years of hard work into an intervention trial, study investigators also have a natural human tendency to interpret trial results in the most positive light. We need to find better ways to guard against both types of bias if we are to make true progress in the field of behavioral intervention research. The challenge for both the subject and the investigator is formidable when diet change is the behavioral target in a trial. Considerable controversy surrounds the question of the validity of self-reports of diet. Despite errors in all methods of selfreport, it seems clear that we can generate a useful estimate of diet via various methods of asking people what they eat (1). Misclassification of persons can be tolerated if errors in diet reports are random, which has implications only for power and sample size. However, we can get wrong answers if diet reports are biased by the principal factor under study. This is a particular risk in the context of nutritional intervention studies, in which subjects receive repeated messages about the desirability of their dietary change. The findings of the Pathways Study that are reported by Caballero et al (2) in this issue of the Journal raise questions about the difficulty in separating biased reporting from true intervention effects in behavioral intervention trials. The Pathways Study was a very ambitious and important 3-y trial that targeted changes in both nutrition and physical activity in 41 elementary schools serving Native American children. The 24-h dietary recalls suggested that the children in the intervention schools consumed 12% fewer calories per day than did the children in the control schools. Self-reported activity measures suggested that the children in the intervention schools expended 12% more activity than did the children in the control schools. Despite this apparent improvement in caloric balance, body fat was unchanged by the intervention. How are we to best interpret these findings? Caballero et al mention reporting bias as a possibility, but they also speculate about other possibilities and conclude that a greater amount of change is needed to have an effect on body fat. Just how much change the Pathways intervention really created is an important unanswered question, the answer to which will influence estimates of how much change is necessary to yield a meaningful effect from school policy. Both subjects and investigators in other studies have had difficulty separating biased reports from true effects. The 5-ADay Study reported to have increased fruit and vegetable intakes in fourth-grade students by 74% (3). However, the direct observation of diet behavior in the lunchroom indicated no intervention effect, even though the 24-h dietary recalls of lunchroom eating collected the very next day indicated an intervention effect. It seems likely that the children in that trial had recalled their diet on the previous day in a biased way, yet the investigators interpreted the intervention as having created a true diet change. On the basis of self-reports that estimated a 55% increase in fruit and vegetable consumption among the intervention group, the Polyp Prevention Trial investigators concluded that they were successful in achieving dietary change, even though serum carotenoid concentrations increased by only 1% (4). Whether diet really changed as much as the self-reports suggested in the Pathways Study, the 5-ADay Study, or the Polyp Prevention Trial is information that is critical to our understanding of the relations between diet change and the important outcomes studied in those trials. The consequence of reporting bias in intervention trials is that we can misinterpret the amount of behavioral change that is needed to produce the desired health outcome. More objective measures of dietary change should be built into future intervention studies. It can be difficult and costly to collect nutritional biomarkers or to assess diet in a way that incorporates reports from both subjects and other independent observers, but such efforts may be necessary in the future. I am not saying that we should scrap behavioral intervention trials, that self-reports of diet cannot be useful, or that investigators should not look for silver linings in the disappointing dark clouds of trial results. However, we need to be much more wary of the hazards inherent in the way both subjects and investigators perceive and report behavioral changes in the context of behavioral intervention trials.