Conventional analyses of data from dietary validation studies may misestimate reporting accuracy: illustration from a study of the effect of interview modality on children's reporting accuracy.

Validation and relative validation studies of dietary-reporting methods are conducted to evaluate the extent to which those methods elicit accurate reports; this involves comparing reports obtained by such methods as 24-hour recalls and food-frequency questionnaires to reference information obtained by such methods as direct observation, duplicate portion collection, or food records. The reported and reference information each consists of a set of food items and their respective amounts. (Table 1 shows terms used in this article and their definitions.) Table 1 Terminology Assessment of reporting accuracy for a group of participants often begins with converting both reported and reference information to energy and nutrients. After cumulating, for each participant, the energy and nutrients within each set, analyses involve several descriptive measures and inferential tests: e.g., 1–14 First, paired t-tests may be used to compare the mean of the participant-specific differences between reported and reference energy and nutrients to 0. Second, the correlation, over participants, between reported and reference energy and nutrients may be calculated, with correlations near 1 presumed to indicate higher reporting accuracy. Third, the ratio of reported to reference energy and nutrients, multiplied by 100, may be calculated. We call this the report rate; others have labeled it “percentage recalled” and “percentage over- (or under-) reported”.e.g., 3,5,6 Values close to 100%, found when reported and reference energy and nutrients are approximately equal, are presumed to represent high reporting accuracy. 12–14 We call these analyses, taken together, the “conventional approaches” to evaluating reporting accuracy. These conventional approaches are generally indifferent to whether food items and amounts are reported correctly—reported information is converted to energy and nutrients regardless of whether items were actually eaten. In this article, we demonstrate that conventional analyses of data from dietary-reporting validation studies may mask the complexity of dietary reporting error and overestimate reporting accuracy. Consider, for example, mean differences between reported and reference information. For any dietary measure, the mean difference would be 0 in a validation study in which all participants reported items and amounts with perfect accuracy. However, the mean difference could also be 0 if, on average, participants reported items and amounts that they did not eat that offset their failures to report items and amounts that they did eat. Suppose that an individual was observed eating a serving of hash browns, and subsequently reported eating a serving of applesauce. Converted to energy and macronutrients, the individual was observed eating 100 kcal, 1 g protein, 12 g carbohydrate, and 5 g fat; and reported eating 96 kcal, 0.2 g protein, 25.4 g carbohydrate, and 0.2 g fat. (Composition information for a 57 g serving of hash browns is from the product label, and for a 120 ml serving of sweetened applesauce is from the Nutrition Data System for Research [NDSR; version 4.03, Nutrition Coordinating Center, University of Minnesota, Minneapolis, MN, 2000] database.) The report rates for energy (96%), protein (20%), carbohydrate (212%), and fat (4%) would be interpreted as essentially perfect reporting of energy, underreporting of protein and fat, and overreporting of carbohydrate. However, all reported energy and macronutrients were from a reported item that was, in fact, not eaten by that individual. The conventional approaches, by their indifference to whether reported items and amounts were actually eaten, may misrepresent dietary-reporting accuracy. We introduce some terminology to facilitate our discussion of this problem and of our recommendations concerning how to address it. We then present an illustration using data from a validation study of dietary recall, so frame the discussion in terms of dietary recall. However, the concerns pertain also to relative validation of food-frequency questionnaires. In dietary recall, the reported information is a set of item-amount pairs (e.g., [apple, 1 medium]; [whole milk, 1 serving]) reported by a participant. The reference information is also a set of item-amount pairs. Reporting accuracy may be evaluated by assessing the congruence between the reported and reference sets of information, which involves partitioning the sets as illustrated in the Figure and discussed here: For any participant, reported items are partitioned into matches and intrusions, and reference items are partitioned into (the same) matches and omissions.15,16 A match is a reported item that is in the reference set. An intrusion is a reported item that is not in the reference set. An omission is an item in the reference set that is not reported. Amounts—whether of servings or of such quantities as kilocalories and grams of protein—are classified as corresponding, overreported, or unreported. For a match, the reported amount equals, exceeds, or falls short of the reference amount. If the reported amount equals the reference amount, the reported amount is a corresponding amount. If the reported amount exceeds the reference amount, the amount by which it does so is an overreported amount; if the reported amount falls short of the reference amount, the amount by which it falls short is an unreported amount. In each case, the overlap between the reported and reference amounts is a corresponding amount. (For example, if an individual was observed consuming¼ serving of milk, and reported consuming a whole serving, the reported serving consists of a corresponding ¼ serving and an overreported¾ serving. If an individual was observed consuming 1 serving of milk and reported consuming ¼ serving, the reported ¼ serving is corresponding, and ¾ serving is unreported.) For an omission, the entire amount is unreported. For an intrusion, the entire amount is overreported. From this, it follows that ReportedAmount=CorrespondingAmount+OverreportedAmount,and ReferenceAmount=CorrespondingAmount+UnreportedAmount, in which each term is an arithmetic quantity and + represents addition. Letting rep, ref, c, and o represent reported, reference, corresponding, and overreported amounts, respectively, we can abbreviate the former expression to rep = c + o. The report rate of energy or of a nutrient, a conventional measure of reporting accuracy, is the reported amount divided by the reference amount, multiplied by 100. A report rate has a lower bound of 0%, which would indicate that nothing was reported, and no upper bound, because there is no limit on what an individual might report. Algebraic expression of the report rate, as: (rep/ref)×100=[(c+o)/ref]×100=[(c/ref)×100]+[(o/ref)×100], shows that it consists of two parts. We call [(c/ref) × 100] the correspondence rate: It is the percentage of the reference amount to which the reported amount corresponds. It is a genuine measure of accuracy with a value between 0%, which would indicate that no reference item had been reported, and 100%, which would indicate that all reference items and their amounts had been reported correctly. We call [(o/ref) × 100] the inflation ratio: It is a non-negative augmentation to correctly reported information based on inaccurate reporting; it has no upper bound, because there is no limit on what can be reported. Although the report rate is treated conventionally as a measure of accuracy, it includes a component based on reporting error: Because the inflation ratio is nonnegative, any overreporting—whether it is reporting intrusions or amounts of matches that exceed reference values—inflates the report rate. Consider again the individual who was observed eating one serving of hash browns and reported eating one serving of applesauce, and assume that this individual ate only hash browns and reported only applesauce. This individual’s report rates for energy, protein, carbohydrate, and fat, are 96%, 20%, 212%, and 4%, respectively, but the correspondence rate for each is 0%, because none of what was reported was from a match. The inflation ratios for energy and each macronutrient are equal to the respective report rates, because all of what was reported was from an intrusion. In this article, we demonstrate that conventional analyses of data from dietary-reporting validation studies that convert reported and reference information to energy and nutrients, and that do not distinguish between reported matches and intrusions or between corresponding and overreported amounts, may mask the complexity of dietary reporting error and overestimate reporting accuracy. We use data from a study conducted to determine whether children’s dietary-recall accuracy depends on interview modality (in-person; telephone).17