Expertise and the Illusion of Comprehension

For almost 20 years the findings of Glenberg and Epstein (1987) have been offered as evidence that experts are less accurate than novices in judging the state of their understanding. In this paper we point to a number of difficulties with this finding, and report a modified replication of the original study. Although we find that the possession of domain knowledge is positively related to both comprehension and judgments of comprehension, domain knowledge was not significantly related to relative metacomprehension accuracy. In terms of absolute accuracy, however, higher-knowledge individuals actually appear better calibrated. Altogether, we find that domain knowledge accounts for an extremely slight proportion of the variance in the accuracy of comprehension judgments. Expertise and the Illusion of Comprehension Close to twenty years of research has cited the finding of Glenberg and Epstein (1987) that experts can be less accurate about their own state of understanding than novices. In the original study, Glenberg and Epstein found a negative relationship between the accuracy of metacomprehension judgments (readers’ predictions of whether they would be able to answer comprehension questions about texts they had just read) and the number of courses they had taken in the content area. To our knowledge these results have not been replicated, yet they form the basis for over 100 citations of the claim that there is a negative effect of expertise or familiarity with content on comprehension monitoring judgments (c.f. Glenberg, Sanocki, Epstein, & Morris, 1987). One of the main reasons for the popularity of Glenberg and Epstein’s finding is likely that it stands in contrast to most findings related to domain knowledge and performance on domain-related tasks, i.e. that the possession of domain knowledge usually improves performance on domain-related text comprehension and problem solving (Chi, Glaser & Farr, 1988; Ericsson & Kintsch, 1995; Spilich, Vesonder, Chiesi & Voss, 1979). This conclusion suggests that experts may be less likely to review information that they fail to comprehend accurately on their first pass through a novel text. Moreover, if highknowledge individuals assume that they understand novel domain-related information on the basis of their general proficiency in the domain, they may inaccurately encode this information, or fail to encode it altogether. This may increase the probability that these individuals will commit undesirable and potentially costly errors. A Closer Look at the Illusion Despite the popularity of Glenberg and Epstein’s (1987) conclusions, there are a number of concerns that can be raised about their study. A principal concern is the way in which Glenberg and Epstein analyzed their data on the relation between expertise (number of courses taken) and the accuracy of metacomprehension judgments. The Goodman-Kruskal Gamma (G) was used as an index of the association between metacomprehension judgments and comprehension scores. Higher values for G imply better relative accuracy of comprehension judgments. For each participant in their study, a separate G was computed for each domain of texts presented, music and physics. Separate regression analyses were then conducted to predict the mean G for each domain, with number of music courses (MC) and number of physics courses (PC) serving as the two predictor variables in each equation. These regression equations are reproduced in Table 1. Table 1: Regression equations for resolution of comprehension (from Glenberg & Epstein, 1987, p.87)