Four Measures of Topical Knowledge: A Study of Construct Validity.

This study was an effort to establish the construct validity of measures designed to assess topical knowledge. We began with the assumption that the best way to find out how much people know about a topic is to interview them. The interview then became the criterion for validating three paper and pencil tests of topical knowledge. Elementary and junior high school students were interviewed to ascertain their knowledge about four topics; two weeks later they responded to three tests of topical knowledge. The results were analyzed quantitatively and qualitatively. The correlations between the interview scores and the paper and pencil measures did not reveal a strong and clear relationship between the students' performance on the interviews and their performance on the tests of topical knowledge. This was an unexpected finding which led to further analyses (conditional probability analysis and case studies). The conditional probability analysis revealed that the interview and the paper and pencil tests provided different information regarding an individual's knowledge of a topic. These differences seem to be related to the difference between a recall and a recognition task. There was a high probability that if students gave information in the interviews, they would get that information correct on the paper and pencil tests (if it appeared there). Conversely, the probability was low that students would have mentioned in the interviews all, or even most, of the information that they got correct on the topical knowledge tests. We conclude from these findings that the information one gets of topical knowledge differs between interviews and paper and pencil measures. If the goal is to get the most complete picture possible regarding an individual's topical knowledge, then both interview and paper and pencil measures are necessary. If the goal is to assess only a specific body of information, then a paper and pencil measure might suffice; and, if the goal is to open a broader window on a student's knowledge, then an interview seems preferable. Looking across the few select cases that we analyzed in depth, we found that the interview approach captured individual differences more clearly and dramatically than did the paper and pencil tests of topical knowledge. Topical Knowledge 1 Valencia, Stallman, Commeyras, Pearson, & Hartman FOUR MEASURES OF TOPICAL KNOWLEDGE: A STUDY OF CONSTRUCT VALIDITY In their daily routines teachers have acknowledged for years the importance of topical knowledge for reading comprehension by teaching vocabulary, building background for a selection, and even setting purposes for reading a particular text. But it took an event as major as the cognitive revolution in the psychology of learning to convince researchers to begin to study the knowledge-comprehension relationship. Between 1970 and the present, numerous studies have demonstrated that the experiences and knowledge a person brings to a text influence how he or she comprehends and recalls the text (e.g., Anderson, Reynolds, Schallert, & Goetz, 1977; Bransford & Johnson, 1973; Dooling & Lachman, 1971; Pichert & Anderson, 1977). The finding that topical knowledge affects comprehension was a natural bridge to lines of research attempting to calibrate more subtle aspects of the topical knowledgecomprehension relationship (e.g., Alvermann & Hynd, 1987; Alvermann, Smith, & Readence, 1985; Anderson & Smith, 1984; Callahan & Drum, 1984; Chou Hare, 1982; Chou Hare & Devine, 1983; Davey & Kapinus, 1985; Domaracki, 1984; Langer, 1984; Langer & Nicolich, 1981; Lipson, 1982; Maria & Blustein, 1986; Marr & Gormley, 1982; Stevens, 1980). An important outgrowth of these attempts to examine this relationship in more complex ways is that researchers have had to focus their energies on developing valid and reliable measures of readers' knowledge of text topics. In short, we have moved from broad definitions of topical knowledge and a general acceptance of the topical knowledgecomprehension connection to more specific research questions regarding the breadth and depth of that knowledge. In many of the early studies, topical knowledge tended to be manipulated rather than measured. For example, subjects would be given texts with vague and/or ambiguous terms in order to see what knowledge they voluntarily used to make sense of them (Anderson et al., 1977). Other researchers examined the effect of disambiguating pictures or titles on comprehension of these vague passages; allegedly, the titles or pictures served as bridges to subjects' prior knowledge (Bransford & Johnson, 1973; Dooling & Lachman, 1971). Later, more direct measures of topical knowledge were used in order to try to quantify the knowledge-comprehension relationship (Davey & Kapinus, 1985; Domaracki, 1984; Langer, 1984). These studies included investigations of the breadth and depth of knowledge, and misconceptions that might detrimentally affect comprehension. Although a variety of formats have been used in these studies (e.g., vocabulary, multiple-choice, free response), few have been accompanied by any documentation of efforts to evaluate their validity and reliability. Validity and reliability are important characteristics of any measure from which educators wish to draw inferences about either individuals or groups. Given the complexity of the topical knowledge construct (see, for example, Spiro, Feltovich, Coulson, & Anderson, in press), the diversity of the measures used across studies, and the widespread application of the topical knowledge-comprehension research findings, the need for validity studies is clear. If, in fact, measures that have been used to assess topical knowledge are not valid, then the assumptions and conclusions that have been based on this body of research are brought into question. Some scholars (e.g., Johnston, 1984a; Messick, 1981) have argued that construct validity is one of the most important concerns in assessment. In this study, we began with our best definition of the construct and then tried to determine what aspects of topical knowledge could be assessed by each of several candidate measures. The studies of most interest to us are those that have attempted to measure topical knowledge directly. Within these studies, the types of measures, the scoring systems, the kinds of topical knowledge assessed, and the types of texts vary considerably. These differences demand critical evaluation of various operational definitions of topical knowledge and the ways in which these implicit definitions have led to theories regarding the role of topical knowledge in comprehension. And thus, it is these differences that serve as the framework for our discussion of various topical knowledge assessments. Topical Knowledge 2 Valencia, Stallman, Commeyras, Pearson, & Hartman Formats for Measuring Topical Knowledge Researchers have used at least seven different formats for assessing topical knowledge. The formats range from those with a recognition orientation, such as multiple-choice questions, to those with a recall orientation, such as written statements of what an individual "knows" about a topic. We have placed these various formats on a continuum that illustrates the type of demand the assessment task places on the student (see Figure 1). Recognition tasks reflect the body of knowledge that the researcher has decided is critical to the comprehension task at hand. The respondents' task is to react to the stimulus by choosing an answer from a set of items. In contrast, the recall tasks are more openended and provide greater latitude for those who are asked to demonstrate their knowledge. On the recall end of the continuum, typically students are presented with some general topic label or situation and asked to tell or write what they remember about the topic. In between the multiple-choice knowledge test and the "what do you know about X" recall task are many variations each depicted in Figure 1 and discussed below. [Insert Figure 1 about here] Multiple-choice content questions. This type of topical knowledge measure was used in 6 of the 19 studies (see Table 1). These questions require students to recognize correct information about the content of the specified topic. Alvermann, Smith, and Readence (1985) developed a 20-item multiplechoice test with one correct answer and two distractors per item. The following sample typifies multiple-choice recognition assessment of topical knowledge: Which of the following is true about rattlesnakes? a. Rattlesnakes hide for protection. b. Rattlesnakes chase people. c. Rattlesnakes can strike over 3 times their length. [Insert Table 1 about here] While some studies use three or even four distractors, the basic multiple-choice recognition format is the same. It is not surprising that these types of questions constitute the most frequently used type of topical knowledge measure. They are also the most common question type used for assessing comprehension in instructional materials (Armbruster & Ostertag, 1989; Foertsch & Pearson, 1987). Multiple-choice vocabulary tests. This type of topical knowledge measure was used in two studies. Typical of these vocabulary tests, Johnston (1984b) gave subjects 33 content-specific multiple-choice questions, each presenting a word and four possible definitions or a definition and four possible words. For example: