Comparing Word-based and LSA Systems 3 Evaluating Self-Explanations in iSTART : Comparing Word-based and LSA Systems Interactive Strategy Training for Active

This chapter compares the effectiveness of LSA and non-latent word-based algorithms in assessing the quality of self-explanations in iSTART, an automated tutor for improving students’ self-explanations of science texts. We have compared various methods of using LSA and wordbased methods to guide the responses the tutor makes to the students concerning the quality of their explanations. This chapter examines the effectiveness of 7 systems using word-based algorithms, LSA, and combinations of both, that vary in the degree of manual preparation of the target text required. Their effectiveness is measured in terms of their match to human ratings of the explanations. Our results indicate that the most successful method is a combined system with no hand coding. This fully automated system will make it possible for us to more easily expand iSTART's repertoire to include a wide variety of practice texts. Comparing Word-based and LSA Systems 3 Evaluating Self-Explanations in iSTART: Comparing Word-based and LSA Systems Interactive Strategy Training for Active Reading and Thinking (iSTART) is a web-based application that provides young adolescent to college-aged students with self-explanation and reading strategy training (McNamara, Levinstein, & Boonthum, 2004). Although untutored selfexplanation -that is, explaining the meaning of text to oneself -has been shown to improve text comprehension (Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Chi, de Leeuw, Chiu, & LaVancher, 1994), many readers explain text poorly and gain little from the process. iSTART is designed to improve students’ ability to self-explain by teaching them to use reading strategies such as comprehension monitoring, making bridging inferences, and elaboration. In the final phase of training, students practice using reading strategies by typing self-explanations of sentences from science texts. The computational challenge is to provide appropriate feedback to the students concerning their self-explanations. To do so requires capturing some sense of both the meaning and quality of the self-explanation. LSA is an important component in that process. Indeed, an important contribution of LSA is that it allows researchers to automatically capture meaning in text (see also, E. Kintsch et al., this volume; Graesser et al., this volume). Interpreting text is critical for intelligent tutoring systems, such as iSTART, that are designed to interact meaningfully with, and adapt to, the users’ input. One question, however, regards the extent to which LSA enables or enhances the accuracy of self-explanation evaluation in iSTART. Thus, in this chapter, we compare various systems of self-explanation evaluation that differ in terms of whether the algorithms are word-based, incorporate LSA, or use a combination of algorithms. Because we want to increase the number of texts available for practice in iSTART, we sought to develop evaluation systems that required less human preparation of the included Comparing Word-based and LSA Systems 4 texts, so an important characteristic of the systems discussed is the amount of ‘hand-coding’ required. This chapter describes iSTART and our evaluation of these feedback systems. iSTART: The Intervention iSTART is modeled after a human-delivered reading strategy intervention called SelfExplanation Reading Training, or SERT (McNamara, 2004; McNamara & Scott, 1999; O’Reilly, Best, & McNamara, 2004). SERT training can be administered to a small group of students in about two hours. It consists of three phases, an introduction with definitions and examples of self-explanation and six reading strategies, a demonstration of the strategies by a student on a videotape, and practice using the strategies with science texts. The six reading strategies include (a) comprehension monitoring, being aware of understanding; (b) paraphrasing, or restating the text in different words; (c) elaboration, using prior knowledge or experiences to understand the text (i.e., domain-specific knowledge based inferences); (d) logic or common sense, using logic to understand the text (i.e., domain-general knowledge based inferences); (e) predictions, predicting what the text will say next; and (f) bridging, understanding the relation between separate sentences of the text. During the introduction phase, a description of the strategy and examples of self-explanations using the strategies are provided. Comprehension monitoring is presented as a strategy that should be used all of the time. Paraphrasing is presented as a basis or jumpstart for self-explanation, but not as means for self-explaining text because it does not go beyond the text. The remaining strategies are various forms of inferences (i.e., domain specific, domain-general, predictive, and bridging) that improve the students’ ability to make sense of difficult text. Although SERT has been found to be effective both in terms of improving comprehension scores and course performance (McNamara, 2004; McNamara & Scott, 1999; Comparing Word-based and LSA Systems 5 O’Reilly, Best, & McNamara, 2004), there are scale-up problems in administering SERT with human tutors to students in many classrooms. First, it is expensive to provide and train the tutors. Second, the delivery of training may vary from one class to the next, despite best efforts to maintain consistency. Third, human tutors cannot be made accessible to all students who need it. Finally, the training is delivered to students in groups and therefore cannot be tailored to the individual needs of the learner. iSTART alleviates these shortcomings while at the same time providing comprehension gains equivalent to SERT (Magliano et al., 2004; O’Reilly, Sinclair, & McNamara, 2004; O’Reilly, Best, & McNamara, 2004). Since iSTART is web-based, it has the potential to make the training available to any school in the country with internet access. Furthermore, as an automated tutoring system, it can deal with students individually, which affords them self-paced learning. Automated tutors can also keep track of students’ performance and provide adaptive scaffolding. Web-based systems can also be easily accessed outside the classroom, which potentially increases the time that students can spend on training. Finally, systems that incorporate pedagogical agents and automated linguistic analysis can engage the student in interactive dialog and thereby induce an active learning environment (e.g., Bransford, Brown, & Cocking, 2000; Graesser, Hu, & Person, 2001; Graesser, Hu, & McNamara, in press; Graesser et al., this volume; Louwerse, Graesser, & Olney, 2002). In iSTART, pedagogical agents instruct trainees in the use of self-explanation and other active reading strategies to comprehend the meaning of text while they read. It consists of three modules, including an introduction, a demonstration, and a practice module. The introduction module presents reading strategy concepts in the form of a classroom discussion among three animated characters (an instructor and two students) that interact with each other, provide Comparing Word-based and LSA Systems 6 information, pose questions, and give explanations and examples of the reading strategies. These characters are full-body figures with heads that are slightly exaggerated to make mouth movements and facial expressions more visible. The characters speak using a text-to-speech synthesizer. Their speech is audible and appears as text in a speech bubble. They possess a repertoire of gestures and can move about the screen. The interactions between the characters vicariously simulate the active processing necessary to learn the strategies. After each section of the introduction, the students complete brief multiple-choice quizzes to assess their learning of the strategies. The quizzes are designed as learning tools to guide the student to a better understanding of each SERT strategy by providing hints, prompts, and explanations for incorrect choices. In the demonstration module, one animated character (Merlin) guides the trainees in analyzing the explanations produced by a second animated character (Genie). Genie (representing a student) reads aloud each sentence from a science text and produces a selfexplanation that appears both in spoken form and as text in a box on the web page. Merlin continues by asking the student using the program to indicate which strategies Genie employed in producing his self-explanation. Merlin follows up by asking the student to identify and locate in the text box the various reading strategies contained in Genie’s self-explanation. Merlin may follow up further by asking the trainee to identify the sentence referenced from the text when Genie uses a bridging strategy. Merlin then gives Genie verbal feedback on the quality of his self-explanation. This feedback mimics the interchanges that the student will encounter in the practice module. For example, sometimes Merlin states that the self-explanation is too short, prompting Genie to add to his self-explanation. Of course, Merlin also gives the trainee feedback, for example, applauding a correct identification. Comparing Word-based and LSA Systems 7 In the practice module, Merlin plays coach to the trainees and provides feedback to them while they practice self-explanation using the repertoire of reading strategies. The goal is to help the student acquire the skills necessary to integrate knowledge from various sources to understand a target sentence. This knowledge may come from something they have read in the passage (e.g., the previous sentence), general knowledge, or domain knowledge. For each sentence, Merlin reads the sentence aloud before asking the student to self-explain it by typing a self-explanation. Merlin gives feedback, sometimes asking the student to modify unsatisfactory self-explanations. Once the self-explanation is satisfactory, Merlin asks the student to identify what strategies were used and where they were used in the explanatio