AutoTutor's Coverage of Expectations during Tutorial Dialogue

AutoTutor is a learning environment with an animated agent that tutors students by holding a conversation in natural language. AutoTutor presents challenging questions and then engages in mixed initiative dialogue that guides the student in building an answer. AutoTutor uses latent semantic analysis (LSA) as a major component that statistically represents world knowledge and tracks whether particular expectations and misconceptions are expressed by the learner. This paper describes AutoTutor, reports some analyses on the adequacy of the LSA component, and proposes some improvements in computing the coverage of particular expectations and misconceptions. Tutorial Dialogue with AutoTutor AutoTutor is a computer tutor that holds conversations with learners in natural language (Graesser et al. 2004; Graesser, Person, & Harter 2001; Graesser, VanLehn, Rose, Jordan, & Harter 2001). AutoTutor simulates the discourse patterns of human tutors and is augmented with a number of ideal tutoring strategies. The tutor presents a series of challenging questions from a curriculum script and engages in a collaborative mixed initiative dialog while constructing answers. AutoTutor was designed to be a good conversational partner that comprehends, speaks, points, and displays emotions, all in a coordinated fashion. AutoTutor “speaks” by utilizing a speech engine developed at Microsoft (www.microsoft.com/products/msagent) or SpeechWorks (www.speechworks.com). For some topics and versions of AutoTutor, there are graphical displays, animations of causal mechanisms, or interactive simulation environments, with AutoTutor talking about and pointing to various components (Graesser, Chipman, Haynes, & Olney, in press). The initial versions of AutoTutor were on the topic of computer literacy. A later version, called Why/AutoTutor, helps college students learn Newtonian physics (Graesser, VanLehn et al. 2001) by asking them why-questions on difficult problems. There are a number of distinctive features of AutoTutor that set it apart from most other intelligent tutoring systems. One strength of AutoTutor is that the tutoring domain can be changed quickly with lesson authoring tools, without the need to rebuild any of the conversational or pedagogical components of the system. A second strength is that the dialogue quality is sufficiently robust Copyright © 2005, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. that bystander evaluators cannot tell the difference between a speech act generated by AutoTutor and a speech act generated by a human tutor; such bystander Turing tests have been reported in several experiments by Person and Graesser (2002). A third strength is that AutoTutor shows impressive learning gains of nearly a letter grade improvement compared with pretest measures and suitable comparison conditions (Graesser et al. 2003; Graesser, Lu et al. 2004; VanLehn, Graesser et al. 2004). A fourth strength is that AutoTutor uses a hybrid of structured knowledge representations and a statistical representation of world knowledge called Latent Semantic Analysis (LSA, Landauer, Folts, & Laham 1998; Graesser et al. 2000). It is this LSA component that will be examined in the present article. Specific Goals and Scope of AutoTutor AutoTutor presents a series of difficult questions to be answered, or problems to be solved. An answer to a typical question is 3-7 sentences, or approximately a paragraph of information. When students are asked questions that require paragraph-length answers and deep reasoning, initial answers to these questions are typically only 1 or 2 sentences in length. AutoTutor helps the student fill in missing pieces of the answer by managing a mixed initiative dialogue with different categories of dialogue acts. AutoTutor provides feedback to the student on what the student types in (positive, neutral, or negative feedback), pumps the student for more information (“What else? ”), prompts the student to fill in missing words, gives the student hints, fills in missing information with assertions, identifies and corrects erroneous ideas and misconceptions, answers the student’s questions, and summarizes answers. These dialogue acts of feedback, pumps, prompts, hints, assertions, corrections, answers, and summaries eventually lead to a full correct answer. The tutorial dialog patterns of AutoTutor were motivated by research in discourse processing, cognitive science, and intelligent tutoring systems. Constructivist theories of learning emphasize the importance of learners actively constructing explanations (Aleven & Koedinger 2002; Chi, de Leeuw, Chiu, & LaVancher 1994; McNamara 2004). Researchers have developed intelligent tutoring systems that adaptively respond to the learner’s knowledge and help construct explanations (Anderson Corbett, Koedinger, & Pelletier 1995; VanLehn et al. 2003). Empirical research in discourse processing has documented the collaborative constructive activities that frequently occur during human tutoring (Chi, Siler, Jeong, Yamauchi, & Hausmann 2001; Fox 1993; Graesser, Person, & Magliano 1995). Surprisingly, the dialog moves of most human tutors are not particularly sophisticated from the standpoint of today’s pedagogical theories and intelligent tutoring systems. Human tutors rarely implement bona fide Socratic tutoring strategies, modeling-scaffolding-fading, reciprocal training, building on prerequisites, and other sophisticated pedagogical techniques. Instead, human tutors normally coach the student by filling in missing pieces of information in an expected answer and by fixing bugs and misconceptions expressed by the student. We refer to this tutoring mechanism as Expectation and Misconception Tailored Dialog. AutoTutor was designed to simulate the dialog moves of human tutors who coach students in constructing explanations and answers to openended questions. A Concrete Example of AutoTutor AutoTutor’s curriculum script consists of a set of questions (or problems) and answers that require deep reasoning. Listed below are the important forms of content affiliated with each main question. Main Question. If a lightweight car and a massive truck have a head-on collision, upon which vehicle is the impact force greater? Which vehicle undergoes the greater change in its motion? Explain why. Ideal Answer. The force of impact on each of the colliding bodies is due to interaction between them. The forces experienced by these bodies are thus an action/reaction pair. Thus, in terms of Newton's third law of motion, these forces will be equal in magnitude and opposite in direction. The magnitude of the acceleration produced by a force on different objects is inversely proportional to their masses. Hence, the magnitude of the car's acceleration due to the force of impact will be much larger than that of the more massive truck. A larger magnitude of acceleration implies a larger rate of change of velocity, which may be interpreted as greater change in motion. Therefore, the car undergoes greater change in its motion. Expectations. (E1) The magnitudes of the forces exerted by the two objects on each other are equal. (E2) If one object exerts a force on a second object, then the second object exerts a force on the first object in the opposite direction. (E3) The same force will produce a larger acceleration in a less massive object than a more massive object. Misconceptions. (M1) A lighter/smaller object exerts no force on a heavier/larger object. (M2) A lighter/smaller object exerts less force on other objects than a heavier/larger object. (M3) The force acting on a body is dependent on the mass of the body. (M4) Heavier objects accelerate faster for the same force than lighter objects. (M5) Action and reaction forces do not have the same magnitude. Functionally Equivalent Concepts. car, vehicle, object truck, vehicle, object The learner might possibly articulate the ideal answer to this question, but such a complete answer rarely if ever occurs. Natural language is much too imprecise, fragmentary, vague, ungrammatical, and elliptical to anticipate such semantically well-formed and complete responses. LSA is used to evaluate, probabilistically, the extent to which the information within the student turns (i.e., an individual turn, a combination of turns, or collective sequence of turns) matches the ideal answer. AutoTutor requires that the learner articulate each of the expectations before it considers the question answered. The system periodically identifies a missing expectation during the course of the dialogue and posts the goal of covering the expectation. When expectation E is missed and therefore posted, AutoTutor attempts to get the student to articulate it by generating hints and prompts that encourage the learners to fill in missing content words and propositions. Expectation E is considered covered if the content of the learners’ turns meet or exceed a threshold T in its LSA cosine value. More specifically, E is scored as covered if the cosine match between E and the student input I is high enough: cosine (E, I) ≥ T. The threshold has varied between . 40 and . 75 in previous versions and evaluations of AutoTutor. Each expectation E has a family of prompts and hints that potentially may be recruited in AutoTutor’s dialogue acts in order to get the student to fill in every content word and proposition in E. A particular prompt or hint from the family is selected to maximize an increase in the LSA cosine match score when the prompt or hint is expressed by the learner. Students may express a misconception during the dialogue. This happens when the student input I matches a misconception M with a sufficiently high match score. AutoTutor corrects the misconception and goes on. AutoTutor systematically manages the dialogue when it attempts to get the learner to articulate an expectation E that gets posted. AutoTutor stays on topic by completing the sub-dialog that covers