Estimating the Effectiveness of Conversational Behaviors in a Reading Tutor that Listens (2002)

Project LISTEN’s Reading Tutor listens to children read aloud, and helps them learn to read. Besides user satisfaction, a primary criterion for tutorial spoken dialogue agents should be educational effectiveness. In order to learn to be more effective, a spoken dialogue agent must be able to evaluate the effect of its own actions. When evaluating the effectiveness of individual actions, rather than comparing a conversational action to nothing, an agent must compare it to reasonable actions. We describe a methodology for analyzing the immediate effect of a conversational action, and some of the difficulties in so. We also describe some preliminary results on evaluating the effectiveness of conversational behaviors in a reading tutor that listens. Introduction The idea of getting computers to listen to children read has been around for years. However, early attempts [Bernstein and Rtischev 1991, Kantrov 1991, Phillips, McCandless, and Zue 1992] were hampered by slow hardware and the restrictions of isolated word recognizers. With the advent of affordable consumer-level machines that can recognize continuous speech in near-realtime, the time is ripe for reading tutoring as a testbed for conversational computing for children. Here we report on Project LISTEN’s Reading Tutor [Mostow and Aist AAAI 1997, Mostow et al. UIST 1995, Mostow et al. AAAI 1994]. Related efforts are underway at, for example, IBM (1997) and DERA Malvern (Russell et al. 1996). In 1996-1997, a pilot test of the Reading Tutor at an urban elementary school produced dramatic results [Mostow and Aist WPUI 1997]. The subjects were several third graders who started out reading nearly three years below grade level. They used the Reading Tutor for eight months, supervised individually by a school aide. Each student had 30-60 sessions, averaging 14 minutes per session. The six preand post-tested subjects advanced by an average of about two years in instructional reading level, defined as the grade level of material they could read with at least 75% accuracy and 75% comprehension. One student could read and comprehend sixth grade material, but another student showed little or no improvement. The Reading Tutor is now deployed in classroom field studies in order to evaluate and improve it. We extended the Reading Tutor to provide a wider range of assistance, such as sounding out a word or giving a rhyming hint. We deployed this extended version of the Reading Tutor at a month-long Reading Clinic at the same school during July 1997, where 62 students used the Reading Tutor on eight Pentium ProTM machines in a lab setting. Since August 1997, the Reading Tutor has been deployed in regular classrooms and used regularly by over 100 students. Students in these field tests have ranged from kindergarten through fourth grade, with a few fifth graders and one sixth grader. This wider range of age groups and ability levels offers new challenges for instruction, including how to make the Reading Tutor usable in ordinary classrooms with a minimum of adult assistance. How can we improve the Reading Tutor to make it work better for all children? If the Reading Tutor could evaluate the effectiveness of its own actions, it could adjust From: AAAI Technical Report SS-98-01. Compilation copyright © 1998, AAAI (www.aaai.org). All rights reserved. its behavior to work better overall, and even adapt to fit individual students. How can a spoken dialogue agent monitor its own effectiveness? The PARADISE model for evaluating spoken dialogue agents [Walker, Litman, Kamm and Abella ACL 1997] predicts user satisfaction based on a number of factors including task success and total dialogue time. However, since the PARADISE model works over global measures of dialogue success, it is not clear how to assign credit or blame for user satisfaction to individual dialogue acts -just the agent as a whole. Rather than user satisfaction, the ultimate measure of the Reading Tutor’s success is student learning. By listening to children read, the Reading Tutor can estimate overall improvement in student performance unobtrusively and objectively [Mostow and Aist AAAI 1997]. In this paper, we estimate local effectiveness of tutorial interventions by looking at the differential accuracy before and after intervention opportunities how a child reads a word before getting (or not getting) help on that word, and how the same child reads the same word just afterwards. A Reading Tutor that Listens Project LISTEN’s Reading Tutor adapts the Sphinx-II speech recognizer [Huang et al. 1993] to listen to children read aloud. The Reading Tutor runs on a single standalone PentiumTM. The child uses a noise-cancelling headset or handset microphone and a mouse, but not a keyboard. Roughly speaking, the Reading Tutor displays a sentence, listens to the child read it, provides help in response to requests or on its own initiative based on student performance. [Aist 1997] describes how the Reading Tutor decides when to go on to the next sentence. The student can read a word aloud, read a sentence aloud, or read part of a sentence aloud. The student can click on a word to get help on it. The student can click on Back to move to the previous sentence, Help to request help on the sentence, or Go to move to the next sentence. The student can click on Story to pick a different story, or on Goodbye to log out. The Reading Tutor can choose from several communicative actions, involving digitized and synthesized speech, graphics, and navigation [Aist and Mostow 1997]. The Reading Tutor can provide help on a word (e.g. by speaking the word), provide help on sentence (e.g. by reading it aloud), backchannel (mmhmm), provide just-in-time help on using the system, and navigate (e.g. go on to the next sentence). With speech awareness central to its design, interaction can be natural, compelling, and effective [Mostow and Aist WPUI 1997]. Fair Alternatives to Conversational Behaviors If a conversational agent is to evaluate the effectiveness of its own behaviors, what should it compare a given behavior against? A key observation is that one cannot compare against doing nothing. Doing in a conversation is in itself an action, and if done too frequently or for too long, violates the unwritten rules of conversational behavior. For example, if an agent takes a turn because it is time to take a turn, it is unreasonable to compare the turn’s effectiveness to the alternative of saying nothing when saying nothing might be seen as a system failure. Thus, a more reasonable methodology is to compare the effectiveness of a given behavior against the effectiveness of other equally felicitous behaviors. Machine learning paradigms such as active learning or reinforcement learning [Kaelbling, Littman, and Moore 1996, Sutton and Barto, forthcoming] hold promise for agents learning behavior by active exploration of the environment. However, in mixed-initiative conversation, where a strict turn-taking assumption (T speaks, speaks, T speaks, S speaks) does not hold, evaluating the effectiveness of conversational behavior presents additional challenges. First, time is not discrete as in many machine learning paradigms; it is continuous, and the time that an action took place may be just as important as when it took place. For example, consider a hypothetical case of two ways of giving the same kind of help, where one way causes the user to take twice as long to respond as the other. Clearly here time is an important factor in evaluating conversational behavior. How should two outcomes that occur at different times be weighted? Secondly, more than one Reading Tutor intervention may take place before the student responds. How should individual Reading Tutor actions be credited if more than one of them takes place between student turns? For the purposes of this paper, we defer this issue by considering only whether any of the Reading Tutor’s actions between student turns were relevant to a given word, without distinguishing among series of such actions. Estimating the Effectiveness of a Conversational Behavior How can a conversational agent evaluate the effectiveness of a conversational behavior? In this paper, we describe how we evaluated the effectiveness of the Reading Tutor’s conversational behavior (intervention) based solely the student’s reading before and after the intervention in question. Assigning credit or blame in this (or some other) way is a necessary prelude to learning which