Towards explaining effective tutorial dialogues

Towards explaining effective tutorial dialogues Barbara Di Eugenio and Davide Fossati {bdieugen, dfossa1}@uic.edu Department of Computer Science, University of Illinois at Chicago, USA Stellan Ohlsson and David Cosejo {stellan, dcosej1}@uic.edu Department of Psychology, University of Illinois at Chicago, USA Abstract We present a study of human tutorial dialogues in a core Computer Science domain that: focuses on indi- vidual tutoring sessions, rather than on contrasting dif- ferent types of tutors; uses multiple regression analysis to correlate features of those sessions with learning out- comes; and highlights the effects of two types of tutor moves that have not been studied in depth so far, direct instruction and positive feedback. Keywords: Artificial Intelligence; Education; Dis- course; Interactive Behavior; Human Experimentation; Intelligent Tutoring Systems. Introduction One-on-one tutoring has been shown to be a very ef- fective form of instruction compared to other educa- tional settings. For more than twenty years, researchers have worked on uncovering features of tutorial inter- action that engender learning in students (Fox, 1993; Graesser, Person, & Magliano, 1995; Lepper, Drake, & O’Donnell-Johnson, 1997; Chi, 2001; Moore, Porayska- Pomsta, Varges, & Zinn, 2004; Evens & Michael, 2006; Litman et al., 2006; Di Eugenio, Fossati, Haller, Yu, & Glass, 2008). This research is partly motivated by the search for computational models 1 of effective tutoring strategies, which are an essential component of dialogue interfaces to Intelligent Tutoring Systems, or ITSs. From such an extensive body of work, many find- ings have arisen, but a unifying explanation has yet to emerge, partly because tutorial interactions are so rich that it is hard to characterize them fully. One line of at- tack, pursued by us as well as by others in the past, has tried to understand what expert tutors do (Lepper et al., 1997; Evens & Michael, 2006; Di Eugenio, Kershaw, Lu, Corrigan-Halpern, & Ohlsson, 2006; Cade, Copeland, Person, & D’Mello, 2008). However, it is not clear what expert tutors are to start with; expertise does not nec- essarily mean, ’with extensive experience’. Another ap- proach is to imbue the notion of expertise with what we know from pedagogy, and equate expert tutors with idealized, Socratic human tutors. However, as Graesser and collaborators pointed out in a series of papers (Per- son, Graesser, Magliano, & Kreuz, 1994; Graesser et al., 1995), the idealized Socratic human tutor is difficult to We mean computational in the algorithmic sense coming from computer science, rather than in the sense of mental computation coming from cognitive psychology. come by. In the course of our own tutorial dialogues collection and analysis (Di Eugenio et al., 2006, 2008), we have come to believe that even many experienced tu- tors do not behave socratically most of the times. Still, students learn with them, sometimes but not necessarily more than what they learn with inexperienced tutors. As we discuss in Ohlsson et al. (2007), this brought us to advocate a different approach to tutorial dialogue analysis. We eschew defining who an expert tutor is a priori, or casting one type of tutor against another, and analyzing differences between them. Rather, we pool the data from all the tutors together, and we focus on the effectiveness of the individual tutoring session, by run- ning regression analyses that correlate features of those sessions with learning outcomes. The next point concerns which features of the tutor- ing session should be entered in the regression analysis. The body of work we described above has proposed a variety of coding schemes to annotate tutorial dialogues. In Ohlsson et al. (2007), we noted that most often such coding schemes are motivated by pedagogical tenets, by linguistic theories of speech acts, and often by what hap- pens in the tutoring dialogues themselves. What many schemes are missing is being informed by cognitive in- sights into how people learn. In this paper, we present a corpus analysis which em- bodies those earlier proposals of ours: it pools together tutoring sessions collected in a study that had originally started as a contrast between a more experienced and a less experienced tutor; because it is motivated by what is cognitively plausible for learning, it focuses on two fea- tures that have not been studied in depth so far, direct instruction and positive feedback; and uses regression analysis. In particular, we show that positive feedback correlates with learning, and that our methodology al- lows us to further annotate the data in a controlled, prin- cipled way. We also very briefly discuss the ITS we have built, partly based on these results. Human tutoring in Computer Science Our domain of interest is Computer Science (CS), specif- ically, introductory data structures such as linked lists, stacks and binary search trees, and the algorithms that manipulate them. This domain is partly motivated by the interests of some of us, as educators in CS. Basic