Classroom success of an intelligent tutoring system for lexical practice and reading comprehension.

We present an intelligent tutoring system called REAP that provides reader-specific lexical practice for improved reading comprehension. REAP offers individualized practice to students by presenting authentic and appropriate reading materials selected automatically from the web. We encountered a number of challenges that must be met in order for the system to be effective in a classroom setting. These include general challenges for a system that uses authentic materials, as well as more specific challenges that arise from integrating the system with pre-existing classroom curricula. We discuss how these challenges were met, and present evidence that REAP has gained acceptance into the classroom at the English Language Institute at the University of Pittsburgh. 1. System Description We begin with brief descriptions of the REAP intelligent tutoring system and its primary users. For a more detailed description of the REAP project, please see [1] and [2]. The REAP project’s goal is to provide appropriate, authentic reading materials to students learning to read. It gathers and selects documents automatically from the web, which raises a number of concerns that will be discussed in this paper. The system has focused on English language so far, but future developments could extend the scope of the project to other languages. REAP incorporates a variety of statistical language modeling and information retrieval methods in order to model students’ knowledge and find useful reading passages for them. Recent work on the REAP system includes creating a system for the University of Pittsburgh’s English Language Institute (ELI) Reading 4 course, an upper-level course for English as a Second Language (ESL) that focuses on reading skills. A study on usability of REAP is currently in progress at the ELI. In this study, which we will refer to as the Spring ’06 ELI Study, thirty-three students use the system once a week for forty minutes over the course of the semester, reading documents containing target unknown vocabulary identified from a pre-test. REAP gathers documents from the Web in order to find useful, authentic reading material for these students. The documents are analyzed according to syntactic features, readability, length, and the occurrence of target vocabulary. The tutor uses an extended version of the Lemur Toolkit for Language Modeling and Information Retrieval [3] to annotate the documents and create an index for language-model based retrieval. When a student uses REAP, the system searches among this set of documents for those that satisfy a number of constraints, including the student’s target vocabulary list, document length, his or her user model, and the target reading level for the course, which is sixth to eighth grade. After reading a document, usually from one to two pages in length, the student works through a series of automatically generated exercises based on the target vocabulary found in the reading. The student model is updated after every reading so that the optimal document can be retrieved for the next reading passage. By using authentic reading materials the REAP system offers realistic training and individualized curricula to students. Reading textbooks and hand-selected materials are usually wellcontrolled, appropriate, and contain high-quality input, yet such materials are also static, difficult to produce, and very limited in quantity. In a classroom setting, it is typical that all students see the same material from a textbook, regardless of the state of their lexical or grammatical development. Also, reading materials for use in most classrooms must meet a wide variety of syntactic and lexical constraints in order for students of a given reading proficiency to be able to read them without confusion. Teachers or textbook authors often have to heavily edit or even produce the reading materials themselves in order to meet these constraints, introducing some amount of artificiality into the materials. Intelligent tutoring systems such as REAP can examine large corpora such as the Web in order to automatically select materials that meet these various criteria. Students using REAP work toward their ultimate goal of reading real text by actually reading real text. Intelligent tutoring systems also provide students with individualized practice rather than static sets of exercises. Students go through readings at very different rates, and so faster students need a greater number of more difficult reading passages than do slower students. In a current study ten students using the REAP system had completed fewer than ten reading passages, while twelve students had completed twenty or more, despite having the same time on task. The average number completed was just under seventeen. REAP selects as many documents as are necessary for each student, and these documents satisfy certain lexical, syntactic, and readability constraints based on a model of the current student’s knowledge. Finding a large number of appropriate documents for an entire classroom of students can in many cases only be accomplished by an intelligent tutoring system. Such a system is therefore very valuable to language teachers. The value of the system is demonstrated in results from an exit survey taken toward the end of a recent study, shown in Figure 1. The students (N=33) were asked to respond on a Likert scale from 1 to 5 indicating the degree to which they agree to given statements about the system. The results indicate that students feel that the REAP system is easy to use, valuable for learning both target and non-target vocabulary, and worth using in future classes. Students wanted more personalization and choice of the reading topics, however, to make the system more engaging.