Evaluating the Effectiveness of a Cognitive Tutor for Fundamental Physics Concepts

Evaluating the Effectiveness of a Cognitive Tutor for Fundamental Physics Concepts Patricia L. Albacete (albacete@isp.pitt.edu) Intelligent Systems Program; 607 Dixie Drive Pittsburgh, PA 15235 USA Kurt A. VanLehn (Vanlehn@cs.pitt.edu) Learning, Research and Development Center; University of Pittsburgh Pittsburgh, PA 15260 USA Abstract In this article we describe and analyze the evaluation of the Conceptual Helper, an intelligent tutoring system that uses a unique cognitive approach to teaching qualitative physics. The results of the evaluation are encouraging and suggest that the proposed methodology can be effective in performing its task. Introduction Several studies (e.g. Hake, 1998; Halloun & Hestenes, 1985a, 1985b) have revealed that solving physics problems of a qualitative nature, such as the one presented in figure 1, pose a great cognitive challenge for most students taking elementary mechanics classes. They uncover naive conceptions that are seldom removed or modified while completing their courses. Several attempts have been made to improve this situation though none has met with great success (Hake, 1998). Given that mechanics is a required course for most science majors, there is a clear need to improve its instruction. Toward this end we developed an intelligent tutoring system called the Conceptual Helper that follows a cognitive teaching strategy which is deployed emulating effective human tutoring techniques as well as successful pedagogical techniques and less cognitive demanding methods (Albacete, 1999; Albacete & VanLehn, 2000). In this article we describe the evaluation of the system and discuss its implications. Two steel balls, one of which weights twice as much as the other, roll off of a horizontal table with the same speeds. In this situation: a) both balls impact the floor at approximately the same horizontal distance from the base of the table. b)the heavier ball impacts the floor closer to the base of the table than does the lighter. c) the lighter ball impacts the floor closer to the base of the table than does the heavier. Figure 1. Example of a qualitative problem Brief description of the Conceptual Helper The Conceptual Helper is an intelligent tutoring system (ITS) designed to coach students through physics homework problem solving of a qualitative nature, i.e., those problems that do not require the use algebraic manipulation to be solved but so require the application of conceptual knowledge. The tutor is basically a model-tracing ITS enhanced by the use of probabilistic assessment to guide the remediation. As a model-tracing ITS it contains a cognitive model that is capable of correctly solving any problem assigned to the student. Model tracing consists of matching every problem-solving action taken by the student with the steps of the expert’s solution model of the problem being solved. This matching is used as the basis for providing immediate feedback to students as they progress through the problem. The system also has a student model which is represented by a Bayesian network. Each node in the network represents a piece of conceptual knowledge that the student is expected to learn or a misconception that the tutor can help remedy. Each node has a number attached to it that indicates the probability that the student will apply the piece of knowledge when it is applicable. As the student solves a problem, the probabilities are updated according to the actions taken by the student. The challenge for the tutor is to decide when to intervene and what to say when it does so. This task is particularly challenging in this domain because tutoring of qualitative knowledge usually takes the form of verbal discussions, which given the state of the art of natural language processing is not an option for the computer tutor. To take care of the issue of when to intervene, we emulated human tutors in two ways: first, by giving immediate feedback (red for incorrect; green for correct) on each student entry (Merrill et al., 1992) and second, by helping the student with post-problem reflection (Katz & Lesgold, 1994; Katz et al., 1996). However, most of our work went into the second issue—deciding what to say when intervening. Novel approaches were developed in three areas: 1) the teaching strategy, 2) the manner in which the knowledge is deployed, and 3) the way in which misconceptions are handled.