General and Efficient Cognitive Model Discovery Using a Simulated Student

General and Efficient Cognitive Model Discovery Using a Simulated Student Nan Li (nli1@cs.cmu.edu) Eliane Stampfer (estampfe@cs.cmu.edu) William W. Cohen (wcohen@cs.cmu.edu) Kenneth R. Koedinger (koedinger@cs.cmu.edu) School of Computer Science , Carnegie Mellon University 5000 Forbes Ave., Pittsburgh PA 15213 USA Abstract important instructional details may still be overlooked. Au- tomated search methods such as Learning Factor Analysis (LFA) (Cen, Koedinger, & Junker, 2006) are more objective: the algorithm searches through the space of human-provided factors to find a cognitive model that best matches with hu- man data. Although automated search methods have found better models than manual construction, the quality of the dis- covered model depends on the quality of the human-provided factors. If there is a better model that can not be expressed by known factors, LFA will not be able to uncover it. In Li, Matsuda, Cohen, and Koedinger (2011), we have proposed to use the state-of-the-art learning agent, SimStu- dent (Matsuda, Lee, Cohen, & Koedinger, 2009), to auto- matically discover cognitive models without depending on human-provided factors. SimStudent learns skill knowledge from demonstration and problem solving experience. Each skill SimStudent acquires corresponds to a KC in the cogni- tive model. To demonstrate the generality of this approach, we present evaluations of the SimStudent-generated models in three domains: algebra, stoichiometry, and fraction addi- tion. We validate the quality of the cognitive models using hu- man student data as in Koedinger and MacLaren (1997). In- stead of matching with performance data, we use the discov- ered cognitive model to predict human learning curve data. Experimental results show that for algebra and stoichiometry, SimStudent directly finds a better cognitive model than hu- mans. For fraction addition, SimStudent results assist LFA in finding a better cognitive model than a domain expert. We have also carried out an in-depth study using Focused Bene- fits Investigation (FBI) (Koedinger, McLaughlin, & Stamper, 2012) to better understand this machine learning approach, and discussed possible ways of further improvements. In order to better understand how humans acquire knowledge, one of the essential goals in cognitive science is to build a cognitive model of human learning. Moreover, a cognitive model that better matches student behavior will often yield bet- ter instruction in intelligent tutoring systems. However, man- ual construction of such cognitive models is time consuming, and requires domain expertise. Further, manually-constructed models may still miss distinctions in learning which are impor- tant for instruction. Our prior work proposed an approach that finds cognitive models using a state-of-the-art learning agent, SimStudent, and we demonstrated that, for algebra learning, the agent can find a better cognitive model than human experts. To ensure the generality of that proposed approach, we further apply it to three domains: algebra, stoichiometry, and frac- tion addition. To evaluate the quality of the cognitive models discovered, we measured how well the cognitive models fit to student learning curve data. In two of those domains, SimStu- dent directly discovers a cognitive model that predicts human student behavior better than the human-generated model. In fraction addition, SimStudent supported discovery of a better cognitive model in combination with another automated cog- nitive model discovery method. Keywords: cognitive model, machine learning, simulated stu- dent Introduction One of the fundamental goals in cognitive science is to un- derstand human knowledge acquisition. A cognitive model of human learning that fits data would be a significant achieve- ment. This goal also complements with another goal in edu- cation, which is to provide individualized instruction based on students’ abilities, learning styes, etc. Cognitive mod- els provide intelligent tutoring systems with useful informa- tion on the learning task difficulties and transfer of learning among similar problems. A better cognitive model often leads to more effective tutoring. A cognitive model is a system that can solve problems in the various ways human students can. One common way of representing a cognitive model is a set of knowledge components (KC) (Koedinger & McLaughlin, 2010). The set of KCs includes the component skills, con- cepts, or percepts that a student must learn to be successful on the target tasks. For example, a KC “divide” in algebra encodes how to proceed given problems of the form Nv = N (e.g., −3x = 6), where N stands for a number, and v stands for a variable. Nevertheless, manual construction of cognitive models re- mains time consuming and error prone. Traditional ways to construct cognitive models include structured interviews, think-aloud protocols, and rational analysis. Manual con- struction of cognitive models requires domain expertise, and A Brief Review of SimStudent SimStudent is an intelligent agent that inductively learns skills to solve problems from demonstrated solutions and from problem solving experience. It is a realization of pro- gramming by demonstration (Lau & Weld, 1998) using a variation of the version space algorithm (Mitchell, 1982), in- ductive logic programming (Muggleton & Raedt, 1994), and iterative-deepening depth-first search as underlying learning techniques. For more details, please refer to Matsuda et al. (2009). Recently, in order to build a more human-like in- telligent agent, we have developed a model of representation learning, and integrated it into SimStudent’s skill acquisition mechanism.