Progress toward a multi activity Bayesian student modeler

Assessment is a necessary part of education and training Student modeling is an assessment method that avoids some of the problems of conventional tests However there are many obstacles to e ective student modeling In this paper we address three A student modeling system should a analyze data in a statistically sound defensible manner b augment data on a person s performance while they work with data from other tasks and c provide assessments at multiple grain sizes We present Olae a computer assistant to a human assessor that collects data about problem solving in elementary physics analyzes that data with sound probabilistic methods and exibly presents the results of analysis Like it or not assessment is a part of education and training Decisions must be made about students teachers curricula and schools Making these decisions intelligently ultimately requires knowing what a student has learned This amounts to forming a model of the student s knowledge This model may represent a student s knowledge at varying levels of detail Fine grained models indicate whether a student has mastered a particular rule concept or subskill whereas large grained models indicate whether a student has mastered a more comprehensive piece of knowledge say a particular chapter in a textbook In general a ne grained student model is needed for decisions that only a ect the student for a short time such as deciding which exercise to pick next or how to explain a di cult concept Large grained models are needed for decisions that a ect the student for longer periods such as deciding which physics course to take or which summer job to take Student modeling is a form of assessment that was developed to help computer based tutoring systems decide which exercise to give to a student when to interrupt what level of explanation to give and so on This technology can also aid human decision makers who would otherwise rely on traditional assessment instruments such as multiple choice tests which are sometimes inadequate Collins For example multiple choice tests are often not representative of real work in a subject area where solving a problem can take hours or days Ideally the assessor would observe a person as they worked and over time obtain an accurate model of that person s strengths and weaknesses This is exactly what student modeling systems are designed to do However signi cant problems impede the adoption of student modeling as a bona de assessment tech nology We have identi ed three and are building a system that embeds solutions to all of them Most student modelers even the best ones e g Burton s Debuggy use heuristics to analyze the raw behavioral data However when important decisions must be made on the basis of assessments heuristics should be minimized One must be able to defend the conclusions of the student modeler to students employees employers and others Testing organizations are often taken to court and their analyses must meet or exceed the standards used by scientists in interpreting their data Thus building a student model that uses a statistically sound defensible data analysis is our rst goal for the system which we call Olae Our solution is to use Bayesian networks for the data analysis This technology which was originally developed for medical diagnosis is statistically sound The second problem is that merely monitoring a person s performance while they work does not always provide enough data Often the most interesting thinking occurs when the person is silent and motionless For instance when solving a problem people often pause for a while in order to mentally represent or idealize the problem and to formulate a solution plan Often it is impossible to infer what they are thinking from their observable behavior even if they are asked to talk aloud as they go Our solution to this problem is to include several tasks from the expert novice literature that have been shown to be sensitive to the problem representing and solution planning expertise of subjects The data from these tasks is integrated with performance data from ordinary tasks in order to form a more accurate assessment of the person s knowledge The third problem is that assessments are needed at multiple grain sizes in order to support decision making of many kinds A curriculum developer might want to see exactly which rules and concepts a subject mastered in order to determine if her instruction was e ective A student might want to see if she understands chapter well enough to go on to chapter A commander for a satellite tracking station might want to nd someone who knows celestial mechanics or could learn it quickly Our solution to this problem is to conduct the assessment at the nest grain size that we think the users will want and provide simple graphical tools for users to build coarser assessments on top of these ne grained ones The initial version of Olae is being developed to address these three problems with college physics as the task domain Physics was chosen for three reasons First many physics educators feel that traditional assessment instruments overrate student s understanding in physics e g Hestenes Wells Swackhamer We expect Olae to provide a more accurate assessment than traditional instruments and in par ticular to reveal how students can correctly answer traditional questions while having serious knowledge Olae is an acronym for On Line Assessment of Expertise because from the point of view of traditional paper and pencil based testing Olae looks like an on line system However Olae does not deliver its analyses in real time The students behavior is recorded and analyzed later Thus from the student modeling point of view the system does o line data analysis Fortunately the same acronym works for both on line and o line assessment of expertise Problem Solving Example Studying Solution Planning Conceptual Problem Solving Problem Classifying Assessor’s Interface Actions Rules . Rules Factor Hierarchy Bayesian Data Analyzer Task Interfaces Figure An overview of the three components of Olae Interface Tasks Bayesian data analysis and an Assessor s interface de cits Second physics was chosen because it has both procedural and conceptual content Third physics was chosen because many of the expert novice studies were conducted with physics as their task domain Using physics as the initial task domain for Olae reduces the risk in adopting these expert novice tasks as measures of expertise The remainder of this paper describes our design forOlae its current status and our plans for developing it further We conclude with an assessment of whether our three goals have been met