Subgroup Discovery with User Interaction Data: An Empirically Guided Approach to Improving Intelligent Tutoring Systems

Introduction Learner modeling is a critical component in the process of adapting instruction to the specific needs of different learners with computerized systems. Adaptive instructional systems may be defined as a systematic process which consists of four steps: (1) capturing information about the learner; (2) analyzing learner interactions through a model of learner characteristics in relation to the domain; (3) selecting the appropriate instructional content and resources; and (4) delivering the content to the learner (Shute & Zapata-Rivera, 2012). The analytical function of the learner model can be further classified in terms of processes conducted at both the macro and micro levels (VanLehn, 2006). At the macro-level, a representation of the path towards competency within the domain is updated for each task with the aim of selecting the next task that is the most appropriate for the learner. At the micro-level, instructional materials such as hints and feedback are delivered to the learner on the basis of a representation that is repeatedly updated over the duration of task performance. Intelligent tutoring systems have been shown to improve upon typical classroom instruction in several disciplines, including mathematics (e.g., Cognitive Tutors; Koedinger & Corbett, 2006), computer science (e.g., Constraint-Based Tutors; Mitrovic, 2003), microbiology (e.g., Narrative-Based Tutors; Rowe, Shores, Mott, & Lester, 2011), as well as physics and computer literacy (e.g., Dialogue-Based Tutors; Graesser, VanLehn, Rose, Jordan, & Harter, 2001). The challenge in modeling learning in the context of solving ill-structured problems is that there are multiple paths towards attaining the correct solution (Lajoie, 2003, 2009). These paths should be documented in order to represent common misconceptions or impasses in moving along the trajectory towards competency. In the medical domain, experts have been shown to diagnose patient diseases in different ways, while at the same time, justifying plausible hypotheses on the basis of common evidence items, including patient symptoms and lab test information (Gauthier & Lajoie, 2014; Lajoie, Gauthier, & Lu, 2009). An intelligent tutoring system such as BioWorld can represent these evidence items in terms of a novice-expert overlay system, which compares novice solution paths to those of the experts in order to individualize feedback. A similar modeling technique has been applied in other systems, such as SlideTutor in diagnosing dermatopathology (Feyzi-Behnagh, Azevedo, Legowski, Reitmeyer, Tseytlin, & Crowley, 2014) as well as the MedU virtual patient cases (Berman, Fall, Chessman, Dell, Lang, Leong, Nixon, & Smith, 2011). On the basis of the user interactions that are recorded by BioWorld, the system will highlight areas of similarities and differences with the expert solution path, allowing novices to self-reflect on their own approach to resolving the problem (Lajoie & Poitras, 2014). Several studies investigating the novice-expert overlay model have been carried out with the aim of capturing linguistic features from written case summaries to tailor feedback content (Poitras, Doleck, & Lajoie, 2014; Lajoie, Poitras, Doleck, & Jarrell, 2015) as well as the impact of goal-orientations and affective reactions towards attention given to feedback (Lajoie, Naismith, Poitras, Hong, Panesso-Cruz, Ranelluci, & Wiseman, 2013). In this study, we examine the use of subgroup discovery for the induction of rules that characterize the relationship between impasses in problem-solving and lab-tests ordered in BioWorld. Whereas our previous research characterized how experts converged in their paths to solving a problem, the present study captures how novices diverged from an expert solution path. In doing so, we claim that subgroup discovery algorithms are particularly well suited towards describing the multiple paths that characterize problem-solving in ill-structured domains. …