Creating Bridges : T he Role of Exploratory Design

Designers of Intelligent Tutoring Systems (ITS) have long been interested in delivering personalised teaching to individual students, typically by ensuring that the student receives content appropriate to their skills and knowledge. Nonetheless, a more holistic view on what constitutes good teaching practice has challenged whether this approach to user modelling is sufficient. Teaching is not only defined by what is taught, but also by how it is taught. In this paper, we demonstrate that exploratory design research can support this view by generating a more inclusive set of user attributes for purposes of user modelling. Through a case study, we show that design research for user modelling can function as a boundary object serving three important roles, that underpin more specifically the design of user modelling and more broadly ITS design. First, design research can establish common ground by encapsulating domain knowledge in an accessible form. This can support diverse project stakeholders to make decisions on what is to be modelled. Second, design research can reveal a wide range of teaching and learning perspectives that in turn introduce transparency to the decision-making process of user modelling and provoke a sense of criticality and accountability amongst project stakeholders. Third, design research can build new bridges between the design of the technology and the user model that underpins it. To this end, user attributes deemed important, yet too complex or cumbersome to develop, can become design principles in the context of the overall ITS design. INTRODUCTION Design thinking begins with the understanding and exploration of a problem. This exploratory, early stage of design enables designers to break free from existing ways of thinking and knowing (Dorst, 2011). Design research – encompassing any research activity that supports the process of design – has been instrumental both in this early stage of design and in refining technology design once a direction has been decided (Vines et al., 2013). Even though this view on design has in many ways become ingrained in mainstream HCI, it has not always been adopted by communities that have overlapping interests with the HCI community. A prominent example is user modelling, an area of interest for researchers working on Artificial Intelligence (AI) and HCI. Within the field of user modelling, AI researchers have largely occupied themselves with the production of better algorithms through which to understand and model human intelligence. Design research has typically featured as a way to enrich researchers’ theories and perspectives, rather than to define or prioritise them. Meanwhile, HCI research has typically conversed with AI in improving the use of AI algorithms after they have been developed (Grudin, 2009). In cases where HCI researchers have sought to influence AI in the design stage, the process of doing so has not been straightforward. For example, reporting on their experience of designing an adaptive serious game, Vasalou and Khaled (2013) point out the difficulties they encountered when introducing generative design research in the project. Rooted in a tradition of data-driven design methods, AI researchers were hesitant to embrace the subjectivity that design methods invited. Thus, even though both HCI and AI researchers recognized the reciprocal interaction between user modeling and design, their divergent goals, epistemology, and related practices became a barrier to their collaboration. As the authors explain, the consequence of this was that design decisions between teams were frequently disjointed. While acknowledging these challenges, in this paper we advocate the view that design thinking can play an important role in the design of user modelling. In particular, we propose that exploratory design research for user modelling can act as a ‘boundary object’ (Star and Griesemer, 1989). In this capacity, it can help frame, inform and justify user modelling decisions, while at the same time it can influence a broader cohort of design decisions involved in the development of intelligent technology. This proposal is explored through a case study of an intelligent tutoring system (ITS) designed to teach reading (word decoding) skills to primary school children with dyslexia. The work we report contributes to our knowledge of design by demonstrating that design research can play three different roles that directly support design decision-making processes for user modelling and ITS. Firstly, design research can encapsulate domain knowledge in an accessible form that in turn creates common ground between diverse project stakeholders, providing all team members with the knowledge and confidence to make decisions. Secondly, given its inductive nature, design research can reveal a wide range of teaching and learning perspectives that introduce transparency to the design process and as a consequence provoke a sense of criticality and accountability amongst project stakeholders. Thirdly, design research for user modelling captures core teaching and learning knowledge that is of relevance more broadly to the design of ITS and can thus directly inform other design decisions and activities. In the following section we give an overview of current user modelling approaches used within ITS and their relationship to theories of learning. Next, we elaborate on why the introduction of design research in the exploratory stages of design can inspire new ways of approaching user modelling, as well as inform broader design decisions within ITS. We go on to present the application of design research within a case study of a project whose aim is to develop an ITS to support children with dyslexia in acquiring reading skills. The implications of this case study are discussed, focussing on how our design research acted as a boundary object within our multidisciplinary team collaboration to inform user modelling and ITS design. BACKGROUND It has long been recognised that personalised education is more effective than a one-size-fits-all teaching approach. Bloom showed that an average student who received one-to-one tutoring from an expert tutor scored two standard deviations higher than an average student taught in a traditional group-based instructional setting (Bloom, 1984). Cohen and colleagues (1982) found a similar result (though with a weaker effect size) based on a meta-analysis of tutoring more broadly. While personalised education has recently been forecast as one of the global challenges we face in coming years (LaFontaine, 2013), for decades it has been a mainstay objective in the field of ITS, which has aspired to construct personalised learning experiences that “make the benefits of individualized instruction available to all students at affordable costs” (Anderson, 1992, p. 3). The main approach to delivering personalised education with technology is through a user model. User modelling is based on a simple idea: By having information about a specific individual, technology can make decisions that are best suited to that individual. Any technology that behaves differently for different users employs a user model. User models can be big or small, complex or simple, rich or sparse. Orwant (1996) captures the essence of user modelling by explaining that, “user modelling is nothing more than a fancy term for automated personalisation. Humans model each other all the time. I am modelling you as I write; my topics, presentation, and language are all aimed at a hypothetical, average reader of this journal. If I have guessed well, you will enjoy this essay. If not, you will skip to the next one. That is what user modelling systems do – they make guesses, and hopefully educated ones, about their users” (Orwant, 1996, p. 398). ITS researchers have focused their modelling efforts on a number of key areas. Some researchers have modelled student knowledge. For example, Anderson (1992) developed a high school mathematics tutor using pre-existing maths curriculum as a foundation for progressing students through the software. Taking a different approach to modelling knowledge, Radlinski and McKendree (1992) created the COBOL tutor based on an understanding of the level of difficulty required to solve particular problems. COBOL was thus based on a production system model of an “ideal student”, suggesting what coding errors a student has made and what type of code should follow what they have written. A second area of research has been concerned with the modelling of student affect. Conati, for example, worked on detecting students’ affect during the use of educational game, using a combination of physical sensors and log files to detect their emotions while playing the game. Depending on which emotions were detected, a game character appeared and changed the form of feedback it gave to the player (Conati et al., 2003; Conati and Maclaren, 2009). A second representative example is work by Mota and Picard (2003) who developed a model that relied on a student’s posture to infer their interest during a learning task, which they proposed could be used in turn to provide assistance or encouragement at the right time. A third area of development has focused on modelling motivation. Costagliola et al. (2010) detailed a method for modelling attention from body posture. Given the student’s motivation, the system changed the topic and level of content being presented. These examples indicate consensus regarding the requirement to tailor learning experiences to students’ individual characteristics and needs. However, despite this common ground, the diversity of these applications provides evidence to the different approaches taken to solving the same design problem of how to best personalise an educational experience through technology. For example, when developing a user model for a maths tutor, a fundamental difference exists between modelling the acquisition of knowledge (e.g. the