Rethinking the Role of Representation in HCI

Although critiques of computational models and rigi d representation schemes have been quite vocal in the past decades, by and large current system design still overwhelmi ngly relies on the equivalence between entities and processes from the outside world and their digital counterparts. My research d raws on and extends existing critiques to address the demands o f today’s technological trends and the limitations of traditi onal system design. My contribution is two fold: 1) it recasts the role of representation within system design to accommodate the growing interest in experience design and the accompanying shift towards the subjective, situated, personal and idiosyncrati c and 2) it argues for a holistic approach that seriously engages the limits of representation and permeates every aspect of system building (conceptualization, design, technical implementatio n, evaluation, etc). INTRODUCTION Computational technologies are inherently represent ational. In order to reason about the world, systems typically have access to relevant aspects of their environment via sensors o r input modalities; these aspects are typically represented internally through variables and models associated to objects, ntities or processes. From these representations outcomes are derived, which must hold in the outside world. As such, repr esentation gives computers the means to participate in their e nvironment. As expected, any representation is a simplification : t highlights certain features which appear to be general or rele vant and discards less frequent as well as harder to formali ze aspects. Computational models are certainly very useful when all the aspects relevant for the system’s success are captu red in the model. However, problems arise when the system’s fo cus is on aspects that are less general or the ones that resi s formalization. Indeed, this is old news in CSCW as the limitations f modeling users, users’ actions and work processes have been thoroughly discussed in the past two decades e.g. in the work of Suchman [10] and Robinson and Bannon [8]. In spite of these significant contributions, new an d old issues related to formalization surface in current HCI pra ctice. On the one hand, the growing interest in experience-focuse d technology brings out new tensions between the subjective, idi osyncratic aspects of human experience that are the focus of s uch ystems and the objective representations required by the t echnology. On the other hand, mismatches between internal represe ntations and the outside world are a recurrent source of difficu lt es in HCI and related fields, as evidenced by the omnipresent cal l for future work on better, more accurate models. This suggests that the lessons of Suchman and others have been incorporate d only superficially in current practice as current system s are still fundamentally constrained by the power, as well as the limitations, of such models. More specifically, alt hough these hard learned lessons have informed the conceptual a spects of the design process, they have not made their way into t he technical design and implementation level. As such, especiall y from a technical perspective, systems continue to be built w th the implicit assumption that representation is in stric t one-to-one correspondence with reality. While corrective meas ure may be taken through interface and interaction design, oft en the mismatches are deeply rooted in the technical imple entation and often little can be done to prevent their impact on the overall experience of the system. Consequently, my research focuses on 1) how to reposition representation to better accom m date both new demands and existing critiques and 2) advancing a holistic approach to system design, in which this reorientat ion of representation permeates every aspect of system dev elopment, including the technical level. EXPECTED CONTRIBUTION My PhD thesis aims to advance current research in H CI by repositioning the role of representation in system design: moving away from representation as a one-to-one mapping to reality to an understanding of representation as partial, incompl ete, subjective (i.e. influenced by the researcher’s assumptions, b eliefs, and epistemological commitments) and always situated in a particular socio-cultural context. Moreover, the gaps between r ality and representation should not be hidden deep in the sys t m’ technical design. Instead, shortcomings related to representa tio should be transparent and a central concern in every aspect o f system design: understanding representation as a digital s c ffolding of reality and reorienting the entire system design ar ound this understanding. One example of doing so is to use th e uncertainty and ambiguity which accompanies any representation s a way to engage users in meaning making as well as reflectio n [5], and allowing multiple meanings to emerge [9]. My research confronts these issues across the conti nuum of practices associated with system design: conceptual , design, ethnography, algorithm and technical. It is informe d by critiques pertaining to representation and formalization. Mor eover, while grounded in subfields of HCI (e.g. affective comput ing) in which these problems are most evident, my work proposes a methodology for approaching such issues through ill ustrative case studies demonstrating heuristics for design. CASE STUDIES Over the years, a number of researchers have tackle d the issues pertaining to representation and formalization. As such my work builds on conceptual critiques such as Winograd and Flores [11] and Agre [1] in AI, Robinson and Bannon in CSCW [8] , Brooks Copyright is held by the author/owner. CSCW’08, November 8-12, 2008, San Diego, California, USA. in robotics [3], and Dourish in HCI [4]. Perhaps mo st influential in HCI has been the work of Lucy Suchman [10] regar ding the fundamental mismatch between the models of action u nderlying planning research in AI in the 1980’s and actual, s ituated human activity in the world. Although concerned with simi lar issues, the debates within HCI and CSCW around representation h ave developed in parallel with debates within AI (with t e exception of ‘Plans and Situated Actions’ [10]). My work attempts to tap these resources outside HCI. One first project for my dissertation discusses the tactics developed within interactioni st approaches in AI with respect to representation and discusses the ir rel vance for today’s HCI [7]. These tactics were developed to av oid the conceptual and technical pitfalls associated with c omplete, one-toone representations. They demonstrate that one can achieve more with less: less in terms of effort spent representi ng and reasoning about the world and more in terms of the delivered experience and utility. One such tactic is to design for engaged audiences, thu making the believability of the system a central feature. This translates into a simplified system design, driven largely by the way the system will be interpreted by friendly users, rathe r than by its behavior when analyzed either in a vacuum or by a h ostile user. That is, system design can become simpler and more effective by making the experience and interpretation of the sys tem by users central to every design decision (including technic al), rather than only considering this when it comes to design the u s r interface. As a continuation, my current research attempts to sh w how these tactics can be used in today’s HCI. As such t he focus is on an every day technology such as GPS navigation devi ces. Currently, I am doing fieldwork in order to documen t breakdowns occurring in usage. These findings will then be use d to inform the redesign of such devices using more flexible repres entation techniques and incorporating the above mentioned ta ctics. A central tenet of my research is the belief that t e world is not completely available to us or to a computational sy stem. Consequently, it is crucial for the design of real world applications that representations inside the system are understood to be partial and not always in direct corresponden ce to the outside world. This becomes particularly obvious in today’s HCI focus on experience and also in light of the paradi gmatic shifts experienced by the field [2]. One area of HCI affec ted by such choices is affective computing (AC). Here the centr al challenge is how to address the subjective, idiosyncratic, perso nal nature of emotions within the rigid framework offered by tech nology. One difficulty lies in mapping sensor readings to t heir meaning. Indeed, this is not a problem specific to this area , but the limitations of such mappings are quite noticeable h ere: e.g. how to get from one’s heart rate to the felt emotion? My previous work shows how objective measures, such as physiological s gnals, do not directly map to affective meaning (as often ass umed in the literature). Through a participative study, my work uncovers a complex relationship between sensor readings and th eir significance and suggests methodologies based on us er interpretation to allow complex meanings to emerge [6]. Another difficulty in this area relates to statistical mode ls of emotions – emotion recognition by way of sensor readings. Concerned with scenarios in which user interpretati on s not feasible due to cognitive constraints placed on the us r, my current work analyses the ways such models are obta ined and raises epistemological questions with respect to sa id models ability to generalize. My current work in this area , suggests a reflective incorporation of such models in system d esign, one that carefully accounts for the models’ uncertainty and i accuracy, with respect to emotion recognition, as these model s can offer at best an educated guess regarding the emotions felt by the user, rather than an oracle which magically accesses the user’s affect. These findings are relevant beyond AC to other area s that fundamentally rely on the accur