MOG 2010 3 rd Workshop on Multimodal Output Generation

Most extant models of verbal reference to objects in a shared domain of conversation, specifically in the field of Natural Language Generation, focus on description: the use of symbolic means to uniquely identify a referent. Generation of multimodal referring acts requires a change in focus. In particular, demonstratives combined with pointing gestures are primarily a form of indicating. In my talk, I will discuss two issues which this change in emphasis brings with it. Firstly, I will examine the evidence for two varieties indicating: neutral and intense indicating, which, I will argue, are associated with the distal and proximal form of demonstrative noun phrases, respectively. Secondly, I will examine how pointing gestures can be incorporated into existing work on the generation of referring expressions. I will show that in order to add pointing, the notion of salience needs to play a pivotal role. After distinguishing two opposing approaches: salience-first and salience-last accounts, I will discuss how a salience-first account nicely meshes with a range of existing empirical findings on multimodal reference. A novel account of the circumstances under which speakers choose to point is described that directly links salience with pointing. The account is placed within a multi-dimensional model of salience for multimodal reference. References Piwek, P. (2009). Salience in the Generation of Multimodal Referring Acts. In Proceedings of the 2009 International Conference on Multimodal Interfaces (ICMI-MLMI), pages 207–210, Cambridge, MA. ACM Press. Piwek, P., Beun, R. , and Cremers, A. (2008). ‘Proximal’ and ‘Distal’ in language and cognition: evidence from deictic demonstratives in Dutch. Journal of Pragmatics, 40(4):694–718. Collaborative Design of Multimodal Output Gavin Doherty Trinity College Dublin Dublin, Ireland Gavin.Doherty@scss.tcd. ie Abstract A major theme of Human-Computer Interaction (HCI) research has been on facilitating user participation in the design of interactive computer systems, including the use of participatory design processes in which users form part of the design team. A further development is the emergence of a range of informative systems in which the content or information being delivered is generated by end-users or domain experts. This content may be delivered in a multimodal fashion, and hence we must consider the future of multi-modal output generation (MOG) technologies to be one in which the final design depends on the technology developers, domain experts and end users. To illustrate, I discuss work in the area of technology support for mental health care, where we have made extensive use of collaborative design methods. The systems developed made use of virtual characters in 3D computer games, video (including animations), mobile phones, Internet charts, and (importantly) paper. The model used was one in which development of the platform was separated from development of content, but each was a collaborative process, one led by the technology developers, the other by domain experts. The user experience emerges from the combination of the two, but the focus of each design effort is different. While I reflect on the potential use of MOG in this area, the focus of the talk will be on the nature of the design problem facing those trying to develop produce informative, affective and engaging experiences using multimodal output, and how this may impact on the future of MOG.A major theme of Human-Computer Interaction (HCI) research has been on facilitating user participation in the design of interactive computer systems, including the use of participatory design processes in which users form part of the design team. A further development is the emergence of a range of informative systems in which the content or information being delivered is generated by end-users or domain experts. This content may be delivered in a multimodal fashion, and hence we must consider the future of multi-modal output generation (MOG) technologies to be one in which the final design depends on the technology developers, domain experts and end users. To illustrate, I discuss work in the area of technology support for mental health care, where we have made extensive use of collaborative design methods. The systems developed made use of virtual characters in 3D computer games, video (including animations), mobile phones, Internet charts, and (importantly) paper. The model used was one in which development of the platform was separated from development of content, but each was a collaborative process, one led by the technology developers, the other by domain experts. The user experience emerges from the combination of the two, but the focus of each design effort is different. While I reflect on the potential use of MOG in this area, the focus of the talk will be on the nature of the design problem facing those trying to develop produce informative, affective and engaging experiences using multimodal output, and how this may impact on the future of MOG. A Preprocessing System to Include Imaginative Animations According to Text in Educational Applications ∗ Éric Charton, Michel Gagnon, Benôıt Ozell École Polytechnique de Montréal 2900 boulevard Édouard-Montpetit, Montréal, QC H3T 1J4, Canada {eric.charton|michel.gagnon|benoit.ozell}@polymtl.ca