An Operational Model of Joint Attention - Timing of Gaze Patterns in Interactions between Humans and a Virtual Human

An Operational Model of Joint Attention - Timing of Gaze Patterns in Interactions between Humans and a Virtual Human Nadine Pfeiffer-Lessmann (nlessman@techfak.uni-bielefeld.de) Thies Pfeiffer (tpfeiffe@techfak.uni-bielefeld.de) Ipke Wachsmuth (ipke@techfak.uni-bielefeld.de) Artificial Intelligence Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany Abstract We constructed an operational model of joint atten- tion (Pfeiffer-Lessmann & Wachsmuth, 2009) for our vir- tual human Max (Lessmann, Kopp, & Wachsmuth, 2006) to create a more natural and effective interaction partner. The model covers four phases: the initiate-act (1), the respond-act (2), the feedback phase (3), and the focus-state (4). However, for Max to apppear believable and to use the same behavior patterns in the phases as humans do, investigations on time- frames, human expectations and insights on how humans ac- tually perceive his behavior are indispensable. The topic of concrete reaction and duration times of feedback behaviors during the joint attention process has to our knowledge not been discussed in the area of human-computer interaction yet. The time-frames and expectations of humans for natural in- teractions are central subject of this paper. In the section to follow, we provide an overview on related work covering research on joint attention in human-human interaction and in the area of technical systems. In the sub- sequent ”Model” section, a brief summary of our own defini- tion of joint attention is provided. Next, we present a study in immersive virtual reality concerning the exact timing of the first phase, the initiate-act, of our joint attention model. Thereafter, results are discussed and the paper ends with our conclusions and future work. Joint attention has been identified as a foundational skill in human-human interaction. If virtual humans are to engage in joint attention, they have to meet the expectations of their human interaction partner and provide interactional signals in a natural way. This requires operational models of joint at- tention with precise information on natural gaze timing. We substantiate our model of the joint attention process by study- ing human-agent interactions in immersive virtual reality and present results on the timing of referential gaze during the ini- tiation of joint attention. Keywords: joint attention; virtual humans; social interaction Introduction Attention has been characterized as an increased aware- ness (Brinck, 2003) and intentionally directed perception (Tomasello, Carpenter, Call, Behne, & Moll, 2005) and is judged to be crucial for goal-directed behavior. Joint atten- tion builds on attentional processes and has been identified to be a foundational skill in communication and interaction. The term joint attention is often used confusably with shared at- tention. We follow Kaplan and Hafner (2006) and Tomasello et al. (2005) in using the term joint attention for the phe- nomenon which presupposes a higher level of interactivity re- quiring intentional behavior and an awareness of the interac- tion partner. Joint attention can be defined as simultaneously allocating attention to a target as a consequence of attending to each other’s attentional states (Deak, Fasel, & Movellan, 2001). In contrast, we see shared attention (as well as shared gaze) as the state in which interactants are just perceiving the same object simultaneously without further constraints con- cerning their mental states or their interaction history. Mundy and Newell (2007) differentiate joint attention be- haviors into two categories: responses to the bids of others and spontaneous initiations. Responding to joint attention refers to the ability to follow the direction of gaze and ges- tures of others in order to share a reference. On the other hand, to initiate joint attention humans use gestures and eye contact to direct the attention of others to objects, events, and to themselves. For joint attention, interlocutors have to deliberatively fo- cus on the same target while being mutually aware of shar- ing their focus of attention (Tomasello et al., 2005; Hobson, 2005). To this end, respond and feedback behaviors are nec- essary. Tasker and Schmidt (2008) argue that to establish joint attention a sequence of behaviors is required which has to meet certain time constraints. Related Work Staudte and Crocker (2011) raise the question whether joint- attention-like behavior is unique to human-human interaction or whether such behaviors can play a similar role in human- robot interaction. They conclude that their own findings sug- gest that humans treat artificial interaction partners similar to humans and that it is therefore valid to investigate joint atten- tion in settings with artificial agents. These artificial agents can consist, on the one hand, of robots (Deak et al., 2001; Imai, Ono, & Ishiguro, 2003; Breazeal et al., 2004; Doniec, Sun, & Scassellati, 2006; Na- gai, Asada, & Hosoda, 2006; Yu, Schermerhorn, & Scheutz, 2012; Huang & Thomaz, 2011; Staudte & Crocker, 2011) and, on the other hand, of virtual humans (Peters, Asteri- adis, & Karpouzis, 2009; Zhang, Fricker, & Yu, 2010; Bailly, Raidt, & Elisei, 2010). Kaplan and Hafner (2006) point out that research in robotics concentrates only on partial and isolated elements of joint attention (e.g. gaze following, simultaneous looking or simple coordinated behavior) covering solely the surface of the process but not addressing the deeper, more cognitive