# Reactive Virtual Agents: A Viewpoint-Driven Approach for Bodily Nonverbal Communication

Pierre Raimbaud, Alberto Jovane, Katja Zibrek, Claudio Pacchierotti, Marc Christie, Ludovic Hoyet, Julien Pettré, Anne-Hélène Olivier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Pierre Raimbaud, Alberto Jovane, Katja Zibrek, Claudio Pacchierotti, Marc Christie, et al. Reactive Virtual Agents: A Viewpoint-Driven Approach for Bodily Nonverbal Communication. IVA 2021: ACM International Conference on Intelligent Virtual Agents, Sep 2021, Virtual Event Japan, France. pp.164-166, 10.1145/3472306.3478351. hal-03349919

HAL Id: hal-03349919

https://hal.science/hal-03349919

Submitted on 20 Sep 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Reactive Virtual Agents: A Viewpoint-Driven Approach for Bodily Nonverbal Communication

Pierre Raimbaud*

pierre.raimbaud@inria.fr

Inria, Univ Rennes, CNRS, IRISA

Rennes, France

Alberto Jovane

alberto.jovane@inria.fr

Inria, Univ Rennes, CNRS, IRISA

Rennes, France

Katja Zibrek

katja.zibrek@inria.fr

Inria, Univ Rennes, CNRS, IRISA

Rennes, France

Claudio Pacchierotti

claudio.pacchierotti@irisa.fr

CNRS, Univ Rennes, Inria, IRISA

Rennes, France

Marc Christie

march.christie@irisa.fr

Univ Rennes, CNRS, IRISA, INRIA

Rennes, France

Ludovic Hoyet

ludovic.hoyet@inria.fr

Inria, Univ Rennes, CNRS, IRISA

Rennes, France

Julien Pettré

julien.petre@inria.fr

Inria, Univ Rennes, CNRS, IRISA

Rennes, France

Anne-Hélène Olivier

anne-helene.olivier@univ-rennes2.fr

Univ Rennes, Inria, CNRS, IRISA, M2S

Rennes, France

<figure><img src="image_3.png" /><figcaption>**Figure 1:** Waving case from left to right: two Intelligent Virtual Agents (IVA) – A and B – observe another one; visual motion features – body orientation, gesture amplitude – are computed on the observed IVA’s motions from each viewpoint; B reacts</figcaption></figure>

## ABSTRACT

Non-verbal communication body cues are paramount to interact. In this preliminary work, we explore ways to let Intelligent Virtual Agents (IVAs) simulating nonverbal communication capabilities. We propose an approach to control IVAs’ reactive behaviour from the analysis of other agents’ apparent motions, in a situation of “observed” IVAs that act and “observers” that react. For that, first a viewpoint-driven analysis of the observed agent’s motion is done, and then a synthesis of this analysis induces the observers’ reaction.

## CCS CONCEPTS

* **Computing methodologies → Simulation environments; Motion processing; Procedural animation.**

## KEYWORDS

virtual agents, reactivity, animation, motion features, visual features

## ACM Reference Format:

Pierre Raimbaud, Alberto Jovane, Katja Zibrek, Claudio Pacchierotti, Marc Christie, Ludovic Hoyet, Julien Pettré, and Anne-Hélène Olivier. 2021. Reactive Virtual Agents: A Viewpoint-Driven Approach for Bodily Nonverbal Communication. In *21th ACM International Conference on Intelligent Virtual Agents (IVA '21), September 14-17, 2021, Virtual Event, Japan*. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3472306.3478351

## 1 INTRODUCTION

Intelligent Virtual Agents (IVA) should be designed to perform realistic and expressive interactions [3] that convey emotions and express personalities [23, 26]. These are notably expressed by humans through nonverbal behaviours in everyday interactions [8, 12]. Thus, the ability to perform nonverbal communication is paramount for IVA design [6, 21]. Nonverbal communication, notably reflected by proxemics and kinesics [17], contributes to the social realism of IVA interactions. In this regard, user studies have been conducted to evaluate generative models of IVAs’ expressive facial motions [11, 14, 22] or body motions [18]. These last ones in particular – divided into postures and gestures [5, 16, 19] – appear to be the key to expressivity in nonverbal communication between humans [2, 10], and with IVAs [1, 27, 29]. Ravenet et al. [24] proposed a model to generate bodily nonverbal communication motions from the agents’ roles in a conversation and some social attitudes and

<footnote>Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

IVA '21, September 14-17, 2021, Virtual Event, Japan

© 2021 Copyright held by the owner/author(s).

ACM ISBN 978-1-4503-8619-7/21/09.

https://doi.org/10.1145/3472306.3478351</footnote>
norms such as group formations. Their results confirmed the benefits of such nonverbal cues for more expressive social virtual agents. In line with this, Oshita et al. [20] focused on the design and use of body postures for attractive IVAs, and Zacharatos et al. [28] analysed body gestures through motion features such as velocity, derived from Laban Movement Analysis, to link them to emotions.

Nonetheless, ensuring the expressivity of IVAs' actions is not the only challenge related to bodily nonverbal communication. IVAs should also embed the ability to react to humans or other IVAs through adapted bodily communication [4]. In the literature, few authors focused on this aspect. Garcia-Rojas et al. [13] proposed a reactive motion model for reflexes according to stimuli and human individual characteristics – catch or dodge a ball – from real human motions analysis, to improve IVAs' individualisation through the personalising of reactions. In the context of conflicts, Campos et al. [7] studied the link of the IVAs' perception of emotions through body, facial motions and speech with reactive behaviours triggering.

Relying on the perception-action loop involved during human interactions [15], this work introduces a new concept to design IVAs with reactive body motions based on a viewpoint-driven approach. We present a new paradigm for reactive behaviours simulation, based on the visual analysis of body movements. In this short paper, we focus on situations where the poses of all agents are known. However, the same idea can easily be extended to more complex scenarios where only partial information about the agents' poses is available. Our viewport-driven approach first analyses the “observed” IVA’s motions from the viewpoint of the “observer” IVA, and then adjusts the observer IVA’s reactions. Section 2 presents this approach, Section 3 a case study, and Section 4 our future work.

## 2 EGOCENTRIC VIEWPOINT APPROACH

Our main contribution is the development of a new approach to improve reactive interaction simulations – either between virtual agents or between real and virtual agents, based on the visual analysis of motions in nonverbal communication situations. Our overall approach is described in Figure 2 and encompasses two steps: an egocentric perspective step, which provides an analysis of the “observed” IVA’s motions from the “observer” IVAs’ viewpoint – the apparent motions as for them – and a synthesis step about this visual analysis to induce reactive motions of “observer” IVAs.

<figure><img src="image_4.png" /><figcaption>Figure 2: Our viewpoint-driven approach for IVAs' reactions</figcaption></figure>

### 2.1 Visual motion features

Our first step consists in expressing the observed IVAs' motions in the egocentric viewpoint of the observer IVAs, fitting thus their visual perception of the current interaction between the IVAs. Our

originality here is thus the reference frame that we use for the motion feature analysis: rather than considering motions as they are performed [28], we focus on how they are visually perceived, i.e. the motions – postures and gestures – as they are apparent from the observers’ point of view. Here visual motion features are extracted from the positions and orientations of the observed IVA bodies, as they appear in the view space of the observers, using 2D projections. For example, we analyse the visual extension of the arms as they are apparent for the observers, rather than computing their absolute extension from the body current state in a 3D basis.

According to nonverbal communication motion classifications [5, 16, 19], we define here two categories of visual motion features, the apparent postures ones and the apparent gestures ones. Features related to posture are: *body orientation, body coverage, body self-coverage – amount of overlapping body parts on others – and body horizontal and vertical extension*. Features related to gesture are: *the amplitude and the velocity of the limbs used in a gesture*.

### 2.2 Synthesis of visual motion features results and reactive motions generation

The second step of our approach consists of analysing the results obtained about the visual motion features and in making a synthesis of it to induce reactive motions of the observers. Our key idea is to use the perception-action loop approach [15] for reactions [9, 15, 25], and to link the motion analysis from the observers’ perception to the generation of reactive behaviours for them, in response to the other agents’ body motions. Here we have addressed situations where this synthesis and generation step has been simplified and adapted for the specific case study presented in the next section, as a selection and then a triggering of a predefined reaction.

## 3 A CASE STUDY: WHO WILL WAVE BACK?

To exhibit our viewpoint-driven approach, we present an illustrative case study with the following nonverbal communication situation: an observed IVA waves at two other IVAs that will be the observers. This situation is shown on the left part of Figure 1. We aim to show that our approach can allow for the detection of which observer must react with a waving back gesture, by determining towards whom the observed IVA’s waving motion is more directed to.

In this case study, in our first step we used one postural visual motion feature, the apparent body orientation – specifically the one of the head and of the waving hand, and one gesture visual motion feature, the apparent amplitude of the waving hand. The apparent body part orientation is computed as the difference of alignment between the observed IVA’s body part orientation and the observer IVA’s view orientation, 0° thus being the value associated to a maximal interaction between the two IVAs. The apparent amplitude of the waving hand is computed as the area covered by this gesture in the view space of the observer. The bigger the area, the more directed towards an observer the observed IVA’s gesture is.

Figure 1 shows on its central part the comparison between the observed IVA’s motions perception of the two observers from their viewpoint: the apparent body orientation is shown by yellow vectors, and the apparent amplitude of the waving hand by yellow areas. The following values have been computed over a 1.7sec waving motion, previously obtained through motion capture. In terms
of posture, orientation results show that the observed IVA is more oriented towards observer B (head: 19.5° ± 4, hand: 20.6° ± 8) than observer A (head: 38.2° ± 4, hand: 40.4° ± 8). Similarly, waving hand amplitude results reveal that the waving is more extensively perceived by observer B (5.47 · 10⁻³%) than observer A (3.99 · 10⁻³%). According to these visual motion features results, a waving back reaction is triggered in observer B, as shown on the right of Figure 1.

## 4 LIMITATIONS AND FUTURE WORK

One limitation here is the omniscient comparative aspect that we used in our case study for the synthesis of reactive motions, which was adapted for such a predefined situation. We plan to develop a computational approach that could be used for environments where we only know partial information, and also for the creation of new reactive motion animations by modifying them depending on the computed features. In a multi-agents scenario, this would allow for the design of not fully predefined non-player character reactive behaviours, e.g. in games. Additionally, on more "spontaneous" cases, the observers' reactions should be independent and guided by visual thresholds over the viewpoint-driven motion features, instead of the omniscient point of view and agents' control used here. For such future work, threshold levels should be determined, e.g. through data about real humans' interactions, and also to manage the non-trivial issue of combining the results obtained from different features. Moreover, we could try to generalise our approach to scenarios with crowds. In such cases, body orientation and amplitude of moving limbs may not be considered as sufficient to induce a reaction, because of the crowd's effect on observers' perception: the agent's postural space in the observers' view space should be evaluated. Thus, in our approach it would consist in computing additional visual features such as horizontal or vertical body extension and body coverage. Finally, our viewpoint-driven approach aims to be applied for IVA - real agent interactions, in a virtual reality environment notably, with real-time tracking of user movements, where agents would be reacting to the embodied user.

## ACKNOWLEDGMENTS

This work received funding from the European Union under grant agreement No 856879 (H2020 ICT-25 RIA PRESENT project), as well as the Brittany Region, project SAD-2019-REACTIVE, No 19023.

## REFERENCES

[1] Simon Alexanderson, Carol O'sullivan, Michael Neff, and Jonas Beskow. 2017. Mimebot—Investigating the Expressibility of Non-Verbal Communication Across Agent Embodiments. *ACM Trans. Appl. Percept.* 14, 4, Article 24 (Sept. 2017), 13 pages. https://doi.org/10.1145/3127590

[2] Michael Argyle. 2013. *Bodily communication*. Routledge, London, UK.

[3] N. Badler. 1997. Real-time virtual humans. In *Proceedings The Fifth Pacific Conference on Computer Graphics and Applications*. IEEE, USA, 4–13.

[4] Atef Ben Youssef, Mathieu Chollet, Hazaël Jones, Nicolas Sabouret, Catherine Pelachaud, and Magalie Ochs. 2015. Towards a Socially Adaptive Virtual Agent. In *Intelligent Virtual Agents*, Willem-Paul Brinkman, Joost Broekens, and Dirk Heylen (Eds.). Springer International Publishing, Cham, 3–16.

[5] Peter E Bull. 2016. *Posture & gesture*. Vol. 16. Elsevier, Oxford, United Kingdom.

[6] Judee K Burgoon and Aaron E Bacue. 2003. *Nonverbal communication skills*. Lawrence Erlbaum Associates Publishers, Mahwah, New Jersey, USA.

[7] Henrique Campos, Joana Campos, Carlos Martinho, and Ana Paiva. 2012. Virtual Agents in Conflict. In *Intelligent Virtual Agents*, Yukiko Nakano, Michael Neff, Ana Paiva, and Marilyn Walker (Eds.). Springer, Berlin, Heidelberg, 105–111.

[8] W. Congreve and Robert Gifford. 2006. *Personality and nonverbal behavior: A complex conundrum*. SAGE Publications, Inc, Thousand Oaks, USA, 159–180.

https://doi.org/10.4135/9781412976152.n9

[9] E. Datteri, G. Teti, C. Laschi, G. Tamburrini, G. Dario, and E. Guglielmelli. 2003. Expected perception: an anticipation-based perception-action scheme in robots. *Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453)* 1 (2003), 934–939 vol.1. https://doi.org/10.1109/IROS.2003.1250748

[10] Beatrice De Gelder. 2009. Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. *Philosophical Transactions of the Royal Society B: Biological Sciences* 364, 1535 (2009), 3475–3484.

[11] Marc Fabri, Moore DJ, and Hobbs DJ. 2002. Expressive Agents: Non-verbal Communication in Collaborative Virtual Environments. In *Proceedings of Autonomous Agents and Multi-Agent Systems (Embodied Conversational Agents)*. Autonomous Agents and Multi-agent Systems - AAMAS, USA.

[12] Robert Stephen Ed Feldman and Bernard Ed Rimé. 1991. *Fundamentals of nonverbal behavior*. Editions de la Maison des Sciences de l'Homme, Paris, France.

[13] Alejandra García Rojas, Frédéric Vexo, and Daniel Thalmann. 2006. Individualized Reaction Movements for Virtual Humans. In *Proceedings of the 4th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia (Kuala Lumpur, Malaysia) (GRAPHITE '06)*. Association for Computing Machinery, New York, USA, 79–85. https://doi.org/10.1145/1174429.1174442

[14] Alejandra García-Rojas, Frédéric Vexo, Daniel Thalmann, Amaryllis Rauzaioui, Kostas Karpouzis, S Kollias, Laurent Moccozet, and Nadia Magnenat-Thalmann. 2006. Emotional face expression profiles supported by virtual human ontology. *Computer Animation and Virtual Worlds* 17, 3-4 (2006), 259–269.

[15] James Jerome Gibson and James J Gibson. 1986. *The ecological approach to visual perception*. Vol. 1. Psychology Press, New York, USA.

[16] Anthony Guye-Vuillème, Tolga K Capin, S Pandzic, N Magnenat Thalmann, and Daniel Thalmann. 1999. Nonverbal communication interface for collaborative virtual environments. *Virtual Reality* 4, 1 (1999), 49–59.

[17] Jinni Harrigan, Robert Rosenthal, Klaus R Scherer, and Klaus Scherer. 2008. *New handbook of methods in nonverbal behavior research*. Oxford University Press, Oxford, United Kingdom.

[18] Jing Huang and Catherine Pelachaud. 2012. Expressive Body Animation Pipeline for Virtual Agent. In *Intelligent Virtual Agents*, Yukiko Nakano, Michael Neff, Ana Paiva, and Marilyn Walker (Eds.). Springer, Berlin, Heidelberg, 355–362.

[19] Warren Lamb. 1965. *Posture and gesture: An introduction to the study of physical behaviour*. G. Duckworth, London, UK.

[20] Masaki Oshita, Kei Yamamura, and Aoi Honda. 2017. Expressive Human Pose Deformation Based on the Rules of Attractive Poses. In *Interactivity, Game Creation, Design, Learning, and Innovation*. Springer International Publishing, Cham, 162–171.

[21] C. Pelachaud. 2009. Modelling multimodal expression of emotion in a virtual agent. *Philosophical Transactions of the Royal Society B: Biological Sciences* 364 (2009), 3539–3548.

[22] Catherine Pelachaud. 2009. Studies on gesture expressivity for a virtual agent. *Speech Communication* 51, 7 (2009), 630–639.

[23] Tanmay Randhavane, Aniket Bera, Kyra Kapaskis, Rahul Sheth, Kurt Gray, and Dinesh Manocha. 2019. EVA: Generating Emotional Behavior of Virtual Agents Using Expressive Features of Gait and Gaze. In *ACM Symposium on Applied Perception 2019* (Barcelona, Spain) (SAP '19). Association for Computing Machinery, New York, USA, Article 6, 10 pages. https://doi.org/10.1145/3343036.3343129

[24] Brian Ravenet, Angelo Cafaro, Beatrice Biancardi, Magalie Ochs, and Catherine Pelachaud. 2015. Conversational behavior reflecting interpersonal attitudes in small group interactions. In *International Conference on Intelligent Virtual Agents*. Springer International Publishing, Cham, 375–388.

[25] Anna Schubö, Gisa Aschersleben, and Wolfgang Prinz. 2001. Interactions between perception and action in a reaction task with overlapping SR assignments. *Psychological Research* 65, 3 (2001), 145–157.

[26] Maayan Shvo, Jakob Buhmann, and Mubbasir Kapadia. 2019. An Interdependent Model of Personality, Motivation, Emotion, and Mood for Intelligent Virtual Agents. In *Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents* (Paris, France) (IVA '19). Association for Computing Machinery, New York, USA, 65–72. https://doi.org/10.1145/3308532.3329474

[27] Shawna CG Tan and Alexander Nareyek. 2009. Integrating facial, gesture, and posture emotion expression for a 3D virtual agent. In *Proceedings of the 14th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGames 2009)*. CGames, USA, 23–31.

[28] Haris Zacharatos, Christos Gatzoulis, Yiorgos Chrysanthou, and Andreas Aristidou. 2013. Emotion Recognition for Exergames Using Laban Movement Analysis. In *Proceedings of Motion on Games* (Dublin 2, Ireland) (MIG '13). Association for Computing Machinery, New York, USA, 61–66. https://doi.org/10.1145/2522628.2522651

[29] Katja Zibrek, Ludovic Hoyet, Kerstin Ruhland, and Rachel McDonnell. 2015. Exploring the Effect of Motion Type and Emotions on the Perception of Gender in Virtual Humans. *ACM Trans. Appl. Percept.* 12, 3, Article 11 (July 2015), 20 pages. https://doi.org/10.1145/2767130