Multimodal Networks of Interpersonal Interaction and Conversational Contexts

Multimodal Networks of Interpersonal Interaction and Conversational Contexts Alexandra Paxton (paxton.alexandra@gmail.com) Rick Dale (rdale@ucmerced.edu) Cognitive and Information Sciences, University of California, Merced Merced, CA 95341 USA Abstract In interpersonal interaction, the terms synchrony or alignment refer to the way in which communication channels like speech or body movement become intertwined over time, both across interlocutors and within a single individual. A recent trend in alignment research has targeted multimodal alignment, exploring how various communication channels affect one another over time (e.g., Louwerse et al., 2012). While existing research has made significant progress in mapping multimodal alignment during task-based or positively valenced interactions, little is known about the dynamics of multimodal alignment during conflict. We visualize multimodal alignment during naturalistic affiliative and argumentative interactions as networks based on analyses of body movement and speech. Broadly, we find that conversational contexts strongly impact the ways in which interlocutors’ movement and speech systems self-organize interpersonally and intrapersonally. Keywords: alignment; conflict; conversation; interaction; movement; network; speech; synchrony Introduction Interpersonal communication is a multimodal activity. Conversation incorporates multiple channels of communication that enrich the interaction, like hand gestures, facial expressions, posture, and speech. To effectively communicate with one another, interlocutors cue in to each of these channels simultaneously, often without realizing the importance placed on each of them. Considerable work has surveyed multimodal qualities of interaction (e.g., Norris, 2004). However, in general, experimental study of interpersonal communication in cognitive science tends to target single behavioral channels. This has led to significant advances in our understanding of these specific channels, but there is still much work to be done in investigating the connections among them. Continued multimodal research will likely yield interesting—and possibly unexpected—relationships among channels that have been extensively explored unimodally. For this reason, the current research explores multimodal alignment situated within different conversational contexts. Specifically, the present research examines interpersonal communication through alignment dynamics. Research on interpersonal alignment focuses on how affect, behavior, and cognition of interacting individuals affect one another over time. Over the past several decades, researchers have explored interpersonal alignment over a range of channels, from movement (e.g., Richardson et al., 2007) to speech (e.g., Garrod & Pickering, 2004) to cognition (e.g., Brennan, Galati, & Kuhlen, 2010). As aforementioned, the majority of this work centers on one or two behavioral channels, but a growing body of literature has begun to investigate how multiple channels align during communication (e.g., Louwerse, Dale, Bard, & Jeuniaux, 2012). A distinction can be drawn between alignment in a general sense and synchrony. 1 We use the term alignment to refer broadly to the concept that individuals, over time, change their affect, behavior, and cognition as a direct result of their interaction with another individual. This umbrella term encompasses everything from mimicry, in which individuals are performing highly similar behaviors to their interaction partner (e.g., Chartrand & Bargh, 1999), to more complementary behavior patterns like synergy (e.g., Riley, Richardson, Shockley, & Ramenzoni, 2011). Synchrony, on the other hand, can be considered a specific pattern of alignment and refers exclusively to the in-phase entrainment of behavior or communication channels. In the spirit of intrapersonal alignment research spanning the last several decades (e.g., Haken, Kelso, & Bunz, 1985), the current research explores the dynamics of interpersonal alignment as a self-organizing property of human interaction. The present research focuses specifically on two channels of communication—speech and body movement— and the ways in which these channels are affected by each other, by conversation partners, and by the conversational context. If it is true that human interaction self-organizes around, for example, conversational goals, we should expect to see multimodal alignment patterns changing across these different contexts. Body Movement and Speech Alignment Previous work on speech and body movement demonstrates rich interactivity between and within individuals. Some of the earliest work in alignment research presents evidence for interpersonal and intrapersonal multimodal alignment between movement and speech channels (Condon & Ogston, 1966). Since then, research has continued to explore intrapersonal alignment, both in speech (Reitter, Moore, & Keller, 2010) and movement (Beek, Peper, & Daffertshofer, 2002). However, body movement and speech have both been studied extensively within the interpersonal alignment literature as well, generally within affect-neutral, positively valenced, or task-oriented settings. Studies of body movement alignment have spanned a wide variety of behaviors, including gesture (Bernieri & These specific terminologies are laid out here for the purpose of the present paper, rather than trying to resolve the emerging terminological debate within the field.