Virtual Humans: A New Toolkit for Cognitive Science Research

Virtual Humans: A New Toolkit for Cognitive Science Research Jonathan Gratch (gratch@ict.usc.edu), Arno Hartholt (hartholt@ict.usc.edu), Morteza Dehghani (morteza@ict.usc.edu), Stacy Marsella (marsella@ict.usc.edu) Institute for Creative Technologies, University of Southern California 12015 Waterfront Drive, Playa Vista, CA 90094, USA Social Cognition: People interact socially through their bodies and VHs allow researchers to systematically examine and model the cognitions underlying social interaction. VHs can act as “virtual confederates” (Blascovich et al., 2002), allowing systematic manipulation of visual appearance, speech type, and contextual graphical environments. This makes VHs a convenient platform to isolate unique socio- cultural characteristics and realize them through simulation. Along with enhanced experimental control, ease of manipu- lations, consistency and controlled measurements, these features make VHs useful and reliable tools for studying social cognitions. In the tutorial, we will review several ex- amples, including how expressions of emotion by VHs can influence decision making in negotiations tasks and social dilemmas (e.g. de Melo et al., 2012; Dehghani et al., 2012); the role of accent in cultural cognition (Dehghani et al., 2012) and the role of rapport and gender in enhancing par- ticipants’ performance (Karacora et al., 2012). Keywords: Virtual humans, embodied cognition, social cog- nition, virtual confederates Tutorial Objectives Virtual humans (VHs) are digital anthropomorphic charac- ters that exist within virtual worlds but are designed to per- ceive, understand and interact with real-world humans. Alt- hough typically conceived as practical tools to assist in a range of application (e.g., HCI, training and entertainment), the technology is gaining interest as a methodological tool for studying human cognition. VHs not only simulate the cognitive abilities of people, but also many of the embodied and social aspects of human behavior more traditionally studied in fields outside of cognitive science. By integrating multiple cognitive capabilities (e.g., language, gesture, emo- tion, and the control problems associated with navigating and interacting with a simulated virtual world) and requiring these processes to support real-time interactions with peo- ple, VHs create a unique and challenging environment with- in which to develop and validate cognitive theories. In this tutorial, we will review recent advances in VH technologies, demonstrate examples of use of VHs in cognitive science research and provide hands on training using our Virtual Human Toolkit (http://vhtoolkit.ict.usc.edu/). The Virtual Human Toolkit The University of Southern California’s Institute for Crea- tive Technologies (ICT) is recognized as a leader in the de- velopment of VH technology (Gratch et al., 2002) and in applying this research to application domains including “vir- tual role players” for interpersonal-skills training (e.g., Campbell et al. 2011), informal science education (e.g. Swartout et al., 2010), intelligent tutoring, (e.g. Lane et al., 2011), and as “virtual confederates” to study cognitive and social processes (e.g. de Melo et al., 2012; Dehghani et al., 2012). One goal of the institute is to foster research in VH by making this technology freely available for research pur- poses through the Virtual Human Toolkit. The research underlying the toolkit draws heavily on cognitive science research. For example, VH “brains” are inspired by psychological theories of human cognition (e.g. Swartout, Gratch et al., 2006), language (e.g. Traum, 2008) and emotion (Gratch & Marsella, 2005), VH bodies are in- formed by knowledge of physiological and biomechanical processes (e.g. Honglun, et al. 2007; Thiebaux et al., 2008) and the relation between the VH’s brain and body is in- formed by social psychology research (Lee & Marsella, 2006, Wang et al, 2013). Translating these theories and findings into working software requires the integration of advanced capabilities from a number of domains of comput- er science research including machine perception, artificial intelligence, cognitive modeling, graphics and animation. The complexity of creating a VH can appear daunting. Fortunately, considerable research has focused on the de- Virtual Humans and Cognitive Science In helping to define the field of cognitive science, Herb Si- mon emphasized the importance of “understanding by simu- lating” (Simon 1969, 17-22). From the perspective of cogni- tive science, VHs provide the opportunity to understand the mind by simulating the body. Although still limited in their capabilities, VHs combine a rich set of capabilities for ex- ploring how cognitive processes manage interactions with the physical and social world. In this sense, they comple- ment recent interest in robotics as a tool for cognitive sci- ence, and address many of the limitations of physical robots. Embodied Cognition: Embodied theories argue that the brain and body are tightly linked: the configuration and state of one’s body profoundly influence cognitive processes and vice versa. For example, posture can impact how easily we are persuaded (Petty, et al. 1983); gestures and language are closely coupled, often grounded in shared metaphors (McNeill 2005); and facial expressions can influence our emotions (Niedenthal, et al., 2010). VH technology is in- creasingly used to unpack this relationship between mind and body (e.g., Sprague, et al., 2007) and in the tutorial we will review research in this growing area.