Joint inferences of belief and desire from facial expressions

Joint inferences of belief and desire from facial expressions Yang Wu (yangwu@mit.edu), Chris L. Baker (clbaker@mit.edu), Joshua B. Tenenbaum (jbt@mit.edu), Laura E. Schulz (lschulz@mit.edu) Department of Brain and Cognitive Sciences, MIT 77 Massachusetts Avenue, Cambridge, MA 02139 USA Abstract Theory of mind research has looked at how learners infer an agent’s unobservable mental states from observable actions. However, such research has tended to neglect another observable source of data: the agent’s reactions to events. In particular, the agent’s facial reactions might provide important information about her mental states that are otherwise ambiguous given her actions. Here we present a Bayesian framework and a behavioral study testing how adults use an agent’s facial reactions to reason backward about her beliefs and desires. We found that participants’ joint inferences of belief and desire from facial expressions were predicted by a Bayesian model analysis, based on integrating the likelihoods of the observed facial reactions and the observed action with their prior over mental states. We argue that people’s naive theory of emotional reactions is structurally and causally intertwined with theory of mind in a way that allows forward prediction and backward inference. Keywords: Theory of mind; appraisal theory; emotion; facial expression; Bayesian inference Introduction Human beings are adept at inferring others’ mental states given sparse observations. One of the mysteries that has intrigued cognitive scientists for decades is what representations make this inference so efficient and accurate. Many studies in theory of mind have focused on how people infer beliefs and desires from observed actions. For example, if Sally reaches for a container, what can we infer about her beliefs and desires? A powerful basis for such inferences is the assumption of rational action – that agents act to fulfill their desires as efficiently as possible in accordance with their beliefs about the world. If we know that Sally believes there are cookies within the container, we may infer that Sally wants cookies based on her reaching behavior; conversely, if we know that Sally wants cookies, it is plausible that she believes there are cookies in the container. Studies suggest that even infants can infer the desire underlying an observed action (when the belief is directly or indirectly given by the context) or the belief underlying an action (when the desire is directly or indirectly given) (e.g. Csibra, Biro, Koos, & Gergely, 2003; Gergely, Nadasdy, Csibra, & Biro, 1995; see Gergely & Csibra, 2003 for a review). Other work suggests that adults and older children can jointly infer an agent’s desires and beliefs given a sequence of actions in which the agent approaches and retreats from potential goals (Baker, Saxe, & Tenenbaum, 2011; Richardson, Baker, Tenenbaum, & Saxe, 2012). Often, however, the information available to observers about agents’ actions, beliefs, and desires may be much more limited. We may for example arrive in the middle of a scene, seeing someone we do not know engage in a single action. For instance, we might see someone look up as another person approaches. This information is relatively sparse; inferring the agent’s belief and desire from the action is nearly impossible, much like trying to solve one equation with two unknown variables. The observation of a simple action is not informative enough to discriminate different mental states. However, these kinds of actions are typically accompanied by another kind of observable response: an emotional reaction. Emotional reactions – often manifest as facial expressions – intuitively seem to provide rich evidence about agents’ mental states, arguably simplifying the theory of mind inference. If, for example, the agent frowns, we might infer that she knows the person and doesn’t like him; if she smiles we might infer that she knows and likes him, and if she has no emotional response, we will probably infer that she doesn’t know the approaching person at all. There is of course a large literature looking at emotion per se (e.g. Ekman, 1992; Lazarus, 1991; Vuilleumier, 2005). However, this literature has remained relatively disconnected from the theory of mind literature. Perhaps the most relevant work connecting emotion to other cognitive states comes from appraisal theory, a theory of emotion suggesting that an individual’s evaluation of events plays a crucial role in eliciting and differentiating emotions (e.g., Ellsworth & Scherer, 2003). Although different appraisal theories differ in the appraisal dimensions that are at stake (e.g. desirability, certainty, causal attribution, coping potential), most of these theories make reference to the agents’ beliefs and desires (either explicitly or implicitly) as influences on people’s evaluation of events and thus the generation of their emotional reactions (e.g. Lazarus, 1991; Ortony, 1990; Scherer, 1984). Critically however, appraisal theory is a scientific theory of how emotions are generated within the individual. It does not attempt to describe the analogous intuitive theory—how either the individual herself might think about the causes of her emotional states, or how observers might use an agents’ emotional reactions to reason backward about the mental states (the beliefs and desires) that generated them via appraisal. This is our goal here. We hypothesize that people have an intuitive theory of emotional responses that is at least coarsely analogous to appraisal theory, and they can