Perception and Emotion How We Recognize Facial Expressions
(2006)

Perception and emotion interact, as is borne out by studies of how people recognize emotion from facial expressions. Psychological and neurological research has elucidated the processes, and the brain structures, that participate in facial emotion recognition. Studies have shown that emotional reactions to viewing faces can be very rapid and that these reactions may, in turn, be used to judge the emotion shown in the face. Recent experiments have argued that people actively explore facial expressions in order to recognize the emotion, a mechanism that emphasizes the instrumental nature of social cognition. KEYWORDS—face perception; faces; emotion; amygdala An influential psychological model of face processing argued that early perception (construction of a geometric representation of the face based on its features) led to subsequently separate processing of the identity of the face and of the emotional expression of the face (Bruce & Young, 1986). That model has received considerable support from neuroscience studies suggesting that the separate processes are based on separable neuroanatomical systems: In neuroimaging studies, different parts of the brain are activated in response to emotional expressions or identity changes; and brain damage can result in impairments in recognizing identity but not in recognizing emotional expressions, or the reverse. SOME PROCESSING IS RAPID AND AUTOMATIC AND CAN OCCUR IN THE ABSENCE OF AWARENESS OF THE STIMULI Some responses in the brain to emotional facial expressions are so rapid (less than 100 ms) that they could not plausibly be based on conscious awareness of the stimulus, although the responses, in turn, may contribute to conscious awareness. Evidence comes from studies using event-related potentials, measures of the brain’s electrical activity recorded at the scalp (or, much more rarely, directly from the brain in surgical patients). In those experiments, the responses to many presentations of emotional stimuli are averaged across stimuli, and the changes in electrical potential can be accurately timed in relation to stimulus onset. Evidence has also come from studies in which viewers were shown facial expressions subliminally. Especially salient aspects of faces are most potent in driving nonconscious responses: For instance, subliminal presentation of only the whites of the eyes of fearful faces results in measurable brain activation (Whalen et al., 2004). One specific structure involved in such rapid and automatic neural responses is the amygdala, a structure in the medial temporal lobe that is known to be involved in many aspects of emotion processing. Findings such as these have been of interest to psychologists, as they provide a potential mechanism consistent with two-factor theories of human behavior. For instance, the theory that affect and cognitive judgment are separate processes, and that affect can precede cognitive judgment, receives some support from the neuroscience findings (Zajonc, 1980). The data also add detail to theories of visual consciousness. In one set of studies, emotional facial expressions were shown to neurological patients who, because of their brain damage, were unable to report seeing the stimuli. An individual with damage to the visual cortex had ‘‘blindsight’’ for emotional faces: He could discriminate the emotion shown in faces by guessing, even though he reported no visual experience of seeing the faces. A neuroimaging experiment in this same individual revealed that the amygdala was activated differentially by different emotional faces, despite the absence of conscious visual experience (Morris, deGelder, Weiskrantz, & Dolan, 2001). These and other studies have suggested a distinction between subcortical processing of emotional visual stimuli (e.g., involving the amygdala and brainstem nuclei such as the superior colliculus), which may be independent of conscious vision (Johnson, 2005), and cortical processing, which is usually accompanied by conscious experience. It is interesting to note that amphibians and reptiles have only subcortical visual processing, since they lack a neocortex. One broad interpretation of these observations is thus that the subcortical route for processing emotional stimuli is the more ancient one, and that in Address correspondence to Ralph Adolphs, HSS 228-77, Caltech, Pasadena, CA 91125; e-mail: radolphs@hss.caltech.edu. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE 222 Volume 15—Number 5 Copyright r 2006 Association for Psychological Science mammals an additional, cortical route has evolved that probably allows more flexible behaviors based on learning and conscious deliberation. A final wrinkle has come from psychological studies of the relationship between attention and emotion. It has been known for some time that emotionally salient stimuli can capture attention, an interaction that makes much adaptive sense (Ohman, Flykt, & Esteves, 2001). But more recent studies have also shown the reverse: that volitionally allocating attention to stimuli can influence their emotional evaluation. Inhibiting attention to visual stimuli that are distractors in a search task, for example, results in devaluation of those stimuli when subjects are asked to rate them on affective dimensions (Raymond, Fenske, & Tavassoli, 2003). Emotions thus represent the value of stimuli—what people approach or avoid, cognitively or behaviorally, volitionally or automatically. FEAR AND DISGUST MAY BE PROCESSED SPECIALLY Several studies have found evidence that the amygdala, the subcortical structure discussed earlier, is disproportionately important for processing facial expressions of fear, although by no means exclusively so. The only other emotion for which a similar neuroanatomical specificity has been reported is disgust. A region of the cortex called the insula represents motor and sensory aspects of that emotional response (Calder, Lawrence, & Young, 2001). In lesion studies, damage to the amygdala is shown to result in impairment in the ability to recognize fear from facial expressions. Similarly, damage to the insula can result in impairment in the ability to recognize disgust from facial expressions. However, questions remain regarding exactly which emotion category or dimension is being encoded in these brain regions, since other emotions are often also variably impaired. Especially informative have been studies in a rare patient, SM, who has total damage to the amygdala on both sides of the brain. The subject can accurately judge age and gender from faces, and she has no difficulty recognizing familiar individuals from their faces. She also has little difficulty recognizing most facial expressions, with the notable exception of fear. When asked to judge the fearfulness of faces, SM is selectively and severely impaired (Adolphs, Tranel, Damasio, & Damasio, 1994). However, other patients with similar damage are impaired on a wider array of emotions; in functional neuroimaging studies, activation of the amygdala is seen in response to several emotions in addition to fear; and even SM’s highly specific impairment depends on the questions she is asked. For example, when asked to classify faces into categories of basic emotions, SM is rather selectively impaired on fear; but when asked to rate how arousing the emotion shown in the face is, she is impaired on all emotions of negative valence (Adolphs, Russell, & Tranel, 1999). These data suggest that, while the amygdala is critical for recognizing fear in faces, its role encompasses a broader, more abstract, and perhaps dimensional rather than categorical aspect of emotions, for which fear is but one instance—an issue I take up again below. MANY BRAIN STRUCTURES ARE INVOLVED Just as there is good evidence that the amygdala does more than solely detect fear, there are of course brain structures other than the amygdala that participate in perceiving emotion from faces. Brain-imaging studies, in particular, have found evidence for a large number of other brain structures that may come into play, depending on the emotion shown in the face and on the demands of the experimental task. One framework that summarizes these data runs as follows. Visual areas in the temporal cortex encode the face’s features and bind them into a global perceptual representation of the face. Subcortical visual areas (such as the superior colliculus, prominent in animals such as frogs) carry out coarser but faster processing of the face. Both routes, the cortical and the subcortical, feed visual information about the face into the amygdala. The amygdala then associates the visual representation of the face with an emotional response in the body, as well as with changes in the operation of other brain structures. For instance, it likely triggers autonomic responses (such as skin-conductance response, the sympathetic autonomic response of sweating on the palms of the hands) to the face, and it also modulates visual attention to the face. Tracing the path from initial perception of the face to recognizing the emotion it expresses is complicated by feedback loops and by multiple pathways that can be engaged. One class of proposals, as influential as they are controversial, argues that the emotional response elicited by the amygdala can in fact be used by the viewer to reconstruct knowledge about the emotion shown in the face. Roughly: if I experience a pang of fear within myself upon seeing a fearful face, I can use the knowledge of my own emotion to infer what the emotional state of the person whose face is shown in the stimulus might be. Broader theories in a similar vein do not restrict themselves to the amygdala or to fear, but more generally propose that we make inferences about other people’s emotional states by simulating within ourselves aspects of those states (Goldman & Sripada, 2005). Emotional contagion and imitation may be the earliest aspects of such a mechanism that can already be seen in infants