c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

Available online at www.sciencedirect.com

ScienceDirect
Journal homepage: www.elsevier.com/locate/cortex

Research report

Reconstructing dynamic mental models of facial
expressions in prosopagnosia reveals distinct
representations for identity and expression
Anne-Rapha€elle Richoz a, Rachael E. Jack b, Oliver G.B. Garrod b,
Philippe G. Schyns b and Roberto Caldara a,*ab

Department of Psychology, University of Fribourg, Switzerland
Institute of Neuroscience and Psychology, University of Glasgow, United Kingdom

article info

abstract

Article history:

The human face transmits a wealth of signals that readily provide crucial information for

Received 18 November 2013social interactions, such as facial identity and emotional expression. Yet, a fundamental

Reviewed 5 February 2014question remains unresolved: does the face information for identity and emotional

Revised 7 May 2014expression categorization tap into common or distinct representational systems? To

Accepted 25 November 2014

address this question we tested PS, a pure case of acquired prosopagnosia with bilateral

Action editor Jason Barton

occipitotemporal lesions anatomically sparing the regions that are assumed to contribute

Published online 24 December 2014

to facial expression (de)coding (i.e., the amygdala, the insula and the posterior superior
temporal sulcus e pSTS). We previously demonstrated that PS does not use information

Keywords:

from the eye region to identify faces, but relies on the suboptimal mouth region. PS's

Prosopagnosia

abnormal information use for identity, coupled with her neural dissociation, provides a

Facial expressions of emotion

unique opportunity to probe the existence of a dichotomy in the face representational

Reverse correlation

system. To reconstruct the mental models of the six basic facial expressions of emotion in
PS and age-matched healthy observers, we used a novel reverse correlation techniquetracking information use on dynamic faces. PS was comparable to controls, using all facial
features to (de)code facial expressions with the exception of fear. PS's normal (de)coding ofdynamic facial expressions suggests that the face system relies either on distinct representational systems for identity and expression, or dissociable cortical pathways to accessthem. Interestingly, PS showed a selective impairment for categorizing many static facial
expressions, which could be accounted for by her lesion in the right inferior occipital gyrus.
PS's advantage for dynamic facial expressions might instead relate to a functionally distinct
and sufficient cortical pathway directly connecting the early visual cortex to the spared
pSTS. Altogether, our data provide critical insights on the healthy and impaired face systems, question evidence of deficits obtained from patients by using static images of facial
expressions, and offer novel routes for patient rehabilitation.
© 2014 Elsevier Ltd. All rights reserved.

* Corresponding author. Department of Psychology, University of Fribourg, Faucigny 2, 1700 Fribourg, Switzerland.
E-mail address: roberto.caldara@unifr.ch (R. Caldara).
http://dx.doi.org/10.1016/j.cortex.2014.11.015
0010-9452/© 2014 Elsevier Ltd. All rights reserved.

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 41.

Introduction

The human face transmits a wealth of visual signals relevant
for the identification and the categorization of facial expressions of emotion. The brain, as a decoder, flexibly filters theincoming visual information transmitted by the face to rapidly
achieve complex perceptual categorizations (Schyns, Petro, &Smith, 2009). For example, the uniqueness of facial featurescharacterizing a given individual, and their overall organization in the face, constitute the core information for identification and also for dissociating familiar from unfamiliar faces.
Other signals can also be extracted from faces, such as thecues disclosing age (e.g., George & Hole, 1995), gender (e.g.,
Brown & Perrett, 1993; Ekman & Friesen, 1976, 1978; Schyns,
Bonnar, & Gosselin, 2002; Tranel, Damasio, & Damasio,1988), race (e.g., Caldara & Abdi, 2006; Caldara, Rossion,
Bovet, & Hauert, 2004; Vizioli, Foreman, Rousselet, &Caldara, 2010; Vizioli, Rousselet, & Caldara, 2010) andemotional state (e.g., Bruce & Young, 1986; Calder & Young,
2005; Ekman & Friesen, 1976, 1978; Smith, Cottrell, Gosselin,& Schyns, 2005). Overt emotional states can also be extracted from face signals; they are mostly conveyed by facial expressions of emotion. The basic signals (i.e., “happy,”“surprise,” “fear,” “disgust,” “anger,” and “sad”) are only
weakly correlated with each other to minimize confusions fortheir decoding (Smith et al., 2005), and we recently reported
cross-cultural tunings in the way the emotion signals aretransmitted and decoded (Jack, Blais, Scheepers, Schyns, &Caldara, 2009; Jack, Caldara, & Schyns, 2012; Jack, Garrod,
Yu, Caldara, & Schyns, 2012). Yet, a fundamental question
remains unresolved: does the face information used torecover identity and emotional expressions tap into common
or distinct representational systems?
According to influential cognitive (Bruce & Young, 1986)
and neuroanatomical (Haxby, Hoffman, & Gobbini, 2000)
models of face processing, two distinct functional and neuralsystems accomplish the recognition of facial identity andfacial expression. The first system e performing facial identification (Haxby et al., 2000) e is proposed to mainly involvethe inferior occipital gyri and lateral fusiform gyrus, whereas
the second system e performing facial expression categorization e is proposed to involve the inferior occipital gyri, theposterior superior temporal sulcus (pSTS) and the amygdala(for a review see, Calder & Young, 2005; Pessoa, 2008). However, some authors have questioned the idea of independence
between those systems, by mainly relying on results fromcomputational modelling and neuroimaging evidence (Calder,2011; Calder & Young, 2005). A single model based on a Principal Component Analysis (PCA) can achieve independent
coding of facial identity and facial expression, suggesting thepossible existence of a multidimensional system, with a morepartial than absolute independence (Calder, Burton, Miller,
Young, & Akamatsu, 2001). These simulations have thus
challenged the view of an independence between the coding
for identity and expression, at least suggesting that thosemodels are less strongly supported than what is often
assumed (Calder & Young, 2005). In line with this position,
Palermo, O'Connor, Davis, Irons, and McKone (2013) haverecently put forward the theory of a first common step in the51

processing of expression and identity, and the occurrence of a
splitting at a later stage; a view that is in agreement with thefunctional involvement of the inferior occipital gyrus as theentry level for both tasks (Calder & Young, 2005; Haxby et al.,
2000; Pitcher, 2014). However, even though a neural dissociation for the processing of identity and emotional expression is
supported by electrophysiological studies in primates (e.g.,
Hasselmo, Rolls, & Baylis, 1989) functional neuroimaging inhumans (e.g., Winston, Henson, Fine-Goulden, & Dolan, 2004)
and brain-damaged patients (Haxby et al., 2000), recent evidence suggests that the neural computations occurring in theinferior occipital gyrus and the right pSTS are functionallydistinct and have a causal involvement in processing facial
expressions (Pitcher, Duchaine, & Walsh, 2014). To sum up,
more evidence is necessary to clarify this debate and, as
acknowledged by Calder and Young (2005), further studies
with brain-damaged patients are necessary to probe the hypothesis of distinct visuoperceptual systems for facial identityand facial expression categorization.
Following brain lesions, some patients lose the ability torecognize facial identity, despite no other obvious impairments of the visual system and a preserved identification via
other modalities (e.g., voice, gait and so forth). The specificityof this face recognition deficit is spectacular, rare and has
elicited considerable attention within the neuropsychologicalliterature since the first clinical observations (Quaglino, 1867;
Wigan, 1844) and the introduction of the term prosopagnosia
by Bodamer (1947). Acquired prosopagnosia typically follows
brain damage to bilateral occipitotemporal areas (e.g.,
Damasio, Damasio, & Van Hoesen, 1982; Farah, 1990; Landis,
Regard, Bliestle, & Kleihues, 1988; Sergent & Signoret, 1992).
Anatomical descriptions of prosopagnosia endorse thenecessary and sufficient role of the right hemisphere (Landis
et al., 1988; Sergent & Signoret, 1992) in the occipitotemporalpathway of face processing (for a review see, Bouvier & Engel,2004). The clinical and anatomical conditions of prosopagnosia have always received great interest in cognitive
neuroscience, as they clarify the neurofunctional mechanisms of normal face processing. The different sub-functionsof the cognitive architecture of face processing have been
isolated by the occurrence of distinct double dissociations inbrain-damaged patients, for instance: a functional segregation between the ability to recognize unfamiliar and familiar
faces (e.g., Malone, Morris, Kay, & Levin, 1982) and between lip
reading and face identification (Campbell, Landis, & Regard,1986). Yet, the neuropsychological literature remains controversial on the spared ability of prosopagnosic patients to
identify facial expressions despite their impairment torecognize facial identity, and on patients showing impaired
facial expression recognition with preserved facial identity
recognition (for a detailed review see, Calder, 2011). Some acquired prosopagnosic patients showed a marked impairment
in the categorization of facial expressions (Bowers, Bauer,
Coslett, & Heilman, 1985; De Gelder, Pourtois, Vroomen, &Bachoud-Levi, 2000; De Renzi & Di Pellegrino, 1998; Humphreys, Donnelly, & Riddoch, 1993). Other studies reported
preserved recognition of emotion in acquired prosopagnosia(Bruyer et al., 1983; Cole & Perez-Cruet, 1964; Mattson, Levin, &Grafman, 2000; Sergent & Villemure, 1989; Shuttleworth,
Syring, & Allen, 1982; Tranel et al., 1988; Young, Newcombe, de

52

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4Haan, Small, & Hay, 1993). In addition, as pointed out by
Calder and Young (2005) and Calder (2011), the decoding offace identity, as well as facial expressions of emotion, activates a similar network of regions in the occipitotemporal
cortex. Facial expression impairments in patients are often
correlated with a deficit to decode emotions from other modalities, which suggests a general, multimodal deficit in thosepatients, rather than a selective impairment of facial expression representations. In addition, a better understanding of
the patients' information use (i.e., representations) for bothtasks is necessary to clearly understand the very nature of the
deficits in the face processing system (Calder & Young, 2005;
Calder et al., 2011). Consequently, the question of dissociation between the identity and expression systems with acquired cases of prosopagnosia remains unclear.
To address this issue, we tested PS e a pure case of acquired prosopagnosia. PS is a 64-year-old woman (born in1950) who sustained a closed-head injury in 1992. PS shows
normal object recognition (e.g., Busigny, Graf, Mayer, &Rossion, 2010; Rossion et al., 2003) and relies on atypicalcues to determine the identity of a person, such as voice,clothes, or other salient non-face features (e.g., glasses,
haircut, beard, posture). She has major lesions on the left midventral and the right inferior occipital cortex. Minor lesions of
the left posterior cerebellum and the right middle temporalgyrus were also detected (for a complete anatomical description see, Rossion, 2008; Sorger, Goebel, Schiltz, & Rossion,2007), whereas the regions that are assumed to be critical forthe decoding of emotional expressions (i.e., the amygdala, theinsula and the pSTS) are anatomically spared. Note that even
if the occipital temporal regions are not playing a central rolefor facial expression decoding, the right inferior occipitalgyrus is damaged in PS and represents the entry level for
expression and identity in posited neuroanatomical models(Haxby et al., 2000; Pitcher, 2014). Thus, it remains to be clarified whether these lesions in the patient have also an impact
on the processing of facial expressions. Of interest, we previously used a response classification technique e Bubbles e toreveal the diagnostic information used by PS for face identification (Caldara et al., 2005). Bubbles is a response classification technique sampling the information in 3-D space (2Dimage  spatial frequencies) (Gosselin & Schyns, 2001), topresent sparse versions of the faces as stimuli. Observers
categorize the sparse stimuli, and Bubbles keeps track of thesamples of information that lead to correct and incorrect
identification responses. From this information, we canestablish how each region of the input space contributed to
face identification performance and depict the diagnostic information used to effectively decode the stimulus. In contrast
to healthy observers, PS did not use information from the eyeregion to identify familiar faces, but instead the lower part of
the face, including the mouth and the external contours. To
sum up, PS's well-established bias to use information from themouth to identify faces and her anatomical neural dissociation provide a unique opportunity to probe the existence of adichotomy in the representations used for facial identity andexpression categorization.
Here, we first assessed her categorization performance ofthe six facial expressions of emotion using the classical Ekman
and Friesen (1976) FACS (Facial Action Coded System) static face

database. The FACS provides an anatomical taxonomy of thehuman muscles activated during the transmission of facial
expressions of emotion (Ekman & Friesen, 1978), by quantifying
facial movements for every expression in terms of so-called
Action Units (AUs e each of them relating to a particular
muscle). We then modelled PS's 3D dynamic mental representations of the six classic facial expressions by using a dynamic FACS-based Generative Face Grammar (GFG, see Fig. 1,
the methods section and Yu, Garrod, & Schyns, 2012) on theAUs combined with a reverse correlation technique (see themethods and also Jack, Caldara, et al., 2012). The use of dynamic facial expressions provides a more ecologically valid
approach to study the perception and processing of facial expressions, as our natural environment is surrounded with dynamic, temporal and multimodal information (Johnston,
Mayes, Hughes, & Young, 2013). Pertinently, it has also been
recently demonstrated that there is a causal involvement of theright pSTS in the processing of dynamic facial information(Pitcher et al., 2014), a region anatomically spared in PS.
The main goal of our study was to test whether the representations for identity and facial expressions are distinct orcommon. Mapping out the facial features used by a prosopagnosic patient to perform facial expression categorization isnecessary to achieve this goal. As PS shows a suboptimal use
of facial information for identity (i.e., by using the mouth region and external contours), we put forward the hypothesisthat if she adequately uses all facial features for expression
categorization, this observation would support the hypothesisof distinct sets of representations for identity and emotion
recognition.

2.

Participants

2.1.

PS's case report

PS is a 64-year-old case of acquired prosopagnosia withnormal object recognition. Despite the multiple and extensive
brain lesions in the occipitotemporal cortex, PS recovered wellin the months following her accident and with the support ofneuropsychological rehabilitation she restarted working as akindergarten teacher. PS's low-level vision is well preservedwith a good visual acuity in both eyes, except to a small rightparacentral scotoma. She reads normally (although slowly)
and has normal object perception and recognition, even for
subordinate-level discriminations (Rossion et al., 2003). However, as a consequence of the lesions, she reports her facerecognition is severely impaired, even for close relatives(husband, children, mother, father) and her own. PS can
categorize a face as a face, discriminate faces from objects andfrom a complex scene background, even at brief presentations(Schiltz et al., 2006). Her ability to categorize gender isimpaired, in both accuracy and sensitivity (Rossion et al.,2003). Her performance in categorizing facial expressions is
not as good as those of controls. In a 3AFC expression categorization task where the stimuli were presented for a
maximum of 10 sec, PS's reaction times were significantly
slower than those of controls and her response accuracy was
below range for the first block of expressions comprising joy,fear and anger. She was normal for the second block testingc o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

53

Fig. 1 e Stimulus generation e On each trial, the generative face grammar (GFG) randomly selected a subset of action units(AUs; AU17, AU10L, and AU9 are shown here with colour-coded labels) and values for six temporal parameters (see thecolour-coded AU curves, which illustrate the amplitude and acceleration or deceleration of movement over time). The
colour-coded vector at the bottom of the figure represents the 3 (of 41) randomly selected AUs that make up the stimulus on
this illustrative experimental trial. We then applied the random facial animation to one of eight neutral-expression face
identities using the procedure described in Yu et al. (2012). Mental representations e Observers categorized each random
facial animation according to the six basic emotion categories (plus “don't know”) and rated the emotional intensity on a
five-point scale. Observers will interpret the random facial animation as a meaningful facial expression (here, “disgust,”“medium intensity”) when the facial movements correspond to the observer's mental representation of that facial
expression.

disgust, sadness and surprise (Rossion et al., 2003). However
her performance was collapsed across the expressions presented within a block and she was normal for the second block
testing disgust, sadness and surprise (Rossion et al., 2003).
Therefore, we had to assess PS's categorization accuracy ofstatic facial expressions properly by using a 7AFC task
(Experiment 1). PS is able to draw correctly a schematic faceand perfectly points out all the single features and estimatesage in the normal range. This latter result contrasts with her
inability to recognize previously seen or familiar faces and tomatch unfamiliar faces, including changes of viewpoints(Rossion et al., 2003). As reported by Rossion et al. (2003) she isranked as highly impaired on the Benton Face Matching Test
(Benton & Van Allen, 1972) scoring 27/54 (percentile 1). She isalso impaired on the Short Recognition Memory Test forFaces, a set of the Camden Memory Tests (Warrington, 1984)
scoring 18/25 (percentile 3). When confronted with the pictures of 60 famous people (all known by the patient), she wasable to classify 14 of them as familiar, and correctly classified
all the unfamiliar ones (Rossion et al., 2003). Nevertheless,when asked to report the individual names of the faces classified as familiar, as well as their semantic information, she
was correct for only four of them. Finally, PS has been tested
extensively with simultaneous and delayed face and non-face(cars and novel objects) matching tasks in previous studies(Rossion et al., 2003). Although she is consistently impaired
and slowed down for the face conditions, her performancewith the non-face objects is in the normal range. Given the

54

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

restriction of her deficit to the face category and that PS isalert, cooperative, and without any learning difficulties(Caldara et al., 2005) she represents an ideal case to isolate thenature of the facial information extracted by an impaired facesystem for the processing of facial expressions.

2.2.

Table 1 e Recognition accuracy of PS and age-matched
controls on Ekman and Friesen's facial expression
recognition test. PS showed impaired recognition of anger,
fear, surprise and sad compared to healthy controls.

Score %

Control participants

The age-matched healthy observers who voluntarily took partin our experiments had normal or corrected to normal vision
with no neurological or psychiatric history. The number ofhealthy observers and their age is reported in the method
section of each experiment. For all our experiments, PS as wellas the control participants signed a consent form describing
the main goals of our experiments. The Ethical Committee of
the Department of Psychology of the University of Fribourg
approved all the studies reported here.

PS

Age-matched controls

Modified t-test

Mean %

SD %

71.95
79.45
96.11
72.40
67.22
70.00
70.00

8.46
6.16
3.12
4.77
9.30
8.16
7.24

Ekman and Friesen (1978)
Anger
36.33
4.38*
Disgust
88.55
1.53
Happy
97.78
.55
Neutral
79.17
1.47
Fear
45.67
2.41*
Surprise48.89
2.69*
Sad
47.22
3.27**p < .05.

3.2.
Experiment 1 e reconstructing PS's dynamic mental
models of facial expressions of emotion

3.

Experiments

3.1.
Preliminary experiment: categorization of the sixclassic facial expressions of emotion
We first assessed categorization accuracy of static facial expressions using a standard set of posed facial expression
stimuli e the Pictures of Facial Affect series (POFA) (Ekman &Friesen, 1976, 1978).

3.1.1. Material and methods
3.1.1.1. CONTROL PARTICIPANTS. Twelve age-matched healthy
control subjects (8 female) participated in the experiment(mean age ¼ 59.41; SD ¼ 3.98).

3.1.1.2. STIMULI. We selected seven posed facial expression
images from 22 identities (11 female) e one per emotion
category (happy, surprise, fear, disgust, anger, sad, andneutral) e from the Pictures of Facial Affect (POFA) database(Ekman & Friesen, 1976, 1978). The pictures were in grayscaleand not cropped.

3.1.1.3. PROCEDURE. Observers categorized each stimulus according to emotion (happy, surprise, fear, disgust, anger, sadand neutral) in a 7AFC task using a computer keyboard inwhich we labelled the keys accordingly. Each image was presented for 2000 msec in random order in the centre of theobservers' visual field with 30 repetitions of each expression,
resulting in a total of 210 trials. Faces subtended a visual angleof 9.54 (vertical) and 8.11 (horizontal) on the screen.
3.1.2.

Results and discussion

To determine whether PS's average accuracy is significantly
different from that of the age-matched healthy participants,
we used a modified independent samples t-test for single casestudies (Crawford & Garthwaite, 2002). The statistical level ofsignificance is p < .05. As shown in Table 1, PS's categorizationof static, posed, FACS-coded facial expressions is significantly
impaired for anger [t(11) ¼ 4.38, p < .05], fear [t(11) ¼ 2.41,
p < .05], surprise [t(11) ¼ 2.69, p < .05] and sad [t(11) ¼ 3.27,
p < .05] compared to the age-matched healthy controls.

Using a novel method, we modelled the 3D dynamic mental
representations of the six facial expressions plus neutral of PS
and control participants, by using the FACS-based
GFG computer graphics platform and a reverse correlation
technique.

3.2.1. Material and methods
3.2.1.1. CONTROL PARTICIPANTS. Five healthy age-matched controls (3 women) participated in the experiment (meanage ¼ 60.2; SD ¼ 3.27).

3.2.1.2. STIMULI. Using the GFG (Jack, Garrod, et al., 2012; Yu
et al., 2012) we synthesized a series of photorealistic facial
animations by randomly selecting a subset of groups ofmuscles moving in synergy on the face e AUs (Ekman &Friesen, 1976, 1978). Practically, they were measured andmodelled in 4D (3D face plus time), from the mapping of actors
trained by Ekman to produce them. Existing as mathematicalmodels in the GFG, we can animate each AU over time independently, using 6 temporal parameters (onset, acceleration,
peak amplitude, peak latency, deceleration, offset) (see Fig. 1
and Movie C1 for an example stimulus).
Supplementary video related to this article can be found at
http://dx.doi.org/10.1016/j.cortex.2014.11.015.
Each animation is displayed on one of 8 unfamiliar 3D
photorealistic white Caucasian face identities (4 female),which were acquired with a 3D photorealistic capture system(Dimensional Imaging). On any given trial, the GFG selects
amongst the 41 core AUs a subsample of AUs using a binomial
distribution (n ¼ 5, p ¼ .6, median ¼ 3). The mental representation of each facial expression was then modelled foreach observer by reverse correlating the random AUs andtheir temporal parameters with the observer's emotion responses. This technique has been validated in previous
studies (Gill, Garrod, Jack, & Schyns, 2014; Jack, Caldara, et al.,2012; Jack, Garrod, et al., 2012).

3.2.1.3. PROCEDURE. The experiment consisted in 12 sessionsof 4 blocks that ran on a computer using a program written

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

with Adobe Flash. Each block included 50 trials, consisting of
4D facial animations (3D þ time) displaying a random subsetof AUs movements. All the observers categorized a total of2400 of such animations, comprising 30 frames (24 frames/
second) of 1.25 sec duration. The facial animations subtended approximately 9.54 (vertically) and 6.68 (horizontally) of visual angle. On each trial, observers viewed thefacial animation and categorized it according to the 6 classic
emotions: happy, surprise, fear, disgust, anger and sad and a“don't know” response. Furthermore, observers rated theintensity of emotion perceived on a 5 point-rating scale(“Very weak”, to “Very strong”). We adapted the response to
our senior population using the Geneva Emotion Wheel(GEW) introduced by Scherer (2005). We labelled the emotion
categories in the outer boundaries of a circle, with the “don'tknow” response option in the centre. Additionally, we used 5
circles gradually increasing from the centre towards therespective emotional category placed on the border to allow
participants to report the intensity of the perceived facial
expression. Observers navigated with a mouse to select thebasic emotion and its intensity. We did not provide feedback
and did not place any time pressure on participants(including PS) to respond. All of the observers were familiar
with using a computer and did not have difficulty with thisinterface. The whole experiment lasted for about 6 h, over a
period of 6 weeks.

3.2.2. Results
3.2.2.1. REPRESENTATION OF THE SIX BASIC EMOTIONS. For eachparticipant, we modelled their mental representations of thesix basic emotions, by reverse correlating the AUs randomly
selected on each trial with the responses of the observers. For
each observer, this resulted in a 41-dimensional (one dimension per AU) ON/OFF vector, with ON AUs being significantly
correlated with an emotion category. The p-value of .05 wasused to determine whether the correlation coefficient between each AU's presence or absence and the participant's
emotional response was significantly larger than zero. For the
final models only those AUs whose correlation coefficient wassignificantly larger than zero were displayed, but the regression coefficients for each AU were not affected by whether or
not other AU were deemed significantly correlated or not. We
then regressed for each ON AU, the six temporal parameterswith the intensity responses of observers to model the activation dynamics (Jack, Caldara, et al., 2012; Jack, Garrod, et al.,2012; Yu et al., 2012). Figs. 2 and 3 report the results. Fig. 2 illustrates the static version of the dynamic stimuli, computedby collapsing the highest amplitude of each AU involved in themodels of the “Very Strong” intensity judgement. Fig. 3 represents a static rendering of the facial parts that move (withamplitude of movement represented in millimetres).
Figs. 2 and 3 convey that the models of PS are within therange of those of controls. That is, for each individual
expression, at least one control is very similar to that of PS(e.g., as Fig. 3 reveals PS's “happy” is similar to all controls,“surprise” to control 5, “fear” to 5 again, “disgust” to at least
control 3, “anger” to control 4 and “sad” to control number 3).
Movie 2 presents the 4D dynamic version of the mental
models of PS and Movies 3, 4, 5, 6, 7 those of the 5 control
subjects”. To test whether the patients' AUs preferences for55

each emotion fell within the normal range of the population
we computed Pearson correlations between each of the patient's AUs ON/OFF vectors and the corresponding vectors forthe control group (Table 2, column 1).
Supplementary video related to this article can be found at
http://dx.doi.org/10.1016/j.cortex.2014.11.015.
A permutation test confirmed that the 95% confidence interval of the expected correlation (between any control andthe remaining controls, Table 2, column 2) for the null hypothesis (Patient ¼ Controls) contained the correlation between the patient and controls for each of the six emotions.
Table 2, column 3 contains the difference between thepatientecontrols correlations and the mean of the permutation test correlations expressed as Z-scores. PS fell in thenormal range for AU preference. Fig. 3 represents a static
rendering of the facial parts that move (with amplitude of
movement represented in millimetres) for every expression
and observer e the deviation maps.

3.2.3.

Discussion

The reconstructed mental models of PS show that her representations of facial expressions are comparable to those of theage-matched controls. Of interest, PS uses information fromthe eye region to represent basic emotions as demonstrated
by the analysis on the deviation maps. These results conflictwith her inability to accurately categorize most of the static,
posed facial expressions of the Ekman and Friesen (1978) face
database (i.e., anger, fear, surprise and sad). How can wereconcile such a discrepancy?

3.3.
Experiment 2 e assessing categorization of facial
expressions of emotion with PS's static and dynamic
reconstructed mental models
To clarify our previous observations, we carried out a verification task in which we presented PS and a new group of agematched controls with static and dynamic mental models of
the patient, and asked all observers to categorize the facial
expressions of these stimuli.

3.3.1. Material and methods
3.3.1.1. CONTROL PARTICIPANTS. A new group of 10 age-matched
controls (6 women) and PS participated in the experiment(mean age ¼ 58.4; SD ¼ 4.19).

3.3.1.2. STIMULI
3.3.1.2.1. STATIC RECONSTRUCTED MODELS. This experiment
comprised static and dynamic versions of the same facial
expressions. That is, for each dynamic 4D model (3D þ time),
we created a static version by collapsing time, keeping only
the highest amplitude of each AU involved in the model (seeFig. 2 and Movies 2 & 3 for examples). We used PS's mental
models of each of the basic emotions (happy, surprise, fear,
disgust, anger and sad) at three levels of intensity (low, medium and high). The judgement of intensity was only kept tomaintain attention and provide observers with a fine-grained
scale, but its analysis is beyond the scope of the present work.
This resulted in 18 facial expressions presented 6 times, for agiven total of 108 images. The faces were subtending the same56

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

Fig. 2 e PS's and controls' reconstructed mental models of happy, surprise, fear, disgust, anger and sad. Positive correlation
coefficients between active Action Units and emotional responses are represented as AU intensity. Active AUs for a given
observer and emotion correspond to those whose correlation coefficient is significantly greater than zero (p < .05, onetailed). Rows e observer (top row PS, bottom rows controls). Columns e emotion (happy, surprise, fear, disgust, anger, sad).

visual angles as in the previous experiment and were presented for 1250 msec.

3.3.1.2.2. DYNAMIC RECONSTRUCTED MODELS. Each dynamic
facial expression consisted of 30 frames (24 frames/second)and lasted for 1250 msec. The facial animations and theirstatic version were covering the same visual angle of theprevious experiment.
3.3.1.3. PROCEDURE
3.3.1.3.1. STATIC RECONSTRUCTED MODELS. The stimuli were
presented in a random order, one at a time on the computer
screen. PS and the controls were instructed to categorize eachstimulus as accurately as possible by pressing one of six
labelled keys on the computer keyboard (one per expression).
No feedback was provided.

3.3.1.3.2. DYNAMIC RECONSTRUCTED MODELS. We used 12 animations for each emotion (4 per intensity) for a total of 72
trials. These animations were taken from the same identities
used for the static presentation. After each presentation, observers categorized the dynamic facial expression according
to the six classic emotions (i.e., happy, surprise, fear, disgust,anger and sad), as accurately as possible by pressing one of six
labelled keyboard keys (one per expression). No feedback wasprovided.

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

57

Fig. 3 e Deviation maps of PS's and controls' reconstructed mental models of happy, surprise, fear, disgust, anger, and sad.
Colour scale represents peak magnitude of facial movement relative to neutral over the course of the expression at thecorresponding position on the face. Facial movements are combinations of active AUs weighted by positive Pearson
correlation coefficients (negative correlations are not shown) with emotional responses. Active AUs for a given observer andemotion correspond to those whose correlation coefficient is significantly greater than zero (p < .05, one-tailed). Rows eobserver (top row PS, bottom rows controls). Columns e emotion (happy, surprise, fear, disgust, anger, sad).

3.3.2. Results and discussion
3.3.2.1. STATIC AND DYNAMIC RECONSTRUCTED MODELS. PS could not
categorize the static version of many of her own mental
models of facial expressions (see Fig. 4).
As shown in Table 3, PS was significantly impaired in therecognition of static images of her own mental models ofanger [t(9) ¼ 2.04, p < .05], disgust [t(9) ¼ 3.008, p < .05] andsurprise [t(9) ¼ 2.57, p < .05]. In contrast, controls categorized
the static models of PS successfully, with the exception of“fear.”In addition, PS categorized her own dynamic mental
models (with the exception of fear) without difficulty. She was
perfectly accurate for “anger, disgust, happiness, surprise andsad”. A one-way repeated-measures analysis of variance(ANOVA) was performed to examine whether dynamic

presentations facilitated performance for controls in the same
way as for PS. There was no significant effect of display presentation (static vs dynamic) [F(1,18) ¼ .74, p ¼ .4] and no significant interaction of emotion  display presentation[F(1,18) ¼ .14, p ¼ .98].

3.4.
Experiment 3 e assessing categorization of facial
expressions of average static and dynamic reconstructed
mental models
To verify whether PS's ability to categorize dynamic emotional
expressions is restricted to the recognition of the expressions
reconstructed from her mental model, we assessed her ability
to categorize average static and dynamic reconstructed mental
models. Given the limited number of mental models from58

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

Table 2 e For each expression, Pearson correlations
between PS's ON/OFF AU vectors and age-matched
controls (column 1). Permutations of one control with othercontrols produced the expected correlations (column 2). Zscore difference between patientecontrols correlations
and the expected correlations (column 3). None of thosedifferences was significant.

Score %

PS versus Age-matched controls,
Pearson correlations
PS versus
Controls
Static reconstructed model
Anger
.3028
Disgust
.4131
Happy
.5027
Fear
.0175
Surprise
.4185
Sad
.1040

Table 3 e Percentage accuracy score of PS and agematched controls for PS's static and dynamic reconstructed
mental models. The scores are given for the recognition ofanger, disgust, happy, fear, surprise and sad. (Experiment2).

Expectedcorrelation

Z-scoreddifference

.2020
.3585
.5679
.0745
.3638
.1053

.9096
.3630
.8744
.4265.5040
.0096

healthy controls in the present experiment, we reconstructed
an average mental model by using the data of 30 Westernerobservers of our previous study (Jack, Garrod, et al., 2012). In
line with experiment 2, this comprised the categorization of
static and dynamic versions of the same facial expressions.

3.4.1. Material and methods
3.4.1.1. CONTROL PARTICIPANTS. PS and 12 age-matched control
subjects (7 women) took part in this experiment (meanage ¼ 59; SD ¼ 3.71).

3.4.1.2. STIMULI
3.4.1.2.1. AVERAGE STATIC AND DYNAMIC 4D MENTAL MODELS. We
used an average of the mental models of 30 Westerners foreach of the basic emotions (Jack, Garrod, et al., 2012), allPS

Age-matched controls

Modified t-test

Mean %

SD %

80.03
90.00
96.67
10.00
83.33
73.33

32.19
22.49
10.54
31.62
23.57
40.98

81.67
93.33
96.67
0
74.97
68.27

33.75
14.05
10.54
0
34.52
38.95

Static reconstructed model
Anger
16.67
2.04*
Disgust25.00
3.008*
Happy
86.11
1.04
Fear
8.33
.05
Surprise
25.00
2.57*
Sad
41.67
.80
Dynamic reconstructed model
Anger
100
.56
Disgust
100
.49
Happy
100
.33
Fear
17
e
Surprise
100
.76
Sad
100
.85
*p < .05.

presenting the highest level of intensity. We then renderedthese models on 10 Caucasian actors (5 female). The faces
were randomly expressing the six basic emotional expressions of fear, happy, anger, disgust, sad and surprise. As in theprevious experiment, we created a static version of theaverage models by collapsing the time factor and keeping only
the highest amplitude of each AU involved in the model (seeFig. 5). All observers were required to categorize both the staticand dynamic facial expressions of the stimuli. We thenrandomly sampled 10 times all the mental models of each of
the basic emotions (happy, surprise, fear, disgust, anger and

100%
90%

Accuracy score

80%
70%
60%
50%

Static40%

Dynamic

30%
20%
10%
0%
Happy

Surprise

Fear

Disgust

Anger

Sad

Facial expression
Fig. 4 e PS's percentage accuracy score for the recognition of happy, surprise, fear, disgust, anger and sad, from her staticand dynamic reconstructed mental models. With the exception of fear, PS showed the maximum accuracy score of 100% for
the recognition of all the dynamic emotions.

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

59

Fig. 5 e Average static reconstructed mental model for the expressions of happy, surprise, fear, disgust, anger and sad. The
static version of the average models was created by collapsing the time factor and keeping only the highest amplitude ofeach AU involved in the model.

sad) on each face, which resulted in a total of 60 trials in boththe static and dynamic conditions. The faces subtended thesame visual angle as in the previous experiment and eachstimulus was presented for 1250 msec in random order at thecentre of the screen.

3.4.1.3. PROCEDURE. We followed the same procedure as theone in experiment 2.

3.4.2.

Results and discussion

PS was impaired in categorizing the static emotions of fear[t(11) ¼ 2.15, p < .05], as well as sad [t(11) ¼ 2.70, p < .05] fromaverage mental models compared to the age-matched healthy
control observers (see Table 4).
However, PS was accurate for categorizing all the dynamic
emotions. Of interest, her performance for decoding fear fromthe dynamic average of healthy mental models waspreserved.
For the controls, a repeated-measures ANOVA yielded astatistically significant main effect of the presentation (static
vs dynamic) [F(1, 22) ¼ 9.48, p < .05], reflecting that the average

Table 4 e Percentage accuracy score of PS and agematched controls for average static and dynamic
reconstructed mental models. The scores are given for therecognition of anger, disgust, happy, fear, surprise andsad. (Experiment 3).

Score %

PS

Age-matched controls

Modified t-test

Mean %

SD %

60.00
80.83
97.50
71.67
39.17
78.33

.20
.23
1.72
2.15
.05
2.70

79.17
73.33
98.33
75.00
89.17
86.66

25.39
21.03
3.89
23.93
13.79
19.23

Static reconstructed models
Anger
53.33
.20
Disgust
76.67
.23
Happy
90.00
1.72
Fear
33.33
2.15*
Surprise
40.00
.05
Sad
56.57
2.70*
Dynamic reconstructed models
Anger
90
.40
Disgust
100
1.21
Happy
100
.41
Fear
90
.60
Surprise
100
.75
Sad
100
.66
*p < .05.

performance with dynamic presentations was higher(M ¼ 83.61%, SD ¼ 9.55%) than with static ones (M ¼ 71.25%,
SD ¼ 19.94%). A highly significant emotion  presentationinteraction [F(1, 22) ¼ 7.45, p < .001], indicated that accuracy
scores for dynamic presentations were higher for some emotions, but not for others. Post-hoc t-tests revealed that thiseffect was driven only by ‘surprise’, which was significantly
higher with the dynamic presentation (M ¼ 89.17%,
SD ¼ 13.79%) than with its static version (M ¼ 39.17%,
SD ¼ 17.30%) [t(11) ¼ 7.54, p < .001].

4.

General discussion

Influential theoretical (Bruce & Young, 1986) and neuroanatomical (Haxby et al., 2000) models of face processing havesuggested the existence of distinct cortical pathways for face
identification and expression categorization, a view that has
also been challenged (Calder & Young, 2005). The main goal of
the present study was to clarify whether the face informationof identity and emotional expression categorization tap into
common or distinct representational systems. To tackle thisissue, brain-damaged patients can be very informative, asfrom their lesions, their specific behavioural impairments andinformation use it is possible to infer the critical role played by
the damaged regions in the healthy operating system. We firstassessed the ability of PS e an acquired case of pure prosopagnosia e to categorize static facial expressions. PS was
impaired for many facial expressions of the Ekman and Friesen's faces. Secondly, we modelled her mental representations of happy, surprise, fear, disgust, anger and sad. Our
overarching goal was thus to map out the facial features usedby the patient to achieve this biologically relevant task, giventhat we previously reported a suboptimal use of the mouth forface identification in the same patient (Caldara et al., 2005).
Surprisingly, her dynamic mental models of facial expressions
revealed an appropriate use of all facial features and Action
Units (AUs e facial muscles), for all facial expressions, withthe exception of fear. Since PS is using more than the mouth
during this task, these results suggest that the mental representations of facial expressions are anatomically separate, orthey are common but can be accessed from a distinct (cortical)
route from face identification, as shown in memory models(Banks, 2000). Regardless of either potential explanation, our60

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

data support the view of a discrete (neural) coding for
expression and identity.
The normal representations of dynamic facial expressions
in PS, coupled with the impairment in the categorization ofmany of the static Ekman faces, came as a surprise to us andraised unexpected novel questions. How could PS haveappropriate dynamic models of facial expressions and not beable to recognize them when presented in static images? Wethus designed subsequent verification tasks in which the patient was asked to categorize her and average static and dynamic
reconstructed mental models of facial expressions ofemotion. These experiments revealed a clear-cut dissociation.
PS was selectively impaired in the categorization of many facial
expressions of her and the average static reconstructed
models. In stark contrast, she excelled with the very same
dynamic models, a pattern of results that can be explained byher particular set of lesions. Moreover, in general, this result
points to the use of dynamic stimuli as being critical in heassessment of facial expression recognition with braindamaged patients and questions the sole use of static face
images to this aim. We now discuss in turn each of thesefindings and their implications.

4.1.
Does the face system rely on common or distinct
representations for identity and expression?
In order to investigate whether facial identity and expression
decoding is tapping into common or distinct representational
systems, we mapped out the different facial features/AUs
used by a pure case of acquired prosopagnosia to categorize
facial expressions. PS showed abnormal performance in categorizing most of the classic facial expressions (i.e., anger,
fear, surprise and sad) of the well known Ekman and Friesen
database (Ekman & Friesen, 1976, 1978), whereas the agematched control group showed normal range performance(Calder et al., 2003). The analysis of the reconstructed models
clearly revealed that PS is comparable to the age-matched
control observers and uses similar facial muscles (i.e., AUs)
and similar temporal dynamics to represent the six classic
emotions. Thus, contrary to our previous observations for facerecognition (Caldara et al., 2005), PS used all facial features/
AUs to reconstruct facial expressions, favouring the view of
distinct representations for expression and identity.1 In linewith these positions, neuroimaging studies have shown thatthe fusiform gyrus is sensitive both to facial identity and facial
expression information (e.g., Cohen Kadosh, Henson, Cohen
Kadosh, Johnson, & Dick, 2010; Fox, Moon, Iaria, & Barton,
2009; Narumoto, Okada, Sadato, Fukui, & Yonekura, 2001).
Calder (2011) suggested that the fusiform gyrus contributes
primarily to the analysis of the visual form of faces for bothtasks, whereas the pSTS would be involved in the coding of
the changeable aspects of faces (Haxby et al., 2000). PS would
thus rely on the fusiform gyrus during face identification and1

According to this scenario, it could be possible that there areshared mechanisms represented in a multiple node network,with nodes being responsible for the processing of identity andothers for the recognition of facial expressions. Some of thenodes related to face identification would be strongly damaged inPS.

the pSTS during facial expression recognition. This explanation could be partially accounted by a previous neuroimaging
study using a neural adaptation paradigm with the patient.
Schiltz et al. (2006) showed that contrary to normal observers,
the neural activations of PS's right fusiform gyrus could notdiscriminate between the repetition of identical and different
faces. This result suggests that the fine-grained visual form
analysis necessary to effectively perform face identification
requires the integrity of the face network and/or an optimaluse of all the facial features. In contrast, the pSTS isanatomically and functionally spared in PS (Sorger et al., 2007)
and could account for her normal facial information use for
facial expression categorization. Despite favouring this view,
we cannot completely and firmly rule out an alternative scenario in which the face system has a common set of representations, as the access could occur from distinct corticalpathways e one for face identification (through the inferior
occipital gyrus) and one for expressions (through the pSTS).
Importantly, both explanations suggest the existence of adiscrete coding for expression and identity, and a future
neuroimaging study is necessary to provide a clear-cut pictureof those scenarios and precisely isolate the brain regions
dedicated to the decoding of static and dynamic faces in PS, forboth face identity and expression categorization.
On a general note, the use of dynamic reverse correlation
techniques represents a unique tool for the understanding ofpatient impairments and their rehabilitation. For instance,
growing studies indicate that schizophrenic patients are
impaired in the recognition of the six basic facial emotional
expressions (e.g., Kohler, Bilker, Hagendoorn, Gur, & Gur, 2000;
Kohler et al., 2003; Kohler, Turner, Gur, & Gur, 2004; Martin,
Baudouin, Tiberghien, & Franck, 2005; Sachs, Steger-Wuchse,
Kryspin-Exner, Gur, & Katschnig, 2004). A selective impairment in recognizing fear and disgust has also been shown for
unaffected relatives (Mendoza et al., 2011). Revealing the
precise (defective) facial information use in these populations
might help to tailor rehabilitation-training programs. Gazecontingent eye tracking paradigms (Caldara, Zhou, & Miellet,
2010; Miellet, Caldara, & Schyns, 2011; Miellet, He, Zhou, Lao,
& Caldara, 2012) will be well suited for this scope, as they
can force the viewing of a particular diagnostic features during a categorization task (see also Adolphs et al. 2005), hopefully improving performance.

4.2.
Advantage in decoding dynamic facial expression
in prosopagnosia
Our study clearly points to an advantage in decoding dynamic
versus static images for the categorization of facial expressions in PS only. Dynamic facial expressions provide observers
with richer, unique and ecologically valid representations,which should facilitate their processing (e.g., Giard &Peronnet, 1999; Johnston et al., 2013; LaBar, Crupain,
Voyvodic, & McCarthy, 2003; Paulmann, Jessen, & Kotz, 2009;
Sato, Kochiyama, Yoshikawa, Naito, & Matsumura, 2004;
Trautmann, Fehr, & Herrmann, 2009). However, the advantage in the processing of dynamic over static emotional
stimuli in healthy observers is controversial (for a review see,Fiorentini & Viviani, 2011). Some studies have showed anadvantage (e.g., Ambadar, Schooler, & Cohn, 2005;

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

Cunningham & Wallraven, 2009; Giard & Peronnet, 1999;
Knappmeyer, Thornton, & Bulthoff, 2003; Paulmann et al.,2009; Wehrle, Kaiser, Schmidt, & Scherer, 2000), whereas
others have revealed that the benefits of dynamic properties
in processing facial information might be minimal (e.g., Goldet al., 2013) or inexistent (e.g., Fiorentini & Viviani, 2011). A
study by Palermo et al. (2011) also revealed that congenital
prosopagnosic patients showed normal recognition of static
facial expressions but less efficient and weaker holistic coding
for both identity and expressions. These results suggest thatcompensatory strategies could play a role in the normal
recognition of static facial expressions and that the perceptual
representations of identity and expression may not be separate at all stages. But more importantly, this observation canbe accounted for by the set of lesions in the patient PS. It has
been recently shown that the right inferior occipital gyrus eanatomically damaged in PS e is causally engaged in theprocessing of static face images (Pitcher et al., 2014) and expressions (Pitcher, 2014). On the contrary, the pSTS eanatomically spared in PS e has been related to the processingof dynamic faces (Pitcher et al., 2014) and expressions (Pitcher,2014). Consequently, our data suggest that the inferior occipital gyrus plays a critical role in the decoding of static facial
expressions (as the patient shows an impairment), but thepSTS is sufficient to effectively decode facial expressions fromdynamic visual inputs (for which the patient shows normal
performance). Greater responses for dynamic compared tostatic facial expressions have been found in the fusiform
gyrus and the pSTS in the right hemisphere (e.g., Johnston
et al., 2013; Kessler et al., 2011; LaBar et al., 2003; Sato et al.,
2004; Schultz & Pilz, 2009), as well as the visual motion areain occipitotemporal regions (more commonly referred to as
V5/MT) (Johnston et al., 2013). Our observations also reinforce
the existence of a direct and functionally distinct corticalpathway connecting the early visual cortex to the pSTS, which
would not require structural information from the right inferior occipital gyrus (since this region is damaged in PS) to
decode expressions effectively. This advantage for directly
processing dynamic visual inputs seems to be specific to facial
expressions, as the patient cannot recover identity throughdynamic visual information in everyday life. However, thishypothesis remains to be clarified with a future functional
neuroimaging study measuring the extent to which the activations observed in the pSTS in PS (Sorger et al., 2007) would
be significantly modulated by the presentations of dynamic
faces, as well as static facial expressions.
Our results also feed the debate of the dynamic versus
static advantage with new data, by suggesting that dynamic
information might give an advantage to patients only. In fact,
our results are in line with those by Humphreys et al. (1993)
with an agnostic patient. This patient was impaired at identifying facial identity and facial expressions when exposed tostatic images, whereas he was performing normally when
asked to judge a subset of facial expressions (i.e., smiling,frowning, or surprise) and gender from dynamic faces
animated by light dots. However, whether this agnostic patient could correctly categorize all the basic facial expressions
and his fine-grained information use was not assessed. In
spite of these methodological and theoretical differences, theobservation that emerges from both studies is that the61

recognition of emotions seems to imply a complex mechanism being facilitated by dynamic information. In fact, patients suffering from depression or schizophrenia also benefit
from dynamic presentation of facial emotional expressions(e.g., Atkinson, Dittrich, Gemmell, & Young, 2004; Kan,
Mimura, Kamijima, & Kawamura, 2004; Schaefer, Baumann,Rich, Luckenbaugh, & Zarate, 2010). Dynamic stimuli contain
information that cannot be completely rendered by static
representations and force the observers to shift their attention
to different facial features. This might enhance attention andmotor simulations particularly in fragile or neurologically
impaired face systems, explaining increased performance fordynamic faces in those populations.
We should acknowledge that we did not objectively assess
whether the amount of physical information conveyed bystatic and dynamic stimuli is identical or different for facial
expressions categorization. We therefore cannot completely
rule out the hypothesis that temporal properties provide a
wealth of information that cannot be completely rendered bystatic facial cues. Interestingly, Gold et al., 2013 have shownthat dynamic stimuli do not seem to provide additional information for the recognition of facial expressions than what
is already offered by static facial cues in normal observers, bymeasuring the amount of information carried by static anddynamic facial expressions. In our study we created a static
version of the stimuli by keeping only the highest amplitudesof each AU involved in the model, a procedure that shouldhave led to the representation of ‘optimal’ static signals. In
fact, by using a comparable approach, Fiorentini and Viviani(2011) have shown the absence of an advantage between the
categorization of static and dynamic facial expressions innormal observers. In line with those results, our findings show
equal performance for our healthy control observers between
static and dynamic stimuli, with the exception of surprise.
Thus, PS's ability to correctly categorize dynamic facial expressions does not seem to stem from the physical information available, but rather from an adequate psychologicalability to make use of this information; a process that is mostprobably occurring in the pSTS.

4.3.

The special case of “fear”

Our results support the hypothesis of a defective processing offear in brain-damaged patients. PS was strongly impaired inrecognizing the dynamic facial expressions of fear from her
mental model, the static version from Ekman and Friesen's
faces, and average mental model. It is worth pointing out that
our age-matched control group showed an effective categorization of emotions from PS's static and dynamic reconstructed models, for all facial expressions but fear. Thisobservation suggests defective internal representations forthis expression in the patient. However, when PS was stimulated by optimal fear dynamic inputs (i.e., the average dynamic
mental models) she achieved correct categorization.
The impairment for fear categorization with static images
resonates with findings obtained with another brain-damaged
patient. Adolphs et al. (2005) found selective impairment in fear
recognition with a patient presenting bilateral damage of theamygdala: SM. SM did not spontaneously use the eyes during
fear decoding in static images, but the mouth, which resulted

62

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

in impairment in fear recognition. SM was able to categorize
fear only when forced to look at the diagnostic region for thisexpression: the eye (Smith et al., 2005). Similarly, the analysis
comparing PS's model for fear with the Ekman model showed aclear tendency to emphasize information from the mouth region in PS, compared to the eyes. However, PS's lesions areanatomically sparing the amygdala (Sorger et al., 2007) andcannot account for this deficit. In fact, Adolphs (2013) suggeststhat the processing of fear involves a complex distributed
network featuring interactions among diverse cortical regions,rather than a single, localized “fear center”. Very little is however known about which structures play a key role in this brain
circuit and how these different regions interact together. Thus,
PS's selective impairment in fear recognition could also stemfrom her brain lesions, micro-lesions such as axonal shearing(Sorger et al., 2007) or the missing interactions between her
lesions and other cortical structures.
PS's impairment for fear might also appear in contrast with aprevious fMRI result on her emotional attention (Peelen, Lucas,
Mayer, & Vuilleumier, 2009). In a visual search task, a target face
was presented among an array of distractor faces. The target
differed from the distractor either by identity, identity andemotional expression (fear or happy), or identity and colour. PS
and the controls showed a faster detection for fear and happycompared to the neutral face conditions. PS showed also asimilar advantage for fear in a second change detection task,
detecting significantly more changes when the changed face
was fearful, as compared to when it was neutral. This behavioural pattern of results was parallelled with according neural
responses biases indicating emotional attention in the brain
areas coding for faces, suggesting a normal processing for fear.
However, paying attention to a particular facial expression ofemotion does not necessarily involve an explicit correct categorization for this expression. PS could have reached thispattern of results by using low-level cues distinguishing fear
from neutral or happy signals, as she is relatively preserved for
those expressions. PS's deficit for fear might also be clarified byrecording her brain activation during the decoding of her andaverage models with in a future neuroimaging study.

5.

Conclusions

The adequate categorization of facial expressions is a criticalfeature for adaptive social interactions. Our general goal was tounderstand whether face information used for identity andemotional expression categorization tap into common or
distinct representational systems. We isolated information use
for facial expressions in a pure case of acquired prosopagnosia
with a lesion encompassing the right inferior occipital gyrus.
PS's reconstructed mental models showed a normal use of all of
the face features and muscles (i.e., AUs of the FACS coding
system) for the representation of facial expressions, with the
exception of fear. This is in stark contrast with the suboptimal
information she uses for retrieving face identity (i.e., the mouth
and the external contours). These data suggest that the facesystem does not rely on a unique representational system tocode face features for identity and expression, or at least it
relies on distinct cortical pathways to access them, flexibly
adapting to visual and task constraints. In addition, ourobservations indicate that those cortical routes are modulatedby the use of dynamic information, which facilitates the correct categorization of facial expressions in the patient. The
inferior occipital gyrus plays a critical role for the decoding ofstatic images, and the patient presents a selective impairment
in the decoding of static expressions. On the contrary, the patient shows normal performance to effectively decode facial
expressions from dynamic faces. The pSTS, which is spared in
the patient, would be sufficient to effectively achieve this task.
This result reinforces the view of the existence of a corticalpathway carrying out directly face signals from the early visualcortex to the pSTS, thus providing novel insights on the normal
face operating system. Altogether, our data question also theconclusions obtained with patients by using unnatural static
images and emphasize the need for a future neuroimaging
study on the same patient to consolidate and provide a finegrained picture of the present findings.

Acknowledgements
We are extremely grateful to PS for her devoted participationin this research as well as to all our healthy participants. This
study was supported by the National Center of Competence in
Research (NCCR) Affective sciences financed by the Swiss
National Science Foundation (n 51NF40-104897).

references

Adolphs, R. (2013). The biology of fear. Current Biology, 23(2),79e93.
Adolphs, R., Gosselin, F., Buchanan, T. W., Tranel, D., Schyns, P.,& Damasio, A. R. (2005). A mechanism for impaired fear
recognition after amygdala damage. Nature, 433(7021), 68e72.
Ambadar, Z., Schooler, J. W., & Cohn, J. F. (2005). Deciphering theenigmatic face: the importance of facial dynamics ininterpreting subtle facial expressions. Psychological Science,16(5), 403e410.
Atkinson, A. P., Dittrich, W. H., Gemmell, A. J., & Young, A. W.
(2004). Emotion perception from dynamic and static body
expressions in point-light and full-light displays. Perception,
33(6), 717e746.
Banks, W. P. (2000). Recognition and source memory as
multivariate decision processes. Psychological Science, 11(4),267e273.
Benton, A., & Van Allen, M. (1972). Prosopagnosia and facial
discrimination. Journal of the Neurological Sciences, 15, 167e172.
Bodamer, J. (1947). Die Prosopagnosie. Archiv fuer Psychiatrie und
Nervenkrankheiten, 179, 6e54.
Bouvier, S. E., & Engel, S. A. (2004). Patterns of cortical damage inachromatopsia and prosopagnosia. Journal of Vision, 4(8), 205a.
Bowers, D., Bauer, R. M., Coslett, H. B., & Heilman, K. M. (1985).
Processing of faces by patients with unilateral hemispherelesions. Brain and Cognition, 4(3), 258e272.
Brown, E., & Perrett, D. I. (1993). What gives a face its gender?
Perception, 22(7), 829e840.
Bruce, V., & Young, A. (1986). Understanding face recognition.
British Journal of Psychology, 77(3), 305e327.
Bruyer, R., Laterre, C., Seron, X., Feyereisen, P., Strypstein, E.,
Pierrard, E., et al. (1983). A case of prosopagnosia with somepreserved covert remembrance of familiar faces. Brain andCognition, 2(3), 257e284.

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4

Busigny, T., Graf, M., Mayer, E., & Rossion, B. (2010). Acquired
prosopagnosia as a face-specific disorder: ruling out the generalvisual similarity account. Neuropsychologia, 48(7), 2051e2067.
Caldara, R., & Abdi, H. (2006). Simulating the 'other-race' effectwith autoassociative neural networks: further evidence infavor of the face-space model. Perception, 35(5), 659e670.
Caldara, R., Rossion, B., Bovet, P., & Hauert, C. A. (2004). Eventrelated potentials and time course of the “other-race” faceclassification advantage. NeuroReport, 15(5), 905e910.
Caldara, R., Schyns, P., Mayer, E., Smith, M. L., Gosselin, F., &Rossion, B. (2005). Does prosopagnosia take the eyes out offace representations? Evidence for a defect in representing
diagnostic facial information following brain damage. Journalof Cognitive Neuroscience, 17(10), 1652e1666.
Caldara, R., Zhou, X., & Miellet, S. (2010). Putting culture under the'spotlight' reveals universal information use for facerecognition. PLoS One, 5(3), e9708.
Calder, A. J. (2011). Does facial identity and facial expression
recognition involve separate visual routes? In A. J. Calder,
G. Rhodes, M. H. Johnson, & J. V. Haxby (Eds.), The Oxford
handbook of face perception (pp. 427e448) Oxford: Oxford
University Press.
Calder, A. J., Burton, A. M., Miller, P., Young, A. W., & Akamatsu, S.
(2001). A principal component analysis of facial expressions.
Vision Research, 41(9), 1179e1208.
Calder, A. J., Keane, J., Manly, T., Sprengelmeyer, R., Scott, S.,
Nimmo-Smith, I., et al. (2003). Facial expression recognition
across the adult life span. Neuropsychologia, 41(2), 195e202.
Calder, A. J., & Young, A. W. (2005). Understanding the recognitionof facial identity and facial expression. Nature Reviews
Neuroscience, 6(8), 641e651.
Campbell, R., Landis, T., & Regard, M. (1986). Face recognition andlipreading. A neurological dissociation. Brain, 109, 509e521.
Cohen Kadosh, K., Henson, R. N., Cohen Kadosh, R.,
Johnson, M. H., & Dick, F. (2010). Task-dependent activation of
face-sensitive cortex: an fMRI adaptation study. Journal of
Cognitive Neuroscience, 22(5), 903e917.
Cole, M., & Perez-Cruet, J. (1964). Prosopagnosia. Neuropsychologia,
2, 237e246.
Crawford, J. R., & Garthwaite, P. H. (2002). Investigation of thesingle case in neuropsychology: confidence limits on theabnormality of test scores and test score differences.
Neuropsychologia, 40(8), 1196e1208.
Cunningham, D. W., & Wallraven, C. (2009). Dynamic information
for the recognition of conversational expressions. Journal of
Vision, 9(13), 1e17.
Damasio, A. R., Damasio, H., & Van Hoesen, G. W. (1982).
Prosopagnosia: anatomic basis and behavioral mechanisms.
Neurology, 32(4), 331e341.
De Gelder, B., Pourtois, G., Vroomen, J., & Bachoud-Levi, A. C.
(2000). Covert processing of faces in prosopagnosia isrestricted to facial expressions: evidence from cross-modalbias. Brain and Cognition, 44, 425e444.
De Renzi, E., & Di Pellegrino, G. (1998). Prosopagnosia and alexia
without object agnosia. Cortex, 34(3), 403e415.
Ekman, P., & Friesen, W. V. (1976). Pictures of facial affect. Palo Alto,
CA: Consulting Psychologists Press.
Ekman, P., & Friesen, W. V. (1978). Facial action coding system. Palo
Alto, CA: Consulting Psychologists Press.
Farah, M. J. (1990). Visual agnosia. Cambridge: MIT Press.
Fiorentini, C., & Viviani, P. (2011). Is there a dynamic advantage
for facial expressions? Journal of Vision, 11(3), 1e15.
Fox, C. J., Moon, S. Y., Iaria, G., & Barton, J. J. (2009). The correlatesof subjective perception of identity and expression in the face
network: an fMRI adaptation study. NeuroImage, 44(2), 569e580.
George, P. A., & Hole, G. J. (1995). Factors influencing the accuracyof age estimates of unfamiliar faces. Perception, 24(9),1059e1073.

63

Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration
during multimodal object recognition in humans: a behavioral
and electrophysiological study. Journal of Cognitive
Neuroscience, 11(5), 473e490.
Gill, D., Garrod, O. G., Jack, R. E., & Schyns, P. G. (2014). Facial
movements strategically camouflage involuntary social signals
of face morphology. Psychological Science, 25(5), 1079e1086.
Gold, J. M., Barker, J. D., Barr, S., Bittner, J. L., Bromfield, W. D.,
Chu, N., et al. (2013). The efficiency of dynamic and static
facial expression recognition. Journal of Vision, 13(5), 1e12.
Gosselin, F., & Schyns, P. G. (2001). Bubbles: a technique to reveal
the use of information in recognition tasks. Vision Research,
41(17), 2261e2271.
Hasselmo, M. E., Rolls, E. T., & Baylis, G. C. (1989). The role ofexpression and identity in face-selective responses of neurons
in the temporal visual cortex of the monkey. Behavioural Brain
Research, 32, 203e218.
Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2000). The distributed
human neural system for face perception. Trends in Cognitive
Sciences, 4(6), 223e233.
Humphreys, G. W., Donnelly, N., & Riddoch, M. J. (1993).
Expression is computed separately from facial identity, and it
is computed separately for moving and static faces:
neuropsychological evidence. Neuropsychologia, 31(2), 173e181.
Jack, R. E., Blais, C., Scheepers, C., Schyns, P. G., & Caldara, R.
(2009). Cultural confusions show that facial expressions are
not universal. Current Biology, 19(18), 1543e1548.
Jack, R. E., Caldara, R., & Schyns, P. G. (2012a). Internal
representations reveal cultural diversity in expectations offacial expressions of emotion. Journal of Experimental Psychology
General, 141(1), 19e25.
Jack, R. E., Garrod, O. G., Yu, H., Caldara, R., & Schyns, P. G.
(2012b). Facial expressions of emotion are not culturally
universal. Proceedings of the National Academy of Sciences of theUnited States of America, 109(19), 7241e7244.
Johnston, P., Mayes, A., Hughes, M., & Young, A. W. (2013). Brain
networks subserving the evaluation of static and dynamic
facial expressions. Cortex, 49, 2462e2472.
Kan, Y., Mimura, M., Kamijima, K., & Kawamura, M. (2004).
Recognition of emotion from moving facial and prosodic
stimuli in depressed patients. Journal of Neurology Neurosurgery& Psychiatry, 75(12), 1667e1671.
Kessler, H., Doyen-Waldecker, C., Hofer, C., Hoffmann, H.,
Traue, H. C., & Abler, B. (2011). Neural correlates of theperception of dynamic versus static facial expressions ofemotion. Psycho-social Medicine, 8, 1e8.
Knappmeyer, B., Thornton, I. M., & Bulthoff, H. H. (2003). The useof facial motion and facial form during the processing ofidentity. Vision Research, 43(18), 1921e1936.
Kohler, C. G., Bilker, W., Hagendoorn, M., Gur, R. E., & Gur, R. C.
(2000). Emotion recognition deficit in schizophrenia:
association with symptomatology and cognition. BiologicalPsychiatry, 48(2), 127e136.
Kohler, C. G., Turner, T. H., Bilker, W. B., Brensinger, C. M.,
Siegel, S. J., Kanes, S. J., et al. (2003). Facial emotion recognitionin schizophrenia: intensity effects and error pattern. American
Journal of Psychiatry, 160(10), 1768e1774.
Kohler, C. G., Turner, T. H., Gur, R. E., & Gur, R. C. (2004).
Recognition of facial emotions in neuropsychiatric disorders.
CNS Spectrums, 9(4), 267e274.
LaBar, K. S., Crupain, M. J., Voyvodic, J. T., & McCarthy, G. (2003).
Dynamic perception of facial affect and identity in the humanbrain. Cerebral Cortex, 13(10), 1023e1033.
Landis, T., Regard, M., Bliestle, A., & Kleihues, P. (1988).
Prosopagnosia and agnosia for noncanonical views. An
autopsied case. Brain, 111(6), 1287e1297.
Malone, D. R., Morris, H. H., Kay, M. C., & Levin, H. S. (1982).
Prosopagnosia: a double dissociation between the recognition

64

c o r t e x 6 5 ( 2 0 1 5 ) 5 0 e6 4of familiar and unfamiliar faces. Journal of Neurology
Neurosurgery & Psychiatry, 45(9), 820e822.
Martin, F., Baudouin, J. Y., Tiberghien, G., & Franck, N. (2005).
Processing emotional expression and facial identity in
Schizophrenia. Psychiatry Research, 134, 43e53.
Mattson, A. J., Levin, H. S., & Grafman, J. (2000). A case of
prosopagnosia following moderate closed head injury withleft hemisphere focal lesion. Cortex, 36(1), 125e137.
Mendoza, R., Cabral-Calderin, Y., Dominguez, M., Garcia, A.,
Borrego, M., Caballero, A., et al. (2011). Impairment ofemotional expression recognition in schizophrenia: a Cuban
familial association study. Psychiatry Research, 185, 44e48.
Miellet, S., Caldara, R., & Schyns, P. G. (2011). Local Jekyll andglobal Hyde: the dual identity of face identification.
Psychological Science, 22(12), 1518e1526.
Miellet, S., He, L., Zhou, X., Lao, J., & Caldara, R. (2012). When East
meets West: gaze-contingent Blindspots abolish cultural
diversity in eye movements for faces. Journal of Eye Movement
Research, 5(2), 1e12.
Narumoto, J., Okada, T., Sadato, N., Fukui, K., & Yonekura, Y.
(2001). Attention to emotion modulates fMRI activity inhuman right superior temporal sulcus. Cognitive Brain Research,
12(2), 225e231.
Palermo, R., O'Connor, K. B., Davis, J. M., Irons, J., & McKone, E.
(2013). New tests to measure individual differences inmatching and labelling facial expressions of emotion, andtheir association with ability to recognise vocal emotions andfacial identity. PLoS One, 8(6), 1e16.
Palermo, R., Willis, M. L., Rivolta, D., McKone, E., Wilson, C. E., &Calder, A. J. (2011). Impaired holistic coding of facial
expression and facial identity in congenital prosopagnosia.
Neuropsychologia, 49(5), 1226e1235.
Paulmann, S., Jessen, S., & Kotz, S. A. (2009). Investigating themultimodal nature of human communication: an ERP study.
Journal of Psychophysiology, 23(2), 63e76.
Peelen, M. V., Lucas, N., Mayer, E., & Vuilleumier, P. (2009).
Emotional attention in acquired prosopagnosia. Social
Cognitive and Affective Neuroscience, 4(3), 268e277.
Pessoa, L. (2008). On the relationship between emotion andcognition. Nature Reviews Neuroscience, 9(2), 148e158.
Pitcher, D. (2014). Facial expression recognition takes longer inthe posterior superior temporal sulcus than in the occipital
face area. Journal of Neuroscience, 34(27), 9173e9177.
Pitcher, D., Duchaine, B., & Walsh, V. (2014). Combined TMS andfMRI reveal dissociable cortical pathways for dynamic andstatic face perception. Current Biology, 24(17), 2066e2070.
Quaglino, A. (1867). Emiplegia sinistra con amaurosid
GuarigionedPerdita totale della percezione dei colori e dellamemoria della configurazione degli oggetti. Giornaled'Oftalmologia Italiano, 10, 106e112.
Rossion, B. (2008). Constraining the cortical face network by
neuroimaging studies of acquired prosopagnosia. NeuroImage,40(2), 423e426.
Rossion, B., Caldara, R., Seghier, M., Schuller, A. M., Lazeyras, F., &Mayer, E. (2003). A network of occipito-temporal face-sensitive
areas besides the right middle fusiform gyrus is necessary fornormal face processing. Brain, 126, 2381e2395.
Sachs, G., Steger-Wuchse, D., Kryspin-Exner, I., Gur, R. C., &Katschnig, H. (2004). Facial recognition deficits and cognitionin schizophrenia. Schizophrenia Research, 68(1), 27e35.
Sato, W., Kochiyama, T., Yoshikawa, S., Naito, E., &Matsumura, M. (2004). Enhanced neural activity in response to
dynamic facial expressions of emotion: an fMRI study.
Cognitive Brain Research, 20(1), 81e91.
Schaefer, K. L., Baumann, J., Rich, B. A., Luckenbaugh, D. A., &Zarate, C. A., Jr. (2010). Perception of facial emotion in adults
with bipolar or unipolar depression and controls. Journal of
Psychiatric Research, 44(16), 1229e1235.

Scherer, K. (2005). What are emotions? And how can they bemeasured? Social Science Information, 44(4), 695e729.
Schiltz, C., Sorger, B., Caldara, R., Ahmed, F., Mayer, E., Goebel, R.,
et al. (2006). Impaired face discrimination in acquired
prosopagnosia is associated with abnormal response to
individual faces in the right middle fusiform gyrus. Cerebral
Cortex, 16(4), 574e586.
Schultz, J., & Pilz, K. S. (2009). Natural facial motion enhances
cortical responses to faces. Experimental Brain Research, 194(3),465e475.
Schyns, P. G., Bonnar, L., & Gosselin, F. (2002). Show me thefeatures! Understanding recognition from the use of visual
information. Psychological Science, 13(5), 402e409.
Schyns, P. G., Petro, L. S., & Smith, M. L. (2009). Transmission of
facial expressions of emotion co-evolved with their efficient
decoding in the brain: behavioral and brain evidence. PLoS
One, 4(5), e5625.
Sergent, J., & Signoret, J. L. (1992). Varieties of functional deficits
in prosopagnosia. Cerebral Cortex, 2(5), 375e388.
Sergent, J., & Villemure, J. G. (1989). Prosopagnosia in a righthemispherectomized patient. Brain, 112, 975e995.
Shuttleworth, E. C. J., Syring, V., & Allen, N. (1982). Further
observations on the nature of prosopagnosia. Brain andCognition, 1(3), 307e322.
Smith, M. L., Cottrell, G. W., Gosselin, F., & Schyns, P. G. (2005).
Transmitting and decoding facial expressions. Psychological
Science, 16(3), 184e189.
Sorger, B., Goebel, R., Schiltz, C., & Rossion, B. (2007).
Understanding the functional neuroanatomy of acquired
prosopagnosia. NeuroImage, 35(2), 836e852.
Tranel, D., Damasio, A. R., & Damasio, H. (1988). Intact recognitionof facial expression, gender, and age in patients with impaired
recognition of face identity. Neurology, 38(5), 690e696.
Trautmann, S. A., Fehr, T., & Herrmann, M. (2009). Emotions inmotion: dynamic compared to static facial expressions of
disgust and happiness reveal more widespread emotionspecific activations. Brain Research, 1284, 100e115.
Vizioli, L., Foreman, K., Rousselet, G. A., & Caldara, R. (2010a).
Inverting faces elicits sensitivity to race on the N170
component: a cross-cultural study. Journal of Vision, 10(1),1e23.
Vizioli, L., Rousselet, G. A., & Caldara, R. (2010b). Neural repetition
suppression to identity is abolished by other-race faces.
Proceedings of the National Academy of Sciences of the United Statesof America, 107(46), 20081e20086.
Warrington, E. K. (1984). Recognition memory test. Windsor,
Berkshire: NFER.
Wehrle, T., Kaiser, S., Schmidt, S., & Scherer, K. R. (2000). Studying
the dynamics of emotional expression using synthesizedfacial muscle movements. Journal of Personality and Social
Psychology, 78(1), 105e119.
Wigan, A. L. (1844). The duality of the mind: Proved by the structure,functions, and diseases of the brain, and by the phenomena of mental
derangement, and shown to be essential to moral responsibility.
London: Longman, Brown, Green, and Longmans.
Winston, J. S., Henson, R. N., Fine-Goulden, M. R., & Dolan, R. J.
(2004). fMRI-adaptation reveals dissociable neuralrepresentations of identity and expression in face perception.
Journal of Neurophysiology, 92(3), 1830e1839.
Young, A. W., Newcombe, F., de Haan, E. H., Small, M., &Hay, D. C. (1993). Face perception after brain injury. Selective
impairments affecting identity and expression. Brain, 116,941e959.
Yu, H., Garrod, O. G. B., & Schyns, P. G. (2012). Perception-driven
facial expression synthesis. Computer & Graphics, 36(3),152e162.