Cerebral Cortex, 2024, 34, bhae211
https://doi.org/10.1093/cercor/bhae211
Advance access publication date 24 May 2024

Original Article

Neural computations in prosopagnosia
Simon Faghel-Soubeyrand
1

1,2, Anne-Raphaelle Richoz3 , Delphine Waeber3 , Jessica Woodhams4 , Roberto Caldara3 ,1,

Frédéric Gosselin , Ian Charest *2 Department of Experimental Psychology, University of Oxford, Anna Watts Building, Woodstock Rd, Oxford OX2 6GG
3 Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
4 School of Psychology, University of Birmingham, Hills Building, Edgbaston Park Rd, Birmingham B15 2TT, UK*Corresponding authors: 90 av. Vincent D’indy, Montréal, Qc, Canada. Email: ian.charest@umontreal.ca

We report an investigation of the neural processes involved in the processing of faces and objects of brain-lesioned patient PS, a welldocumented case of pure acquired prosopagnosia. We gathered a substantial dataset of high-density electrophysiological recordings
from both PS and neurotypicals. Using representational similarity analysis, we produced time-resolved brain representations in aformat that facilitates direct comparisons across time points, different individuals, and computational models. To understand how
the lesions in PS’s ventral stream affect the temporal evolution of her brain representations, we computed the temporal generalizationof her brain representations. We uncovered that PS’s early brain representations exhibit an unusual similarity to later representations,
implying an excessive generalization of early visual patterns. To reveal the underlying computational deficits, we correlated PS’ brain
representations with those of deep neural networks (DNN). We found that the computations underlying PS’ brain activity bore acloser resemblance to early layers of a visual DNN than those of controls. However, the brain representations in neurotypicals became
more akin to those of the later layers of the model compared to PS. We confirmed PS’s deficits in high-level brain representations bydemonstrating that her brain representations exhibited less similarity with those of a DNN of semantics.
Key words: prosopagnosia; EEG; RSA; artificial neural networks; semantic representations.

Introduction
The human brain is equipped with sophisticated machinery optimized to quickly and effectively recognize faces in a series ofcomputations unfolding within tens of milliseconds. A dramatic
contrast to this typically efficient process has been revealedin brain-lesioned patients with an inability to recognize faces,
individuals called acquired prosopagnosics (Bodamer 1947). Findings from these patients have refined the functional role andthe distributed nature of the face-sensitive brain regions in theventral stream, such as the fusiform gyrus (fusiform face area
[FFA]; Bobes et al. 2003; Kanwisher et al. 1997) and the lateral
portion of the inferior occipital gyrus (Occipital Face Area, OFA;Dricot et al. 2008; Gauthier et al. 2000; Rossion et al. 2003; Sorger
et al. 2007). This literature has generally contributed to the ideathat specialized and category-selective neural modules are necessary for functional aspects of face processing (Cohen et al.
2019). Brain imaging findings from individuals born with deficits
in face recognition (developmental prosopagnosics; (McConachie
1976; Avidan et al. 2005; Kaltwasser et al. 2014; Rosenthal et al.
2017; Jiahui et al. 2018) have revealed finer-grained functional
neural differences in the processes associated with deficits in facerecognition (Avidan et al. 2014; Rosenthal et al. 2017; Jiahui et al.
2018; Zhao et al. 2018). Overall, the cumulation of these neuropsychological, neuroanatomical, and functional components
of prosopagnosia (Busigny et al. 2010; Dricot et al. 2008; B. C.
Duchaine and Nakayama 2006; Rossion 2018; Rossion et al. 2003)
has significantly contributed to neural models of face perception

in the last two decades (B. Duchaine and Yovel 2015; Haxby et al.
2000; White and Mike Burton 2022). Yet, little is known on thenature of face representations of those patients (e.g. Caldara et al.
2005; Fiset et al. 2017), and next to nothing is known on the natureof brain dynamics and neural computations affected in prosopagnosia. Here, we report an investigation of the neural computations
involved in the processing of faces and objects of patient PS, awell-documented case of pure acquired prosopagnosia (Rossion
et al. 2003; Sorger et al. 2007), using Representational Similarity
Analysis (RSA; Charest et al. 2014; Kriegeskorte, Mur, and Bandettini, 2008; Kriegeskorte and Kievit 2013; Nili et al. 2014) applied tobrain imaging and computational models.
Patient PS is a right-handed woman who suffered a closed head
injury in 1992, resulting in extensive bilateral occipitotemporallesions that encompasses the right occipital face area (OFA), left
FFA, and a small region of the right middle temporal gyrus (Sorger
et al. 2007; Dricot et al. 2008). She stands out as one of the mostextensively studied cases of acquired prosopagnosia, with over 32
scientific publications dedicated to her in the past two decades,
as highlighted in recent reviews by Rossion (2022a, 2022b). The
prominence of this case stems from the relatively focal nature ofher lesions in the face network and the highly specific impairment
she experiences in face identification (Busigny et al. 2010). Despite
the comprehensive examination of her condition, including its
impact on various perceptual mechanisms like holistic processes
and the visual content of face representations (Caldara et al.
2005; Ramon et al. 2016; Fiset et al. 2017), there has been no
direct attempt, to the best of our knowledge, to characterize the

Received: December 27, 2022. Revised: April 30, 2024. Accepted: May 3, 2024
© The Author(s) 2024. Published by Oxford University Press.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which
permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

1 Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada

2|

Cerebral Cortex, 2024, Vol. 34, No. 5
to images from several categories. Using a well-established multivariate technique, RSA (Kriegeskorte, Mur, and Bandettini, 2008),
we generated functional brain representations in a versatile format that enables direct comparisons across time, individuals withdiffering neuroanatomical structures, and computational models(Golarai et al. 2015; Popal et al. 2019). To understand how thelesions in PS’s ventral stream affect the temporal evolution ofthese brain representations, we conducted a comparison of thetemporal generalization of these representational geometries inboth PS and a control group. Additionally, to gain insights into
the specific computational deficits in PS’s brain, we conducted
a comparative analysis of her brain representations with thosegenerated by artificial models designed to perform various typesof computations. These models consisted in deep neural networks(DNN) specializing in vision and in semantics, providing a valuable
perspective on the neural computations involved in PS’s deficits(see Faghel-Soubeyrand et al. 2024).

Materials and methods
Patient PS and neurotypical participants
A total of 20 participants were recruited for this study. The firstgroup consisted of 19 neurotypicals individuals that included 15
young controls (9 female, Mage = 22.9 years old) as well as fourage-matched controls (three female, Mage = 67.5). This sample size
was chosen according to the effect sizes described in previous
multivariate pattern analysis studies (Carlson et al. 2013; Cichy
et al. 2014; Hebart et al. 2018; Faghel et al. 2022), as well as previous
studies on prosopagnosia (Humphreys et al. 2007; Richoz et al.
2015; Liu-Shuang et al. 2016; Gao et al. 2019). Data from 10 ofthese young controls (#1–10) have been reported in a previous
study (Faghel-Soubeyrand et al. 2024). One participant from theage-matched group (#2) was rejected due to faulty EEG recordings
and poor behavioral performance during the one-back task and
CFMT+. This study was approved by the Ethics and ResearchCommittee of the University of Birmingham, The University of Fribourg, and informed consent was obtained from all participants.

PS’s case report
Patient PS was born in 1950 and is a pure case of acquired
prosopagnosia. She was hit by the side mirror of a London’sbus in 1992 while crossing the road. This closed head injury
led to major lesions in the left middle fusiform gyrus, where
the left Fusiform Face Area (lFFA) is typically located, and in theright inferior occipital gyrus, which typically locates the rightOccipital Face Area (rOFA; see Gao et al. 2019) for converging fMRI
evidence). Both regions play a critical functional role within theface cortical network (Rossion 2022a, 2022b). She also reported
minor damages in the right middle temporal gyrus and left
posterior cerebellum (for an exhaustive anatomical description
and an illustration of her brain damages (see Sorger et al. 2007,
Figs. 2 and 3). Patient PS is a very well-documented and describedcase of acquired prosopagnosia. She has been extensively studied
over the last 20 years, leading to impactful scientific contributions
that significantly enriched the theoretical models on human faceperception (Rossion 2008, 2014; for a complete case report see,
Rossion 2022a, 2022b; Rossion et al. 2003).
Patient PS recovered remarkably well from initially significant cognitive deficits with the support of medical treatment
and neuropsychological rehabilitation. A couple of months after
her injury she performed within the normal range at different
non-visual tasks for which she was slightly impaired after theaccident (e.g. calculation, short and long-term memory, visual

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

neural computations characteristic of her deficits. The assessment of the nature and extent of brain computations affected inprosopagnosia has traditionally relied on examining the temporal
dynamics and face-selectivity of neural activity. For instance,
event-related potential differences occurring later in processing
have been viewed as indicative of higher-level processes thanthose occurring earlier (Eimer et al. 2012; Bentin and Deouell 2000;
Herzmann et al. 2004; Gosling and Eimer 2011; Alonso Prieto et al.
2011; Simon et al. 2011; Tanaka et al. 2006; Liu-Shuang et al. 2016;
Wiese et al. 2019).
Associations have been found between prosopagnosia andwell-established neural markers of face processing, such as theface-sensitive N170 (Bentin et al. 1996) and face-selective fMRI
activation (Towler and Eimer 2012; Bobes et al. 2003; Alonso
Prieto et al. 2011; Gao et al. 2019). It is worth noting that despite
her significant lesions and difficulties in face identification, PS
still exhibits typical face-selectivity in spared regions of the righthemisphere, including the presence of a right FFA (Rossion et al.
2003; Gao et al. 2019), as well as a typical N170 component in
the right hemisphere, but not in the left (Alonso-Prieto, 2011;
Bobes et al. 2003; Dalrymple et al. 2011). Similarly, developmental
prosopagnosics demonstrate typical activation in the coreposterior regions of the face-processing system, including the
OFA and FFA (Avidan et al. 2014). Recent advances in experimental
techniques, such as fast periodic visual stimulation (Liu-Shuang
et al. 2016b), have shed light on the crucial deficits in neuralface individuation observed in PS. However, characterizing
the underlying computations involved in these neural andperceptual processes remains a difficult endeavor. Describing
neural computations is inherently challenging due to signalto-noise ratio (SNR) issues, which are even more pronounced
when recording brain activity from patients with brain lesions(Liu-Shuang et al. 2016a). Brain damage can substantially alter thef low of brain activity compared to typical observers, potentiallydistorting event-related potential components (Alonso Prieto et al.
2011) and requiring additional repetitions of conditions. Moreover,relying solely on temporal evidence has limitations as it only
partially reveals the nature of the computations that the brain
depends on (Lamme and Roelfsema, 2000; McDermott et al. 2002).
Individuals performing different neural computations on faces
may very well exhibit identical activity at a given latency, as
indicated by univariate event-related potentials.
In recent years, innovative techniques have gained popularity for exploring the nature of brain representations by linking
functional and multivariate brain activity with computationalmodels (Dwivedi et al. 2021; di Oleggio Castello et al. 2021; Popham
et al. 2021; Kriegeskorte and Diedrichsen 2016; Doerig et al. 2022;
Faghel-Soubeyrand et al. 2024). The previously mentioned SNR
concerns may explain why most studies on prosopagnosia havetraditionally relied on a limited set of stimuli conditions, blockdesigns, and univariate methods such as averaging and subtraction. Although these methods are associated with relatively good
statistical power, they have limited our understanding of the brain
computations that underlie prosopagnosia. To gain a more comprehensive understanding, researchers have begun investigating
brain processing using condition-rich designs (Allen et al. 2022;
Charest et al. 2014a; Kriegeskorte and Kievit 2013; Naselaris et al.
2021). These approaches emphasize examining diverse models on
a whole-brain basis, thus providing a broader description of theunderlying brain mechanisms (Dwivedi et al. 2021; Kriegeskorte
and Diedrichsen, 2019; Popham et al. 2021).
Here, we adopted a data-driven approach. We recorded thebrain activity of patient PS and neurotypical controls in response

Faghel-Soubeyrand et al. |Behavioral tasks

Cambridge face memory test +All participants were administered the CFMT long-form, or
CFMT+ (Russell et al. 2009). In the CFMT+, participants are
required to memorize a series of face identities, and to subsequently identify the newly learned faces among three faces.
It includes a total of 102 trials of increasing difficulty. The
duration of this test is about 15 min. EEG was not recorded while
participants completed this test.

One-back task
The stimuli used in the main experiment consisted of 49 imagesof faces, animals (e.g. giraffe, monkey, puppy), plants, objects(e.g. car, computer monitor, f lower, banana), and scenes (e.g. city
landscape, kitchen, bedroom). The 24 faces (eight identities (fourfemales) and three expressions: neutral, happy, and fearful) weretaken from the Radboud Face dataset (Langner et al. 2010). For further details on stimulus processing steps, see (Faghel-Soubeyrand
et al. 2024). In total, ∼8 exemplars of each category (e.g. eightanimals, eight objects, nine scenes, eight joyful/fearful/neutral
faces) were presented to the participants during the EEG task.
These stimuli were presented during a one-back task where
we measured high-density electroencephalographic (EEG) activity(Fig. 1b,c). Participants performed ∼3200 trials in two recording
sessions, which were separated by at least 1 day and by a maximum of 2 weeks. Participants were asked to press a computer
keyboard key only on trials where the image was identical to theprevious one (repetitions occurred with a 0.1 probability). They
were asked to respond as quickly and accurately as possible. A
trial unraveled as follows: a white fixation dot was presented ona gray background for 500 ms (with a jitter of ±50 ms); followed
by a stimulus presented on a gray background for 600 ms; and,
finally, by a white fixation dot on a gray background for 500 ms.
Participants had a maximum of 1100 ms following stimulus onset
to respond.

Electroencephalography recording andpreprocessing
High-density electroencephalographic data were continuously
recorded at a sampling rate of 1024 Hz using a 128-channel
BioSemi ActiveTwo headset (Biosemi B.V., Amsterdam, Netherlands). Electrodes’ impedance was kept below 20 μV. Data werecollected at the University of Fribourg. Data were preprocessedusing FieldTrip (Oostenveld et al. 2011) and in-house MATLAB
code: continuous raw signal was first re-referenced relative to A1(Cz), filtered with a band-pass filter [.01–80 Hz], segmented into
trial epochs from −200 ms to 1100 ms relative to image onset, anddown-sampled at 256 Hz. These EEG recordings were completed
during the one-back task only.

Representational similarity analysis
Brain representational dissimilarity matrices
For every participant, we trained a Fisher linear discriminant(5-fold cross-validation, five repetitions; Treder 2020) to distinguish pairs of stimuli from every 4 ms intervals of EEG responseto these stimuli from −200 to 800 ms after stimulus onset (Cichy
and Oliva 2020; Graumann et al. 2022). All 128 channels served
as features in these classifiers. Cross-validated area under the
curve (AUC) served as pairwise classification dissimilarity metric.
By repeating this process for all possible pairs (1176 for our
49 stimuli), we obtained a representational dissimilarity matrix(RDM; Fig. 1c. see also Supplementary Fig. 1b).

Time generalization
We systematically characterized the differences in representational trajectories between PS and controls using a proceduresimilar to temporal generalization (King and Dehaene 2014).
Specifically, we cross-correlated the RDMs across all time points
after image onsets, creating a time × time temporal generalization
matrix (TGM), indicating how similar is an RDM at a specific time
window to other RDMs at other time windows, and compared thisTGM between PS and controls using Howell-Crawford t-tests (seeFig. 1d, and Fig. 2).

Brain and deep neural networks comparisons
We compared our participants’ brain representations to thosefrom visual and caption deep neural networks (DNN) using RSA(Charest et al. 2014; Kriegeskorte, Mur, and Bandettini, 2008;

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

imagery). She even resumed working as a kindergarten teacher
only 2 years after her traumatic accident. Yet, her fine-grained
visual discrimination abilities remained slower compared to controls, and she also presented reduced contrast sensitivity to highspatial frequency information (>22 cycles/degree) and a profound prosopagnosia with massively impaired face recognition
abilities (Rossion et al. 2003). The patient complains of a severe
difficulty at recognizing faces, including the ones of her kindergarten children, close relatives (husband, children, friends), as
well as her own face. PS can correctly categorize (and draw)
faces as a unique visual object and discriminate faces fromother non-face objects or scenes, even when the images arebrief ly presented (Schiltz et al. 2006). She shows no difficultyat object recognition, even for subordinate-level discriminations(Rossion et al. 2003; Schiltz et al. 2006). Patient PS is perfectat all tests from the Birmingham Object Recognition Battery(BORB—(Riddoch and Humphreys 2022) showing preserved processing of low-level aspects of visual information (i.e. matching ofbasic elementary features), intact object matching from different
viewpoints, and normal performance for object naming (Rossion
et al. 2003; Table 1). Her reading abilities are also well preserved
although slightly slowed down, her visual acuity (0.8 bilaterally) is
within the normal range, and her visual field almost intact apartfrom a small left paracentral scotoma. As reported by Rossion
et al. (2003), she is highly impaired on the Benton Face Matching
Test (BFRT—(Benton and Van Allen 1972) scoring 27/54 (percentile1). She performs also poorly on the Warrington Recognition Memory Test (WRMT—(Warrington and Shallice 1984), scoring 18/25(percentile 3) a performance that characterizes her as impaired
compared to controls. Over the years, patient PS developed strategies to infer a person’s identity by relying on external cues suchas haircut, clothes, beard, glasses, gait, posture, or a person’s
voice. Moreover, as revealed by the Bubbles response classification
technique, patient PS uses suboptimal diagnostic information torecognize familiar faces, relying on the lower part of the face(i.e. the mouth region and external contours) instead of the mostinformative eye area (Caldara et al. 2005). A similar bias towards
the mouth has been observed for the recognition of static facial
expressions (Fiset et al. 2017) for which she is strongly impaired.
Her ability to recognize the dynamic versions of the same facial
expressions is nevertheless preserved (Richoz et al. 2015). Overall,
PS is a very cooperative patient with extraordinarily preserved
cognitive functions, sensory and motor skills and without any
attentional deficits. She therefore represents an exemplary caseto investigate the functional models of typical face processing.

3

4

|

Cerebral Cortex, 2024, Vol. 34, No. 5

Kriegeskorte, Mur, Ruff, et al. 2008; Kriegeskorte and Kievit,
8/2013).
Visual convolutional neural networks RDMs. We used AlexNet(Krizhevsky et al. 2012) trained on ecoset, an ecologically valid
image training dataset with faces, objects, etc. (Mehrer et al. 2021)
one model of the visual computations along the ventral stream(Mehrer et al. 2021). We also used two other visual DNNs: AlexNet
trained on ImageNet (Krizhevsky et al. 2012) and VGG-Face, amodel specialized for faces (Parkhi et al. 2015). In all cases, our
49 stimuli were input to the DNN, and layer-wise RDMs wereconstructed comparing the unit activation patterns for each pairof images using Pearson correlations. These DNN process visualfeatures of gradually higher complexity and abstraction along
their layers (Güçlü and van Gerven 2015), from low-level (i.e.

orientation, edges in shallow layers) to high-level features (e.g.
objects and object parts in deeper layers).
Caption-level semantic RDM. We also used the caption-level
semantic model derived by (Faghel-Soubeyrand et al. 2024). They
asked five participants to provide a sentence caption describing
each stimulus (e.g. “a city seen from the other side of the forest”),
including those of faces (e.g. “a neutral female face”, “an unsure
man’s face”), using the Meadows online platform (www.meadowsresearch.com). The sentence captions were fed as inputs in
Google’s universal sentence encoder (GUSE; (Cer et al. 2018)
resulting in a sentence embedding with 512 dimensions for eachof our 49 stimuli. GUSE was trained to predict semantic textual
similarity from human judgments, and its embeddings generalize
to an array of other semantic judgment tasks (Cer et al. 2018).

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

Fig. 1. Overview of the experiment. a) the histogram shows the Cambridge face memory test long-form (CFMT+, (Russell et al. 2009) scores for PS, ourtypical recognizers (dark gray bars), and an additional 332 neurotypical observers from three independent studies for comparison (Faghel-Soubeyrand
et al. 2019; Tardif et al. 2019; Fysh et al. 2020). b) Participants were engaged in a one-back task while their brain activity was recorded with highdensity electroencephalography. The stimuli included objects from various categories, such as faces, objects, and scenes. Note that the face drawings
shown here are anonymized representations used as substitutes for the actual face stimuli presented to our participants. c) Representational similarity
analyses consisted in constructing brain representational dissimilarity matrices (RDMs) by comparing representational patterns (as characterized by
EEG topographies) for all pairwise comparisons of stimuli, independently for each time-point and participants. Specifically, RDMs were constructed using
cross-validated decoding performance between the EEG topographies at 4 ms intervals, providing a dynamic account of representational geometries
unfolding after stimulus onset. d) To evaluate the temporal evolution of brain representations, a temporal generalization matrix was computed for eachparticipant. This involved calculating all pairwise correlations between a participant’s time-resolved brain RDMs. A specific time-resolved brain RDM isconsidered to “generalize” to later time-resolved brain RDMs when it exhibits a positive correlation with them.

Faghel-Soubeyrand et al. |5

Faghel-Soubeyrand et al. (2024) then computed the dissimilarities(cosine distances) between the sentence embeddings acrossall pairs of captions, resulting in a caption-level semantic
RDM for each of their participants. The GUSE embeddings
forming the basis for the RDMs had good levels of agreement

between the five captioners (r range across participants = 0.65–.71; mean r = 0.6823), and in turn, the resulting RDMs showed
an average RDM spearman correlation between participantsof r = 0.7116. Therefore, we used the average RDM for ouranalyses.

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

Fig. 2. Temporal generalization of EEG representations across time in PS and controls. a) Temporal generalization over all pairwise stimulus comparisons.
To assess the temporal evolution of brain representations, we computed a temporal generalization matrix for each participant. This process involved
calculating pairwise correlations between time-resolved brain representational dissimilarity matrices (RDMs). The leftmost column displays the mean
temporal generalization matrix of control participants. The yellowish square in the upper right section of the matrix indicates temporal generalization
within the N170 time window. The central column illustrates the temporal generalization matrix of PS, which resembles that of the controls but isassociated with earlier brain RDMs. This is most evident in the rightmost column, representing the difference between PS’s and controls’ temporal
generalization matrices. Statistically significant regions in this contrast matrix are outlined in black (P < 0.05, uncorrected), with only positive differences
reaching the threshold. b) Similar to a), temporal generalization matrices were computed, but this time specifically for a subset of time-resolved brain
RDMs comparing pairs of face stimuli. c) Similar to a), temporal generalization matrices were computed, but this time specifically for a subset of timeresolved brain RDMs comparing pairs of face and nonface stimuli. d) Similar to a), temporal generalization matrices were computed, but this time
specifically for a subset of time-resolved brain RDMs comparing pairs of nonface stimuli.

6|

Cerebral Cortex, 2024, Vol. 34, No. 5

Representational dissimilarity matrices comparisonsGroup comparison and inferential statistics

We compared our participants’ brain RDMs to those from thevision (Fig. 3a, see also Supplementary Fig. 2) and caption-level
semantic description (Fig. 3b) models described in the previous section using partial Spearman correlations (also see
Supplementary Fig. 3 for comparisons with additional visualmodels). We accounted for the correlation between semantic
and categorical models by partialling out the correlation
with the last layer of AlexNet, and vice-versa. Specifically,
for the brain-AlexNet correlations, each layer-brain RDM wascorrelated using partial Spearman correlation, partialling out
the semantic RDM variance. For the brain-semantic model
correlations, each semantic-brain RDM correlation was alsodone using partial Spearman correlation, partialling out thelast layer of AlexNet RDM. Simple Spearman correlations(unconstrained on a third variable) were also computed andshown in supplementary materials (Supplementary Figs. 4
and 5).

All contrasts between PS and neurotypical controls werecomputed using Crawford-Howell modified t-tests for casecontrols comparisons (Crawford and Howell 1998; Crawford andGarthwaite 2012). Non-parametric tests in single case studies
have been demonstrated to be less reliable considering that
there is no equivalent for non-parametric statistics that are
reliable (Crawford et al. 2006). All time-resolved contrasts werecomputed from 0 to 800 ms after image-onset. To assess statisticalsignificance in cases where we do not compare PS and controls weused nonparametric permutation tests (e.g. Nichols and Holmes
2002; Li et al. 2022).

Results
One-back task
Accuracies did not differ between age-matched and young controls subgroups either for face stimuli (t(16) = −0.3099, P = 0.761;

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

Fig. 3. Comparison of brain representations with those of artificial neural networks of visual and semantic processing. a) Partial spearman correlation
between brain RDMs and ecoset-trained AlexNet RDMs (removing shared correlation between brain and semantic model) is shown for PS (pink curve)and controls (gray curve). Each column shows different layer RDMs in ascending order from left to right. See Supplementary Fig. 2 for the same analysis
on all the deep neural networks (DNN) layers. We found lower similarity of visual computations within the brain of PS compared to controls in layers 6
and 7 (black dots indicate significant contrasts in favor of controls, Howell-Crawford modified t-tests, P < 0.05; uncorrected), with differences peaking
in higher-level DNN layer 7. We observed the opposite in layers 1 to 3 and, to a lesser extent in layer 5, at early time points (red dots indicate significant
contrasts in favor of PS, Howell-Crawford modified t-tests, P < 0.05; uncorrected). Similar results were observed when comparing brains and DNN models
without removing the shared information between brains and the semantic (caption-level) model (Supplementary Fig. 4). See Supplementary Fig. 3aand b for partial Spearman correlations for AlexNet trained on ImageNet and of VGGFace, respectively. b) Partial Spearman correlation with RDMs of
the semantic model (excluding shared information between brain and AlexNet) was significantly lower in the brain of PS compared to controls (cyancurve; black dots indicate significant contrasts, P < 0.05; uncorrected). Similar results were observed when comparing brains and DNN models without
removing the shared information between brains and the semantic (caption-level) model (see Supplementary Fig. 5). The shaded areas of all curves
represent the standard error for the controls.

Faghel-Soubeyrand et al. |t(16) = −0.9607, P = 0.3510) or non-face stimuli (t(16) = 1.2925,
P = 0.215; t(16) = −.09704, P = 0.3463). Therefore, their data were
aggregated into a single neurotypical control group.

Face-specific behavioral index

Cambridge face memory test long-form
Within the control participants, CFMT+ scores did not differ
between aged-matched and non–aged-matched control groups(t(16) = −0.8058, P = 0.4322). Their data were aggregated into asingle control group. PS significantly differed from controls on
this standard face identification ability score (t(17) = −2.7623,
P = 0.0133).

Stability of neural code across time
A predominant assumption in cognitive neuroscience is that temporally early brain signal refers to low-level computations whilelater brain signal refers to higher-level computations (e.g. DiCarloet al. 2012; Wiese et al. 2019). To investigate whether PS presents
abnormal profiles of computations related to the lesioned corticalsites, we performed a variant of temporal generalization analyses(King and Dehaene 2014) using cross-correlated EEG RDMs acrosstime. Specifically, correlation of EEG-RDMs across all time points
after image onsets creates a (symmetric) time × time TGM. Thissymmetric temporal generalization matrix (TGM) indicates how
representational geometries elicited at different time-points are
similar (see Fig. 1d). We compared this TGM between PS andcontrols using Howell-Crawford t-tests (see Fig. 2). If the lesioned
cortical sites perform critical computations on brain signals fed
forward from early visual areas, we should observe excessive
generalization of the representational geometries encoded inearly time-points after stimulus onset for PS. Indeed, compared
to controls, PS showed significantly higher correlation between
her early representations (i.e. ∼80-100 ms, i.e. around P100
(Luck et al. 1990) and late representations from around 230 ms toaround 900 ms after image onset, as shown by the long trail ofsignificant contrasts (Fig. 2a; black outline, P < 0.05; uncorrected).
A similar and larger cluster was found between PS’s mid-latency
representations around 300 ms, which generalized more thancontrols to late representations from ∼400 ms until around
900 ms. Overall, thus, these results indicate that PS has morestable/less dynamic neural representations, with late EEG activity
ref lecting more similar representations to early EEG activity
compared to neurotypical controls. We found essentially the same
pattern of results when the TGM was computed on the face vs.
face stimuli condition (Fig. 2b) and on face vs. non-face stimuli
condition (Fig. 2c) but not on the non-face vs. non-face stimuli
condition (Fig. 2d).

Altogether our temporal generalization results indicate that
PS shows relatively less transformations in neural computations
from early to late stages of her visual processing stream, especially for face stimuli (Cichy and Oliva 2020).

Similarity with visual and semantic
computational models
Our results suggest that the lesions in specific high-level corticalsites in PS contribute to an excessive temporal generalization of
the face representations encoded in early time-points. Late stage
processing must rely on inputs fed forward, and if those inputs
come from early low-level processes, the high-level abstract representation relies on features that are weaker given the task at hand.
But temporal generalization results do not explicitly reveal whichkinds of computations may be impaired as a result of critical
cortical sites being lesioned. To better understand the differences in computational trajectories between PS and controls, we
compared brain RDMs to state-of-the-art computational modelsof vision and caption-level semantics. We assessed visual brain
computations in PS and neurotypical controls by comparing theirbrain RDMs to those of the AlexNet-ecoset deep neural networks(DNNs).
The time courses of the partial Spearman correlation between
brain RDMs and DNN layers 3, 5, and 7 (removing sharedcorrelation between brain and semantic model) for PS andcontrols are shown in Fig. 3a (the partial Spearman correlations
for all eight layers are shown in Supplementary Figs. 2 and4, including individual control traces). Direct contrasts of PS’correlation time courses with those of controls indicated reduced
similarity with the RDMs of the visual DNN’s final layers (P < 0.05;
uncorrected; layer 5: ∼130–150 ms, layer 6: ∼310–400 and 700–
800 ms, layer 7: ∼300–500 and 700–800 ms). These significant
contrasts were present relatively late after image onset, andpeaked at layer 7, which represent a higher proportion ofhigh-level visual features (e.g. whole objects and object parts,(Güçlü and van Gerven 2015; Long et al. 2018). Interestingly,
however, while similarity to these late visual representations
were reduced in PS, similarity to earlier visual representationsof the DNN (layers 1–5) were increased earlier on in this patient(see Supplementary Fig. 2, first two panels, P < 0.05; uncorrected;
layer 1: 72-100 ms, layers 2–4: 72–84 ms, layer 5: 72–80 ms). This
indicates that PS shows reduced similarity with high-level visual
representations late after image onset and increased similarity
with early visual representations early after image onset. We
further tested the similarity to visual DNN representations ofPS and controls by comparing them with similar convolutional
networks trained either on faces-only (VGGface; (Parkhi et al.
2015); see Supplementary Fig. 3a) or objects-only (imagenettrained AlexNet; (Krizhevsky et al. 2012); see Supplementary
Fig. 3b). Overall this confirmed the impaired representational
similarity to visual DNNs in the brain of PS in late stages of visual
processing. However, the better alignment observed for PS early
after image onset with early layers of ANNs was not observedin VGGface nor in AlexNet-imagenet. This discrepancy could
potentially be explained by the images used in training these ANN
models. VGGFace was trained with thousands of face identities in∼1,000,000 images. AlexNet-imagenet was trained with imagenet(Deng et al. 2009) which has been criticized for its lack of realworld distribution of objects and categories (Beyer et al. 2020). The
AlexNet-ecoset model was trained on a well-controlled diversified
set of images aiming to improve the model’s ecological validity(Mehrer et al. 2021).

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

To assess the face-specific performance of PS and controls in asingle individual score, we combined performance in the oneback task accuracies and response times of face and non-facetrials using Principal Component Analysis (PCA). Specifically,
face-specific performance in the one-back tasks was computed
as a face vs. non-face performance contrast score ([face—nonface]/[face + non-face]) separately for accuracy and RTs, foreach participant. We used PCA to extract projections explaining
variance across these two variables (Calder et al. 2001; Calderand Young 2005). The first component, which explained 83.86%of the variance in performance across participants, is henceforthreferred to as the face-specific performance score. PS significantly
differed from neurotypical controls on this score (t(17) = −7.1571,
P = 1.6053e-06; see Supplementary Fig. 1a), indicating typical facespecific behavioral deficits in this patient.

7

8

|

Cerebral Cortex, 2024, Vol. 34, No. 5

Discussion
Finding brain correlates for the face individuation deficits seen inprosopagnosia has proven to be quite challenging, as highlighted
in previous research (Anzellotti et al. 2014; Alonso Prieto et al.
2011; though there are exceptions, as seen in Liu-Shuang et al.
2016). In this study, we addressed this issue embracing a datadriven approach. We conducted high-density EEG recordings bothon an individual with prosopagnosia (PS) and neurotypical controls while presenting various images from different categories.
We employed RSA (Kriegeskorte, Mur, and Bandettini, 2008) toassess how the time-resolved representations of the visual stimuli
relate to one another in the brain of PS and in the brains ofcontrols, as well as how they relate to the representations ofcomputational models. Our findings revealed that the temporal

evolution of visual representations in PS’s brain follows an abnormal trajectory. Additionally, by comparing PS’s brain representations with those of computational models of vision and semantics,
we gained insights into the nature of her computational deficits.
To investigate the changes in PS’s brain representations overtime, we conducted a variant of temporal generalization analyses
developed by King and Dehaene (2014). This involved correlating
the time-resolved brain Representational Dissimilarity Matrices(RDMs) for an individual with all the RDMs of that same individual. In the case of neurotypical participants, we observed the
expected temporal generalization of the brain RDMs within the
N170 temporal window. This indicates that the brain’s visual
representation at this point in time is relatively stable as time
progresses. In other words, additional computations after 170 ms
do not significantly alter the brain representations of control
participants. However, in the case of PS, we noted atypical generalization patterns. Specifically, the brain representations aroundthe P100 time window were abnormally similar to later representations, suggesting an excessive generalization of early visual
representations. This overgeneralization was primarily associatedwith the similarity between brain activity in trials involving a faceand a nonface stimulus and, to a lesser extent, between two faces,
but not between two non-face stimuli. This implies that the key
differences lie in how the brain represents faces between 80 and
100 ms in PS, possibly due to critical computations in corticalsites that affect the feedforward signals from early visual areas
responsible in the typical brain for achieving a stable N170 face
representation.
To investigate this possibility, we conducted a comparative
analysis of the RDMs derived from PS’s brain with those generated
by a deep neural network (DNN) trained to distinguish objects
and faces. We assumed that this DNN model represents a nearly
optimal sequence of visual stimuli representations for object
recognition. Our findings revealed that the neural computations
underlying PS’s brain activity in the early layers exhibited a closeralignment with the model of vision than those of neurotypicalindividuals. This suggests that PS’s early brain representations
are, indeed, better building blocks than those of control participants for recognizing faces and nonface visual stimuli. As wedelved into the middle layers of the DNN, both PS and neurotypicals showed a relatively similar alignment, and in the later layers, the brain representations in neurotypical individuals became
more akin to those of the model, that is, more efficient.
It is worth noting that late layers in visual DNNs have previously been associated with processing in the human inferotemporal cortex (hIT; Güçlü and van Gerven 2015; Khaligh-Razavi
and Kriegeskorte 2014; Jiahui et al. 2023), with a peak in the FFA(Khaligh-Razavi and Kriegeskorte 2014). These layers are functionally connected to higher-level visual feature representations,encompassing various aspects of objects, including parts, whole
objects, and viewpoint-invariant features. These findings are thus
in agreement with the previously documented impairments inboth whole face (Ramon et al. 2016) and feature representations(Caldara et al. 2005; Ramon et al. 2016; Fiset et al. 2017) in patient
PS. One interesting observation from the model to brain comparisons performed here is that the different model layers show
similar onsets of peak correlations. This is an observation that has
previously been documented (Cichy et al. 2016). Neural traces, asassessed through EEG, can exhibit diverse components of feedforward and feedback signals. In contrast, deep convolutional neuralnetworks, such as those employed in our study, are intentionallydesigned to be hierarchical, featuring progressively abstracted
information from pixel inputs as we move deeper into the network

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

To reveal whether semantic computations (Barton et al. 2009;
Schweinberger and Neumann 2016) could be affected in thebrain of PS, we used a deep averaging network (Google Universal
Sentence Encoder, GUSE (Cer et al. 2018)) to transform humanderived captions of our stimuli (e.g. “a city seen from the other sideof the forest”) into semantic embeddings (points in a semantic
space; for more details see Faghel-Soubeyrand et al. 2024). We
computed partial Spearman correlations between the RDMs
derived from this semantic model and the brain RDMs (excluding
shared correlation between brain and the visual model) of PS andcontrol participants. Direct contrasts revealed reduced partial
Spearman correlations with these semantic computations in thebrain of PS compared to controls (Fig. 3b, P < 0.05; uncorrected).
This reduced similarity with semantic representations appeared
as early as ∼90 ms after image onset, with effects appearing
until as late as around 400 ms. Note that similar results wereobserved when comparing brains and the semantic model
without removing this shared information between brain andvisual model (see Supplementary Figs. 4 and 5).
To better understand the features encoded in the different layers of the visual and semantic models, we compared the models
with models constructed around the categorical features of our
stimulus set. We constructed one-hot and multihot categoricalmodels by encoding several dimensions as binary encoding vectors (including face gender, face emotion, face identity, animacy,
object categories, etc.), and measuring the distance between theseencoding vectors across all image pairs. The multihot RDM peakedin similarity with the semantic model. Correlation of categoricalmodels with DNN RDMs, on the other hand, showed overall lower
correlations with finer-grained categorical distinctions (e.g. theone-hot model encoding face gender information) and stronger
dissociations between non-face stimuli along its latest layers.
The correlation between the GUSE RDMs and relevant categoricalmodels can be found in Supplementary Fig. 6. These correlations,as well as those with additional behavioral data on high-level
human judgments (n = 32; see Supplementary Fig. 7), indicatedthat the semantic model contains rich information about animacy, face vs. non-face distinction, function and meaning ofsentence description of images, as well as more fine-grained
face-specific information about gender, emotion, and meaning at
the level of sentence description. In other words, the semantic
model encapsulates rich and diversified high-level conceptual
information about images, including information about faces.
Overall, thus, contrasts with computational models showedthat PS’s brain processing stream exhibits impairments peaking inhigher-level visual (DNN layers 6–7) and semantic (caption-level)
representations.

Faghel-Soubeyrand et al. |Acknowledgments
We sincerely thank PS for her precious contribution and participation in this study.

Author contributions
Simon Faghel-Soubeyrand (Conceptualization, Methodology,
Software, Formal Analysis, Investigation, Data curation, Writing—original draft, Visualization, Supervision, Project administration,

Funding acquisition), Anne-Raphaelle Richoz (Project administration, Investigation, Writing—review and editing), Jessica Woodhams (Funding acquisition), Delphine Waeber (Investigation),
Roberto Caldara (Resources, Project administration, Writing—review and editing), Frédéric Gosselin (supervision, Writing—review and editing), Ian Charest (Supervision, Methodology,
Resources, Formal Analysis, Writing—review and editing, Project
administration, Funding acquisition).

Supplementary material
Supplementary material is available at Cerebral Cortex online.

Funding
Funding for this project was supported by an ERC Starting Grant[ERC-StG-759432] to I.C, an ERSC-IAA grant to J.W., I.C. and S.F.S.,
by a Swiss National Science Foundation grant (10001C_201145) to
A.-R.R. and R.C., and by a NSERC and IVADO graduate scholarships
to S.F.S.
Conf lict of interest statement: None declared.

Competing interests
The authors declare no competing interests.

Data availability
Data are available from the corresponding authors upon request.

Code availability
The MATLAB and Python codes used in this study will be available
upon request.

References
Allen EJ, St-Yves G, Wu Y, Breedlove JL, Prince JS, Dowdle LT,
Nau M, Caron B, Pestilli F, Charest I, et al. A massive 7T fMRI
dataset to bridge cognitive neuroscience and artificial intelligence. Nat Neurosci. 2022:25(1):116–126. https://doi.org/10.1038/
s41593-021-00962-x.
Alonso Prieto E, Caharel S, Henson R, Rossion B. Early (N170/M170)
face-sensitivity despite right lateral occipital brain damage inacquired prosopagnosia. Front Hum Neurosci. 2011:5:138. https://doi.org/10.3389/fnhum.2011.00138.
Anzellotti S, Fairhall SL, Caramazza A. Decoding representationsof face identity that are tolerant to rotation. Cereb Cortex.
2014:24(8):1988–1995. https://doi.org/10.1093/cercor/bht046.
Avidan G, Hasson U, Malach R, Behrmann M. Detailed explorationof face-related processing in congenital prosopagnosia: 2. Functional neuroimaging findings. In. J Cogn Neurosci. 2005:17(7):1150–1167. https://doi.org/10.1162/0898929054475145.
Avidan G, Tanzer M, Hadj-Bouziane F, Liu N, Ungerleider LG,
Behrmann M. Selective dissociation between core and extended
regions of the face processing network in congenital prosopagnosia. Cereb Cortex. 2014:24(6):1565–1578. https://doi.org/10.1093/
cercor/bht007.
Barton JJS, Hanif H, Ashraf S. Relating visual to verbal semantic
knowledge: the evaluation of object recognition in prosopagnosia.
Brain J Neurol. 2009:132(12):3456–3466. https://doi.org/10.1093/
brain/awp252.

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

layers. Therefore, the absence of a strictly monotonic relationshipbetween peak onset and model layer is to be expected.
Associations between brain activity and higher-level semantic
computations have only recently gained attention in the field(Doerig et al. 2022; Dwivedi et al. 2021; Faghel-Soubeyrand et al.
2024; Popham et al. 2021). Here, we demonstrated that the
representational geometry of PS exhibits a substantial, albeit
significantly reduced, correlation with those of a model ofcaption-level semantics (Cer et al. 2018; Doerig et al. 2022; Frisby
et al. 2023; Faghel-Soubeyrand et al. 2024) when compared tocontrols. Further analyses suggest that this semantic model
encapsulates a wealth of information about various aspectsof face categories, including gender and emotion, differentiation between faces and non-face stimuli, as well as thefunctional aspects of objects, faces, and scenes. Our findings
unequivocally illustrate a strong connection between thesesemantic brain computations and significant alterations in theability to recognize faces, underscoring the importance of thesecomputations in the context of face recognition (Bruce andYoung, 1986; Duchaine and Yovel 2015; Faghel-Soubeyrand et al.
2024).
In conjunction with similar computational characterizationsof brain representations in individuals with “super-recognition”
of faces (Faghel-Soubeyrand et al. 2024), our findings suggest agradient of neural computations spanning from the lower end
to the higher end of face-recognition abilities. For instance, inthe study by Faghel-Soubeyrand et al. (2024), super-recognizers
exhibited increased similarity with mid-level visual and semantic
computations around the N170 and P600 time windows, respectively. In contrast, our results demonstrate that PS shows reduced
similarity with visual and semantic computations. Specifically,
PS’s neural computations, especially in the realm of semantic
processing, appear to be affected at a much earlier stage and toa greater extent than those of super-recognizers. While semantic
brain computations were enhanced around the P600 time window
in super-recognizers, PS exhibited reduced similarity as early as
the P100, persisting throughout the N170 and N400 time windows. These observations indicate that PS’s deficits in neuralcomputations commence relatively early along the conventional
processing pathway.
To the best of our knowledge, this study presents the first
analysis of the fine-grained temporal progression of brain
representations in prosopagnosia, alongside a state-of-the-art
computational characterization of these representations. This
comprehension of impaired perceptual representations not only
paves the way for novel approaches to patient rehabilitation
but also holds promise in uncovering and potentially diagnosing
subtle deficits in perception and cognition across diverse clinicalpopulations. These advancements have been facilitated by recent
technological progress, which has significantly contributed to thefindings of this study.

9

10

|

Cerebral Cortex, 2024, Vol. 34, No. 5
prosopagnosia. Brain J Neurol. 2019:142(12):3975–3990. https://doi.
org/10.1093/brain/awz332.
Crawford JR, Garthwaite PH. Single-case research in neuropsychology: a comparison of five forms of t-test for comparing a case tocontrols. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2012:48(8):1009–1016. https://doi.org/10.1016/j.
cortex.2011.06.021.
Crawford JR, Howell DC. Comparing an Individual’s test score against
norms derived from small samples. Clin Neuropsychol. 1998:12(4):482–486. https://doi.org/10.1076/clin.12.4.482.7241.
Crawford JR, Garthwaite PH, Azzalini A, Howell DC, Laws KR. Testing for a deficit in single-case studies: effects of departuresfrom normality. Neuropsychologia. 2006:44(4):666–677. https://doi.
org/10.1016/j.neuropsychologia.2005.06.001.
Dalrymple KA, Oruç I, Duchaine B, Pancaroglu R, Fox CJ, Iaria
G, Handy TC, Barton JJS. The anatomic basis of the rightface-selective N170 IN acquired prosopagnosia: a combined
ERP/fMRI study. Neuropsychologia. 2011:49(9):2553–2563. https://doi.org/10.1016/j.neuropsychologia.2011.05.003.
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. ImageNet: alarge-scale hierarchical image database. 2009 IEEE Conference on
Computer Vision and Pattern Recognition. 2009:248–255. https://doi.
org/10.1109/CVPR.2009.5206848.
di Oleggio Castello MV, Haxby JV, Ida GM. Shared neural codes forvisual and semantic information about familiar faces in a common representational space. Proc Natl Acad Sci USA. 2021:118(45).
https://doi.org/10.1073/pnas.2110474118.
DiCarlo JJ, Zoccolan D, Rust NC. How does the brain solve
visual object recognition? Neuron. 2012:73(3):415–434. https://doi.
org/10.1016/j.neuron.2012.01.010.
Doerig A, Kietzmann TC, Allen E, Wu Y, Naselaris T, Kay K,
Charest I. Semantic scene descriptions as an objective of humanvision. arXiv [csCV] arXiv. 2022. https://doi.org/http://arxiv.org/
abs/2209.11737.
Dricot L, Sorger B, Schiltz C, Goebel R, Rossion B. The roles of“face” and “non-face” areas during individual face perception:evidence by fMRI adaptation in a brain-damaged prosopagnosic
patient. NeuroImage. 2008:40(1):318–332. https://doi.org/10.1016/
j.neuroimage.2007.11.012.
Duchaine BC, Nakayama K. Developmental prosopagnosia: a window to content-specific face processing. Curr Opin Neurobiol.
2006:16(2):166–173. https://doi.org/10.1016/j.conb.2006.03.003.
Duchaine B, Yovel G. A revised neural framework for face processing. Annual Review of Vision Science. 2015:1:393–416. https://doi.
org/10.1146/annurev-vision-082114-035518.
Dwivedi K, Bonner MF, Cichy RM, Roig G. Unveiling functions of
the visual cortex using task-specific deep neural networks. PLoS
Comput Biol. 2021:17(8):e1009267. https://doi.org/10.1371/journal.
pcbi.1009267.
Eimer M, Gosling A, Duchaine B. Electrophysiological markers
of covert face recognition in developmental prosopagnosia.
Brain J Neurol. 2012:135(2):542–554. https://doi.org/10.1093/brain/
awr347.
Faghel-Soubeyrand S, Ramon M, Bamps E, Zoia M, Woodhams
J, Richoz A-R, Caldara R, Gosselin F, Charest I. The neuralcode behind face recognition abilities. bioRxiv. 2022:2022 March
19.484245. https://doi.org/10.1101/2022.03.19.484245.
Faghel-Soubeyrand S, Alink A, Bamps E, Gervais R-M, Gosselin F,
Charest I. The two-faces of recognition ability: better face recognizers extract different physical content from left and rightsides of face stimuli. J Vis. 2019:19(10):136d–136d. https://doi.
org/10.1167/19.10.136d.

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

Bentin S, Allison T, Puce A, Perez E, McCarthy G. Electrophysiological studies of face perception in humans. J Cogn Neurosci.
1996:8(6):551–565. https://doi.org/10.1162/jocn.1996.8.6.551.
Bentin S, Deouell LY. Structural encoding and identificationin face processing: erp evidence for separate mechanisms. Cogn
Neuropsychol. 2000:17(1):35–55. https://doi.
org/10.1080/026432900380472.
Benton AL, Van Allen MW. Prosopagnosia and facial discrimination.
J Neurol Sci. 1972:15(2):167–172. https://doi.org/10.1016/0022-510
x(72)90004-4.
Beyer L, Hénaff OJ, Kolesnikov A, Zhai X, van den Oord A. Are we donewith ImageNet? arXiv [csCV] arXiv. 2020. https://doi.org/10.48550/
arXiv.2006.07159.
Bobes MA, Lopera F, Garcia M, Díaz-Comas L, Galan L, Valdes-Sosa
M. Covert matching of unfamiliar faces in a case of prosopagnosia: an ERP study. Cortex; a Journal Devoted to the Study of
the Nervous System and Behavior. 2003:39(1):41–56. https://doi.
org/10.1016/s0010-9452(08)70073-x.
Bodamer J. Die Prosop-Agnosie. Arch Psychiatr Nervenkr. 1947:179(1–2):6–53. https://doi.org/10.1007/BF00352849.
Bruce V, Young A. Understanding face recognition. Br J Psychol.
1986:77(Pt 3):305–327. https://doi.org/10.1111/j.2044-8295.1986.
tb02199.x.
Busigny T, Graf M, Mayer E, Rossion B. Acquired prosopagnosia asa face-specific disorder: ruling out the general visual similarity account. Neuropsychologia. 2010:48(7):2051–2067. https://doi.
org/10.1016/j.neuropsychologia.2010.03.026.
Caldara R, Schyns P, Mayer E, Smith ML, Gosselin F, Rossion B. Does
prosopagnosia take the eyes out of face representations? Evidence for a defect in representing diagnostic facial information
following brain damage. J Cogn Neurosci. 2005:17(10):1652–1666.
https://doi.org/10.1162/089892905774597254.
Calder AJ, Young AW. Understanding the recognition of facial identity and facial expression. Nat Rev Neurosci. 2005:6(8):641–651.
https://doi.org/10.1038/nrn1724.
Calder AJ, Burton AM, Miller P, Young AW, Akamatsu S. A principal
component analysis of facial expressions. Vis Res. 2001:41(9):1179–1208. https://doi.org/10.1016/s0042-6989(01)00002-5.
Carlson TA, Tovar DA, Alink A, Kriegeskorte N. Representational
dynamics of object vision: the first 1000 ms. J Vis. 2013:13(10):1–1.
https://doi.org/10.1167/13.10.1.
Cer D, Yang Y, Kong S-Y, Hua N, Limtiaco N, St. John R, ConstantN, Guajardo-Cespedes M, Yuan S, Tar C, et al. Universal sentence
encoder. arXiv [csCL] arXiv. 2018. https://doi.org/http://arxiv.org/
abs/1803.11175.
Charest I, Kievit RA, Schmitz TW, Deca D, Kriegeskorte N. Uniquesemantic space in the brain of each beholder predicts perceived
similarity. Proc Natl Acad Sci USA. 2014:111(40):14565–14570.
https://doi.org/10.1073/pnas.1402594111.
Cichy RM, Oliva A. A M/EEG-fMRI fusion primer: resolving humanbrain responses in space and time. Neuron. 2020:107(5):772–781.
https://doi.org/10.1016/j.neuron.2020.07.001.
Cichy RM, Pantazis D, Oliva A. Resolving human object recognitionin space and time. Nat Neurosci. 2014:17(3):455–462. https://doi.
org/10.1038/nn.3635.
Cichy RM, Khosla A, Pantazis D, Torralba A, Oliva A. Comparison of deep neural networks to spatio-temporal cortical
dynamics of human visual object recognition reveals hierarchicalcorrespondence. Sci Rep. 2016:6(1):27755. https://doi.org/10.1038/
srep27755.
Cohen AL, Soussand L, Corrow SL, Martinaud O, Barton JJS, Fox
MD. Looking beyond the face area: lesion network mapping ofFaghel-Soubeyrand et al. |Kaltwasser L, Hildebrandt A, Recio G, Wilhelm O, Sommer W. Neurocognitive mechanisms of individual differences in face cognition: a replication and extension. Cogn Affect Behav Neurosci.
2014:14(2):861–878. https://doi.org/10.3758/s13415-013-0234-y.
Kanwisher N, McDermott J, Chun MM. The fusiform face area:a module in human extrastriate cortex specialized for faceperception. J Neurosci. 1997:17(11):4302–4311 https://www.
ncbi.nlm.nih.gov/pubmed/9151747.
https://doi.org/10.1523/
JNEUROSCI.17-11-04302.1997.
Khaligh-Razavi S-M, Kriegeskorte N. Deep supervised, but not
unsupervised, models may explain IT cortical representation.
PLoS Comput Biol. 2014:10(11):e1003915. https://doi.org/10.1371/
journal.pcbi.1003915.
King J-R, Dehaene S. Characterizing the dynamics of mental
representations: the temporal generalization method. Trends
Cogn Sci. 2014:18(4):203–210. https://doi.org/10.1016/j.tics.2014.
01.002.
Kriegeskorte N, Kievit RA. Representational geometry: integrating
cognition, computation, and the brain. Trends Cogn Sci. 2013:17(8):401–412. https://doi.org/10.1016/j.tics.2013.06.007.
Kriegeskorte N, Mur M, Bandettini P. Representational similarity analysis—connecting the branches of systems neuroscience. Front Syst Neurosci. 2008a:2(4). https://doi.org/10.3389/
neuro.06.004.2008.
Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification withdeep convolutional neural networks. In: Pereira F, Burges
CJC, Bottou L, Weinberger KQ, editors. Advances in neural
information processing systems 25. Lake Tahoe, Nevada, USA:
Curran Associates, Inc; 2012. pp. 1097–1105 http://papers.nips.cc/
paper/4824-imagenet-classification-with-deep-convolutionalneural-networks.pdf
Kriegeskorte N, Diedrichsen J. Inferring brain-computational mechanisms with models of activity measurements. Philos Trans R
Soc Lond Ser B Biol Sci. 2016:371(1705). https://doi.org/10.1098/
rstb.2016.0278.
Kriegeskorte N, Diedrichsen J. Peeling the onion of brain representations. Annu Rev Neurosci. 2019:42(1):407–432. https://doi.
org/10.1146/annurev-neuro-080317-061906.
Kriegeskorte N, Mur M, Bandettini P. Representational similarity
analysis - connecting the branches of systems neuroscience. Front
Syst Neurosci. 2008:2:4. https://doi.org/10.3389/neuro.06.004.2008.
Lamme VA, Roelfsema PR. The distinct modes of vision offeredby feedforward and recurrent processing. Trends Neurosci.
2000:23(11):571–579. https://doi.org/10.1016/s0166-2236(00)016
57-x.
Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, vanKnippenberg A. Presentation and validation of the Radboud
faces database. Cognit Emot. 2010:24(8):1377–1388. https://doi.
org/10.1080/02699930903485076.
Li Y, Zhang M, Liu S, Luo W. EEG decoding of multidimensional
information from emotional faces. NeuroImage. 2022:258:119374.
https://doi.org/10.1016/j.neuroimage.2022.119374.
Liu-Shuang J, Torfs K, Rossion B. An objective electrophysiologicalmarker of face individualisation impairment in acquired
prosopagnosia with fast periodic visual stimulation.
Neuropsychologia.
2016:83:100–113.
https://doi.org/10.1016/j.
neuropsychologia.2015.08.023.
Long B, Yu C-P, Konkle T. Mid-level visual features underlie the highlevel categorical organization of the ventral stream. Proc Natl
Acad Sci USA. 2018:115(38):E9015–E9024. https://doi.org/10.1073/
pnas.1719616115.
Luck SJ, Heinze HJ, Mangun GR, Hillyard SA. Visual event-related
potentials index focused attention within bilateral stimulus

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

Faghel-Soubeyrand S, Ramon M, Bamps E, Zoia M, Woodhams J,
Richoz A-R, Caldara R, Gosselin F, Charest I. Decoding face recognition abilities in the human brain. PNAS Nexus. 2024:3(3):gae095.
https://doi.org/10.1093/pnasnexus/pgae095.
Fiset D, Blais C, Royer J, Richoz A-R, Dugas G, Caldara R. Mapping
the impairment in decoding static facial expressions of emotion
in prosopagnosia. Soc Cogn Affect Neurosci. 2017:12(8):1334–1341.
https://doi.org/10.1093/scan/nsx068.
Frisby SL, Halai AD, Cox CR, Lambon Ralph MA, Rogers TT. Decoding
semantic representations in mind and brain. Trends Cogn Sci.
2023:27(3):258–281. https://doi.org/10.1016/j.tics.2022.12.006.
Fysh MC, Stacchi L, Ramon M. Differences between and within
individuals, and subprocesses of face cognition: implications
for theory, research and personnel selection. R Soc Open Sci.
2020:7(9):200233. https://doi.org/10.1098/rsos.200233.
Gao X, Vuong QC, Rossion B. The cortical face network of
the prosopagnosic patient PS with fast periodic stimulation in fMRI. Cortex. 2019:119:528–542. https://doi.org/10.1016/j.
cortex.2018.11.008.
Gauthier I, Tarr MJ, Moylan J, Skudlarski P, Gore JC, Anderson AW.
The fusiform “face area” is part of a network that processes faces
at the individual level. J Cogn Neurosci. 2000:12(3):495–504https://direct.mit.edu/jocn/article-abstract/12/3/495/3437. https://doi.
org/10.1162/089892900562165.
Golarai G, Liberman A, Grill-Spector K. Experience shapes the development of neural substrates of face processing in human ventral
temporal cortex. Cereb Cortex. 2015:27(2). https://doi.org/10.1093/
cercor/bhv314.
Gosling A, Eimer M. An event-related brain potential study of explicit
face recognition. Neuropsychologia. 2011:49(9):2736–2745. https://doi.org/10.1016/j.neuropsychologia.2011.05.025.
Graumann M, Ciuffi C, Dwivedi K, Roig G, Cichy RM. The spatiotemporal neural dynamics of object location representationsin the human brain. Nat Hum Behav. 2022:6(6):796–811. https://doi.org/10.1038/s41562-022-01302-0.
Güçlü U, van Gerven MAJ. Deep neural networks reveal a gradient in the complexity of neural representations across theventral stream. J Neurosci. 2015:35(27):10005–10014. https://doi.
org/10.1523/JNEUROSCI.5023-14.2015.
Haxby JV, Hoffman EA, Gobbini MI. The distributed human neuralsystem for face perception. Trends Cogn Sci. 2000:4(6):223–233.
https://doi.org/10.1016/s1364-6613(00)01482-0.
Hebart MN, Bankson BB, Harel A, Baker CI, Cichy RM. The representational dynamics of task and object processing in humans. elife.
2018:7. https://doi.org/10.7554/elife.32816.
Herzmann G, Schweinberger SR, Sommer W, Jentzsch I. What’s
special about personally familiar faces? A multimodal approach.
Psychophysiology. 2004:41(5):688–701. https://doi.org/10.1111/
j.1469-8986.2004.00196.x.
Humphreys K, Avidan G, Behrmann M. A detailed investigationof facial expression processing in congenital prosopagnosia as
compared to acquired prosopagnosia. Experimental Brain ResearchExperimentelle Hirnforschung Experimentation Cerebrale. 2007:176(2):356–373. https://doi.org/10.1007/s00221-006-0621-5.
Jiahui G, Yang H, Duchaine B. Developmental prosopagnosics
have widespread selectivity reductions across category-selective
visual cortex. Proc Natl Acad Sci USA. 2018:115(28):E6418–E6427.
https://doi.org/10.1073/pnas.1802246115.
Jiahui G, Feilong M, di Oleggio, Castello M, Nastase SA, Haxby
JV, Gobbini MI. Modeling naturalistic face processing inhumans with deep convolutional neural networks. Proc Natl
Acad Sci USA. 2023:120(43):e2304085120. https://doi.org/10.1073/
pnas.2304085120.

11

12

|

Cerebral Cortex, 2024, Vol. 34, No. 5
Rossion B. Understanding face perception by means of prosopagnosia and neuroimaging. Front Biosci. 2014:6(2):258–307. https://doi.org/10.2741/E706.
Rossion B. Damasio’s error—prosopagnosia with intact withincategory object recognition. J Neuropsychol. 2018:12(3):357–388.
https://doi.org/10.1111/jnp.12162.
Rossion B. Twenty years of investigation with the case of prosopagnosia PS to understand human face identity recognition.
Part I: function. Neuropsychologia. 2022a:173:108278. https://doi.
org/10.1016/j.neuropsychologia.2022.108278.
Rossion B. Twenty years of investigation with the case of prosopagnosia PS to understand human face identity recognition. Part
II: neural basis. Neuropsychologia. 2022b:173:108279. https://doi.
org/10.1016/j.neuropsychologia.2022.108279.
Rossion B, Caldara R, Seghier M, Schuller A, Lazeyras F, Mayer
E. A network of occipito-temporal face-sensitive areas besides
the right middle fusiform gyrus is necessary for normal faceprocessing. Brain J Neurol. 2003:126(11):2381–2395. https://doi.
org/10.1093/brain/awg241.
Russell R, Duchaine B, Nakayama K. Super-recognizers: people withextraordinary face recognition ability. Psychon Bull Rev. 2009:16(2):
252–257. https://doi.org/10.3758/PBR.16.2.252.
Schiltz C, Sorger B, Caldara R, Ahmed F, Mayer E, Goebel R, Rossion
B. Impaired face discrimination in acquired prosopagnosia isassociated with abnormal response to individual faces in theright middle fusiform gyrus. Cereb Cortex. 2006:16(4):574–586.
https://doi.org/10.1093/cercor/bhj005.
Schweinberger SR, Neumann MF. Repetition effects in human ERPs
to faces. Cortex; a Journal Devoted to the Study of the Nervous
System and Behavior. 2016:80:141–153. https://doi.org/10.1016/j.
cortex.2015.11.001.
Simon SR, Khateb A, Darque A, Lazeyras F, Mayer E, Pegna AJ.
When the brain remembers, but the patient doesn’t: converging fMRI and EEG evidence for covert recognition in acase of prosopagnosia. Cortex. 2011:47(7):825–838. https://doi.
org/10.1016/j.cortex.2010.07.009.
Sorger B, Goebel R, Schiltz C, Rossion B. Understanding the functional
neuroanatomy of acquired prosopagnosia. NeuroImage. 2007:
35(2):836–852. https://doi.org/10.1016/j.neuroimage.2006.09.051.
Tanaka JW, Curran T, Porterfield AL, Collins D. Activation of preexisting and acquired face representations: the N250 event-related
potential as an index of face familiarity. J Cogn Neurosci. 2006:
18(9):1488–1497. https://doi.org/10.1162/jocn.2006.18.9.1488.
Tardif J, Morin Duchesne X, Cohan S, Royer J, Blais C, Fiset
D, Duchaine B, Gosselin F. Use of face information varies
systematicallyfromdevelopmental
prosopagnosics
tosuper-recognizers. Psychol Sci. 2019:30(2):300–308. https://doi.
org/10.1177/0956797618811338.
Towler J, Eimer M. Electrophysiological studies of face processing
in developmental prosopagnosia: neuropsychological and neurodevelopmental perspectives. Cogn Neuropsychol. 2012:29(5–6):
503–529. https://doi.org/10.1080/02643294.2012.716757.
Treder MS. MVPA-light: a classification and regression toolbox for
multi-dimensional data. Front Neurosci. 2020:14:289. https://doi.
org/10.3389/fnins.2020.00289.
Warrington EK, Shallice T. Category specific semantic impairments.
Brain J Neurol. 1984:107(3):829–853. https://doi.org/10.1093/
brain/107.3.829.
White D, Mike Burton A. Individual differences and themultidimensional nature of face perception. Nature Reviews
Psychology. 2022:1(5):287–300. https://doi.org/10.1038/s44159022-00041-3.

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024

arrays. II. Functional dissociation of P1 and N1 components. Electroencephalogr Clin Neurophysiol. 1990:75(6):528–542. https://doi.
org/10.1016/0013-4694(90)90139-b.
McConachie HR. Developmental prosopagnosia. A single case report.
Cortex; a Journal Devoted to the Study of the Nervous System and
Behavior. 1976:12(1):76–82. https://doi.org/10.1016/s0010-9452(76)80033-0.
McDermott J, Schiller PH, Gallant JL. Spatial frequency and orientation tuning dynamics in area V1. 2002: Proceedings of the. https://www.pnas.org/content/99/3/1645.short.
Mehrer J, Spoerer CJ, Jones EC, Kriegeskorte N, Kietzmann TC. An
ecologically motivated image dataset for deep learning yields
better models of human vision. Proc Natl Acad Sci USA. 2021:118(8).
https://doi.org/10.1073/pnas.2011417118.
Naselaris T, Allen E, Kay K. Extensive sampling for complete models
of individual brains. Curr Opin Behav Sci. 2021:40:45–51. https://doi.org/10.1016/j.cobeha.2020.12.008.
Nichols TE, Holmes AP. Nonparametric permutation tests for functional neuroimaging: a primer with examples. Hum Brain Mapp.
2002:15(1):1–25. https://doi.org/10.1002/hbm.1058.
Nili H, Wingfield C, Walther A, Su L, Marslen-Wilson W, Kriegeskorte N. A toolbox for representational similarity analysis. PLoS
Comput Biol. 2014:10(4):e1003553. https://doi.org/10.1371/journal.
pcbi.1003553.
Oostenveld R, Fries P, Maris E, Schoffelen J-M. FieldTrip: open source
software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci. 2011:2011:156869.
https://doi.org/10.1155/2011/156869.
Parkhi O, Vedaldi A, Zisserman A. Deep face recognition.
BMVC 2015—Proceedings of the British Machine Vision Conference
2015. 2015: https://ora.ox.ac.uk/objects/uuid:a5f2e93f-2768-45
bb-8508-74747f85cad1.
Popal H, Wang Y, Olson IR. A guide to representational
similarity analysis for social neuroscience. Soc Cogn Affect
Neurosci. 2019:14(11):1243–1253. https://doi.org/10.1093/scan/nsz099.
Popham SF, Huth AG, Bilenko NY, Deniz F, Gao JS, NunezElizalde AO, Gallant JL. Visual and linguistic semantic
representations are aligned at the border of human visual cortex.
Nat Neurosci. 2021:24(11):1628–1636. https://doi.org/10.1038/
s41593-021-00921-6.
Ramon M, Busigny T, Gosselin F, Rossion B. All new kidson the block? Impaired holistic processing of personallyfamiliar faces in a kindergarten teacher with acquired
prosopagnosia. Vis Cogn. 2016:24(5–6):321–355. https://doi.
org/10.1080/13506285.2016.1273985.
Richoz A-R, Jack RE, Garrod OGB, Schyns PG, Caldara R.
Reconstructing dynamic mental models of facial expressions in
prosopagnosia reveals distinct representations for identity andexpression. Cortex; a Journal Devoted to the Study of the Nervous
System and Behavior. 2015:65:50–64. https://doi.org/10.1016/j.
cortex.2014.11.015.
Riddoch JM, Humphreys GW. BORB: Birmingham object recognition battery. Psychology Press; 2022, Hove, East Sussex, United Kingdom.
https://play.google.com/store/books/details?id=kLhmEAAAQBAJ.
Rosenthal G, Tanzer M, Simony E, Hasson U, Behrmann M, Avidan G.
Altered topology of neural circuits in congenital prosopagnosia.
elife. 2017:6. https://doi.org/10.7554/eLife.25069.
Rossion B. Constraining the cortical face network by
neuroimaging studies of acquired prosopagnosia. NeuroImage.
2008:40(2):423–426. https://doi.org/10.1016/j.neuroimage.2007.10.
047.

Faghel-Soubeyrand et al. |Wiese H, Tüttenberg SC, Ingram BT, Chan CYX, Gurbuz Z,
Burton AM, Young AW. A robust neural index of highface familiarity. Psychol Sci. 2019:30(2):261–272. https://doi.
org/10.1177/0956797618813572.

13

Zhao Y, Zhen Z, Liu X, Song Y, Liu J. The neural network forface recognition: insights from an fMRI study on developmental prosopagnosia. NeuroImage. 2018:169:151–161. https://doi.
org/10.1016/j.neuroimage.2017.12.023.

Downloaded from https://academic.oup.com/cercor/article/34/5/bhae211/7681100 by Verdon Christine user on 28 May 2024