# Macaque claustrum, pulvinar and putative dorsolateral amygdala support the cross-modal association of social audio-visual stimuli based on meaning

Mathilda Froesel, Maëva Gacoin, Simon Clavagnier, Marc Hauser, Quentin

Goudard, Suliann Ben Hamed

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Mathilda Froesel, Maëva Gacoin, Simon Clavagnier, Marc Hauser, Quentin Goudard, et al.. Macaque claustrum, pulvinar and putative dorsolateral amygdala support the cross-modal association of social audio-visual stimuli based on meaning. European Journal of Neuroscience, 2024, 59 (12), pp.3203-3223. 10.1111/ejn.16328. hal-04798996

HAL Id: hal-04798996

https://hal.science/hal-04798996v1

Submitted on 22 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Macaque claustrum, pulvinar and putative dorsolateral amygdala support the cross-modal association of social audio-visual stimuli based on meaning

Mathilda Froesel<fnref n="1" /> | Quentin Goudard<fnref n="1" />

Maëva Gacoin<fnref n="1" /> | Suliann Ben Hamed<fnref n="1" />

Simon Clavagnier<fnref n="1" /> | Marc Hauser<fnref n="2" />

## Correspondence

Mathilda Froesel and Suliann Ben Hamed, Institut des Sciences Cognitives Marc Jeannerod, UMR5229 CNRS Université de Lyon, 67 Boulevard Pinel, 69675 Bron Cedex, France.

Email: mathilda.froesel@isc.cnrs.fr; benhamed@isc.cnrs.fr

## Funding information

French National Research Agency, Grant/Award Numbers: ANR-16-CE37-0009-01, ANR-11-IDEX-0007; Université de Lyon, Grant/Award Number: ANR-11-LABX-0042

Edited by: John Foxe

## Abstract

Social communication draws on several cognitive functions such as perception, emotion recognition and attention. The association of audio-visual information is essential to the processing of species-specific communication signals. In this study, we use functional magnetic resonance imaging in order to identify the subcortical areas involved in the cross-modal association of visual and auditory information based on their common social meaning. We identified three subcortical regions involved in audio-visual processing of species-specific communicative signals: the dorsolateral amygdala, the claustrum and the pulvinar. These regions responded to visual, auditory congruent and audio-visual stimulations. However, none of them was significantly activated when the auditory stimuli were semantically incongruent with the visual context, thus showing an influence of visual context on auditory processing. For example,

**List of abbreviations:** %SC, percent signal change in hemodynamic signal; AC, audio-visual task, blocks with only auditory stimulations that are congruent with the general context of the run as set by the visual stimulations (e.g. coos presented alone, in a run in which the visual stimulations are positive lipsmacks); AI, Audio-visual task, blocks with only auditory stimulations that are incongruent with the general context of the run as set by the visual stimulations (e.g. coos presented alone, in a run in which the visual stimulations are negative aggressive faces); AMG, amygdala: note that the activation is in the putative dorsolateral amygdala; Claus, claustrum; F-, audio-visual task, face negative context: runs of the audio-visual tasks presenting only aggressive faces as visual stimuli (lipsmacks) combined with coos or aggressive calls as auditory stimuli; F+, Audio-visual task, face affiliative context: runs presenting only affiliative faces as visual stimuli (lipsmacks) combined with coos or aggressive calls as auditory stimuli; fMRI, functional magnetic resonance imaging; PLdm, dorso-medial lateral pulvinar; PLvl, ventro-lateral lateral pulvinar; PuA, anterior pulvinar; PuI, inferior pulvinar; PuL, lateral pulvinar; PuM, medial pulvinar; ROI, region of interest; S-, combination of negative social context runs (S1- & S2-); S+, combination of positive social context runs (S1+ & S2+); S1-, audio-visual task, social aggressive negative context: runs presenting only aggressive group or individual scenes as visual stimuli combined with coos or aggressive calls as auditory stimuli; S1+, audio-visual task, first social affiliative context: runs presenting only mutual grooming scenes as visual stimuli combined with coos or aggressive calls as auditory stimuli; S2-, audio-visual task, social negative social escape context: runs presenting only fleeing groups or individual scenes as visual stimuli combined with coos or screams as auditory stimuli; S2+, audio-visual task, second social affiliative context: runs presenting only mutual grooming scenes as visual stimuli combined with coos or screams as auditory stimuli; SNR, signal to noise ratio; STS, superior temporal sulcus; V, audio-visual task, blocks with only visual stimulations block; VAC, audio-visual task, blocks with visual and auditory stimulation that match semantically (e.g. coos and lipsmacks); VAI, audio-visual task, blocks with visual and auditory stimulation that do not match semantically (e.g. coos and aggressive faces).

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

© 2024 The Authors. *European Journal of Neuroscience* published by Federation of European Neuroscience Societies and John Wiley & Sons Ltd.

<footnote n="1">¹Institut des Sciences Cognitives Marc Jeannerod, UMR5229 CNRS Université de Lyon, Bron Cedex, France</footnote>

<footnote n="2">²Risk-Eraser, West Falmouth, Massachusetts, USA</footnote>
positive vocalization (coos) activated the three subcortical regions when presented in the context of positive facial expression (lipsmacks) but not when presented in the context of negative facial expression (aggressive faces). In addition, the medial pulvinar and the amygdala presented multisensory integration such that audiovisual stimuli resulted in activations that were significantly higher than those observed for the highest unimodal response. Last, the pulvinar responded in a task-dependent manner, along a specific spatial sensory gradient. We propose that the dorsolateral amygdala, the claustrum and the pulvinar belong to a multisensory network that modulates the perception of visual socioemotional information and vocalizations as a function of the relevance of the stimuli in the social context.

**Significance statement:** Understanding and correctly associating socioemotional information across sensory modalities, such that happy faces predict laughter and escape scenes predict screams, is essential when living in complex social groups. With the use of functional magnetic imaging in the awake macaque, we identify three subcortical structures—dorsolateral amygdala, claustrum and pulvinar—that only respond to auditory information that matches the ongoing visual socioemotional context, such as hearing positively valenced coo calls and seeing positively valenced mutual grooming monkeys. We additionally describe task-dependent activations in the pulvinar, organizing along a specific spatial sensory gradient, supporting its role as a network regulator.

## KEYWORDS

audio-visual, claustrum, dorsolateral amygdala, fMRI, macaque, multisensory, pulvinar, socioemotional

# 1 | INTRODUCTION

In a wide variety of species, social communication often involves sending and receiving systems that must integrate information across different modalities, recruiting multiple processes such as sensory perception, emotion processing and attention. Communication has been extensively described as a multisensory process relying on the association of visual and auditory components of vocal signals based on their temporal and spatial congruence (Murray & Wallace, 2012; Partan, 2013; Partan & Marler, 2005). Macaques are notably successful in matching specific calls, such as coos, with the proper facial expression, thus associating the specific lip and mandible configuration with the matching call (Ghazanfar & Logothetis, 2003; Hauser et al., 1993; Hauser & Ybarra, 1994). This cross-modal association (defined as the binding or association of stimuli or experiences that come from different sensory modalities) of visual information with vocal signals is not fully explained by the dynamic properties of the visual input that are associated

with low level features and is shown to be specific to face processing. At the neuronal level, this cross-modal association is subserved by multisensory integrative mechanisms, such that a matching pair of faces and vocalizations triggers a neuronal response that is significantly different from that to the most effective unimodal stimulus in the pair (Ghazanfar et al., 2005). In addition to low level dynamic visual features such as facial mimics or local colour contrasts, faces and vocalizations also contain higher level information such as emotions and semantics or social meaning. Whether monkeys can associate faces and vocalizations based on their meaning rather than on their low-level matching features was unknown. For example, a scene of escaping monkeys is expected to be associated with a scream, because they describe a same event, but not with a coo. This cross-modal association based on stimulus meaning is expected to build on the knowledge we have from previous experience with this type of stimuli. Macaques scream when they experience fear, triggered by potential danger due to complex social situations from conspecifics or
heterospecifics. Macaques also coo during positive social interactions, such as friendly approach, feeding or group coordinated displacement (Gouzoules et al., 1984; Hauser & Marler, 1993). This raises the question of whether hearing a scream generates a visual representation of antagonistic as opposed to positive social situations and whether seeing an antagonistic situation set up the expectation for screams, but not coos, to be produced. Such a mechanism could be crucial to social cognition and may have subserved the development of language. Indeed, in humans, the core language system is amodal, as our phonology, semantics and syntax do not depend on whether the input is auditory (speech) or visual (sign). In a recent study, we demonstrated using cardiac recordings (Froesel et al., 2020) and functional magnetic resonance imaging (fMRI) that macaques associate communicatively salient audio-visual information based on the social context set by visual information. This process is mediated by a network of face and voice patches in the superior temporal sulcus (STS) and lateral sulcus (LS) (Froesel et al., 2022). We propose that this could correspond to a supramodal representation of social stimuli, that is a representation that does not depend on the encoding sensory modality and that serves to attribute meaning to complex social stimuli.

Given that the task involves functionally meaningful communicative signals that are emotionally salient, we predicted strong amygdala activation. This subcortical nucleus contains face-selective neurons that are globally activated both by face identity and facial expressions (Fitzgerald et al., 2006; Livneh et al., 2012; Nakamura et al., 1992; Pessoa et al., 2006; Sergerie et al., 2008; Todorov, 2012). Specifically, affiliative facial expressions induce a decrease of firing rates in the amygdala, while threatening faces induce an increase in firing rates (Gothard et al., 2007). Note that the part of the amygdala responding to face stimuli in macaque fMRI studies is located in the dorsolateral part of the nucleus, across the white matter from the claustrum (Hadj-Bouziane et al., 2008, 2012; Schwiedrzik et al., 2015). Due to the type of stimuli used in the present study, we thus expect to observe activations specifically in this part of the amygdala. Overall, beyond face processing, the amygdala is part of a social perception network and is proposed to play a central role in emotional encoding during complex social interactions (Barraclough & Perrett, 2011; Leonard et al., 1985), including during cross-modal sensory emotional processing (Dolan et al., 2001; Kuraoka & Nakamura, 2007). In addition to the amygdala, the pulvinar, the largest nucleus of the thalamus, is also involved in face processing, emotion regulation and multisensory integration (Froesel et al., 2021; Moeller et al., 2008;

Pessoa, 2010a). This subnucleus is subdivided into several parts, each exhibiting different activation patterns based on the sensory modality as well as a distinct connectivity with other brain regions. Specifically, the inferior pulvinar is mainly responsive to visual stimuli, the anterior pulvinar, to auditory stimuli, and the medial part, which is strongly connected to cortical multisensory areas, is expected to display multisensory responses (see for review Froesel et al., 2021). Last, the claustrum is also activated following stimulation of cortical face selective areas, also called face patches, and is proposed to play a role in face perception (Moeller et al., 2008). We thus hypothesize that all of these subcortical structures are activated by visual and auditory social stimuli such as macaque faces, macaque group scenes and macaque vocalizations and may play a part in multisensory integration. Social perception also involves integrating contextual, behavioural and emotional information (Freiwald, 2020; Ghazanfar & Santos, 2004). We thus predict that the audio-visual association of social information is impacted by the functional context and mediated by both these subcortical structures.

Here, we describe an fMRI study in awake macaques performing a passive audio-visual task manipulating the congruency of cross-modal audio-visual social information across six different emotional contexts, as well as a passive visual task manipulating monkey facial expressions. Complementing our previous report of the cortical activation patterns, we show that sub-cortical activations are determined by whether monkey vocalizations match or do not match the facial expressions associated with them. As predicted, the subcortical structures identified included the claustrum and putative dorsolateral amygdala that respond to all of visual, auditory and audio-visual stimuli, demonstrating their involvement in audio-visual cross-modal association as well as the pulvinar that is activated along a sensory gradient that is determined by stimulation modality (visual, auditory or audio-visual) or context. The functional activations observed in the medial pulvinar are similar to those observed in the putative dorsolateral amygdala and in the claustrum, thus suggesting a coordinated role of these three sub-cortical structures in multisensory social processing. In addition, the putative dorsolateral amygdala and the medial pulvinar express multisensory integration, such that congruent social audio-visual stimuli trigger a hemodynamic response that is significantly different from that to the most effective unimodal stimulus in the pair. Overall, our observations shed a new light on how multimodal social signals are processed when functionally meaningful, by subcortical structures.
<figure><img src="image_3.png" /><figcaption>**FIGURE 1** Experimental designs. Each sensory stimulation block contained a rapid succession of 500-ms stimuli (with the exception of the lipsmack for the unimodal visual task). Each run started and ended with 10 s of fixation regardless of the task type. (a) Experimental design of the audio-visual task. Example of an aggressive face (F−) context. One run represents one context set up by the visual stimuli and contains three randomized repetitions of six different blocks of 16 s. The six blocks displayed were either visual stimuli only (Vi), auditory congruent stimuli only (AC), auditory incongruent stimuli only (AI), audio-visual congruent stimuli (VAC) or audio-visual incongruent stimuli (VAI) or fixation with no sensory stimulation (Fx). Blocks were pseudo randomized in order that each block was, on average, preceded by the same number of blocks from the other conditions and that each run started with a block of a visual information (V, VAC or VAI). (b) Description of the contexts. Six contexts were displayed. Each context combined visual stimuli of identical social content with either semantically congruent or incongruent monkey vocalizations. Pairs of contexts shared the same auditory stimuli but opposite social visual content (F+ vs. F−; S1+ vs. S1−; S2+ vs. S2−). Each run corresponded to one of the semantic contexts described above. (c) Experimental design of the unimodal visual task. In one run, six different blocks were displayed three times randomly. The six possible 16-s blocks were fixation (Fx), lipsmack (Lip), scared monkey faces (Sca), aggressive monkey faces (Aggr), neutral monkey faces (Neu) and scrambled monkey faces (Scr). Visual stimuli were extracted from videos collected by the Ben Hamed lab, as well as by Marc Hauser on Cayo Santiago, Puerto Rico.</figcaption></figure>

## 2 | MATERIAL AND METHODS

### 2.1 | Subjects and surgical procedures

Two male rhesus monkeys (*Macaca mulatta*) participated in the study (T, 15 years, 10 kg and S, 12 years, 11 kg). The animals were implanted with a Peek MRI-compatible headset covered by dental acrylic. The

anaesthesia for the surgery was induced by Zoletil (Tiletamine-Zolazepam, Virbac, 5 mg/kg) and maintained by isoflurane (Belamont, 1–2%). Post-surgery analgesia was ensured thanks to Temgesic (buprenorphine, .3 mg/ml, .01 mg/kg). During recovery, proper analgesic and antibiotic coverage was provided. The surgical procedures conformed to European and National Institutes of Health Guidelines for the Care and Use of Laboratory
Animals. The project was authorized by the French Ministry for Higher Education and Research (project no. 2016120910476056 and 1588-2015090114042892) in accordance with the French transposition texts of Directive 2010/63/UE. This authorization was based on ethical evaluation by the French Committee on the Ethics of Experiments in Animals (C2EA) CELYNE registered at the national level as C2EA number 42.

## 2.2 | Experimental setup

During the scanning sessions, monkeys sat in a sphinx position in a plastic monkey chair (Vanduffel et al., 2001) facing a translucent screen placed 60 cm from the eyes. Visual stimuli were retro-projected onto this translucent screen. Their head was restrained, and the auditory stimuli were displayed by Sensimetrics MRI-compatible S14 insert earphones. The monkey chair was secured in the MRI with safety rubber stoppers to prevent any movement. Eye position (X, Y, right eye) was recorded thanks to a pupil-corneal reflection video-tracking system (EyeLink at 1000 Hz, SR-Research) interfaced with a program for stimulus delivery and experimental control (EventIDE®). Monkeys were rewarded for maintaining fixation into a 2° × 2° tolerance window around the fixation point.

## 2.3 | General audio-visual run design

On each run, monkeys were required to fixate a central cross on the screen (Figure 1a). Runs followed a block design. Each run started with 10 s of fixation in the absence of sensory stimulation followed by three repetitions of a pseudo-randomized sequence containing six possible 16 s blocks: fixation (Fx), visual (Vi), auditory congruent (AC), auditory incongruent (AI), congruent audio-visual (VAC) and incongruent audio-visual (VAI). Each block (except the fixation block) consisted of alternating 500-ms stimuli (except for lipsmacks, 1-s dynamic stimuli succession) of the same semantic category (see Stimuli section below), in the visual, auditory or audio-visual modalities. Each block ended with 10 s of fixation in the absence of sensory stimulation. Note that within any one run, the visual stimulations were prevalent and always reflecting the same emotional content, thus setting the emotional context of the run. The initial blocks always contained a visual stimulation (V, VAC or VAI) such that unimodal auditory blocks could be defined as congruent or incongruent relative to the visual context set by previous blocks.

## 2.4 | Face and social task design

Six audio-visual contexts were presented to both monkeys, organized in runs as described above (Figure 1b). Each run combined visual stimuli of identical social content with either semantically congruent or incongruent monkey vocalizations (Figure 1b). The face affiliative context (F+) combined lipsmacks with coos and aggressive calls. The face aggressive context (F−) combined aggressive faces with coos and aggressive calls. The first social affiliative context (S1+) combined mutual grooming scenes with coos and aggressive calls. The second social affiliative context (S2+) combined mutual grooming scenes with coos and screams. The social aggressive context (S1−) combined aggressive group or individual scenes with coos and aggressive calls. The social escape context (S2−) combined fleeing groups or individual scenes with coos and screams. Importantly, pairs of contexts (F+ & F−; S1+ & S1−; S2+ & S2−) shared the same auditory conditions but opposite social visual content.

## 2.5 | Unimodal visual task design

The design of the visual runs was similar to that of the audio-visual run design, organized in blocks, except for the fact that all blocks were unimodal visual blocks and varied as a function of facial emotions. The six possible 16-s blocks were fixation (Fx), lipsmack (Lip), scared monkey faces (Sca), aggressive monkey faces (Aggr), neutral monkey faces (Neu) and scrambled monkey faces (Scr). As for the audio-visual runs, each block consisted in an alternation of 500-ms stimuli (except for lip smacks, 1-s dynamic stimuli succession) of the same emotional category (see Figure 1c).

## 2.6 | Social stimuli

Vocalizations were recorded by Marc Hauser from semi-free-ranging rhesus monkeys during naturally occurring situations. Detailed acoustic and functional analyses of this repertoire have been published elsewhere (e.g., Gouzoules et al., 1984; Hauser & Marler, 1993). Field recordings were then processed, restricting selection of experimental stimuli to calls that were recorded from known individuals, in clearly identified situations and that were free of competing noise from the environment. Exemplars from this stimulus set have already been used in several imaging studies (Belin et al., 2007; Cohen et al., 2007; Romanski, 2012; Romanski et al., 2005; Russ et al., 2008). As in our previous study
(Froesel et al., 2022), all stimuli were normalized in luminance and colour, but the frequency ranges varied between the different types of stimuli as shown in Figure S7. For each of the three vocalization categories, we used 10 unique exemplars coming from matched male and female individuals for each category in order to control for possible gender, social hierarchy or individual effects. Coos are affiliative vocalizations, aggressive calls are used as a precursor of a physical attack and screams are produced by subordinate being chased or attacked by a dominant. Facial expression (lipsmacks and aggressive facial expression) and social scene (group grooming, aggressive individual alone or in group/escaping individual or group) stimuli were extracted from videos collected by the Ben Hamed lab, as well as by Marc Hauser on Cayo Santiago, Puerto Rico. Images were normalized for average intensity and size. All stimuli were 4° × 4° in size. We decided to keep them in colour to get closer to natural stimuli even if it produced greater luminosity disparity between the different stimuli preventing us to use pupil diameter as a physiological marker. Only unambiguous facial expressions and social scenes were retained. A 10% blur was applied to all images, in the hope of triggering multisensory integration processes (Stein & Meredith, 1993) (but see Section 3). For each visual category, 10 stimuli were used. The scrambling of the images was performed by EventIDE (https://www.okazolab.com/okazolab.com/) and applied to each visual stimuli displayed from the task.

## 2.7 | Scanning procedures

The in-vivo MRI scans were performed on a 3T Magnetom Prisma system (Siemens Healthineers, Erlangen, Germany).

**Anatomical images:** For the anatomical MRI acquisitions, monkeys were first anesthetized with an intramuscular injection of ketamine (10 mg/kg). Then, the subjects were intubated and maintained under 1–2% of isoflurane. During the scan, animals were placed in a sphinx position in a Kopf MRI-compatible stereotaxic frame (Kopf Instruments, Tujunga, CA). Two L11 coils were placed on each side of the skull, and an L7 coil was placed on the top of it. T1-weighted anatomical images were acquired for each subject using a magnetization-prepared rapid gradient-echo (MPRAGE) pulse sequence. Spatial resolution was set to .5 mm, with TR = 3000 ms, TE = 3.62 ms, Inversion Time (TI) = 1100 ms, flip angle = 8°, bandwidth = 250 Hz/pixel, 144 slices. T2-weighted anatomical images were acquired per monkey, using a Sampling Perfection with Application optimized Contrasts using different flip angle Evolution

(SPACE) pulse sequence. Spatial resolution was set to .5 mm, with TR = 3000 ms, TE = 366.0 ms, flip angle = 120°, bandwidth = 710 Hz/pixel, 144 slices.

fMRI images were acquired on awake macaque monkeys as follows: before each scanning session, a contrast agent, composed of monocrystalline iron oxide nanoparticles, Molday ION™, was injected into the animal's saphenous vein (9–11 mg/kg) to increase the signal to noise ratio (SNR) (Leite et al., 2002; Vanduffel et al., 2001). Monkeys were trained to stay calm to let the experimenter perform this injection without anaesthesia and got rewarded for it. They were also trained to stay calm during the fMRI scans. We acquired gradient-echoechoplanar images covering the whole brain (TR = 2000 ms; TE = 18 ms; 37 sagittal slices; resolution: 1.25 × 1.25 × 1.38 mm anisotropic voxels) using an eight-channel phased-array receive coil and a loop radial transmit-only surface coil (MRI Coil Laboratory, Laboratory for Neuro- and Psychophysiology, Katholieke Universiteit Leuven, Leuven, Belgium, see Kolster et al., 2014). The coils were placed so as to maximize the signal on the temporal lobe.

## 2.8 | Data description

In total, for the audio-visual runs (155 pulses), 76 runs were collected in 12 sessions for monkey T and 65 runs in 9 sessions for monkey S. For the unimodal visual runs (155 pulses), 13 runs were collected in 8 sessions for monkey T and 12 runs in 5 sessions for monkey S. On the basis of the monkey's fixation quality during each run (85% within the eye fixation tolerance window), we selected 60 runs from monkey T and 59 runs for monkey S in total, that is 10 runs per task, except for one task of monkey S for audio-visual runs and 11 unimodal visual runs for S and 13 for T.

## 2.9 | Data analysis

Data were pre-processed and analysed using AFNI (Cox, 1996), FSL (Jenkinson et al., 2012), SPM software (version SPM12, Wellcome Department of Cognitive Neurology, London, UK, https://www.fil.ion.ucl.ac.uk/spm/software/), JIP analysis toolkit (http://www.nitrc.org/projects/jip) and Workbench (https://www.humanconnectome.org/software/get-connectome-workbench). The T1-weighted and T2-weighted anatomical images were processed according to the HCP pipeline (Autio et al., 2020; Glasser et al., 2013) and were normalized into the MY19 Atlas (Donahue et al., 2016). Functional volumes were corrected for head motion, slice
<figure><img src="image_4.png" /><figcaption>**FIGURE 2** Whole-brain FACE blocked conditions (F+ & F−) activations: main contrasts. Whole-brain activation maps of the F+ (face affiliative) and F− (face aggressive) runs, cumulated over both monkeys, for the visual, auditory congruent, audio-visual congruent and audio-visual incongruent contrasts. Darker shades of red indicate level of significance at *p* < .001 uncorrected, *t* score 3.09. Lighter shades of yellow indicate level of significance at *p* < .05 FWE, *t* score 4.6. See Table S3 for ROI coordinates. The precise localization of the identified ROIs relative to the MY19 Atlas is described in Figure S1.</figcaption></figure>

timed referred on the middle image of the run and skull-stripped. They were then linearly realigned on the T2-weighted anatomical image with flirt from FSL; the image distortions were corrected using nonlinear warping with JIP. A spatial smoothing was applied with a 3-mm full width of half maximum (FWHM) Gaussian Kernel.

Fixed effect individual analyses were performed for each monkey, with a level of significance set at *p* < .05 corrected for multiple comparisons (family-wise error [FWE], *t*-scores 4.6) and *p* < .001 (uncorrected level, *t*-scores 3.09). Head motion and eye movements were included as covariate of no interest. Because of the contrast agent injection, a specific MION hemodynamic response function (HRF) (Vanduffel et al., 2001) was used instead of the BOLD HRF provided by SPM. The main effects were computed over both monkeys. In most analyses, face blocked conditions and social blocked conditions were independently pooled.

To identify the precise anatomical location of the activations, we used the Subcortical Atlas of the Rhesus Macaque (SARM) (Hartig et al., 2021) (see Figure S1). The activation maps are coregistered on the MY19 Atlas (Donahue et al., 2016) as well as the SARM Atlas, thus providing a direct overlap between the two.

Region of interest (ROI) analyses were performed as follows. The anterior pulvinar, claustrum ROIs were determined from the auditory congruent contrast (AC vs. Fx) and the medial pulvinar and the amygdala ROIs from

the audio-visual contrast (VAC vs. FX) of face context (Figure 2). PLvl and Pldm ROIs were extracted from the unimodal visual runs (all conditions vs. fixation contrast; Figure 4). ROIs were defined as 1-mm diameter spheres centred around the local peaks of activation. To note, there was no overlap between the different clusters (see Figure S1 for the representation of the activation peaks and selected regions of interest and Table S3 that indicates the exact coordinates of the centre of the ROIs selected for each ROI-based analysis). For each ROI, the activity profiles were extracted with the Marsbar SPM toolbox (marsbar.sourceforge.net) and the mean percent of signal change (%SC) (+/- standard error of the mean across runs) was calculated for each condition relative to the fixation baseline. As the face context includes both aggressive and lipsmack expressions, we decided for unimodal visual runs to combine these two expressions conditions and focus on this combination for the analysis to be comparable with the %SC results from the audio-visual runs. %SC were compared using Friedman non-parametric tests and Wilcoxon non-parametric paired tests.

In the present study, stimuli are associated based on their functions, determined by their naturally occurring context (e.g. coos are associated with mutual grooming) and possible contingencies or predictability (e.g. screams are associated with or predictive of escape scenes). The described audio-visual association thus goes beyond the strict definition of two sensory inputs produced by a
common source. Bimodal and unimodal activations are quantified as follows:

1. Definition of multisensory integration in our fMRI protocol: The criteria used to define multisensory integration at the neuronal level (Avillac et al., 2004, 2007; Murray & Wallace, 2012; Stein et al., 2009) do not apply in fMRI. For example, it is now established that it is quite difficult to reach the superadditive criteria with MRI, that is when the bimodal response is significantly higher than the sum of the two independent modalities. This can be due to the variability of response of multisensory neurons (some exhibiting suppressive responses whereas others exhibiting enhancement and their interdigitated location among substantial populations of unisensory neurons), although some fMRI studies did show superadditivity (Stevenson et al., 2009; Werner & Noppeney, 2010). As a result, in fMRI studies, the maximum and additive criteria are now the most used criteria to investigate multisensory integrative processes on hemodynamic activations (Beauchamp, 2005; Pollick et al., 2011; Tyll et al., 2013; Cléry et al., 2017; see for review Stevenson et al., 2014). In the present study, we use the following criterion for multisensory integration: multisensory activations had to be significantly different from each of the two unisensory responses and significantly different from the control fixation condition. All reported multisensory integrative responses in the present manuscript were significantly higher than each of the two unisensory responses (in other words, none of the reported multisensory integrative responses was significantly lower than any of the two unisensory responses). Note that the identification of multisensory integration at population level should be validated by other methods at single neural level.

2. Definition of cross-modal association in our fMRI protocol: cross-modal association is defined as the effect of one sensory modality on the processing of a second modality, presented either synchronously or asynchronously. Effect of temporal asynchrony has classically been investigated within a range of 100-200 ms of asynchrony (Schormans & Allman, 2018, 2023). However, cross-modal associations have also been reported at longer time ranges, whereby a general context determines whether cross-modal association takes place or not. As an example, auditory stimuli (e.g. coos, which always activate the auditory cortex in the LS) can either result in activations in the STS when congruent with a general visual context (e.g. grooming) or no STS activations when incongruent with the general visual context (e.g. escape)

(Froesel et al., 2022). This type of cross-modal association probably arises from long-term learning and can result in more voxels/regions being recruited to process the stimuli (Archakov et al., 2020) or in a change in degree of activation (Froesel et al., 2022). Here, we manipulated the visual context by showing a single category of visual images of relevant emotional content per run. Setting a specific visual context aimed at identifying the effect of vision on subsequent auditory processing. Cross-modal association was determined as a significant difference in the auditory activations, depending on whether they were congruent (AC) or incongruent (AI) to the context set by the visual stimuli in any given run, thus indicating that how the brain processes the auditory modality depended on the network effects of the preceding visual stimulations.

## 3 | RESULTS

Here, we characterize the activations in the subcortical structures, in the putative dorsolateral amygdala, the claustrum and the pulvinar, underlying the cross-modal association of social visual and auditory stimuli based on their semantic content or meaning in the broad sense. In addition, we show that among these subcortical regions, only the activations observed in the dorsolateral amygdala and the medial pulvinar meet the criteria for multisensory integration in fMRI. Last, we show an effect of both sensory context and sensory modality on the location of pulvinar activations.

### 3.1 | Sub-cortical activations: Dorsolateral amygdala, claustrum and pulvinar

A small group of sub-regions are significantly activated during one or several of the unimodal or bimodal conditions presented to the monkeys in the different types of runs. Indeed, in a general contrast analysis, the claustrum (Claus) and the anterior (PuA) and medial (PuM) pulvinar nuclei are activated by the Auditory congruent versus Fixation contrast (Figure 2, top raw). The dorsal lateral part of the right amygdala (AMG) is activated by the Visual versus Fixation contrast (Figure 2, middle left panel; for precise localization and identification criteria, see supplemental Figure S1B). Given the location of these activations, we will refer to them as putative dorsolateral amygdala. The inferior (PuI) and medial (PuM) pulvinar nuclei are activated by the Audio-visual incongruent versus Fixation contrast
<figure><img src="image_5.png" /></figure>

**FIGURE 3** Percentage of signal change (%SC) for FACE tasks (F+ & F−) and for social tasks (S1+, S1−, S2+ & S2−) across sub cortical regions of interest (ROIs), putative dorsolateral amygdala, claustrum, inferior pulvinar, medial pulvinar and anterior pulvinar of both hemispheres, comparing the auditory, visual and audio-visual conditions. Statistical differences relative to fixation and between conditions are indicated as follows: ***, *p* < .001; **, *p* < .01; *, *p* < .05 (Wilcoxon non-parametric test). See Table S1 and accompanying note for quantitative effect sizes.

(Figure 2, middle right panel). The putative dorsolateral amygdala (AMG), claustrum (Claus) and medial (PuM) pulvinar nucleus are activated by the Audio-visual congruent versus Fixation contrast (Figure 2, bottom raw). All of the activations are bilateral in at least one contrast except for the amygdala. Figure S2 represents the SNR map for the coronal section at the level of the pulvinar.

We defined functional ROIs based on these sub-cortical activations, and we computed the per cent of signal change relative to fixation (%SC) for all conditions, cumulated over identical block conditions of either the face contexts (Figure 3, left), or the social scenes contexts (Figure 3, right). For all ROIs, and both face and social contexts, there was no significant activation relative to the fixation baseline in response to the auditory incongruent blocks. All other blocks led to consistent significant activations, except for inferior and anterior pulvinar, in the visual blocks of both face and social scene contexts, and the claustrum, in the visual blocks of social scene contexts.

## 3.2 | Cross-modal association

The first major result of this study is the impact of the visual context on auditory activations. Although the dorsolateral amygdala, claustrum and pulvinar show activations in response to one or several of the unimodal or bimodal conditions presented to the monkeys in the different types of contexts, no significant activation is reported when the auditory stimuli are incongruent with the visual context (see Figure 3). In the dorsolateral amygdala and the claustrum, %SC in response to the auditory incongruent blocks relative to fixation was statistically significantly lower than the %SC in all other conditions (i.e. V, AC, VAC and VAI), in both the face and the social contexts (Figure 3, two top panels, except for the claustrum, in the social scene where AI was not significantly different from V and AC).

The three different pulvinar ROIs (inferior, medial and anterior) present no significant %SC in response to
the auditory incongruent condition relative to fixation. Quite unexpectedly, in these ROIs, this AI condition was significantly different from the other conditions only in a few cases. In the social context, the anterior and the inferior pulvinar presented a different %SC in the AI condition relative to other conditions. Specifically, AI %SC was significantly different from AC and VAC %SC for the inferior pulvinar and from VAI %SC for the anterior pulvinar (PuI: Friedman non-parametric test, $X_{2(4)} = 19.25$, $p < .001$, $n = 158$, post hoc: AC-AI: $Z = 2.6$, $p = .009$; VAC-AI: $Z = 1.9$, $p = .049$. PuA: Friedman non-parametric test, $X_{2(4)} = 18$, $p = .0012$, $n = 158$, post hoc: AC-AI: $Z = 1.88$, $p = .05$ VAI-AI: $Z = 1.9$, $p = .049$). In the medial pulvinar, in both types of contexts, the %SC in the AI condition was significantly lower than in the VAC and/or VAI conditions (FACE: Friedman non-parametric test, $X_{2(4)} = 19.26$, $p < .001$, $n = 80$, post hoc: VAC-AI: $Z = 1.9$, $p < .01$, VAI-AI: $Z = 1.37$, $p < .01$; SOCIAL: Friedman non-parametric test, $X_{2(4)} = 10.54$, $p = .03$, $n = 158$; post hoc: VAI-AI: $Z = 2$, $p = .048$).

## 3.3 | Multisensory integration

The putative dorsolateral amygdala shows a higher activation following audio-visual congruent stimulation, that is in the bimodal stimulation, than for unimodal stimulation, that is visual only and auditory only contexts, in the social context. In the putative dorsolateral amygdala ROI, VAC conditions in S1-, S2- and S2+ resulted in significant %SC relative to fixation. We did not observe a significant difference in %SC in response to the negative (S-) or positive (S+) social contexts VAC conditions. More specifically, there was no significant difference in %SC for the VAC condition to aggressive visual context (S1-) and the flight visual context (S2-), and the mutual grooming scenes with coos and aggressive calls (S1+)

resulted in significantly lower %SC than the mutual grooming scenes with coos and screams (S2+). In the claustrum ROI, only the mutual grooming scenes with coos and aggressive calls (S1+) and the mutual grooming scenes with coos and screams (S2+) resulted in significant changes in %SC for VAC relative to fixation, although not significantly different from each other. In addition, and paralleling putative dorsolateral amygdala activation, the medial pulvinar showed a higher activation in bimodal (VAC) versus unimodal conditions (V and AC), specifically to the face context (FACE: Friedman non-parametric test, $X_{2(4)} = 19.26$, $p < .001$, $n = 80$, post hoc: V-VAC: $Z = 1.6$, $p < .01$, AC-VAC: $Z = 1.2$, $p = .02$). The putative dorsolateral amygdala and the medial pulvinar thus exhibit a form of multisensory integration.

## 3.4 | Pulvinar audio-visual activations

Three different locations of activations within the pulvinar were identified in the audio-visual task: the inferior, the medial and the anterior pulvinar. The inferior pulvinar is activated in both face and social tasks by auditory congruent stimuli and audio-visual congruent and incongruent stimuli (see Figure 3 panel 3). The anterior pulvinar is significantly activated during AC and VAI blocks in both types of task and in VAC block in the face task (see Figure 3 panel 5). This leads to the noteworthy and unexpected observation that, in this task, the inferior and anterior pulvinar preferentially respond in the auditory conditions (i.e. AC, VAC and VAI) and not in the visual condition.

The medial pulvinar is activated by AC, VAC and VAI stimulations but is in addition the only pulvinar subregion to respond to visual stimulation presented alone (FACE: V: $Z = 2.04$, $p = .04$; SOCIAL: V: $Z = 2.9$, $p = .003$).

<figure><img src="image_6.png" /><figcaption>FIGURE 4 Pulvinar activations in a unimodal visual task: (lipsmacks + aggressive) conditions versus fixation contrasts. Whole-brain activation maps of the unimodal visual task cumulated over both monkeys, for lipsmack + aggressive blocks versus fixation contrast. Darker shades of red indicate level of significance at $p < .001$ uncorrected, $t$ score 3.09. Lighter shades of yellow indicate level of significance at $p < .05$ FWE, $t$ score 4.6. PLvl: ventrolateral lateral pulvinar; PLdm: dorso-medial lateral pulvinar. See Table S3 for region of interest (ROI) coordinates. The precise localization of the identified ROIs relative to the MY19 Atlas is described in Figure S1.</figcaption></figure>
<figure><img src="image_7.png" /></figure>

FIGURE 5 Claustrum activations in a unimodal visual task: (lipsmacks + aggressive) conditions versus fixation contrasts. (a) Whole-brain activation maps of the unimodal visual task cumulated over both monkeys, for lipsmack + aggressive blocks versus fixation contrast. Darker shades of red indicate level of significance at *p* < .001 uncorrected, *t*-score 3.09. Lighter shades of yellow indicate level of significance at *p* < .05 FWE, *t* score 4.6. (b) Percent signal change of lipsmack + aggressive blocks versus fixation contrast in the putative dorsolateral amygdala (AMG) and the claustrum (Claus) region of interest (ROIs) defined in the audio-visual task described in Figure 2. AMG: *Z* = 4.27; *p* < .001; Claus: *Z* = 5.8, *p* < .001. See Table S3 for ROI coordinates. The precise localization of the identified ROIs relative to the MY19 Atlas is described in Figure S1.

## 3.5 | Putative dorsolateral amygdala, claustrum and pulvinar activations in a unimodal visual task

Because macaque pulvinar has been repeatedly observed to be involved in visual processing, we hypothesized that this specific pulvinar response pattern is a task effect and reflects a spatial segregation of pulvinar sensory responses. To test this, we investigated pulvinar activations in a unimodal visual task, in which only monkey faces were presented in blocks of consistent emotional categories. In order to match the visual categories used in the main audio-visual task, the following, we only used the lipsmacks and aggressive categories. Clear lateralized left visual activations could be identified in the lateral pulvinar (Figure 4, medio-dorsally: PLdm and ventro-laterally: PLvl), at a location very distinct from the ROIs activated in the audio-visual tasks. In this same task and same contrast, the claustrum was also significantly activated (Figure 5a) but not the putative dorsolateral amygdala, although a %SC analysis using a priori defined ROIs extracted from the audio-visual task shows that both subcortical structures are highly activated during these unimodal visual runs (Figure 5b; AMG: *Z* = 4.27; *p* < .001; Claus: *Z* = 5.8, *p* < .001). Overall, this indicates that while the putative dorsolateral amygdala and the claustrum are involved in the processing of social cues in both visual and audio-visual contexts, the spatial organization of the pulvinar activations varies between the two types of sensory contexts. This is explored in more detail in the next section.

## 3.6 | Gradient of multi and unimodal pulvinar activations

The pulvinar is involved in several sensory processes (for review, see Froesel et al., 2021). However, this sensory information from multiple modalities is organized remains poorly understood. In the following, we describe a spatial gradient of sensory responses within the pulvinar, from unimodal auditory responses (Figure 6a, red), audio-visual responses (Figure 6a,b, blue, see Figure S3 for individual monkey maps) and unimodal visual responses (Figure 6b, green). More specifically, unimodal auditory activations are located in anterior pulvinar (PuA), anterior and medial to the audio-visual activations that are located in medial pulvinar (PuM), with a slight overlap between the two activated ROIs. The unimodal visual activations in the visual task are located in two distinct lateral pulvinar ROIs (Figure 6b, sagittal view, green), lateral to the PuM audio-visual ROIs (Figure 6b, blue), that coincide with PLvl and PLdm (Figure 4), again with a slight overlap between the audio-visual and the visual ROIs. It is worth noting that this gradient was present in each monkey (Figure S3).

Importantly, this very clear functional sensory gradient within the pulvinar is task specific. Indeed, the visual ROIs identified during the unimodal visual task are not activated during the audio-visual task using the exact same stimuli and vice versa (Figure 7a,b). All pulvinar ROIs except the anterior pulvinar (PuA) are significantly activated relative to fixation during the unimodal visual task (Figure 7a; PuI: *Z* = 3.19, *p* < .001; PuM: *Z* = 2.6,
<figure><img src="image_8.png" /><figcaption>FIGURE 6 Pulvinar (and whole-brain) activations in FACE (F+ & F−) audio-visual and unimodal visual tasks. (a) Whole-brain activation maps of the FACE context cumulated over both monkeys, for auditory congruent (red) and audio-visual congruent (blue) contrasts versus fixation. Axial, sagittal and coronal view are shown, zooming on the pulvinar. The activation outline in red and blue corresponds to activation thresholds at the level of significance *p* < .001 uncorrected, *t* score 3.09.</figcaption></figure>

(b) Whole-brain activation maps of the FACE context cumulated over both monkeys, audio-visual congruent condition of the audio-visual task (blue) and visual condition (aggressive + lipsmack faces) of the unimodal visual task (green).

<figure><img src="image_9.png" /><figcaption>FIGURE 7 Percentage of signal change in selected pulvinar regions of interest (ROIs), the unimodal visual task (a) and the audio-visual task (b). Statistical differences relative to fixation and between conditions are indicated as follows: ***, *p* < .001; **, *p* < .01 (Wilcoxon non-parametric test); see Table S2 and associated note for quantitative effect sizes. Selected ROIs are inferior pulvinar (PuI, bilateral), medial pulvinar (PuM, bilateral), anterior pulvinar (PuA, bilateral) and lateral pulvinar (PLdm, dorso-medial lateral pulvinar and PLvl, ventrolateral lateral pulvinar, both on the left hemisphere). ROIs defined in the audio-visual task are in white. ROIs defined in the unimodal visual task are in grey. V: visual vs fixation; AC: auditory congruent versus fixation and VAC: audio-visual congruent vs fixation. PuM: medial pulvinar; PuA: anterior pulvinar; PuI: inferior pulvinar; PLvl: ventrolateral lateral pulvinar; PLdm: dorso-medial lateral pulvinar. L_!: left hemisphere; R_!: right hemisphere.</figcaption></figure>
$p = .007$; PuA: $Z = 1.86$; $p = .062$). In contrast, only the ventro-lateral pulvinar ROI (PLvl) retains this significant visual response in the visual only condition of the audio-visual task, although both tasks involve the same visual conditions (Figure 7b; $Z = 2.1$, $p = .02$). These results suggest that pulvinar visual responses (except for PLvl) are not fully driven by sensory input and are modulated by general task context.

PLvl also retains significant %SC in the visuo-auditory condition (Figure 7b, VAC: $Z = 2.4$, $p = .01$) but not in the AC condition ($Z = 1.16$, $p = .24$), suggesting that the VAC significance is driven by the visual component of the VAC condition. The inferior and anterior pulvinar ROIs contrast with PLvl and appear to be driven by auditory stimulation as their %SC is significant for both the auditory (Figure 7b, AC: PuI: $Z = 3.2$, $p = .0013$; PuA: $Z = 2.6$, $p < .01$) and visuo-auditory condition (VAC: PuI: $Z = 2.3$, $p = .011$; PuA: $Z = 4.37$, $p < .001$) but not in the visual condition (V: PuI: $Z = .87$, $p = .38$; PuA: $Z = 1.16$, $p = .24$). It is unclear whether PLvl will respond to auditory stimulations. Likewise, it is unclear if the PuI and PuA auditory activations are specific to the audio-visual context. These two points remain to be investigated in a unimodal auditory task, that is a task mirroring the unimodal visual task and containing only auditory stimuli. Medial pulvinar ROI (PuM) stands out in that is activated by all of the visual conditions whether in the unimodal visual task (Figure 7a, V: $Z = 2.6$, $p = .007$) or the audio-visual task (Figure 7b, V: $Z = 2.04$, $p = .04$), the auditory condition of the audio-visual task (Figure 7b, AC: $Z = 2.6$, $p < .01$) and the audio-visual condition of the audio-visual task (Figure 7b, VAC: $Z = 5.25$, $p < .001$). Notably, the %SC in the VAC condition is significantly higher than the %SC in the V and AC conditions of the audio-visual task (Friedman non-parametric test, $X_{2(4)} = 19.26$, $p < .001$, $n = 80$, post hoc: V-VAC: $Z = 1.6$, $p < .01$, AC-VAC: $Z = 1.2$, $p = .02$), strongly suggesting that the medial pulvinar, PuM ROI, in contrast with the other ROIs, is integrating visual and auditory information.

## 4 | DISCUSSION

Overall, three key subcortical structures contribute to some aspect of audio-visual association of functionally significant communicative signals: the putative dorsolateral amygdala, claustrum and pulvinar. The activation of these structures depended on the meaning of the auditory stimuli relative to the visual context. They were activated by auditory stimuli when these were congruent in meaning to the visual context but not when they were

incongruent. This thus reveals a cross-modal audiovisual modulation in these regions. This possibly sheds new light on the functional organization of this subcortical nucleus. Further investigation in humans and non-human primates are needed to further consolidate these observations, including by manipulating context in different ways, as well as to further study the precise functional connectivity of the pulvinar with the cortex (Froesel et al., 2023). In addition to this cross-modal associations of communicative signals, the medial part of the pulvinar and the dorsolateral amygdala show a greater activation to the bimodal audio-visual stimulation relative to each of the visual and auditory unimodal stimulations. They therefore demonstrate multisensory integration similar to what we report in the STS in a previous study using the same data (Froesel et al., 2022). It is worth noting that the putative dorsolateral amygdala, claustrum and pulvinar correspond to the three subcortical regions activated by stimulation of face patch, that is, cortical region selective to responsive to face presentation (Tsao et al., 2008) stimulations. It has been proposed that these regions correspond to bottlenecks for the communication between face patches (Moeller et al., 2008) and consequently in face processing. This interpretation is supported by a tracer studies showing that individual patches receive input from these three subcortical structures (Grimaldi et al., 2016) as well as by resting-state fMRI, alone or in association with single cell recording studies (Schwiedrzik et al., 2015; Zaldivar et al., 2022). Based on our study and the previous literature cited above, the putative dorsolateral amygdala, claustrum and pulvinar are thus a part of a subcortical network involved in face processing. In addition, the amygdala has been shown to be involved in the guidance of social behaviours when specifically involving social interactions (Adolphs & Spezio, 2006). On the other hand, the macaque monkey pulvinar has been shown to respond to visual social stimuli involving faces depicting emotions, even when viewing human faces (Maior et al., 2010) or abstract representations of a face (Nguyen et al., 2013). In this study, we extend their role to the processing of species-specific vocalizations and their association with socioemotional visual information. The degree of specificity is not yet established, requiring tests with other species' vocalizations, facial expressions and social scenes.

### 4.1 | Audio-visual association of social stimuli in the dorsolateral amygdala, the claustrum and the pulvinar

The dorsolateral amygdala and claustrum are activated by all visual, auditory congruent and audio-visual
(congruent and incongruent) stimuli where auditory congruence or incongruence is defined by the visual context set in each presentation. Both these subcortical structures thus follow a response pattern that is similar to that observed at the cortical level (Froesel et al., 2022). In addition, they both are also activated by faces during a unimodal visual task, including faces with high emotional content. This pattern, combined with evidence from prior studies of a selectivity to emotional facial expressions (Fusar-Poli et al., 2009; Wang et al., 2017; Williams et al., 2004), suggests that both structures contribute to context-based social and emotional processing in coordination with the face and voice patches described in Froesel et al. (2022).

The contribution of the amygdala to the processing of social stimuli, whether visual (Hadj-Bouziane et al., 2008, 2012; Nakamura et al., 1992; Pessoa et al., 2006; Sergerie et al., 2008; Todorov, 2012) or auditory (Domínguez-Borràs et al., 2019; Gadziola et al., 2012; Gothard, 2020; Morrow et al., 2019), has been extensively studied by others. Here, we further show that this processing is context dependent, such that a particular stimulus either activates or inactivates the amygdala depending upon the context. Additionally, we show that like the STS (Froesel et al., 2022), the putative dorsolateral amygdala also plays an important role in multisensory integration (see also Ross et al., 2022). In this structure, multisensory integration, defined by the minimal criterion of percent signal change between bimodal and unimodal conditions, is only observed when the visual stimulations are social scenes. This is also the case in the STS (Froesel et al., 2022). This may be due to the fact that these static social scenes were more ambiguous than faces. In this context, associating the auditory information might help resolve this ambiguity. These activations are located in the dorsal and lateral part of the nucleus, close to the claustrum, in a location already described previously in the literature (Hadj-Bouziane et al., 2008, 2012; Schwiedrzik et al., 2015).

The claustrum is, due to its connectivity with inputs from limbic areas and many other cortical and subcortical areas, well positioned as a hub associating sensory and limbic information in order to influence attention via its output to the frontal cortex (see for review Smith et al., 2020). It has recently been hypothesized that in addition to its involvement in the coordination of slow wave sleep, it could serve as a limbic-sensory-motor interface. The claustrum is proposed to integrate limbic and sensory information to guide and sustain attention towards behaviourally relevant and salient stimuli. Electrophysiological recordings on macaques demonstrate a segregation of the sensory responses within the claustrum (Remedios et al., 2010). The central part is

responsive to auditory stimuli; whereas, the ventral part is responsive to visual stimuli. The claustrum is thus described as a multiple sensory structure but not as a multisensory integrator. Last, Remedios and colleagues describe visual responses in the ventral claustrum and auditory responses in the central claustrum (Remedios et al., 2010). While we did not specifically delineate ventral and central claustrum in our images, visual claustrum activations in both audio-visual and unimodal visual contexts appear to be more ventral than the auditory activations. This will have to be further confirmed experimentally. Overall, the activation of the claustrum during vocalizations is strongly modulated by the visual context, as this region is activated when the vocalizations are congruent with the visual context but not when they are incongruent. Moreover, as no multisensory integration was found in this region, this suggests that the perception of vocalization in the claustrum is modulated by context-dependent visual association based on semantic information even in the absence of multisensory integration.

The contribution of the pulvinar to audio-visual association is more complex. Indeed, cross-modal association was observed in the anterior, inferior and medial subnuclei, the auditory activations being driven by the task context set by the visual stimuli. However, the medial pulvinar was the only subnucleus responsive to both auditory and visual stimulation and also the only one that presented multisensory integration. The contribution of the pulvinar to the processing of social auditory and visual stimuli was thus more heterogeneous than the contribution of the putative dorsolateral amygdala and the claustrum.

## 4.2 | Task-dependent sensory gradient in the pulvinar

We show a task-dependent sensory gradient such that auditory activations are located anteriorly, visual activations are located laterally and posteriorly and audio-visual activations are located medially. In Figure 8, we summarize the patterns of activation in the pulvinar. Several points are worth raising about these patterns. Sensory modalities associated with each pulvinar subdivision are defined based on a statistically significant percentage of signal change for that modality relative to the fixation baseline but does not include inference about relative strength of one modality relative to the other. First, a global sensory gradient can be seen, with the anterior pulvinar being dedicated to auditory processing, the medial part to audio-visual and the lateral part of the nucleus to visual. Second, the dorso-medial lateral
<figure><img src="image_10.png" /><figcaption>**FIGURE 8** Summary of the activations within the pulvinar as a function of the task and sensory of stimulation. The circle size is largest in the regions of nucleus that present a higher activation. All circles correspond to significant activations or %SC. PuA: anterior pulvinar; PuM: medial pulvinar; PuI: inferior pulvinar; PLdm: dorso-medial lateral pulvinar; PLvl: ventro-lateral lateral pulvinar.</figcaption></figure>

pulvinar (PLdm) is highly modulated by the task context and is only activated in a unimodal visual task and not in an audio-visual association task. Third, only the medial pulvinar and the ventral lateral pulvinar respond to visual social stimuli irrespective of context. It is also worth noting that this gradient does not follow the classical sub divisions defined based on its cytoarchitectonic properties, that is inferior pulvinar, lateral pulvinar and medial pulvinar (Figures S4 and S5) (Gutierrez et al., 1995; Olszewski et al., 1952; Stepniewska & Kaas, 1997; Walker, 1938). Tracer studies directly addressing the multisensory properties of the pulvinar are scarce. A recent study in the marmoset deserves mention (Homman-Ludiye et al., 2020). Based on retrograde MRI-guided labelling of medial pulvinar, the authors show anatomical connectivity of the pulvinar with the temporal, parietal, frontal, cingulate and orbitofrontal regions, all of which are highly multisensory. Of relevance to our work, reciprocal anatomical connectivity is described between the medial pulvinar and the auditory parabelt (de la Mothe et al., 2012; Homman-Ludiye et al., 2020), possibly accounting for our AC and VAC activations in this part of the pulvinar. Though this work focuses on a New World Monkey, and ours on an Old World monkey, it is likely that similar connectivity has been preserved, if not enhanced, given the greater integration of visual signalling in terrestrial macaques. This proposed pulvinar sensory gradient in the pulvinar is a first attempt to describe the multisensory properties of this complex subcortical structure. This proposal is

inherently limited by the spatial resolution and signal-to-noise limitations of fMRI of subcortical structures. It will have to be confronted with higher resolution methods, including intracortical single neuronal recordings in NHP, as well as with comparative fMRI studies in humans.

## 4.3 | Pulvinar and face perception

The lateral part of monkey pulvinar has been proven to contain face responsive neurons, and the medial part is shown to be responsive to human facial expressions (Maior et al., 2010; Nguyen et al., 2013). Our observations from the unimodal visual task bring full support to this observation (Figure 8, green circles). Using resting-state analysis, it has been found that the dorsal pulvinar (i.e. dorsal lateral and medial pulvinar) is functionally connected with face patches (Schwiedrzik et al., 2015), defined as the cortical regions selectively responsive to faces as compared to other types of stimuli (Tsao et al., 2008). This is also the case for the ventral pulvinar. Generally speaking, the more anterior face patches connect to more anterior parts of the pulvinar thus defining an antero-posterior functional connectivity gradient between the STS and the pulvinar (Grimaldi et al., 2016). In addition, the stimulation of the two face patches AL and ML (respectively, anterior lateral and medial lateral) of the STS elicit activation in the inferior pulvinar (Moeller et al., 2008). Overall, this suggests that the entire
pulvinar is potentially responsive to faces, these face responses being recruited differentially as a function of task and context.

Our study showed that a task implicitly calling for an association between auditory and visual social information activates the medial as well as the ventro-lateral lateral pulvinar (PLvl). The unimodal visual task additionally activates the dorso-lateral and the inferior pulvinar. This is in agreement with the human pulvinar lesion literature, whereby a patient presenting with an entirely damaged unilateral pulvinar was not able to recognize fearful expressions in the contralesional field, and patients with damage limited to the anterior and lateral pulvinar showed no deficits in fear recognition (Ward et al., 2007). These results suggested that fear recognition is mediated by the medial pulvinar, a pattern supported by the observation that the entire nucleus is responsive to fear-related visual stimuli such as snakes (Almeida et al., 2015). We propose to link these observations to the role of the pulvinar in emotional regulation, as discussed next. Additionally, the medial pulvinar is the unique pulvinar nucleus that presents audio-visual integration. This result supports the idea that this subregion implements multisensory integration (Froesel et al., 2021). As is the case for the amygdala, recent studies report multisensory integration in this brain structure. A macaque single cell recording study describes sub-additive and suppressive multisensory integration in the medial pulvinar (Vittek et al., 2023), and a recent human fMRI study of the pulvinar region describes multisensory enhancement during natural narrative speech perception (Ross et al., 2022).

In our present study, activations during the unimodal social visual context are exclusively identified in the left pulvinar. In the pulvinar, a processing bias in the left pulvinar has already been demonstrated in humans, such that left pulvinar activations are more often reported than right pulvinar activations (Padmala et al., 2010). This has been interpreted in the light of the role of the pulvinar in the attentional function. In their monkey rsfMRI study, Schwiedrzik et al. determined that the pulvinar, the dorsolateral amygdala, hippocampus, caudate nucleus, claustrum and other sub cortical structures are functionally connected with face patches (Schwiedrzik et al., 2015). In addition, their left pulvinar activations included significantly more voxels than the right activations. A left hemisphere bias for processing species-specific vocalization was reported in a field study of rhesus monkeys (Hauser & Andersson, 1994). The results, together with the fact that the SNR is not higher on the left than on the right (see Figure S5), raise the possibility of a functional lateralization in the monkey pulvinar. This will have to be further explored.

## 4.4 | Cortical-subcortical network involved in cross-modal audiovisual association of social stimuli

In Froesel et al. (2022), we characterized cortical activations underlying the association of social visual and auditory stimuli based on their semantic content or meaning in the broad sense. Cortical regions were active when visual and auditory information were congruent but inactive when this information was incongruent. This audio-visual, semantic association with contextual information relied on a core cortical functional network involving the STS and the LS. LS ROIs had a preference for auditory and audio-visual congruent stimuli; whereas, STS ROIs responded equally to auditory, visual and audio-visual congruent stimuli. Multisensory integration was only identified in the STS. These cortical regions together with the dorsolateral amygdala, the claustrum and the medial pulvinar thus belong to the same cortical-subcortical network involved in cross-modal audiovisual association of social stimuli (Figure 9). Ross et al. (2022) propose that multisensory enhancement in naturalistic contexts involving understanding of semantics recruits more than the typical multisensory network, notably including the amygdala and the pulvinar as described here. Given the fact that the medial pulvinar is highly connected with the limbic system and areas involved in the regulation of emotions such as the anterior cingulate cortex, the temporal cortex, the temporo-parietal junction, the insula, the frontal parietal opercular cortex (Rosenberg et al., 2009; Yeterian & Pandya, 1997), together with the amygdala with which it is also connected (Jones & Burton, 1976), these structures are proposed to coordinate cortical networks to evaluate the biological significance of affective visual stimuli (Pessoa, 2010b). Their recruitment during audio-visual association based on socioemotional context as described here supports this hypothesis.

In the cortical-subcortical network involved in cross-modal audiovisual association of social stimuli described in Figure 9, three out of the four functional nodes implement multisensory integration such that the activation in response to the audiovisual condition resulted in significantly higher activations than the highest unimodal response. The claustrum was the only node that did not show multisensory integration properties. This possibly suggests that the claustrum plays a distinct role in the processing of audiovisual social stimuli relative to the other nodes of the network it coactivates with.

It is unclear why we do not observe activations in the superior colliculus with the cross-modal association network described in Figure 9. Indeed the superior colliculus has been reported to be connected with both the amygdala and the medial pulvinar and to integrate sensory
<figure><img src="image_11.png" /><figcaption>FIGURE 9 Cortical-subcortical network involved in the cross-modal association of faces and social scenes with vocalizations sharing the same meaning. The nodes of this network are activated by unimodal visual stimuli (white), unimodal auditory stimuli (dark grey) and by audiovisual stimuli. All display multisensory integration (intermediate grey), except the claustrum. STS: superior temporal sulcus; LS: lateral sulcus; AMG: dorsolateral amygdala; Claus: claustrum; PuA: anterior pulvinar; PuM: medial pulvinar; PuL: lateral pulvinar; PuI: inferior pulvinar.</figcaption></figure>

information from multiple modalities (Froesel et al., 2021; Meredith et al., 1987). Based on previous research, it is also hypothesized to provide sensory input to a subcortical network involved in unconscious emotion perception and composed of the superior colliculus, the medial and lateral pulvinar and the amygdala (Almeida et al., 2015; Tamietto et al., 2012). The absence of activation of superior colliculus in the present study could be due to a lower SNR in this deep brain region (Figures S2 and S6). Alternatively, it could be that the specific stimuli used in our study activated a different functional network than the one described above (see Figure 9). Indeed, the subcortical regions described in the present work co-activate with temporal regions closely matching face patches (Froesel et al., 2022), i.e., cortical regions selectively responsive to faces as compared to other types of stimuli (Tsao et al., 2008). Grimaldi et al. (2016) show input to these face patches from the pulvinar, the amygdala and the claustrum, thus directly supporting our observations of a functional co-activation of these subcortical regions with the temporal face patches. However, they do not identify direct projections from the superior colliculus to the temporal face patches, possibly

hinting at the fact that this structure belongs to a different subcortical network not activated by our paradigm.

## 5 | CONCLUSION

In this study, we propose that all of the claustrum, pulvinar and putative dorsolateral amygdala play an essential role in cross-modal association communicatively significant information but that only the dorsolateral amygdala and the medial pulvinar further implement multisensory integration. In addition, we provide evidence towards a sensory audio-visual processing gradient within the pulvinar that is context dependent. We propose that these three subcortical structures are part of a supramodal network that modulates sensory perception as a function of the social context, independently of the sensory modality.

## AUTHOR CONTRIBUTIONS

**Mathilda Froesel:** Conceptualization; data curation; formal analysis; investigation; methodology; resources; validation; visualization; writing—original draft; writing—review and editing. **Maëva Gacoin:** Data curation;
resources; writing—review and editing. **Simon Clavagnier:** Methodology. **Marc Hauser:** Resources; writing—review and editing. **Quentin Goudard:** Methodology; resources; software. **Suliann Ben Hamed:** Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing—original draft; writing—review and editing.

## ACKNOWLEDGEMENTS

S.B.H. was funded by the French National Research Agency (ANR) ANR-16-CE37-0009-01 grant and the LABEX CORTEX funding (ANR-11-LABX-0042) from the Université de Lyon, within the programme Investissements d'Avenir (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR). We thank Fidji Francioly and Laurence Boes for animal care, Julian Amengual and Justine Cléry for their rich scientific exchanges during data collection and analyses, Franck Lamberton and Danièle Ibarrola for their MRI methodological support and Holly Rayson for her help on visual stimuli collection. We thank Serge Pinède for technical assistance on the project.

## CONFLICT OF INTEREST STATEMENT

The authors declare no competing interests.

## PEER REVIEW

The peer review history for this article is available at https://www.webofscience.com/api/gateway/wos/peer-review/10.1111/ejn.16328.

## DATA AVAILABILITY STATEMENT

The codes and data that support the findings of this study are available from the corresponding author upon reasonable request. The code is still being used for other purposes and cannot be made publicly available at this time.

## ORCID

Suliann Ben Hamed https://orcid.org/0000-0003-1510-7284

## REFERENCES

Adolphs, R., & Spezio, M. (2006). Role of the amygdala in processing visual social stimuli. In S. Anders, G. Ende, M. Junghoffer, J. Kissler, & D. Wildgruber (Eds.), *Progress in brain research, understanding emotions* (pp. 363–378). Elsevier. https://doi.org/10.1016/S0079-6123(06)56020-0

Almeida, I., Soares, S. C., & Castelo-Branco, M. (2015). The distinct role of the amygdala, superior colliculus and Pulvinar in processing of central and peripheral snakes. *PLoS ONE*, *10*, e0129949. https://doi.org/10.1371/journal.pone.0129949

Archakov, D., DeWitt, I., Kuśmierek, P., Ortiz-Rios, M., Cameron, D., Cui, D., Morin, E. L., VanMeter, J. W., Sams, M., Jääskeläinen, I. P., & Rauschecker, J. P. (2020). Auditory representation of learned sound sequences in motor regions of the macaque brain. *Proceedings of the National Academy of Sciences*, *117*, 15242–15252. https://doi.org/10.1073/pnas.1915610117

Autio, J. A., Glasser, M. F., Ose, T., Donahue, C. J., Bastiani, M., Ohno, M., Kawabata, Y., Urushibata, Y., Murata, K., Nishigori, K., Yamaguchi, M., Hori, Y., Yoshida, A., Go, Y., Coalson, T. S., Jbabdi, S., Sotiropoulos, S. N., Kennedy, H., Smith, S., ... Hayashi, T. (2020). Towards HCP-style macaque connectomes: 24-channel 3T multi-array coil, MRI sequences and preprocessing. *NeuroImage*, *215*, 116800. https://doi.org/10.1016/j.neuroimage.2020.116800

Avillac, M., Ben Hamed, S., & Duhamel, J.-R. (2007). Multisensory integration in the ventral intraparietal area of the macaque monkey. *The Journal of Neuroscience*, *27*, 1922–1932. https://doi.org/10.1523/JNEUROSCI.2646-06.2007

Avillac, M., Olivier, E., Denève, S., Ben Hamed, S., & Duhamel, J.-R. (2004). Multisensory integration in multiple reference frames in the posterior parietal cortex. *Cognitive Processing*, *5*, 159–166. https://doi.org/10.1007/s10339-004-0021-3

Barraclough, N. E., & Perrett, D. I. (2011). From single cells to social perception. *Philosophical Transactions of the Royal Society B: Biological Sciences*, *366*, 1739–1752. https://doi.org/10.1098/rstb.2010.0352

Beauchamp, M. S. (2005). See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex. *Curr. Opin. Neurobiol Cognitive Neuroscience*, *15*, 145–153.

Belin, P., Fecteau, S., Charest, I., Nicastro, N., Hauser, M. D., & Armony, J. L. (2007). Human cerebral response to animal affective vocalizations. *Proceedings of the Royal Society B: Biological Sciences* *275*, *275*, 473–481. https://doi.org/10.1098/rspb.2007.1460

Cléry, J., Guipponi, O., Odouard, S., Pinède, S., Wardak, C., & Ben Hamed, S. (2017). The prediction of impact of a looming stimulus onto the body is subserved by multisensory integration mechanisms. *The Journal of Neuroscience*, *37*, 10656–10670. https://doi.org/10.1523/JNEUROSCI.0610-17.2017

Cohen, Y. E., Theunissen, F., Russ, B. E., & Gill, P. (2007). Acoustic features of rhesus vocalizations and their representation in the ventrolateral prefrontal cortex. *Journal of Neurophysiology*, *97*, 1470–1484. https://doi.org/10.1152/jn.00769.2006

Cox, R. W. (1996). AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. *Computers and Biomedical Research*, *29*(3), 162–173. https://doi.org/10.1006/cbmr.1996.0014

de la Mothe, L. A., Blumell, S., Kajikawa, Y., & Hackett, T. A. (2012). Thalamic connections of auditory cortex in marmoset monkeys: Lateral belt and parabelt regions. *The Anatomical Record: Advances in Integrative Anatomy and Evolutionary Biology*, *295*(5), 822–836.

Dolan, R. J., Morris, J. S., & de Gelder, B. (2001). Crossmodal binding of fear in voice and face. *Proceedings of the National
*Academy of Sciences*, **98**, 10006–10010. https://doi.org/10.1073/pnas.171288598

Domínguez-Borràs, J., Guex, R., Méndez-Bértolo, C., Legendre, G., Spinelli, L., Moratti, S., Frühholz, S., Mégevand, P., Arnal, L., Strange, B., Seeck, M., & Vuilleumier, P. (2019). Human amygdala response to unisensory and multisensory emotion input: No evidence for superadditivity from intracranial recordings. *Neuropsychologia*, **131**, 9–24. https://doi.org/10.1016/j.neuropsychologia.2019.05.027

Donahue, C. J., Sotiropoulos, S. N., Jbabdi, S., Hernandez-Fernandez, M., Behrens, T. E., Dyrby, T. B., Coalson, T., Kennedy, H., Knoblauch, K., Essen, D. C. V., & Glasser, M. F. (2016). Using diffusion tractography to predict cortical connection strength and distance: A quantitative comparison with tracers in the monkey. *The Journal of Neuroscience*, **36**, 6758–6770. https://doi.org/10.1523/JNEUROSCI.0493-16.2016

Fitzgerald, D. A., Angstadt, M., Jelsone, L. M., Nathan, P. J., & Phan, K. L. (2006). Beyond threat: Amygdala reactivity across multiple expressions of facial affect. *NeuroImage*, **30**, 1441–1448. https://doi.org/10.1016/j.neuroimage.2005.11.003

Freiwald, W. A. (2020). Social interaction networks in the primate brain. *Current Opinion in Neurobiology*, **65**, 49–58.

Froesel, M., Cappe, C., & Ben Hamed, S. (2021). A multisensory perspective onto primate pulvinar functions. *Neurosci. Biobehav. Rev*, **125**, 231–243. https://doi.org/10.1016/j.neubiorev.2021.02.043

Froesel, M., Clavagnier, S., Goudard, Q., Zhu, Q., Vanduffel, W., & Ben Hamed, S. (2023). Multiple focal pulvinar projection fields in the macaque cortex.

Froesel, M., Gacoin, M., Clavagnier, S., Hauser, M., Goudard, Q., & Ben Hamed, S. (2022). Socially meaningful visual context either enhances or inhibits vocalisation processing in the macaque brain. *Nature Communications*, **13**, 4886. https://doi.org/10.1038/s41467-022-32512-9

Froesel, M., Goudard, Q., Hauser, M., Gacoin, M., & Ben Hamed, S. (2020). Automated video-based heart rate tracking for the anesthetized and behaving monkey. *Scientific Reports*, **10**, 17940. https://doi.org/10.1038/s41598-020-74954-5

Fusar-Poli, P., Placentino, A., Carletti, F., Landi, P., Allen, P., Surguladze, S., Benedetti, F., Abbamonte, M., Gasparotti, R., Barale, F., Perez, J., McGuire, P., & Politi, P. (2009). Functional atlas of emotional faces processing: A voxel-based meta-analysis of 105 functional magnetic resonance imaging studies. *J. Psychiatry Neurosci. IPN*, **34**, 418–432.

Gadziola, M. A., Grimsley, J. M. S., Shanbhag, S. J., & Wenstrup, J. J. (2012). A novel coding mechanism for social vocalizations in the lateral amygdala. *Journal of Neurophysiology*, **107**, 1047–1057. https://doi.org/10.1152/jn.00422.2011

Ghazanfar, A. A., & Logothetis, N. K. (2003). Neuroperception: Facial expressions linked to monkey calls. *Nature*, **423**, 937–938. https://doi.org/10.1038/423937a

Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., & Logothetis, N. K. (2005). Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. *The Journal of Neuroscience*, **25**, 5004–5012. https://doi.org/10.1523/JNEUROSCI.0799-05.2005

Ghazanfar, A. A., & Santos, L. R. (2004). Primate brains in the wild: The sensory bases for social interactions. *Nature Reviews. Neuroscience*, **5**, 603–616. https://doi.org/10.1038/nrn1473

Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Andersson, J. L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J. R., Van Essen, D. C., & Jenkinson, M. (2013). The minimal preprocessing pipelines for the human connectome project. *NeuroImage*, **80**, 105–124. https://doi.org/10.1016/j.neuroimage.2013.04.127

Gothard, K. M. (2020). Multidimensional processing in the amygdala. *Nature Reviews. Neuroscience*, **21**, 565–575. https://doi.org/10.1038/s41583-020-0350-y

Gothard, K. M., Battaglia, F. P., Erickson, C. A., Spitler, K. M., & Amaral, D. G. (2007). Neural responses to facial expression and face identity in the monkey amygdala. *Journal of Neurophysiology*, **97**, 1671–1683. https://doi.org/10.1152/jn.00714.2006

Gouzoules, S., Gouzoules, H., & Marler, P. (1984). Rhesus monkey (Macaca mulatta) screams: Representational signalling in the recruitment of agonistic aid. *Animal Behaviour*, **32**, 182–193. https://doi.org/10.1016/S0003-3472(84)80336-X

Grimaldi, P., Saleem, K. S., & Tsao, D. (2016). Anatomical connections of the functionally defined “face patches” in the macaque monkey. *Neuron*, **90**, 1325–1342. https://doi.org/10.1016/j.neuron.2016.05.009

Gutierrez, C., Yaun, A., & Cusick, C. G. (1995). Neurochemical subdivisions of the inferior pulvinar in macaque monkeys. *The Journal of Comparative Neurology*, **363**, 545–562. https://doi.org/10.1002/cne.903630404

Hadj-Bouziane, F., Bell, A. H., Knusten, T. A., Ungerleider, L. G., & Tootell, R. B. H. (2008). Perception of emotional expressions is independent of face selectivity in monkey inferior temporal cortex. *Proceedings of the National Academy of Sciences*, **105**, 5591–5596. https://doi.org/10.1073/pnas.0800489105

Hadj-Bouziane, F., Liu, N., Bell, A. H., Gothard, K. M., Luh, W.-M., Tootell, R. B. H., Murray, E. A., & Ungerleider, L. G. (2012). Amygdala lesions disrupt modulation of functional MRI activity evoked by facial expression in the monkey inferior temporal cortex. *Proceedings of the National Academy of Sciences*, **109**, E3640–E3648. https://doi.org/10.1073/pnas.1218406109

Hartig, R., Glen, D., Jung, B., Logothetis, N. K., Paxinos, G., Garzavillarreal, E. A., Messinger, A., & Evrard, H. C. (2021). The Subcortical Atlas of the Rhesus Macaque (SARM) for neuroimaging. *NeuroImage*, **235**, 117996. https://doi.org/10.1016/j.neuroimage.2021.117996

Hauser, M. D., & Andersson, K. (1994). Left hemisphere dominance for processing vocalizations in adult, but not infant, rhesus monkeys: Field experiments. *Proceedings of the National Academy of Sciences of the United States of America*, **91**, 3946–3948. https://doi.org/10.1073/pnas.91.9.3946

Hauser, M. D., Evans, C. S., & Marler, P. (1993). The role of articulation in the production of rhesus monkey, Macaca mulatta, vocalizations. *Animal Behaviour*, **45**, 423–433. https://doi.org/10.1006/anbe.1993.1054

Hauser, M. D., & Marler, P. (1993). Food-associated calls in rhesus macaques (*Macaca mulatta*): I. Socioecological factors. *Behav. Ecol.*, **4**, 194–205. https://doi.org/10.1093/beheco/4.3.194

Hauser, M. D., & Ybarra, M. S. (1994). The role of lip configuration in monkey vocalizations: Experiments using xylocaine as a nerve block. *Brain and Language*, **46**, 232–244. https://doi.org/10.1006/brln.1994.1014
Homman-Ludiye, J., Mundinano, I. C., Kwan, W. C., & Bourne, J. A. (2020). Extensive connectivity between the medial pulvinar and the cortex revealed in the marmoset monkey. *Cerebral Cortex*, 30(3), 1797-1812. https://doi.org/10.1093/cercor/bhz203

Jenkinson, M., Beckmann, C. F., Behrens, T. E., Woolrich, M. W., & Smith, S. M. (2012). Fsl. Neuroimage, 62(2), 782-790. https://doi.org/10.1016/j.neuroimage.2011.09.015

Jones, E. G., & Burton, H. (1976). A projection from the medial pulvinar to the amygdala in primates. *Brain Research*, 104, 142-147. https://doi.org/10.1016/0006-8993(76)90654-5

Kolster, H., Janssens, T., Orban, G. A., & Vanduffel, W. (2014). The retinotopic organization of macaque occipitotemporal cortex anterior to V4 and caudoventral to the middle temporal (MT) cluster. *Journal of Neuroscience: the Official Journal of the Society for Neuroscience*, 34, 10168-10191. https://doi.org/10.1523/JNEUROSCI.3288-13.2014

Kuraoka, K., & Nakamura, K. (2007). Responses of single neurons in monkey amygdala to facial and vocal emotions. *Journal of Neurophysiology*, 97, 1379-1387. https://doi.org/10.1152/jn.00464.2006

Leite, F. P., Tsao, D., Vanduffel, W., Fize, D., Sasaki, Y., Wald, L. L., Dale, A. M., Kwong, K. K., Orban, G. A., Rosen, B. R., Tootell, R. B. H., & Mandeville, J. B. (2002). Repeated fMRI using iron oxide contrast agent in awake, behaving macaques at 3 tesla. *NeuroImage*, 16, 283-294. https://doi.org/10.1006/nimg.2002.1110

Leonard, C. M., Rolls, E. T., Wilson, F. A., & Baylis, G. C. (1985). Neurons in the amygdala of the monkey with responses selective for faces. *Behavioural Brain Research*, 15, 159-176. https://doi.org/10.1016/0166-4328(85)90062-2

Livneh, U., Resnik, J., Shohat, Y., & Paz, R. (2012). Self-monitoring of social facial expressions in the primate amygdala and cingulate cortex. *Proceedings of the National Academy of Sciences of the United States of America*, 109, 18956-18961. https://doi.org/10.1073/pnas.1207662109

Maior, R. S., Hori, E., Tomaz, C., Ono, T., & Nishijo, H. (2010). The monkey pulvinar neurons differentially respond to emotional expressions of human faces. *Behavioural Brain Research*, 215, 129-135. https://doi.org/10.1016/j.bbr.2010.07.009

Meredith, M. A., Nemitz, J. W., & Stein, B. E. (1987). Determinants of multisensory integration in superior colliculus neurons. *I. Temporal factors*. *J. Neurosci.*, 7, 3215-3229.

Moeller, S., Freiwald, W. A., & Tsao, D. Y. (2008). Patches with links: A unified system for processing faces in the macaque temporal lobe. *Science*, 320, 1355-1359. https://doi.org/10.1126/science.1157436

Morrow, J., Mosher, C., & Gothard, K. (2019). Multisensory neurons in the primate amygdala. *Journal of Neuroscience: the Official Journal of the Society for Neuroscience*, 39, 3663-3675. https://doi.org/10.1523/JNEUROSCI.2903-18.2019

Murray, M. M., & Wallace, M. T. (2012). *The neural bases of multisensory processes*. CRC Press/Taylor & Francis. https://doi.org/10.1201/b11092

Nakamura, K., Mikami, A., & Kubota, K. (1992). Activity of single neurons in the monkey amygdala during performance of a visual discrimination task. *Journal of Neurophysiology*, 67, 1447-1463. https://doi.org/10.1152/jn.1992.67.6.1447

Nguyen, M. N., Hori, E., Matsumoto, J., Tran, A. H., Ono, T., & Nishijo, H. (2013). Neuronal responses to face-like stimuli in the monkey pulvinar. *The European Journal of Neuroscience*, 37, 35-51. https://doi.org/10.1111/ejn.12020

Olszewski, J., Assistant Professor of Neuro-anatomy, D. of N., & Neuro-surgery, M.U. (1952). The thalamus of the Macaca, mulatta. An atlas for use with the stereotaxic instrument. Thalamus Macaca Mulatta Atlas Use Ster. Instrum.

Padmala, S., Lim, S.-L., & Pessoa, L. (2010). Pulvinar and affective significance: Responses track moment-to-moment stimulus visibility. *Frontiers in Human Neuroscience*, 4, 64. https://doi.org/10.3389/fnhum.2010.00064

Partan, S. R. (2013). Ten unanswered questions in multimodal communication. *Behavioral Ecology and Sociobiology*, 67, 1523-1539. https://doi.org/10.1007/s00265-013-1565-y

Partan, S. R., & Marler, P. (2005). Issues in the classification of multimodal communication signals. *The American Naturalist*, 166, 231-245. https://doi.org/10.1086/431246

Pessoa, L. (2010a). Emotion and cognition and the amygdala: From “what is it?” to “what’s to be done?”. *Neuropsychologia*, 48, 3416-3429. https://doi.org/10.1016/j.neuropsychologia.2010.06.038

Pessoa, L., Japee, S., Sturman, D., & Ungerleider, L. G. (2006). Target visibility and visual awareness modulate amygdala responses to fearful faces. *Cerebral Cortex*, 16, 366-375. https://doi.org/10.1093/cercor/bhi115

Pollick, F., Love, S., & Latinus, M. (2011). Cerebral correlates and statistical criteria of cross-modal face and voice integration. *Seeing and Perceiving*, 24, 351-367. https://doi.org/10.1163/187847511X584452

Remedios, R., Logothetis, N. K., & Kayser, C. (2010). Unimodal responses prevail within the multisensory claustrum. *The Journal of Neuroscience*, 30, 12902-12907. https://doi.org/10.1523/JNEUROSCI.2937-10.2010

Romanski, L. M. (2012). Integration of faces and vocalizations in ventral prefrontal cortex: Implications for the evolution of audiovisual speech. *Proceedings of the National Academy of Sciences*, 109, 10717-10724. https://doi.org/10.1073/pnas.1204335109

Romanski, L. M., Averbeck, B. B., & Diltz, M. (2005). Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. *Journal of Neurophysiology*, 93, 734-747. https://doi.org/10.1152/jn.00675.2004

Rosenberg, D. S., Mauguière, F., Catenoix, H., Faillenot, I., & Magnin, M. (2009). Reciprocal Thalamocortical connectivity of the medial Pulvinar: A depth stimulation and evoked potential study in human brain. *Cerebral Cortex*, 19, 1462-1473. https://doi.org/10.1093/cercor/bhn185

Ross, L. A., Molholm, S., Butler, J. S., Bene, V. A. D., & Foxe, J. J. (2022). Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation. *NeuroImage*, 263, 119598. https://doi.org/10.1016/j.neuroimage.2022.119598

Russ, B. E., Ackelson, A. L., Baker, A. E., & Cohen, Y. E. (2008). Coding of auditory-stimulus identity in the auditory non-spatial processing stream. *Journal of Neurophysiology*, 99, 87-95. https://doi.org/10.1152/jn.01069.2007

Schormans, A. L., & Allman, B. L. (2018). Behavioral plasticity of audiovisual perception: Rapid recalibration of temporal sensitivity but not perceptual binding following adult-onset hearing
loss. *Frontiers in Behavioral Neuroscience*, **12**, 256. https://doi.org/10.3389/fnbeh.2018.00256

Schormans, A. L., & Allman, B. L. (2023). An imbalance of excitation and inhibition in the multisensory cortex impairs the temporal acuity of audiovisual processing and perception. *Cerebral Cortex*, **33**, 9937–9953. https://doi.org/10.1093/cercor/bhad256

Schwiedrzik, C. M., Zarco, W., Everling, S., & Freiwald, W. A. (2015). Face patch resting state networks link face processing to social cognition. *PLoS Biology*, **13**, e1002245. https://doi.org/10.1371/journal.pbio.1002245

Sergerie, K., Chochol, C., & Armony, J. L. (2008). The role of the amygdala in emotional processing: A quantitative meta-analysis of functional neuroimaging studies. *Neuroscience and Biobehavioral Reviews*, **32**, 811–830. https://doi.org/10.1016/j.neubiorev.2007.12.002

Smith, J. B., Lee, A. K., & Jackson, J. (2020). The claustrum. *Current Biology*, **30**, R1401–R1406. https://doi.org/10.1016/j.cub.2020.09.069

Stein, B. E., & Meredith, M. A. (1993). *The merging of the senses, the merging of the senses*. The MIT Press.

Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., & Rowland, B. A. (2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. *Experimental Brain Research*, **198**, 113–126. https://doi.org/10.1007/s00221-009-1880-8

Stepniewska, I., & Kaas, J. H. (1997). Architectonic subdivisions of the inferior pulvinar in New World and Old World monkeys. *Visual Neuroscience*, **14**, 1043–1060. https://doi.org/10.1017/S0952523800011767

Stevenson, R. A., Ghose, D., Fister, J. K., Sarko, D. K., Altieri, N. A., Nidiffer, A. R., Kurela, L. R., Siemann, J. K., James, T. W., & Wallace, M. T. (2014). Identifying and quantifying multisensory integration: A tutorial review. *Brain Topography*, **27**, 707–730. https://doi.org/10.1007/s10548-014-0365-7

Stevenson, R. A., Kim, S., & James, T. W. (2009). An additive-factors design to disambiguate neuronal and areal convergence: Measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. *Experimental Brain Research*, **198**, 183–194. https://doi.org/10.1007/s00221-009-1783-8

Tamietto, M., Pullens, P., de Gelder, B., Weiskrantz, L., & Goebel, R. (2012). Subcortical connections to human amygdala and changes following destruction of the visual cortex. *Current Biology*, **22**, 1449–1455. https://doi.org/10.1016/j.cub.2012.06.006

Todorov, A. (2012). The role of the amygdala in face perception and evaluation. *Motivation and Emotion*, **36**, 16–26. https://doi.org/10.1007/s11031-011-9238-5

Tsao, D. Y., Moeller, S., & Freiwald, W. A. (2008). Comparing face patch systems in macaques and humans. *Proceedings of the National Academy of Sciences*, **105**, 19514–19519. https://doi.org/10.1073/pnas.0809662105

Tyll, S., Bonath, B., Schoenfeld, M. A., Heinze, H.-J., Ohl, F. W., & Noesselt, T. (2013). Neural basis of multisensory looming signals. *NeuroImage*, **65**, 13–22. https://doi.org/10.1016/j.neuroimage.2012.09.056

Vanduffel, W., Fize, D., Mandeville, J. B., Nelissen, K., Hecke, P. V., Rosen, B. R., Tootell, R. B. H., & Orban, G. A. (2001). Visual

motion processing investigated using contrast agent-enhanced fMRI in awake behaving monkeys. *Neuron*, **32**, 565–577. https://doi.org/10.1016/S0896-6273(01)00502-5

Vittek, A.-L., Juan, C., Nowak, L. G., Girard, P., & Cappe, C. (2023). Multisensory integration in neurons of the medial pulvinar of macaque monkey. *Cerebral Cortex*, **33**, 4202–4215. https://doi.org/10.1093/cercor/bhac337

Walker, A. E. (1938). The thalamus of the chimpanzee: IV. Thalamic projections to the cerebral cortex. *Journal of Anatomy*, **73**, 37–93.

Wang, S., Yu, R., Tyszka, J. M., Zhen, S., Kovach, C., Sun, S., Huang, Y., Hurlemann, R., Ross, I. B., Chung, J. M., Mamelak, A. N., Adolphs, R., & Rutishauser, U. (2017). The human amygdala parametrically encodes the intensity of specific facial emotions and their categorical ambiguity. *Nature Communications*, **8**, 14821. https://doi.org/10.1038/ncomms14821

Ward, R., Calder, A. J., Parker, M., & Arend, I. (2007). Emotion recognition following human pulvinar damage. *Neuropsychologia*, **45**, 1973–1978. https://doi.org/10.1016/j.neuropsychologia.2006.09.017

Werner, S., & Noppeney, U. (2010). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. *Cerebral Cortex*, **20**, 1829–1842. https://doi.org/10.1093/cercor/bhp248

Williams, M. A., Morris, A. P., McGlone, F., Abbott, D. F., & Mattingley, J. B. (2004). Amygdala responses to fearful and happy facial expressions under conditions of binocular suppression. *Journal of Neuroscience: the Official Journal of the Society for Neuroscience*, **24**, 2898–2904. https://doi.org/10.1523/JNEUROSCI.4977-03.2004

Yeterian, E. H., & Pandya, D. N. (1997). Corticothalamic connections of extrastriate visual areas in rhesus monkeys. *The Journal of Comparative Neurology*, **378**, 562–585. https://doi.org/10.1002/(SICI)1096-9861(19970224)378:4<562::AID-CNE10>3.0.CO;2-L

Zaldivar, D., Koyano, K. W., Ye, F. Q., Godlove, D. C., Park, S. H., Russ, B. E., Bhik-Ghanie, R., & Leopold, D. A. (2022). Brain-wide functional connectivity of face patch neurons during rest. *Proceedings of the National Academy of Sciences*, **119**, e2206559119. https://doi.org/10.1073/pnas.2206559119

## SUPPORTING INFORMATION

Additional supporting information can be found online in the Supporting Information section at the end of this article.

**How to cite this article:** Froesel, M., Gacoin, M., Clavagnier, S., Hauser, M., Goudard, Q., & Ben Hamed, S. (2024). Macaque claustrum, pulvinar and putative dorsolateral amygdala support the cross-modal association of social audio-visual stimuli based on meaning. *European Journal of Neuroscience*, **59**(12), 3203–3223. https://doi.org/10.1111/ejn.16328