[image] [image] Ressource protégée - Archive ouverte HAL Accéder directement au contenu

*[image] [image]*

• [Documentation](#)

• FR
Français (FR)

• Anglais (EN)
Se connecter

• [HAL science ouverte](#)

• Recherche
Loading...

## Recherche avancée

Information de documents

Titres

○ [Titres](#)

○ [Sous-titre](#)

○ Titre de l'ouvrage

○ Titre du volume (Série)

○ Champ de recherche par défaut (multicritères) + texte intégral des PDF

○ [Résumé](#)

○ Texte intégral indexé des documents PDF

○ [Mots-clés](#)

○ Type de document

○ Sous-type de document

○ Tous les identifiants du document

○ Identifiant HAL du dépôt

○ Langue du document (texte)

○ Pays (Texte)

○ Ville

○ À paraître (true ou false)

Ajouter

Auteur
1 Facial cues to anger affect meaning interpretation of subsequent spoken prosody

2 Caterina Petrone¹, Francesca Carbone¹,², Nicolas Audibert³, Maud-Champagne Lavau¹

3

4 ¹Aix-Marseille Université, CNRS, LPL, UMR 7309, Aix-en-Provence, France

5 ²University of Kent, Canterbury, UK

6 ³ Université Paris 3 Sorbonne Nouvelle, Laboratoire de Phonétique et Phonologie, UMR

7 7018, Paris, France

8

9

10 Corresponding author:

11 Caterina Petrone

12 Laboratoire Parole et Langage

13 5, av. Pasteur

14 13100 Aix-en-Provence

15 FRANCE

16 Email: caterina.petrone@univ-amu.fr

17 Tel: +33 (0) 6 19 67 76 39

18

19

20

21

22

23

24
## Auteur (multicritères)

○ Auteur (multicritères)

○ Auteur : Nom complet

○ Auteur : Nom de famille

○ Auteur : Prénom

○ Auteur : Complément de nom, deuxième prénom

○ Auteur : Organisme payeur

○ Auteur : IdHal (chaîne de caractères)

○ Auteur : Fonction

○ Auteur : personID (entier)

○ Auteur : Identifiant de l'organisme payeur

○ Auteur : Identifiant de la structure

○ Directeur de thèse

○ Éditeur

○ Éditeur scientifique

○ Éditeur de la série

## Ajouter

### Structure

### Structure (multicritères)

○ Structure (multicritères)

○ Structure : Acronyme

○ Structure : Nom

○ Structure : Code

○ Structure : Pays

○ Structure : Type

○ Structure : État dans le référentiel

○ Structure : Identifiant HAL de la structure

○ Structure/équipe de recherche : Pays

○ Structure/regroupement d'équipes : Pays

○ Structure/laboratoire : Pays

○ Structure/regroupement de laboratoires : Pays

○ Structure/institution : Pays

○ Structure/regroupement d'institutions : Pays

## Ajouter

## Autres
25 Abstract (199 words)

26 In everyday life, visual information often precedes the auditory one, hence influencing its
27 evaluation (e.g., seeing somebody's angry face makes us expect them to speak to us angrily).

28 By using the cross-modal affective paradigm, we investigated the influence of facial gestures
29 when the subsequent acoustic signal is emotionally unclear (neutral or produced with a
30 limited repertoire of cues to anger). Auditory stimuli spoken with angry or neutral prosody
31 were presented in isolation or preceded by pictures showing emotionally related or unrelated
32 facial gestures (angry or neutral faces). In two experiments, participants rated the valence and
33 emotional intensity of the auditory stimuli only. These stimuli were created from acted speech
34 from movies and delexicalized via speech synthesis, then manipulated by partially preserving
35 or degrading their global spectral characteristics. All participants relied on facial cues when
36 the auditory stimuli were acoustically impoverished; however, only a subgroup of participants
37 used angry faces to interpret subsequent neutral prosody. Thus, listeners are sensitive to facial
38 cues for evaluating what they are about to hear, especially when the auditory input is less
39 reliable. These results extend findings on face perception to the auditory domain and confirm
40 inter-individual variability in considering different sources of emotional information.

41

42 **Keywords:** emotional meaning, spoken prosody, facial gestures, cross-modal affective
43 priming, French.

44

45 Running Head: Visual and auditory cues to emotional meaning

46 Declarations of interest: none

47

48

49
Champ de recherche par défaut (multicritères)

○ <u>Champ de recherche par défaut (multicritères)</u>

○

○ <u>Revues (multicritères)</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Titre abrégé</u>

○ <u>Revue : Titre</u>

○ <u>Revue : Identifiant interne</u>

○ <u>Revue : Date de début de publication</u>

○ <u>Revue : ISSN électronique</u>

○ <u>Revue : ISSN</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Couleur dans SHERPA/RoMEO</u>

○ <u>Revue : État dans le référentiel</u>

○

○ <u>Colloque (multicritères)</u>

○ <u>Colloque : Titre</u>

○ <u>Colloque : Organisateur</u>

○ <u>Colloque : date de début (Année)</u>

○ <u>Colloque : date de fin (Année)</u>

○ <u>Conférence invitée (oui/non)</u>

○

○ <u>Projets ANR (multicritères)</u>

○ <u>Projet ANR : Acronyme</u>

○ <u>Projet ANR : Acronyme du programme</u>

○ <u>Projet ANR : Code décision (référence)</u>

○ <u>Projet ANR : Nom</u>

○ <u>Projet ANR : Identifiant interne</u>

○ <u>Projet ANR : État dans le référentiel</u>

○

○ <u>Projets européens (multicritères)</u>

○ <u>Projet européen : Acronyme</u>

○ <u>Projet européen : Identifiant de l'appel à projet</u>

○ <u>Projet européen : Référence</u>

○ <u>Projet européen : Nom</u>

○ <u>Projet européen : Date de fin</u>

○ <u>Projet européen : Financement</u>

○ <u>Projet européen : Date de début</u>

○ <u>Projet européen : État dans le référentiel</u>
50

# 1. Introduction

51

52

53

54

55

56

57

58

Human emotions are communicated in unimodal or multimodal ways, e.g., through the auditory modality (e.g., raising or lowering the pitch of an utterance, e.g., Scherer, 2003), the visual modality (e.g., smiling or frowning, e.g., Eckman, 1992), or both (e.g., Jessen and Kotz, 2013, 2015). In everyday life, visual information often precedes auditory information (Jessen and Kotz, 2013), thus facilitating language processing and comprehension. For instance, simply seeing somebody's angry facial expression (e.g., via typical facial gestures like lowering and drawing together the eyebrows) makes us expect them to speak to us angrily (Jessen and Kotz, 2013).

59

60

61

62

63

64

65

66

67

68

69

In the literature on multimodal emotion perception, the cross-modal affective priming paradigm has often been adopted to investigate to what extent facial gestures impact the subsequent perception of clear vocal expressions of emotions (in the sense of spoken prosodic patterns that unambiguously convey an emotional meaning via their acoustic cues). Here, we use this paradigm to explore how the visual modality (i.e., facial gestures associated with basic emotion expressions) influences the interpretation of subsequent spoken prosody, which varies in the amount of acoustic information conveying affective (angry) meaning. In particular, we target a specific effect already found in the literature on the perception of emotionally degraded or neutral faces (the “Kuleshov” effect, e.g., Mobbs et al., 2006; Calbi et al., 2017, 2019) to test its potential existence with (emotionally degraded or neutral) audio cues.

70

71

## 1.1. Multimodal perception of emotional prosody

72

73

74

Spoken prosody provides a powerful means to infer other people's emotions (e.g., Banse and Scherer, 1996) that vary in emotional intensity (i.e., strong vs. weak; henceforth 'intensity') and emotional valence (i.e., positive or negative). Multiple phonetic parameters are used to
○ Projet européen : Identifiant interne

○

○ Date de production : année

○ Date de mise en ligne : année

○ Date de publication : année

○ Date d'écriture : année

○ Date de modification du dépôt : année

○ Date de dépôt : année

○ Date de publication électronique : année

○

○ Collection HAL (multicritères)

○ Collection HAL : catégorie

○ Collection HAL : Code

○ Collection HAL : Nom

○ Collection HAL : Identifiant interne

○

○ Identifiant interne du contributeur/déposant

○ Nom complet du contributeur/déposant

○

○ Domaines

○ Domaine primaire

○ Domaine racine

○ Sous-domaine niveau 1

○ Sous-domaine niveau 2

○ Sous-domaine niveau 3

○

○ Statut du document

○ Version du document

○ Type de dépôt

○ Type de document

○ ISBN

○ Numéro - référence

○ Identifiant DOI

○

○ Classification

○ Audience

○ Vulgarisation

○ Comité de lecture - texte (oui ou non)

○ Actes de colloque

○ Référence interne
convey basic emotions (e.g., anger, happiness or sadness), such as fundamental frequency (f0), acoustic intensity, speech rate or voice quality-related characteristics like the harmonic-to-noise ratio, jitter and shimmer (e.g., Banse and Scherer, 1996; Gobl and Ní Chasaide, 2003; Scherer, 2003). For instance, (hot) anger is characterized in many languages by an increase in the f0 range, higher intensity, and a faster speech rate or harsh/tense voice compared to a neutral emotional state (e.g., Scherer, 2003; Gobl and Ní Chasaide, 2003). Language-specific phonological features (e.g., choice of specific pitch accents or changes in prosodic phrasing) may also play a role in conveying basic emotions (Cao et al., 2014).

In everyday life, listeners are exposed to multiple sources of emotional information (e.g., auditory and gestural/facial cues), which they may take into account when perceiving emotions (Paulmann and Pell, 2010). Multimodal emotion perception is more efficient than unimodal perception, as evidenced by faster reaction times, higher accuracy or higher emotional ratings (e.g., de Gelder et al., 1999; Pell, 2005; Zhang et al., 2018). A critical factor when investigating multimodal emotion perception is the timing between the sensory stimuli, e.g., whether emotional information from the auditory and the visual domain are temporally aligned or whether one stimulus precedes the other (Jessen and Kotz, 2013, 2015; Garrido-Vásquez et al., 2018).

Most of previous literature on multimodal emotion perception have focused on multimodal integration. In such studies, information from the auditory and visual modalities are presented simultaneously, with the aim of investigating how listeners integrate different sensory inputs into a coherent percept (e.g., Massaro and Egan, 1996; de Gelder et al., 1999, 2006; de Gelder and Vroomen, 2000). In particular, studies on audiovisual integration of prosody have shed considerable light on how auditory and visual information are used *together* for decisions concerning linguistic and paralinguistic contrasts (e.g., de Gelder and Vroomen, 2000; House
○ Financement

○ Collaborations

Ajouter

Lancer la recherche

Recherche experte (SolR)

Recherche experte (SolR)

Lancer la recherche

Vers la recherche avancée

• Déposer

X

X

X

Loading...

X

Le document que vous demandez est en access restreint
jusqu'au 2025-01-10 !

Revenir a la notice

Retourner à l'accueil

[image]

[image]

[image]

[image]

Contact

• Support

Ressources

• Documentation

• FAQ

• API
99 et al., 2001; Massaro and Beskow, 2002; Snrivasan and Massaro, 2003; Borràs-Comes and Prieto, 2011; Crespo Sendra et al., 2013; Buyok et al., 2022). As for emotion perception, de Gelder and Vroomen (2000) presented an utterance with neutral content which was rendered emotionally ambiguous by manipulating the duration, f0 range and f0 register of the whole utterance between happy and fearful. While the utterance was playing, a picture of a happy or fearful face was simultaneously displayed on the screen. Fearful faces biased the interpretation of the auditory stimuli more towards 'fear', with a stronger effect when the auditory stimuli were ambiguous between the two emotional categories. As de Gelder et al. (2000) argued, the simultaneous presentation of faces and voices pushed listeners to integrate the sensory inputs into a new 'gestalt', with effects mirroring linguistic phenomena of audiovisual integration like the McGurk effect (McGurk and MacDonald, 1976).

110 Some studies have also pointed to the existence of cross-modal affective priming effects for asynchronous sensory inputs, by which information from one sensory modality influences the processing and interpretation of a signal in another modality that comes into play later (e.g., Pell, 2005; Paulmann and Pell, 2010; Paulmann et al., 2012; Jessen and Kotz, 2013; Garrido-Vásquez et al., 2018). Jessen and Kotz (2013) claimed that affective priming is pervasive during multimodal emotion perception, where visual information from the face often precedes the auditory one. Such studies often adopted the cross-modal affective priming paradigm, by which a prime stimulus from one sensory modality is presented for a specified duration, followed by a target stimulus from another sensory modality. Hence, different from multimodal integration studies, visual and auditory information is presented consecutively rather than simultaneously, with the aim of evaluating the effect of one modality over another one. Thus, priming effects have been explained by psycholinguistic mechanisms of speech perception that are specific to asynchronous stimuli only, like spreading activation from the prime to the target or prime-target congruency check mechanisms (Pell, 2005). Pell (2005)
• OAI-PMH

• AuréHAL

## Informations

• À propos

• Données personnelles

• Mentions légales

• Accessibilité

• Conformité RGAA

## Questions juridiques

• Je publie, quels sont mes droits ?

• Loi pour une République numérique

• Stratégie de non-cession des droits

• SHERPA/RoMEO

## Portails

• Portails institutionnels

• HAL

• HAL SHS

• HAL Thèses

• MédiHAL

## CCSD

• CCSD

• Episciences

• Sciencesconf
used the cross-modal affective paradigm to study the impact of the preceding spoken prosody on subsequently presented faces. The spoken prosodic patterns and the faces were either combined in congruent pairs (e.g., happy spoken prosody followed by a happy face) or incongruent pairs (e.g., happy spoken prosody followed by an angry face). Emotion recognition was more accurate for congruent than for incongruent prime-target pairs. In the literature, there is also evidence that the processing of spoken emotional prosody is influenced by the preceding emotional face. Studies combining the cross-modal affective paradigm and ERP measurements have found that emotional congruency effects between vocal expressions and facial gestures can be observed very early in time, i.e., within the first 250 ms after the onset of the auditory stimulus (e.g., Pourtois et al., 2002; Paulmann et al., 2009; Jessen and Kotz, 2013; Garrido-Vásquez et al., 2018, *inter alia*). Garrido-Vásquez et al. (2018) presented angry, happy or neutral faces followed by pseudo-sentences spoken with angry or happy prosody. They found that prime-target congruency affected the N100 and P200 components, indicating faster and more efficient processing of the spoken emotional prosody. Neurosciences literature has interpreted such effects have pointed to the existence of a process of multimodal emotion perception, which is very different from multimodal integration: cross-modal prediction (Jessen and Kotz, 2013). Specifically, facial gestures would help listeners generating predictions about certain characteristics of a subsequent sound (e.g., in terms of its temporal predictability and content), hence facilitating auditory information processing (Jessen and Kotz, 2013).

A limitation of the literature employing the cross-modal affective paradigm is that it has mostly focused on the perception of clear vocal expressions of emotion. To our knowledge, no studies have explored the effects of facial gestures when the subsequent vocal expressions of emotion are less clear. This question will be addressed in this paper.
148 **1.2. The Kuleshov effect**

149 Researchers on face perception agree that basic emotions are signaled via distinctive facial
150 gestures or ‘action units’ (Ekman, 1992; Ekman et al., 2002). For instance, (hot) anger is
151 typically signaled by lowering and drawing together the eyebrows and widening the eyes,
152 while happiness is typically signaled by smiling (Ekman et al., 2002). Research has revealed
153 an interplay between the clarity of the facial gestures associated to emotions and the
154 emotional context accompanying such gestures (e.g., an emotional scene or situation). When
155 facial gestures are clear and prototypical of a basic emotion, the emotion is read out from the
156 face with no influence of the context. When facial gestures are unclear, people rely more on
157 contextual information to infer emotional meanings (e.g., Ekman et al., 1982; Carroll and
158 Russell, 1996; Aviezer et al., 2008).

159 The trade-off relationship between contexts and faces has been corroborated by experimental
160 findings on the influence of emotional scenes in the interpretation of faces which are
161 emotionally unclear (i.e., with facial gestures that are not prototypical of a basic emotion) or
162 neutral (i.e., with no visible facial movements). This line of research is mostly based on the
163 Kuleshov effect, named after an early 20th century Soviet filmmaker (Mobbs et al., 2006;
164 Calbi et al., 2017, 2019; Mullenix et al., 2019, *inter alia*). Lev Kuleshov alternated a close-up
165 of a neutral face with pictures of different emotional scenes (e.g., a dead woman or a little girl
166 playing). The face was perceived as expressing an emotion congruent with the preceding
167 context (sadness and happiness, respectively, cf. Calbi et al., 2017).

168 A first attempt to replicate the original study (Prince and Hensley 1992) did not find any
169 evidence of the Kuleshov effect. Experimental evidence for the Kuleshov effect has been
170 found in more recent years, though the effect appears to be modulated by different factors
171 (e.g., Mobbs et al., 2016; Mullenix et al., 2019; Calbi et al., 2017, 2019). Mobbs et al. (2006)
showed emotional scenes (primes) followed by faces displaying neutral or unclear emotional expressions (targets), consisting of subtle happy and fearful faces. They found that the emotional scenes modulated the ratings of both the neutral and ambiguous faces, but the effects were stronger for emotionally unclear faces than for neutral faces. The authors interpreted this difference as indicating that the effects of the prime were greater when the facial emotion and the emotional scene were emotionally congruent than when a neutral face was paired with an emotional scene. Mullenix et al. (2019) found that the emotional valence of the face was rated more negatively after a negative than a positive or neutral scene. They also reported strong individual variability in the categorization of the neutral face as ‘neutral’, varying from 19% to 94% across participants.

In sum, while emotional prosody can be conveyed by cues in the auditory domain, its interpretation can depend on visual information. However, it is not yet clear whether the influence of facial gestures is modulated by the clarity of the subsequent vocal emotional expression. Literature on face perception has found an effect of emotional images on the interpretation of unclear or neutral facial expressions. Valence and intensity ratings of target faces depend on the valence of the preceding emotional scenes, with the effect being more variable across individuals for neutral faces. In the present study, we adopted the cross-priming affective paradigm to elucidate the effect of the emotional face on emotional or neutral spoken prosody.

## 1.3 Methodological limitations of studies on emotional prosody

Previous experimental studies on emotional prosody present two methodological issues which we aim to overcome in the current study. The first issue is that vocal (and visual) expressions of emotions are often elicited from actors by overtly instructing them (‘say this sentence angrily’; e.g., Campbell, 2000; Juslin and Laukka, 2003; Enos and Hirschberg, 2006;
Bänziger et al., 2012). However, in theater or movies, emotions naturally arise from the actors' deep understanding of their characters using acting techniques (e.g., based on emotional imagination and recall, Enos and Hirschberg, 2006), and they become more and more realistic through rehearsal. This can result in less prototypical, more 'believable' vocal (and visual) expressions than the ones usually obtained in experimental research (Bänziger et al., 2012).

Furthermore, current speech processing methods for isolating acoustic cues to emotional meaning (such as low-pass filters or random splicing) may lead to unnatural and distorted stimuli (Ramus and Mehler, 1999). An alternative method is to use speech synthesis systems. In particular, the MBROLA software (Dutoit et al., 1996) has been used in speech perception research to both dexicalize and manipulate intonational, rhythmic and segmental characteristics of natural speech (Ramus and Mehler, 1999). However, to our knowledge, MBROLA has never been applied to the delexicalization of emotional prosody. We propose to address these methodological limitations in the present paper.

210

## 1.3 Research goals and hypotheses

Our study investigated the influence of facial gestures on the perception of spoken angry and neutral prosody. In particular, we tested a specific effect in the auditory domain, i.e., the Kuleshov effect. In the literature on face perception, this effect has been tested on emotional or on neutral faces (Calbi et al., 2019; Mobbs et al., 2006). Here, we evaluated the Kuleshov effect on the evaluation of both angry and neutral spoken prosody. We focused on anger as it provides a signal of potential danger to listeners, drawing their voluntary and involuntary attention towards a threatening situation (Aue et al., 2011). Two behavioral studies were run, in which we applied the cross-modal affective priming paradigm to study whether the emotional information extracted from the visual modality biases the interpretation of a
subsequent auditory signal. In our experiments, pictures of faces were used as primes and
emotional pseudo-speech stimuli were used as targets. Previous studies using the cross-modal
affective paradigm have focused on prosodic emotional expressions with clear acoustic cues.
Here, we tested for the first time whether the influence of the preceding emotional face
increases as the subsequent auditory stimulus becomes less clear (with a limited repertoire of
cues to anger perception or neutrality, i.e. produced without any emotional connotation,
Bänziger et al., 2012). For this purpose, we used delexicalized utterances in which we
manipulated the amount of acoustic information signalling anger. Specifically, we either
deleted or partially preserved global spectral information contained in the utterances, as this is
an important acoustic cue for anger perception (Gobl and Ní Chasaide, 2003).

The cross-modal affective priming paradigm was combined with rating tasks, in which the
auditory stimuli were evaluated in terms of their valence and emotional intensity. Valence and
intensity have been often used in the literature to describe basic emotions, in line with bi-
dimensional models (Russell, 2003). Previous priming studies have already employed rating
tasks for valence and intensity to investigate the influence of affective primes on the
emotional meaning evaluation of subsequent stimuli (e.g., Flexas et al., 2013; Calbi et al.,
2019).

Our auditory stimuli were resynthesized utterances based on acted speech extracted from
French movies to obtain more naturalistic acted speech (Enos and Hirschberg, 2006).
Prosodic parameters were controlled and manipulated via speech synthesis, which has the
potential to overcome the limitations of low-pass filtering methods (Ramus and Mehler,
1999), still widely applied to the perceptual evaluation of spontaneous emotional speech.

In Experiment 1, we presented angry and neutral auditory stimuli either alone or after a
picture of a congruent facial expression. We hypothesized that listeners would judge auditory
245 stimuli produced with angry prosody as more negative and more intense when preceded by an
246 angry facial expression in comparison to the condition in which the stimuli were presented in
247 isolation. This effect was expected to emerge more strongly when the stimuli were
248 acoustically 'impoverished' so that listeners would more actively exploit facial gesture
249 information for the emotional judgment of the subsequent auditory stimuli (de Gelder et al.,
250 2006).

251 In Experiment 2, we presented angry and neutral auditory stimuli either after a picture of a
252 congruent facial expression (e.g., an angry spoken prosody preceded by an angry face) or an
253 incongruent facial expression (an angry spoken prosody preceded by a neutral face). Neutral
254 facial expressions are uninformative for emotional meaning (Jessen and Kotz, 2013; Garrido-
255 Vásquez et al., 2018). Hence, we expected listeners to judge angry spoken prosody as more
256 negative and more intense when preceded by an angry face than by a neutral face.
257 Furthermore, research on face perception has found that individuals vary strongly in their
258 sensitivity to context when interpreting neutral faces (Mullenix et al., 2019). We thus
259 explored whether individual differences also exist in the perception of neutral spoken
260 prosody. Specifically, we hypothesized that in our sample, some participants would rely more
261 than others on the facial gesture information when judging subsequent neutral prosody.

262 **2. Experiment 1**

263 **2.1 Methods**

264 **2.1.1 Participants**

265 Two hundred native French speakers (99 women and 101 men, mean age: 27.99 y.o., SD:
266 8.01) participated in the experiment. Each participant filled out an online informed consent
267 form before the experiment. Participants were recruited through Prolific and paid five euros.
268 The study was approved by the ethics review board of Aix-Marseille University (2020-12-03-269 011).

270 2.1.2 *Materials*

271 2.1.2.1. *Auditory stimuli (targets)*

272 The auditory stimuli consisted of 36 resynthesized utterances created from 18 natural 273 utterances which were extracted from French movies by a fine arts student. They were 274 selected from a larger set (73 stimuli) after a series of perceptual validation tasks (see par. 275 2.2.2). The auditory stimuli were converted into .wav files (sampling rate: 48 kHz). The 276 natural utterances were produced by nine French actors (four women, five men) either with 277 (hot) angry (nine utterances) or neutral (nine utterances) spoken prosody. These utterances 278 were 6.54 syllables long on average (mean utterance duration = 0.97 seconds, SD = 0.34).

279 They could also contain words carrying negative or positive valence (e.g., *Je suis pas malade moi!* , ‘I am not crazy!’). The natural utterances were of good recording quality and contained 280 very little to no background noise. None of them contained overlapping speech from other 281 actors. 282

283 The auditory stimuli were delexicalized to remove the emotional verbal meaning. We created 284 two versions of each utterance that were either poorer or richer in the quantity of emotional 285 prosodic information. In the first version (morphing- condition), only broad phonotactics, 286 rhythm and intonation contours were preserved from the original sentences. In the second 287 condition (morphing+ condition), the delexicalized stimuli were enriched by partially 288 reconstructing the global spectral characteristics of the original utterances. Delexicalization 289 was obtained through MBROLA (Dutoit et al., 1996; Ramus and Mehler, 1999) which 290 performs a resynthesis through the concatenation of diphones using a database of French 291 diphones. We adopted the ‘saltanaj’ condition (Ramus and Mehler, 1999) in which each
phone in the original utterance is substituted by a phone from the same broad phonologic
category (e.g., /s/ for fricatives and /a/ for vowels). For the morphing+ condition, the spectral
characteristics of the original stimuli were partially reconstructed using a vocal morphing
technique through STRAIGHT (Kawahara, 2006) in Matlab (The MathWorks, Inc., Natick,
MA). We used the original utterances down-sampled at 16 kHz as the source and the
corresponding stimulus in the morphing- condition as the target. Since the segmental
durations of the original utterances were preserved in the morphing- condition, a Short-Term
Fourier Transform procedure was used on 25 ms overlapping frames of both source and target
signals, and the time-aligned morphing process with STRAIGHT was applied to each pair of
frames. The morphing rate was set at 0.5 as a trade-off between a lower rate that would have
discarded most voice quality information and a higher rate that would have rebuilt the
segmental information of the original utterances. For both conditions, we reconstructed the
local variations of intensity of the original utterances via the Vocal Toolkit in Praat (Corretge,
2021). Compared to other existing delexicalization methods based on information removal
such as low-pass filtering or wave modulation (Sonntag and Portele, 1998), or copy synthesis
with phone substitution (Ramus and Mehler, 1999) as implemented in the morphing-
condition, the morphing+ delexicalization method has the advantage of generating more
natural utterances in which part of the spectral information relating to voice quality is
preserved (Audibert et al., 2023).

311 (a)
<figure><img src="image_1.png" /></figure>

312

313 (b)

<figure><img src="image_2.png" /></figure>

314

315 (c)

<figure><img src="image_3.png" /></figure>

316

317 **Figure 1.** Spectrogram, waveform, smoothed f0 contour (in blue), intensity contour (in yellow) and textgrid for the sentence *Rends-le moi* ‘Give it back to me’ in the (a) original, (b)
morphing- and (c) morphing+ conditions. Textgrids contain the orthographic transcription (tier 1), the IPA transcription (tier 2), and the phonological annotation for the f0 contour within the French ToBI system (tier 3). The dash lines indicate segmental boundaries. The three conditions match in terms of their intensity contour, f0 contours and phonological annotation (LHi LH* L-L%). The sentence in the example consists in one Intonational Phrase which contain one Accentual Phrase (the basic prosodic unit in French, composed of an early LHi and a late LH* rise, Jun and Fougeron, 2000).

326

2.1.2.2. *Validation tasks for the auditory stimuli*

As mentioned previously, our 36 auditory stimuli (18 stimuli x 2 morphing conditions) were selected after perceptual validation tasks. We first validated the 18 stimuli in the morphing+ condition through an intelligibility task (Validation Task 1) and an identification task (Validation Task 2). Another identification task was subsequently run on the corresponding 18 stimuli in the morphing- condition (Validation Task 3). Participants in one task did not take part in the other two tasks nor in the main experiments.

The intelligibility task (Validation Task 1) was performed to ensure that the addition of spectral information did not lead to word recognition. Forty-seven native French listeners (38 women and 9 men, mean age: 24.71 y.o., *SD*: 7.64), assigned to two counterbalanced lists, transcribed the auditory stimuli orthographically. The stimuli were presented in random order and heard only once. From this task, we retained only those stimuli in the morphing+ condition which had an intelligibility rate equal or inferior to 30%.

The identification task (Validation Task 2) assessed whether the target emotional states (anger and neutral) were accurately recognized through spoken prosody. Thirty-nine native French listeners (21 women and 18 men, mean age: 30.33 y.o., *SD*: 9.59) identified the emotion
conveyed through spoken prosody by choosing between ‘anger’ and ‘neutral’. From this task, we retained only those stimuli in the morphing+ condition whose identification rate was equal or superior to 70% (mean identification score: 93%, SD: 7%). Based on Validation Tasks 1 and 2, eighteen stimuli from the morphing+ condition were selected as our target auditory stimuli as they simultaneously satisfied the selection criteria of both tasks.

We also ran an identification task on the 18 stimuli in the morphing- condition (Validation Task 3). This task assessed whether our spectral manipulation affected emotion recognition. Twenty native French listeners (13 women and 7 men; mean age: 22 y.o; SD: 8.09) identified the emotion conveyed through spoken prosody by choosing between ‘anger’ and ‘neutral’. Stimuli in the morphing- condition collected lower recognition rates than those in the morphing+ condition (mean identification score: 77%, SD: 15%), with the difference being significant [β=13.97, SE=4.35, t=3.21, p= 0.002].

In total, thirty-six utterances were collected [(9 sentences * 2 emotional prosodies (anger, neutral) * 2 morphing conditions (morphing-, morphing+))].

### 2.1.2.3. Acoustic analyses

We acoustically analyzed both the original utterances and their resynthesized versions in terms of their mean relative acoustic intensity (dB), utterance duration (seconds) and f0 mean (Hz) using custom scripts in PRAAT (Boersma, 2001). Relative acoustic intensity (dB) was calculated as the difference between the mean acoustic intensity of the penultimate vowel of each utterance, expected to be unaccented, and that of the other vowels within the utterance, to enable comparisons between utterances. The f0 mean was calculated over the entire duration of the utterance. We estimated the overall acoustic difference between the original utterances and their resynthesized versions using Euclidean distances in the 12-dimension space of Mel-frequency cepstral coefficients (MFCC; Davis and Mermelstein, 1980; see also
Terasawa et al., 2012 on the relationship between distances in the MFCC space and perceived voice quality). Thirteen coefficients were extracted on 15 ms frames of the speech signal with 5 ms of overlap, before dropping coefficient 0 related to the overall energy in the signal and computing the distance between the original and resynthesized versions on each frame.

Linear regression models showed no differences in acoustic intensity, duration and f0 between the original utterances and the resynthesized stimuli ($p > .05$). The stimuli in the morphing- and morphing+ conditions presented similar values to the original stimuli in terms of their relative acoustic intensity (morphing- vs. original: [$\beta$ = 0.0031; $SE$ = 0.0114, $t$ = 0.277, $p$ = 0.78]; morphing+ vs. original: [$\beta$ = 0.015; $SE$ = 0.011, $t$ = 1.37, $p$ = 0.17]), utterance duration (morphing- vs. original: [$\beta$ = -0.0001; $SE$ = 0.0116, $t$ = -0.013, $p$ = 0.99]; morphing+ vs. original: [$\beta$ = 0.007; $SE$ = 0.011, $t$ = 0.62, $p$ = 0.53]) and f0 mean (morphing- vs. original: [$\beta$ = 1.71; $SE$ = 16.3, $t$ = 0.10, $p$ = 0.91]; morphing+ vs. original: [$\beta$ = 0.88; $SE$ = 16.3, $t$ = 0.05, $p$ = 0.95]). Our spectral manipulation was successful: utterances in the morphing+ condition had lower MFCC coefficients than utterances in morphing- [$\beta$ = -41.174; $SE$ = 3.383, $t$ = -12.17, $p$ = <0.001], which confirmed that the stimuli in the morphing+ condition were closer to the originals than those in the morphing- condition in terms of spectral characteristics.

<table><caption>**Table 1.** Means and standard deviations (in parentheses) of prosodic parameters for utterances from the original set and for stimuli in the morphing+ and morphing- conditions.</caption><thead><tr><th>Stimuli</th><th>Relative acoustic<br>intensity (dB)</th><th>Duration<br>(sec.)</th><th>f0 mean<br>(Hz)</th><th>MFCC distance to<br>original<br>(cepstral magnitude)</th></tr></thead><tbody><tr><td>original</td><td>0.43 (0.51)</td><td>0.96 (0.63)</td><td>265.67 (107.12)</td><td>N/A</td></tr><tr><td>morphing+</td><td>0.47 (0.52)</td><td>0.96 (0.63)</td><td>264.74 (106.63)</td><td>474.83 (115.18)</td></tr><tr><td>morphing-</td><td>0.46 (0.51)</td><td>0.96 (0.63)</td><td>266.75 (107.34)</td><td>516.01 (148.30)</td></tr></tbody></table>
2.1.2.4. *Visual stimuli (primes)*

387 Eighteen pictures of human faces were selected from the standardized corpus ‘Karolinska
388 Directed Emotional Faces database’ (KDEF, Lundqvist et al., 1998). The pictures displayed a
389 frontal view of the facial expressions of nine White amateur actors (four women and five
390 men) expressing either (hot) anger or a neutral expression via typical facial gestures. We
391 selected facial expressions presenting high emotion recognition rates (mean for angry faces:
392 98.21 %, SD: 4.48; mean for neutral faces: 91.32 %, SD: 9.72) from previous validation tasks
393 (Goeleven et al., 2008).

394 2.1.3. *Procedure*

395 We ran an online experiment via Qualtrics (Snow and Mann, 2013), in which the auditory
396 stimuli were either presented in isolation or after displaying a picture of a face. When the face
397 was present, the visual and the auditory modalities were always paired in a congruent manner
398 (an angry face followed by angry spoken prosody or a neutral face followed by neutral spoken
399 prosody). Faces were matched with auditory stimuli according to the sex of the actors (a
400 female face followed by a female voice or a male face followed by a male voice). When the
401 face was absent, the auditory stimuli were preceded by a white background slide. Four
402 counterbalanced lists were constructed which were balanced for spoken prosody (angry and
403 neutral), morphing condition (morphing+ and morphing-), presence of a facial picture
404 (absence and presence) and sex of the actors (female and male). Participants were spread
405 across the four lists so that each auditory stimulus was presented only once. The presentation
406 order was random.

407 Each trial started with a black fixation cross (500 ms) and was immediately followed by the
408 picture of a face (when the face was present) or by a white screen (when the face was absent)
409 for 1500 ms. When the face was present, it disappeared from the screen before the sound was
410 automatically played. Faces were always in a frontal (straight) view, and this was kept
411 constant across the different pictures. The pictures were presented full screen in the center of
412 the screen and their size was automatically adjusted in Qualtrics in proportion to the screen.
413 The size of the pictures was constant across items. The center of each picture coincided
414 spatially with the starting fixation point. When the picture of the face was absent, the white
415 screen remained while the sound was playing. To limit variations in the visual angle,
416 participants were instructed to sit in front of the computer, to maintain a constant position and
417 to look straight ahead at the fixation point.

418 Participants evaluated the emotion expressed by the 'speaker's tone of voice' on the valence
419 and intensity scales so that they focused their attention only on the auditory stimuli. Each
420 auditory stimulus was rated on a continuous scale ranging from 'not at all negative' to
421 'absolutely negative' (for valence) and from 'very weak' to 'very strong' (for emotional
422 intensity). There were no numbers displayed on the slides, but responses were recorded as
423 scores ranging from 1 to 100 (Bhatara et al., 2016), with 1 indicating 'not at all negative' (for
424 valence) and 'very weak' (for intensity) and 100 indicating 'absolutely negative' (for valence)
425 and 'very strong' (for intensity).
<figure><img src="image_4.png" /><figcaption>Figure 2. Illustration of the experimental paradigm.</figcaption></figure>

2.1.4. Statistical analysis

All statistical tests were performed in R version 4.1.1 (R Core Team, 2022). The package *lmerTest* version 3.1.3 (Kutzentsova et al., 2017) was used for running mixed models, while *ggplot2* version 3.3.6 (Wickham, 2016) was used for graphical exploration. Linear mixed effects models were run to analyze valence and intensity scores as a function of SPOKEN PROSODY (neutral vs. angry), FACE (presence vs. absence) and MORPHING (morphing+ vs. morphing-). In addition, PARTICIPANT SEX (female vs. male) and ACTOR SEX (female vs. male) were included as control variables. Interactions among all five fixed factors were included as well. PARTICIPANT (1-200) and ACTOR (1-9) were the random intercepts. We started the statistical analysis by fitting each model with the two intercepts, and by including by-participant and by-actor random slopes for spoken prosody, face and morphing. Since the models showed convergence issues, we simplified the random structure of the models to reduce overparametrization (e.g., by deleting random components with very
little variance). The simplification of the random structure of the model did not change the interpretation of the results. To better understand possible interactions (e.g., effects of MORPHING across SPOKEN PROSODY), we ran the models three times, changing the reference level (intercept) for SPOKEN PROSODY and MORPHING. Thus, the cut-off point for significance was set at 0.016 [$p = 0.05$ divided by the number of models (3) run]. Full model outputs are given in Appendix II.

The final model for both valence and intensity scores was:

Valence (or Intensity) ~ Spoken_Prosody * Morphing * Face * Listener sex * Actor sex + (1 + Spoken_Prosody + Morphing + Face | Participant) + (1+ Spoken_Prosody | Actor)

## 2.2 Results

We found an effect of SPOKEN PROSODY, in that auditory stimuli produced with angry spoken prosody were judged as more negative [$\beta = 20.98, SE = 2.54, t = 8.26, p < .001$] and more intense [$\beta = 19.6, SE = 2.53, t = 7.74, p < .001$] than those produced with neutral spoken prosody. Specifically, angry and neutral spoken prosody scored on average 61.7 and 22.4 on the valence scale, respectively; and they scored 61.5 and 25.5 on the intensity scale, respectively (Figure 3). The factor MORPHING further modulated the judgments of angry spoken prosody, but not those of neutral spoken prosody. Angry spoken prosody in the morphing+ condition was judged as more negative [$\beta = 22.36, SE = 2.96, t = 7.55, p < .001$] and more intense [$\beta = 18.29, SE = 2.45, t = 7.47, p < .001$] than angry spoken prosody in the morphing- condition. In average, angry spoken prosody in the morphing+ and in the morphing- conditions scored 73.6 and 49.7 on the valence scale, respectively, and it scored 72.08 and 51.04 on the intensity scale. Conversely, the morphing manipulation had no significant effects on neutral spoken prosody for valence or intensity. Finally, for the valence
scale only, the factor FACE modulated the judgment of angry spoken prosody, but this effect
was limited to auditory stimuli in the morphing- condition [β = 6.24, SE = 2.24, t = 2.79, p =
.005]. As shown in Figure 3, when an angry face preceded an auditory stimulus with
‘impoverished’ angry spoken prosody (morphing- condition), the valence score was slightly
more negative (mean score = 52.6) compared to the condition in which the same stimulus was
presented in isolation (mean score = 46.8). The re-leveled model showed that the effect of
FACE on valence scores was not significant in the morphing+ condition (p = 0.36).
Furthermore, there was no effect of FACE before neutral spoken prosody (p = 0.41). Finally,
PARTICIPANT SEX and ACTOR SEX were not significant for valence or intensity. Their
interactions with SPOKEN PROSODY, MORPHING and FACE were also not significant.
The output of the linear mixed models is presented in Appendix II.

<figure><img src="image_5.png" /></figure>

477

478
<figure><img src="image_6.png" /><figcaption>**Figure 3.** Boxplots of valence (top) and intensity (bottom) score across MORPHING, split by SPOKEN PROSODY and FACE.</figcaption></figure>

**2.3 Discussion**

Experiment 1 showed that ratings for valence and intensity depended on whether the auditory stimuli were produced with angry vs. neutral prosody. Judgments of angry spoken prosody were further modulated by the morphing manipulation, i.e. whether or not angry auditory stimuli contained spectral information from the original utterances. Crucially, facial expressions affected valence judgments of subsequent angry spoken prosody, but the effect was limited to auditory stimuli containing no spectral information.

The rating task reflects the general assessment of emotions along the valence and intensity dimensions, with angry spoken prosody being judged as more negative and more intense than neutral spoken prosody (Bradley and Lang, 2000). Our auditory stimuli were based on audio
493 excerpts from French movies. This result thus extends findings from acted speech elicited in
494 the lab to acted speech elicited in more ecological settings (Enos and Hirschberg, 2006).
495 Angry spoken prosody was judged as more negative and more intense in the morphing+ than
496 in the morphing- condition, while there was no difference for neutral spoken prosody across
497 the morphing conditions. The crucial difference between these two conditions was the
498 presence vs. absence of voice quality-related acoustic features. This result confirms previous
499 research, which found that voice quality is crucial for the perception of anger (Gobl and Ní
500 Chasaide, 2003). Valence and intensity rating scores for angry auditory stimuli in the
501 morphing- condition were in between neutral auditory stimuli and angry auditory stimuli in
502 the morphing+ condition, providing further evidence that f0 and intensity are also reliable
503 cues to anger (Scherer, 2003).

504 Being presented with a picture of an angry facial expression led to more negative judgments
505 of the valence of ‘impoverished’ angry auditory stimuli. As stimuli in the morphing-
506 condition conveyed the speaker’s emotional state less clearly, the angry facial expression may
507 have resulted in negative visual information affecting the judgment of the following auditory
508 stimulus. Stimuli in the morphing+ condition contained a higher number of acoustic cues for
509 angry spoken prosody. Hence, participants did not need to rely on preceding facial
510 information for judgments of emotional spoken prosody for this subset of stimuli.

511 The observed effect of FACE was rather small in size. In the standard affective priming
512 paradigm, targets are always preceded by either emotionally congruent or incongruent primes.
513 Here, participants were exposed to a mixed block as the auditory stimuli were either preceded
514 by congruent primes or presented in isolation. We considered the absence of a face as our
515 control condition, and we wanted to compare this condition with a condition in which the
516 auditory stimuli were preceded by emotionally-congruent faces. Because the instructions
focused participants' attention on the targets, the mixed block may have further attenuated the impact of task-irrelevant distracting information (the facial gestures). This limitation was overcome in Experiment 2, in which primes always preceded the targets and they could create either congruent or incongruent prime-target pairs. This design is more in line with standard affective priming studies (e.g., Paulmann and Pell, 2010).

## 3. Experiment 2

Experiment 2 broadly followed the experimental procedure of Experiment 1, with two major differences. First, in Experiment 2, we used only the auditory stimuli in the morphing-condition as Experiment 1 showed that facial gestures only have an effect in this condition. Second, the auditory stimuli in Experiment 2 were always preceded by a picture of a face, which could be either emotionally congruent or incongruent.

### 3.1. Methods and materials

#### 3.1.1. Participants

Sixty-seven native French speakers between the ages of eighteen and fifty-five (48 women and 19 men, mean age: 24.74 y.o., SD: 9.86) who did not take part in previous tasks participated in the experiment. Each participant signed an online informed consent form before the start of the experiment.

#### 3.1.2. Materials and procedure

The target stimuli consisted of the 18 resynthesized utterances from the morphing-condition in Experiment 1. The stimuli expressed either anger (9 stimuli) or neutrality (9 stimuli) through spoken prosody. The primes consisted in the eighteen pictures from the KDEF (Lundqvist et al., 1998) employed in Experiment 1, showing facial expressions with typical facial gestures associated to either anger or neutrality. The setting of the experiment and the procedure were identical to Experiment 1. In contrast to Experiment 1, the condition in which
auditory stimuli were presented in isolation was excluded. Each auditory stimulus (target) was preceded by an angry or a neutral face (primes), leading to emotional prime-target pairs which were congruent (angry face/angry spoken prosody or neutral face/neutral spoken prosody) or incongruent (angry face/neutral spoken prosody or neutral face/angry spoken prosody). Facial expressions were matched with the auditory stimuli according to the sex of the actors (female faces were paired with female voices and male faces with male voices). Two counterbalanced lists were constructed, each composed of 18 trials balanced for emotion of spoken prosody (angry and neutral), emotion of face (angry and neutral) and sex of the actors (female and male). Participants were instructed to rate the speakers’ tone of voice on the valence and intensity scales. They were presented with each visual-auditory stimuli pair only once. Visual-auditory pairs were presented in random order within each list.

### 3.1.3. Statistical analyses

Statistical tests were performed in R (R Core Team, 2022). To explore whether some listeners relied more on acoustic cues than on the facial expression to judge spoken prosody, we performed a hierarchical cluster analysis based on valence judgments of neutral spoken prosody. We focused on this condition as we expected larger individual differences in the judgment of neutral spoken prosody based on research on the Kuleshov effect (Mullenix et al., 2019). We first computed a distance matrix using the Euclidean distance to quantify the distance among individuals. Specifically, we computed the distance between mean valence judgments of neutral spoken prosody preceded by angry faces and mean valence judgments of neutral spoken prosody preceded by neutral faces. Then, a hierarchical cluster analysis was conducted on this distance matrix via the Ward algorithm (Ward, 1963; Everitt et al., 2011). The Ward algorithm was used as a linkage method, i.e., it allowed us to classify our sample of participants into subgroups such that similar participants were grouped in the same subgroup. More precisely, the Ward's method is an agglomerative method, which combines clusters
whose grouping leads to the minimum increase in total with in cluster variance. This method
is standard in clustering analyses to model individual variability (e.g., Rivière et al., 2018) and
it is preferred because it minimizes the increase in total within-cluster sum of squared error.
This increase is proportional to the squared Euclidean distance between cluster centers.
Graphical exploration of the data structure was made via a dendrogram. The optimal number
of clusters was defined using the gap statistic (Tibshirani et al. 2001), that compares the total
intracluster variation for different values of *k* with their expected values under a reference null
distribution of the data (i.e. a distribution with no obvious clustering). The reference
distribution is generated using Monte Carlo simulations of the sampling process. In our study,
the gap statistic was applied to the widely used k-mean clustering (Tibshirani et al., 2000).
The gap statistic found two main clusters. The two-cluster subgrouping indicated that our
participants had two main patterns of performance (Appendix III). The two clusters contained
31 and 36 participants, respectively, and they did not differ in age (p > 0.05) or sex (p > 0.05).
Specifically, mean age was 23.7 y.o. (for cluster 1) and 24.5 (for cluster 2). There were 21
women, 9 men and 1 non-binary for cluster 1; and 26 women, 9 men and 1 non-binary for
cluster 2. Linear mixed models were run to analyze valence and intensity scores as a function
of SPOKEN PROSODY (neutral vs. angry), FACE (neutral vs. angry), CLUSTER
.cluster1vs. cluster2) and their interactions. Experiment 1 revealed no effects of the sex of the
participant and of the actor. These two control variables were thus not included in the analysis
of results for Experiment 2. To better understand possible interactions, we ran the models
three times by re-leveling the intercept (alpha level = 0.016). Following simplification
procedures like the ones described in § 2.1.4., the final model was:

Valence (or Intensity) ~ Spoken_Prosody * Face * Cluster + (1 | Participant) + (1 + Face | Actor)
590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

Based on the hierarchical clustering analysis, we found different patterns of performance depending on listeners' use of preceding facial information (Figure 4). Participants in cluster1 judged angry spoken prosody as more negative and more intense when preceded by an angry face (valence score: 64.78; intensity score: 62.31) than by a neutral face (valence score: 44.48; intensity score: 52.24). The effect of FACE on angry spoken prosody was significant both for valence [$\beta$ = 20.67, *SE* = 2.99, *t* = 6.91, *p* < 0.001] and intensity [$\beta$ = 10.59, *SE* = 2.83, *t* = 3.73, *p* < 0.001]. Furthermore, cluster1 rated neutral spoken prosody as more negative and more intense when preceded by an angry face (valence score: 54.76; intensity score: 37.41) than by a neutral face (valence score: 29.65; intensity score: 27.78). The re-leveled model showed that the FACE effect on neutral spoken prosody was significant both for valence [$\beta$ = 25.48, *SE* = 2.99, *t* = 8.52, *p* < 0.001] and intensity [$\beta$ = 10.14, *SE* = 2.83, *t* = 3.57, *p* < 0.001].

The size of the FACE effect was very similar for both neutral and angry spoken prosody. For valence ratings, the difference between congruent and incongruent visual-auditory pairs was 4.81 points higher for neutral than for angry spoken prosody [$\beta$ = 46.16, *SE* = 4.16, *t* = 11.09, *p* < 0.001]. For intensity ratings, the difference between congruent and incongruent visual-auditory pairs was only 0.43 of a point higher for angry spoken prosody than for neutral spoken prosody [$\beta$ = 20.73, *SE* = 3.5, *t* = -5.90, *p* < 0.001]. Participants in the cluster2 subgroup were only sensitive to angry faces before angry spoken prosody for valence ratings, with a smaller effect compared to cluster1 [$\beta$ = -17.04, *SE* = 3.53, *t* = -4.81, *p* < 0.001]. No effect of FACE on angry spoken prosody was found for intensity ratings (*p* = 0.021). Furthermore, ratings of neutral spoken prosody preceded by an angry face did not differ from ratings of neutral spoken prosody preceded by neutral faces for valence (*p* = 0.26) or intensity (*p* = 0.75). The output of the linear mixed models is presented in Appendix IV.
<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 4.** Boxplots of valence (top) and intensity (bottom) scores across SPOKEN PROSODY, split by CLUSTER and FACE.</figcaption></figure>

614

615

616

617
Experiment 1 and 2 showed consistent findings, as the evaluation of (impoverished) angry and neutral spoken prosody was affected by the preceding angry face. Experiment 2 also showed the existence of two clusters of participants, who used facial information either more or less to judge the valence and the intensity of speakers' spoken prosody. Facial gestures associated to anger led participants in cluster1 to judge the subsequent spoken prosody as more negative and more intense, independently of whether the spoken prosody was angry or neutral. For participants in cluster2, the angry face led to more negative judgments of angry spoken prosody only. The magnitude of the effect was smaller than that found for cluster1, and it was limited to the valence scale.

The effect of angry facial gestures on angry spoken prosody confirms several previous findings on cross-modal affective priming (see Garrido-Vásquez et al., 2018 and references therein): facial gestures associated to emotional expressions impact the interpretation of the following target more strongly when the prime-target pair is emotionally congruent. Furthermore, in our study, the angry spoken prosody was less salient than the clear angry facial expression preceding it as it was acoustically impoverished. Hence, cues to angry meaning were 'reinforced' by the preceding congruent visual information. This result parallels findings both on multimodal emotion integration (Massaro and Egan, 1996; de Gelder and Vroomen, 2000; Pourtois et al., 2002) and on multimodal integration of intonational pragmatic variables like incredulity or contrastive focus (e.g., Borràs-Comes and Prieto, 2010; Crespo Sendra et al., 2013; Prieto et al., 2015). All these studies suggest that listeners are sensitive to the relative weight of visual and auditory cues, with the effect of one sensory input being stronger when the other sensory input is more ambiguous.
642

Note though that multimodal integration studies are based on the simultaneous presentation of different sensory inputs, which are thus integrated by listeners into a single percept. The aim of such studies is to evaluate how important cues from one modality are compared to cues from another modality when making, e.g., linguistic or paralinguistic decisions (e.g., for prosody : Snrivasan and Massaro 2003; Borràs-Comes and Prieto, 2011; Crespo Sendra et al., 2013; Massaro and Beskow, 2002; Buyok et al., 2022; House et al., 2001; de Gelder and Vroomen, 2000). Thus, findings on synchronous displays of different sensory inputs have been discussed in connection with integration models for information processing (e.g., the fuzzy logical model, Massaro and Cohen, 1993; the weighted averaging model, Massaro 1998; the additive model, Cutting et al., 1992).

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

Our study, on the other side, is based on the asynchronous presentation of different sensory inputs. Thus, we tested priming effects independent of audiovisual integration, as we showed that gestural faces influence the emotional evaluation of spoken prosody even when information from the two modalities is not temporally aligned. For instance, priming effects can result spreading activation or congruency check mechanisms (Pell, 2005). Visual-auditory congruity effects may reflect the fact that information from the visual modality (e.g., the angry face) activates certain emotional features that are shared to some extent by the subsequent emotional expressions of emotion (e.g., the angry spoken prosody), hence facilitating its accessibility. Pell (2005) and Paulmann and Pell (2010) hypothesized that different sensory inputs (such as angry spoken prosody and angry faces) are organized around the same emotion concepts (e.g., ‘anger’, e.g., Pell, 2005; Paulmann and Pell, 2010) in, e.g., a semantic memory network. The presentation of an emotional or affective prime could induce an automatic affective evaluation which would spread from the primes to the emotionally
congruent targets within the network. Alternatively, affective priming effects could be
explained by listeners' tendencies to judge the compatibility of the affective components of
the prime-target pair (e.g., Calbi et al., 2017). Congruency effects could explain why, in
Experiment 2, we found more systematic effects for anger-anger target pairs (in which the
prime and the target are affectively congruent), while there was much more inter-speaker
variability for anger-neutral target pairs (in which the prime and the target are not affectively
congruent). Congruency mechanisms have been also invoked in the literature on the Kuleshov
effect for interpreting neutral face perception (Calbi et al., 2019).

Another important contribution of our study compared to previous ones is related to our focus
on spectral auditory cues. These cues have been almost neglected in the literature on
multimodal perception and, in particular, in the literature on audiovisual integration
employing simultaneously presented sensory inputs. Mostly of previous studies have focused
on the contribution of auditory cues such as f0, amplitude or duration (e.g., de Gelder and
Vroomen, 2000; Massaro and Beskow, 2002; Bujok et al., 2022). However, spectral cues are
crucial for auditory prosody perception, and, in particular for anger perception (Gobl and Ní
Chasaide, 2003). By combining ecological valid materials (utterances extracted from movies)
and sophisticated delexication methods (MBROLA and STRAIGHT), we were able to show
that nuances in voice quality-related characteristics can modulate the influence of emotional
faces on the evaluation of auditory prosody even when other cues (e.g., f0 and duration) are
preserved.

Interestingly, neuroscience studies claimed that cross-modal affective priming effects can be
due to cross-modal prediction (Jessen and Kotz, 2013). Specifically, emotionally congruent
facial gestures facilitate the processing of clear emotional auditory information, both at the
content and at the temporal levels (e.g., Jessen and Kotz, 2013, 2015; Garrido-Vásquez et al.,
690 2018). In turn, cross-modal emotion prediction is considered an instance of predictive coding
691 (Jessen and Kotz, 2013), allowing faster processing of multiple sources of information at both
692 linguistic and extralinguistic levels (Pickering and Garrod, 2007; Paulmann and Pell, 2010;
693 Huetting and Mani, 2016; van Berkum et al., 2015; Hagoort, 2019; see also Corps, 2018 and
694 references therein). While our study is based on behavioral measures only (rating tasks), it is
695 in line with the idea that emotional facial gestures may predict auditory information at the
696 content level, as the perception of angry faces prime the perception of angry voices (Jessen
697 and Kotz, 2013).

698 Compared to previous cross-modal affective priming studies (thus, with asynchronous
699 presentation of different sensory inputs), we showed for the first time that the influence of
700 facial gestures on the evaluation of the subsequent acoustic signal emerges more strongly at
701 the behavioral level when vocal expressions of emotions are acoustically less clear. In the
702 future, it will be necessary to run electrophysiological studies to investigate to what extent
703 such priming effects for acoustically impoverished/neutral auditory signals results from cross-
704 modal prediction. This would eventually support the claim that listeners use predictions
705 especially when required to compensate for ambiguous inputs, due to strong top-down
706 influences on interpretation in such cases (Pickering and Garrod, 2007; Huetting and Mani,
707 2016). Thus, our findings may be also in line with the constraint satisfaction model (Degen
708 and Tanenhaus, 2019). According to this model, we generate expectations about meaning
709 interpretation based on multiple sources of information from visual or auditory modality
710 processed rapidly in a weighted manner, as soon as they are available. Bottom-up information
711 from the linguistic signal is combined with top-down expectations to determine how incoming
712 information is interpreted (Degen and Tanenhaus, 2019).
Experiment 2 showed that, contrary to participants in cluster1, participants in cluster2 did not use facial gesture information to interpret neutral spoken prosody. These results confirm the existence of inter-individual variability in considering the different sources of information (e.g., visual and acoustic cues), with individuals being more or less sensitive to such cues (Hamann and Canli, 2004; Jun and Bishop, 2015; Rivière et al., 2018). In particular, discrepancies in the evaluation of neutral spoken prosody parallel EEG findings using the cross-modal affective priming paradigm (Jessen and Kotz, 2013; Garrido-Vásquez et al., 2018). Garrido-Vásquez et al. (2018) found that when the visual-auditory pair includes a neutral stimulus, this may either lead to the perception of some kind of audiovisual congruency (with facilitation effects) or audiovisual incongruency (with interference or no facilitation effects). This would explain why cluster1 showed effects for angry face-neutral spoken prosody pairs, which were comparable to those for angry face-angry spoken prosody pairs (as the pair was interpreted as congruent), while cluster2 showed no effects for angry face-neutral spoken prosody pairs (the pair was interpreted as incongruent).

Our results on the emotional evaluation of neutral spoken prosody are in line with the literature regarding the effects of emotional scenes on neutral expressions ('Kuleshov' effect). Results obtained on (impoverished) angry and neutral spoken prosody extend findings from the visual domain to the auditory one, suggesting the existence of an 'auditory' Kuleshov effect on the perception of spoken prosody. We showed that this effect applies even when the sensory inputs come from different domains (see Baranowski and Hecht, 2017 for similar results on faces and music). The fact that some people use facial cues more than others is also consistent with Mullenix et al. (2019) and Kim et al. (2022), who reported strong individual differences for context-dependency in the perception of neutral faces. Note though that it is difficult to make a straightforward comparison between our results on neutral prosody in the present study and results on neutral faces in the Kuleshov literature, as we lack a condition in which
use emotional scenes as primes instead of emotional faces. To our knowledge, there are no
738
studies so far comparing the effects of faces vs. scenes. This kind of comparison is relevant to
739
understand whether the effects we found in the current study are specific to faces and voices
740
during multimodal emotion perception, or whether they are interpretable in terms of more
741
general contextual effects on voice processing.
742

Concerning effect sizes, smaller effects were found in our study compared to multimodal
743
integration studies. There may be at least two different reasons for this. First, the priming
744
paradigm may have attenuated the effects, as we know that behavioural priming effects are
745
usually quite small (Weingarten et al., 2016). Furthermore, in multimodal integration studies,
746
stimuli are often manipulated in a continuum from one category to another across gradual
747
steps of manipulation (e.g., de Gelder and Vroomen, 2000; Crespo Sendra et al., 2013). In our
748
study, on the other hand, ‘impoverished’ stimuli (i.e., with no spectral cues) contained strong
749
f0, durational and intensity cues to anger, that biased judgments towards a more negative and
750
more arousing evaluation even when presented in isolation (as shown in Experiment 1).
751

Finally, the effect of the face was either null (Experiment 1 and cluster2) or smaller (cluster1)
752
in the intensity than in the valence scale. In our study, the intensity scale was presented after
753
participants responded in their own time to the valence scale (following Calbi et al., 2019).
754
Hence, facial emotional features may have faded from participants’ memories within a few
755
seconds, explaining the smaller or null face effects on intensity.
756

It should be noted that the generalizability of our results is limited by the fact that, because of
757
methodological constraints (we used naturalistic acted speech), we focused only on one
758
emotion. It is necessary to compare the relative weight of different cues on priming effects
759
across several emotions. Listeners variably rely on visual vs. vocal cues for emotion
760
recognition, depending on the specific basic emotion (Scherer, 2003). Thus, priming effects
761
may be stronger for emotions that are more clearly expressed via visual cues. On the other side, as we already stated, our study was aimed at investigating a specific effect, i.e., the Kuleshov effect, in the auditory domain. In the literature on face perception, this effect has been tested either on neutral faces (Calbi et al., 2019) or on emotional faces (Mobbs et al., 2006). In a similar vein, we tested the Kuleshov effect on both neutral and emotional (angry) auditory prosody. In this respect, our main result is that angry faces affect the evaluation of both neutral and impoverished emotional auditory signals. The effect was more robust for anger-anger prime-target pairs while it was more listener-dependent for anger-neutral prime-target pairs.

In conclusion, the present study employed a cross-modal affective priming paradigm with pictures of static facial expressions followed by spoken neutral and angry prosody. Facial gestures to anger influenced the evaluation of (impoverished) angry and neutral spoken prosody suggesting that multimodal emotion perception applies even when sensory inputs are not temporally aligned. We also showed that, while all participants use visual information when paired with congruent emotional spoken prosody, only some of them rely on facial gestures to interpret neutral spoken prosody. We think that cross-modal affective priming may play a facilitatory role in everyday communication, e.g., it could enable emotional predictions, based on facial cues, of what we are about to hear. Future studies should clarify which are the neural mechanisms underlying the influence of emotional faces on the processing of neutral and acoustically impoverished emotional prosody, and which is the source of individual variability in using facial- vs. signal-based information.

783 Acknowledgements

This study was supported by an A*MIDEX (Aix-Marseille University) grant to CP and MCL.

785 Data deposition
786 French legislation does not allow us to deposit our audio stimuli, which are film extracts, on
787 an open-access platform. For more information, please contact the first author.

788 References

789 Audibert, N., Carbone, F., Champagne-Lavau, M., Housseini, A.S., and Petrone, C. (2023)

790 Evaluation of delexicalization methods for research on emotional speech. *Proceedings of Interspeech*, 2618–2622, doi: 10.21437/Interspeech.2023-1903

792 Aue, T., Cuny, C., Sander, D., Grandjean, D. (2011). Peripheral responses to attended and unattended angry prosody: A dichotic listening paradigm. *Psychophysiology*, 48, 385–392.

794 https://doi.org/10.1111/j.1469-8986.2010.01064.x

795 Aviezer, H., Hassin, R. R., Ryan, J., Grady, C., Susskind., J., Anderson, A., Moscovitch, M.,
796 and Bentin, S. (2008). Angry, disgusted, or afraid? Studies on the malleability of emotion
797 perception. *Psychological Science*, 19(7), 724–32. https://doi.org/10.1111/j.1467-

798 9280.2008.02148

799 Banse, R., and Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. *Journal
800 of personality and social psychology*, 70(3), 614. https://doi.org/10.1037/0022-3514.70.3.614

801 Bänziger, T., Mortillaro, M., and Scherer, K. R. (2012). Introducing the Geneva Multimodal
802 Expression corpus for experimental research on emotion perception. *Emotion, 12*(5), 1161–
803 1179. https://doi.org/10.1037/a0025827

804 Baranowski, A. M., and Hecht, H. (2017). The auditory Kuleshov effect: multisensory
805 integration in movie editing. *Perception*, 46(5), 624-631.

806 https://doi.org/10.1177/0301006616682754
807

Bhatara, A., Laukka, P., Boll-Avetisyan, N., Granjon, L., Anger Elfenbein, H., and Bänziger

808

T. (2016) Second Language Ability and Emotional Prosody Perception. *PLoS ONE*, 11(6).

809

e0156855. https://doi.org/10.1371/journal.pone.0156855

810

Boersma, P. (2001). Praat, a system for doing phonetics by computer. *Glot*

811

*International*, 5(9), 341-345. ISSN 1381-3439

812

Borràs-Comes, J. and Prieto, P. (2011). ‘Seeing tunes.’ The role of visual gestures in tune

813

interpretation. *Laboratory Phonology*, 2(2), 355–380.

814

https://doi.org/10.1515/labphon.2011.013

815

Bradley, M. M., and Lang, P. J. (2000). Affective reactions to acoustic stimuli.

816

*Psychophysiology*, 37(2). 204–215. https://doi.org/10.1111/1469-8986.3720204

817

Bujok, R., Meyer, A. S., & Bosker, H. R. (2022). Audiovisual Perception of Lexical Stress:

818

Beat Gestures are stronger Visual Cues for Lexical Stress than visible Articulatory Cues on

819

the Face. *PsyArXiv Preprints*. doi:10.31234/osf.io/y9jck.

820

Calbi, M., Heimann, K., Barratt, D., Siri, F., Umiltà, M. A., and Gallese, V. (2017). How

821

Context Influences Our Perception of Emotional Faces: A Behavioral Study on the Kuleshov

822

Effect. *Frontiers in Psychology*, 8, 1684-1078. https://doi.org/10.3389/fpsyg.2017.01684

823

Calbi, M., Siri, F., Heimann, K., Barratt, D., Gallese, V., Kolesnikov, A., and Umiltà, M. A.

824

(2019). How context influences the interpretation of facial expressions: a source localization

825

high-density EEG study on the ‘Kuleshov effect’. *Science Reports*, 9, 2107 (2019).

826

https://doi.org/10.1038/s41598-018-37786-y

827

Campbell, N. (2000). Databases of emotional speech. *Proceedings of the ISCA Workshop on*

828

*Speech and Emotion*, 34–38. https://doi.org/10.1016/s0167-6393(02)00070-5
829 Cao, H., Beňuš, Š., Gur, R. C., Verma R., and Nenkova, A. (2014). Prosodic cues for
830 emotion: analysis with discrete characterization of intonation. *Proceedings of Speech
831 Prosody*, 130–134. https://doi.org/10.21437/SpeechProsody.2014-14

832 Corps, R. E. (2018). Coordinating utterances during conversational dialogue: The role of
833 content and timing predictions. *PhD Thesis*, The University of Edinburgh, Edinburgh.

834 Carroll, J. M., and Russell, J. A. (1996). Do facial expressions express specific emotions?
835 Judging emotion from the face in context. *Journal of Personality and Social Psychology*, 70,
836 205–218. https://doi.org/10.1037/0022-3514.70.2.205

837 Corretge, R. (2012). Praat vocal toolkit: A Praat plugin with automated scripts for voice
838 processing [software package]. http://www.praatvocaltoolkit.com/index.html

839 Crespo Sendra, V., Kaland, C. C. L., Swerts, M. G. J., and Prieto, P. (2013). Perceiving
840 incredulity: The role of intonation and facial gestures. *Journal of Pragmatics*, 47(1), 1–13.
841 https://doi.org/10.1016/j.pragma.2012.08.008

842 Davis, S., and Mermelstein, P. (1980). Comparison of parametric representations for
843 monosyllabic word recognition in continuously spoken sentences. *IEEE transactions on
844 acoustics, speech, and signal processing*, 28(4), 357–366.
845 https://doi.org/10.1109/TASSP.1980.1163420

846 de Gelder, B., and Vroomen, J. (2000). The perception of emotions by ear and by eye.
847 *Cognition and Emotion*, 14, 289–311. https://doi.org/10.1080/026999300378824

848 de Gelder, B., Meeren, H. K., Righart, R., van den Stock, J., van de Riet, W. A., and
849 Tamietto, M. (2006). Beyond the face: exploring rapid influences of context on face
850

processing. *Progress in Brain Research*, 155, 37–48. https://doi.org/10.1016/S0079-

851

6123(06)55003-4

852

de Gelder, B., Vroomen, J., and Pourtois, G. R. C. (1999). Seeing cries and hearing smiles:

853

Cross-modal perception of emotional expressions. In G. Aschersleben, T. Bachmann, and J.

854

Müsseler (Eds.), *Cognitive contributions to the perception of spatial and temporal events*,

855

(Advances in psychology; No. 129, pp. 425–438). North-Holland: Elsevier Science

856

Publishers. https://doi.org/10.1016/S0166-4115(99)80040-5

857

Degen, J., and Tanenhaus, M.K. (2019). Constraint-based pragmatic processing. In C.

858

Cummins and N. Katsos (Eds.), *The Oxford Handbook of Experimental Semantics and Pragmatics* (pp. 21–38). Oxford University Press.

859

860

doi.org/10.1093/oxfordhb/9780198791768.013.8

861

Dutoit, T., Pagel, V., Pierret, N., Bataille, F., and van der Vrecken, O. (1996). The MBROLA

862

project: towards a set of high quality speech synthesizers free of use for non commercial

863

purposes. *Proceedings of the International Conference on Speech and Language Processing*,

864

Philadelphia, 1393–1396. https://doi.org/10.1109/ICSLP. 1996.607874

865

Ekman, P. (1992). An argument for basic emotions. *Cognition and Emotion, 6*(3-4), 169–200.

866

https://doi.org/10.1080/02699939208411068

867

Ekman, P., Friesen, W. V., and Elisworth, P. (1982). What are the relative contributions of

868

facial behavior and contextual information to the judgment of emotion? In P. Ekman (Ed.),

869

*Emotion in the human face* (2nd ed., pp. 111–127). New York: Cambridge University Press.

870

0521239923
Ekman, P., Friesen, W. V., and Hager, J. C. (2002). *The Facial Action Coding System: A Technique for the Measurement of Facial Movement*. San Francisco, CA: Consulting Psychologists Press. 1573387450055380096

Enos, F. and Hirschberg, J. (2006). A Framework for Eliciting Emotional Speech: Capitalizing on the Actor's Process. *Proceedings of LREC Workshop on Emotional Speech*. https://doi.org/10.7916/D88S4ZCG. Retrieved from: https://academiccommons.columbia.edu/doi/10.7916/D88S4ZCG

Everitt, B.S., Landau, S., Leese, M., Stahl, D. (2011). *Cluster Analysis* (5th ed.) Wiley Series in Probability and Statistics. ISBN:9780470749913

Flexas, A., Rosselló, J., Christensen, J. F., Nadal, M., Olivera La Rosa, A., Munar, E. (2013). Affective Priming Using Facial Expressions Modulates Liking for Abstract Art. PLoS ONE 8(11): e80154. https://doi.org/10.1371/journal.pone.0080154

Garrido-Vásquez, P., Pell, M. D., Paulmann, S., Kotz, S. A. (2018). Dynamic Facial Expressions Prime the Processing of Emotional Prosody. *Frontiers in Human Neuroscience*, 12, 244. https://doi.org/10.3389/fnhum.2018.00244

Gobl, C., and Ní Chasaide, A. (2003). The Role of Voice Quality in Communicating Emotion, Mood and Attitude. *Speech Communication*, 40, 189–212. https://doi.org/10.1016/S0167-6393(02)00082-1

Goeleven, E., De Raedt, R., Leyman, L. and Verschuere, B. (2008). The Karolinska Directed Emotional Faces: A validation study. *Cognition and Emotion*, 22(6), 1094–1118. https://doi.org/10.1080/02699930701626582
892 Hagoort P. (2019). The neurobiology of language beyond single-word processing. *Science*, 4;
893 366 (6461), 55–58. doi: 10.1126/science.aax0289.

894 Hamann, S., and Canli, T. (2004). Individual differences in emotion processing. *Current
895 Opinion in Neurobiology*, 14(2), 233–238. doi: 10.1016/j.conb.2004.03.010.

896 House, D., Beskow, J., & Granstrom, B. (2001). Timing and interaction of visual cues for
897 prominence in audiovisual speech perception. *Proceedings of Eurospeech 2001*, 387–390.
898 doi: 10.21437/Eurospeech.2001-61.

899 Huettig, F., and Mani, N. (2016). Is prediction necessary to understand language? Probably
900 not. *Language, Cognition and Neuroscience*, 31(1), 19–31.

901 https://doi.org/10.1080/23273798.2015.1072223

902 Jessen, S., and Kotz, S. A. (2013). On the role of crossmodal prediction in audiovisual
903 emotion perception. *Frontiers in Human Neuroscience*, 7, 369.
904 https://doi.org/10.3389/fnhum.2013.00369

905 Jessen, S., and Kotz, S. A. (2015). Affect differentially modulates brain activation in uni-and
906 multisensory body-voice perception. *Neuropsychologia*, 66, 134–143.
907 https://doi.org/10.1016/j.neuropsychologia.2014.10.038

908 Jun, S. A., and Fougeron, C. (2000) A phonological model of French intonation. In A. Botinis
909 (Ed.), *Intonation. Text, Speech and Language Technology*, 15. Dordrecht, GE: Springer.
910 10.1007/978-94-011-4317-2_10

911 Jun, S. A., and Bishop, J. (2015). Priming implicit prosody: Prosodic boundaries and
912 individual. *Language and speech*, 58(4), 459–473. https://doi.org/10.1177/0023830914563368
913

Juslin, P. N., and Laukka, P. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code?. *Psychological bulletin*, 129(5), 770.
https://doi.org/10.1037/0033-2909.129.5.770

914

915

916

Kawahara, H. (2006). STRAIGHT, exploitation of the other aspect of VOCODER: Perceptually isomorphic decomposition of speech sounds. *Acoustical science and technology*, 27(6), 349–353. https://doi.org/10.1250/ast.27.349

917

918

919

Kim, K. I., Jung, W. H., Woo, C. W., and Kim, H. (2022). Neural signatures of individual variability in context-dependent perception of ambiguous facial expression. *NeuroImage*, 119355.

920

921

922

Kuznetsova, A., Brockhoff, P.B., and Christensen, R.H.B. (2017). lmerTest Package: Tests in Linear Mixed Effects Models. *Journal of Statistical Software*, 82(13), 1–26.
https://doi.org/10.18637/jss.v082.i13

923

924

925

Lundqvist, D., Flykt, A., and Öhman, A. (1998). The Karolinska Directed Emotional Faces—KDEF. Stockholm: Karolinska Institute, Department of Clinical Neuroscience, Psychology Section. https://doi.org/10.1037/t27732-000

926

927

928

Massaro, D. W., and Egan, P. B. (1996). Perceiving affect from the voice and the face. *Psychonomic Bulletin and Review*, 3(2), 215–221. https://doi.org/10.3758/BF03212421

929

930

Massaro, D. W., and Beskow, J. (2002). Multimodal speech perception: A paradigm for speech science. In B. Granstrom, D. House, & I. Karlsson (Eds.), *Multimodality in language and speech systems* (pp.45–71). Kluwer Academic Publishers, Dordrecht, The Netherlands.

931

932

933

McGurk, H., and MacDonald, J. (1976). Hearing lips and seeing voices. *Nature* 264, 746–748. doi: 10.1038/264746a0

934
Mobbs, D., Weiskopf, N., Lau, H. C., Featherstone, E., Dolan, R. J., Frith, C. D. (2006). The Kuleshov Effect: the influence of contextual framing on emotional attributions. *Social Cognitive and Affective Neuroscience*, 1(2), 95–106. https://doi.org/10.1093/scan/nsl014

Mullennix, J., Barber, J., Cory, T. (2019). An examination of the Kuleshov effect using still photographs. *PLoS ONE*, 14(10), e0224623. https://doi.org/10.1371/journal.pone.0224623

Paulmann, S., Jessen, S., and Kotz, S. A. (2009). Investigating the multimodal nature of human communication: insights from ERPs. *Journal of Psychophysiology*, 23, 63–76. https://doi.org/10.1027/0269-8803.23.2.63

Paulmann, S., Pell, M. D. (2010). Contextual influences of emotional speech prosody on face processing: how much is enough? *Cognitive, Affective, and Behavioral Neuroscience*, 10(2), 230–242. https://doi.org/10.3758/CABN.10.2.230

Paulmann, S., Titone, D., and Pell, M. D. (2012). How emotional prosody guides your way: evidence from eye movements. *Speech Communication*, 54, 92–107. https://doi.org/10.1016/j.specom.2011.07.004

Pell, M. D. (2005). Nonverbal emotion priming: evidence from the ‘facial affect decision task’. *Journal of Nonverbal Behavior*, 29, 45–73. https://doi.org/10.1007/s10919-004-0889-8

Pickering, M. J., and Garrod, S. (2007). Do people use language production to make predictions during comprehension? *Trends in Cognitive Sciences*, 11(3), 105–110. https://doi.org/10.1016/j.tics.2006.12.002

Pourtois, G., Debatisse, D., Despland, P. A., and de Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. *Cognitive Brain Research*, 14, 99–105. https://doi.org/10.1016/S0926-6410(02)00064-2
Prieto, P., Pugliesi, C., Borràs-Comes, J., Arroyo, E., and Blat, J. (2015). Exploring the contribution of prosody and gesture to the perception of focus using an animated agent. *Journal of Phonetics*, 49, 41–54. https://doi: 10.1016/j.wocn.2014.10.005

Prince, S., and Hensley, W. E. (1992). The Kuleshov effect: Recreating the classic experiment. *Cinema Journal*, 31(2), 59–75. https://doi.org/10.2307/1225144

R Core Team (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

Ramus, F., Mehler, J. (1999). Language identification with suprasegmental cues: A study based on speech resynthesis. *The Journal of the Acoustical Society of America*, 105(1), 512–521. https://doi.org/10.1121/1.424522

Rivière, E., Klein, M. and Champagne-Lavau, M. (2018). Using context and prosody in understanding irony: Variability amongst individuals. *Journal of Pragmatics*, 138, 165–172. https://doi.org/10.1016/j.pragma.2018.10.006

Russell, J.A. (2003) Core affect and the psychological construction of emotion. *Psychological Review*, 110(1), 145–172. https://doi.org/10.1037/0033-295X.110.1.145

Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. *Speech Communication*, 40(1–2), 227–256. https://doi.org/10.1016/S0167-6393(02)00084-5

Snow, J., and Mann, M. (2013). Qualtrics survey software: handbook for research professionals. *Qualtrics Labs, Inc.*

Sonntag, G. P., and Portele, T. (1998). PURR—a method for prosody evaluation and investigation, *Computer Speech & Language*, 12(4), 437–451.
978 Srinivasan, R. J. & Massaro, D.W (2003). Perceiving from the face and voice: Distinguishing statements from echoic questions in English. *Language and Speech* 46(1), 1–22.
979 https://doi.org/10.1177/00238309030460010201
980

981 Terasawa, H., Berger, J., & Makino, S. (2012). In search of a perceptual metric for timbre: Dissimilarity judgments among synthetic sounds with MFCC-derived spectral envelopes. *Journal of the Audio Engineering Society*, 60(9), 674–685.
982
983

984 Tibshirani, R., Walther, G. and Hastie, T. (2001). Estimating the number of data clusters via the Gap statistic. *Journal of the Royal Statistical Society B*, 63, 411–423.
985
986 https://doi.org/10.1111/1467-9868.00293

987 Van Berkum, J. J., Brown, C. M., Zwitserlood, P., Kooijman, V., and Hagoort, P. (2005).

988 Anticipating upcoming words in discourse: Evidence from ERPs and reading times. *Journal of Experimental Psychology: Learning, Memory, and Cognition*, 31, 443–467.
989

990 https://doi.org/10.1037/0278-7393.31.3.443

991 Ward Jr., J. H. (1963). Hierarchical Grouping to Optimize an Objective Function. *Journal of the American Statistical Association*, 58(301), 236–244.
992
993 https://doi.org/10.1080/01621459.1963.10500845

994 Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., and Albarracin, D. (2016). On Priming Action: Conclusions from a Meta-Analysis of the Behavioral Effects of Incidentally-Presented Words. *Current Opinion in Psychology*, 12, 53–57. doi: 10.1016/j.copsyc.2016.04.015.

998 Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://doi.org/10.1111/j.1541-0420.2011.01616.x
1000 Zhang, H., Chen, X., Chen, S., Li, Y., Chen, C., Long, Q., Yuan, J. (2018). Facial Expression

1001 Enhances Emotion Perception Compared to Vocal Prosody: Behavioral and fMRI Studies.

1002 Neuroscience Bulletin, 34(5), 801-815. https://doi.org/10.1007/s12264-018-0231-9

1003

1004
1005 **Appendix**

1006 **Appendix I.** Set of sentences used in the experiment. Each sentence was extracted from
1007 French movies and spoken by the same actor with neutral or angry emotional prosody.

<table><thead><tr><td>Emotion</td><td>Original verbal content</td><td>Film</td></tr></thead><tbody><tr><td>anger</td><td>Tu comprends pas que c'est un cadeau que je te fais<br/>[Don't you understand that this is a gift I'm giving you]</td><td>L'Humanité<br/>(Bruno Dumont, 1999)</td></tr><tr><td>neutral</td><td>Dis ce soir on va à la brasserie<br/>[Hey tonight we're going to the brewery]</td><td>L'Humanité<br/>(Bruno Dumont, 1999)</td></tr><tr><td>anger</td><td>Pas un voleur<br/>[Not a thief]</td><td>La Drôlesse 'The Hussy' (Jacques Doillon, 1979)</td></tr><tr><td>neutral</td><td>C'est mon beau père<br/>[He's my father-in-law]</td><td>La Drôlesse 'The Hussy' (Jacques Doillon, 1979)</td></tr><tr><td>anger</td><td>Tu tires dans le tunnel<br/>[You're shooting in the tunnel]</td><td>Bande de Fille 'Girlhood' (Céline Sciamma, 2014)</td></tr><tr><td>neutral</td><td>On a qu'à faire un truc qui change<br/>[Let's do something different]</td><td>Bande de Fille 'Girlhood' (Céline Sciamma, 2014)</td></tr><tr><td>anger</td><td>J'te parle<br/>[I'm talking to you]</td><td>Bande de Fille 'Girlhood' (Céline Sciamma, 2014)</td></tr><tr><td>neutral</td><td>Je veux pas faire de CAP<br/>[I don't want to do a CAP]</td><td>Bande de Fille 'Girlhood' (Céline Sciamma, 2014)</td></tr><tr><td>anger</td><td>J'y toucherai quand j'aurai envie d'y toucher<br/>[I'll touch it when I want to touch it]</td><td>Buffet Froid<br/>(Bertrand Blier, 1979)</td></tr><tr><td>neutral</td><td>Boirait pas encore un petit verre<br/>[Wouldn't drink a little more]</td><td>Buffet Froid<br/>(Bertrand Blier, 1979)</td></tr><tr><td>anger</td><td>Rends-le moi</td><td>Rosetta</td></tr></tbody></table>
<table><tr><td></td><td>[Give it back to me]</td><td>(Frères Dardennes, 1999)</td></tr><tr><td>neutral</td><td>J'ai travaillé<br/>[I worked]</td><td>Rosetta<br/>(Frères Dardennes, 1999)</td></tr><tr><td>anger</td><td>Arrête<br/>[Stop it]</td><td>Une étrange affaire 'Strange Affair'<br/>(Pierre Granier-Deferre, 1981)</td></tr><tr><td>neutral</td><td>Tu veux pas je descende Filo<br/>[You don't want me to come down Filo]</td><td>Une étrange affaire 'Strange Affair'<br/>(Pierre Granier-Deferre, 1981)</td></tr><tr><td>anger</td><td>Je suis pas malade moi<br/>[I'm not crazy]</td><td>Buffet Froid<br/>(Bertrand Blier, 1979)</td></tr><tr><td>neutral</td><td>C'est fermé à clef<br/>[It's locked]</td><td>Buffet Froid<br/>(Bertrand Blier, 1979)</td></tr><tr><td>anger</td><td>Des idées de quoi<br/>[Ideas of what]</td><td>Buffet Froid<br/>(Bertrand Blier, 1979)</td></tr><tr><td>neutral</td><td>Pas vraiment<br/>[Not really]</td><td>Buffet Froid<br/>(Bertrand Blier, 1979)</td></tr></table>

1008

1009 **Appendix II.** Output of the linear mixed models estimated for ratings of valence (a) and intensity (b), with PROSODY (neutral vs. angry), FACE (presence vs. absence) and MORPHING (morphing+ vs. morphing-) as fixed effects. PARTICIPANT SEX (female vs. male) and ACTOR SEX (female vs. male) were also included as control variables. Reference level (intercept): Face: absence; Prosody: angry; Morphing: morphing-; Participant Sex male; Actor Sex: male. (n= 3600 for valence and n= 3600). The output is reported only one model (i.e., the one with no reveling of the intercept). Models on the same data sets with relevel of the intercept are not reported for the sake of brevity. Significance codes: p < 0.001 (***), p < 0.01 (**), p < 0.05 (*). Cut-off point for significance: p = 0.016. [p = 0.05 divided by the number of models (3) run].

1019
1020

1021

1022

1023

1024

1025

1026

1027

1028

1029
<table><thead><tr><th>Effect</th><th>β</th><th>SE</th><th>df</th><th>t</th><th>pvalue</th><th></th></tr></thead><tbody><tr><td>(Intercept)</td><td>48.48</td><td>3.54</td><td>14.30</td><td>13.70</td><td>&lt;0.001</td><td>***</td></tr><tr><td>Morphing_morphing+</td><td>22.36</td><td>2.96</td><td>32.40</td><td>7.55</td><td>&lt;0.001</td><td>***</td></tr><tr><td>Face_presence</td><td>6.24</td><td>2.24</td><td>2736.54</td><td>2.79</td><td>0.005</td><td>**</td></tr><tr><td>ParticipantSex_female</td><td>2.78</td><td>2.88</td><td>591.82</td><td>0.97</td><td>0.33</td><td></td></tr><tr><td>ActorSex_female</td><td>-4.96</td><td>4.95</td><td>10.83</td><td>-1.00</td><td>0.34</td><td></td></tr><tr><td>Prosody_neutral</td><td>-20.98</td><td>2.54</td><td>1351.82</td><td>-8.26</td><td>&lt;0.001</td><td>***</td></tr><tr><td>Morphing_morphing+ x Face_presence</td><td>-2.33</td><td>3.15</td><td>2876.77</td><td>-0.74</td><td>0.46</td><td></td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female</td><td>-3.38</td><td>3.30</td><td>2293.37</td><td>-1.03</td><td>0.31</td><td></td></tr><tr><td>Face_presence x ParticipantSex_female</td><td>-3.92</td><td>3.18</td><td>2723.10</td><td>-1.23</td><td>0.22</td><td></td></tr><tr><td>Morphing_morphing+ x ActorSex_female</td><td>5.87</td><td>4.33</td><td>29.18</td><td>1.36</td><td>0.18</td><td></td></tr><tr><td>Face_presence x ActorSex_female</td><td>1.17</td><td>3.33</td><td>2808.36</td><td>0.35</td><td>0.73</td><td></td></tr><tr><td>ParticipantSex_female x ActorSex_female</td><td>-3.18</td><td>3.34</td><td>2804.04</td><td>-0.95</td><td>0.34</td><td></td></tr><tr><td>Morphing_morphing+ x Prosody_neutral</td><td>-22.99</td><td>3.15</td><td>2878.37</td><td>-7.29</td><td>&lt;0.001</td><td>***</td></tr><tr><td>Face_presence x Prosody_neutral</td><td>-8.07</td><td>3.16</td><td>2869.55</td><td>-2.56</td><td>0.01</td><td>*</td></tr><tr><td>ParticipantSex_female x Prosody_neutral</td><td>-4.40</td><td>3.61</td><td>1355.00</td><td>-1.22</td><td>0.22</td><td></td></tr><tr><td>ActorSex_female x Prosody_neutral</td><td>-5.14</td><td>3.32</td><td>2812.68</td><td>-1.55</td><td>0.12</td><td></td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female</td><td>3.40</td><td>4.48</td><td>2877.83</td><td>0.76</td><td>0.45</td><td></td></tr></tbody></table>
<table><tr><td>Morphing_morphing+ x Face_presence x ActorSex_female</td><td>0.87</td><td>4.70</td><td>2817.91</td><td>0.18</td><td>0.85</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female x ActorSex_female</td><td>5.84</td><td>4.73</td><td>2813.53</td><td>1.24</td><td>0.22</td></tr><tr><td>Face_presence x ParticipantSex_female x ActorSex_female</td><td>3.21</td><td>4.73</td><td>2800.67</td><td>0.68</td><td>0.50</td></tr><tr><td>Morphing_morphing+ x Face_presence x Prosody_neutral</td><td>2.15</td><td>4.46</td><td>2961.19</td><td>0.48</td><td>0.63</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female x Prosody_neutral</td><td>4.00</td><td>4.48</td><td>2879.68</td><td>0.89</td><td>0.37</td></tr><tr><td>Face_presence x ParticipantSex_female x Prosody_neutral</td><td>5.20</td><td>4.48</td><td>2870.54</td><td>1.16</td><td>0.25</td></tr><tr><td>Morphing_morphing+ x ActorSex_female x Prosody_neutral</td><td>-3.65</td><td>4.70</td><td>2818.86</td><td>-0.78</td><td>0.44</td></tr><tr><td>Face_presence x ActorSex_female x Prosody_neutral</td><td>3.13</td><td>4.70</td><td>2814.67</td><td>0.67</td><td>0.51</td></tr><tr><td>ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>1.71</td><td>4.72</td><td>2812.48</td><td>0.36</td><td>0.72</td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female x ActorSex_female</td><td>-3.95</td><td>6.68</td><td>2817.94</td><td>-0.59</td><td>0.56</td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female x Prosody_neutral</td><td>-4.93</td><td>6.34</td><td>2962.83</td><td>-0.78</td><td>0.44</td></tr><tr><td>Morphing_morphing+ x Face_presence x ActorSex_female x Prosody_neutral</td><td>-1.17</td><td>6.65</td><td>2858.81</td><td>-0.18</td><td>0.86</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>-3.79</td><td>6.68</td><td>2819.05</td><td>-0.57</td><td>0.57</td></tr><tr><td>Face_presence x ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>-3.33</td><td>6.69</td><td>2814.65</td><td>-0.50</td><td>0.62</td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>2.25</td><td>9.46</td><td>2858.75</td><td>0.24</td><td>0.81</td></tr></table>
<table><thead><tr><th>Effect</th><th>β</th><th>SE</th><th>df</th><th>t</th><th>p</th><th></th></tr></thead><tbody><tr><td>(Intercept)</td><td>52.26</td><td>3.64</td><td>13.74</td><td>14.34</td><td>&lt; 0.001</td><td>***</td></tr><tr><td>Morphing_morphing+</td><td>18.29</td><td>2.45</td><td>53.55</td><td>7.47</td><td>&lt; 0.001</td><td>***</td></tr><tr><td>Face_presence</td><td>-1.52</td><td>2.10</td><td>2599.38</td><td>-0.72</td><td>0.47</td><td></td></tr><tr><td>ParticipantSex_female</td><td>-1.17</td><td>2.88</td><td>494.38</td><td>-0.41</td><td>0.68</td><td></td></tr><tr><td>ActorSex_female</td><td>-1.93</td><td>5.04</td><td>9.99</td><td>-0.38</td><td>0.71</td><td></td></tr><tr><td>Prosody_neutral</td><td>-19.60</td><td>2.53</td><td>979.02</td><td>-7.74</td><td>&lt; 0.001</td><td>***</td></tr><tr><td>Morphing_morphing+ x Face_presence</td><td>4.98</td><td>2.93</td><td>2870.07</td><td>1.70</td><td>0.09</td><td>.</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female</td><td>0.01</td><td>3.01</td><td>2557.46</td><td>0.00</td><td>1.00</td><td></td></tr><tr><td>Face_presence x ParticipantSex_female</td><td>5.67</td><td>2.99</td><td>2591.34</td><td>1.90</td><td>0.06</td><td>.</td></tr><tr><td>Morphing_morphing+ x ActorSex_female</td><td>1.07</td><td>3.61</td><td>50.08</td><td>0.30</td><td>0.77</td><td></td></tr><tr><td>Face_presence x ActorSex_female</td><td>0.26</td><td>3.09</td><td>2807.52</td><td>0.09</td><td>0.93</td><td></td></tr><tr><td>ParticipantSex_female x ActorSex_female</td><td>-0.48</td><td>3.11</td><td>2791.12</td><td>-0.15</td><td>0.88</td><td></td></tr><tr><td>Morphing_morphing+ x Prosody_neutral</td><td>-23.28</td><td>2.93</td><td>2869.72</td><td>-7.94</td><td>&lt; 0.001</td><td>***</td></tr><tr><td>Face_presence x Prosody_neutral</td><td>-1.08</td><td>2.94</td><td>2859.08</td><td>-0.37</td><td>0.71</td><td></td></tr><tr><td>ParticipantSex_female x Prosody_neutral</td><td>-3.67</td><td>3.60</td><td>981.26</td><td>-1.02</td><td>0.31</td><td></td></tr><tr><td>ActorSex_female x Prosody_neutral</td><td>-4.13</td><td>3.09</td><td>2799.33</td><td>-1.34</td><td>0.18</td><td></td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female</td><td>-6.05</td><td>4.17</td><td>2869.17</td><td>-1.45</td><td>0.15</td><td></td></tr></tbody></table>
<table><tr><td>Morphing_morphing+ x Face_presence x ActorSex_female</td><td>0.04</td><td>4.38</td><td>2810.71</td><td>0.01</td><td>0.99</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female x ActorSex_female</td><td>5.43</td><td>4.40</td><td>2796.32</td><td>1.23</td><td>0.22</td></tr><tr><td>Face_presence x ParticipantSex_female x ActorSex_female</td><td>-4.91</td><td>4.40</td><td>2798.99</td><td>-1.12</td><td>0.26</td></tr><tr><td>Morphing_morphing+ x Face_presence x Prosody_neutral</td><td>-3.18</td><td>4.15</td><td>2957.15</td><td>-0.77</td><td>0.44</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female x Prosody_neutral</td><td>-1.31</td><td>4.17</td><td>2869.06</td><td>-0.32</td><td>0.75</td></tr><tr><td>Face_presence x ParticipantSex_female x Prosody_neutral</td><td>-0.68</td><td>4.17</td><td>2858.11</td><td>-0.16</td><td>0.87</td></tr><tr><td>Morphing_morphing+ x ActorSex_female x Prosody_neutral</td><td>3.06</td><td>4.38</td><td>2810.80</td><td>0.70</td><td>0.48</td></tr><tr><td>Face_presence x ActorSex_female x Prosody_neutral</td><td>2.14</td><td>4.38</td><td>2805.77</td><td>0.49</td><td>0.63</td></tr><tr><td>ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>-0.24</td><td>4.40</td><td>2798.53</td><td>-0.06</td><td>0.96</td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female x ActorSex_female</td><td>3.98</td><td>6.22</td><td>2809.86</td><td>0.64</td><td>0.52</td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female x Prosody_neutral</td><td>0.91</td><td>5.90</td><td>2955.24</td><td>0.15</td><td>0.88</td></tr><tr><td>Morphing_morphing+ x Face_presence x ActorSex_female x Prosody_neutral</td><td>-0.85</td><td>6.19</td><td>2853.28</td><td>-0.14</td><td>0.89</td></tr><tr><td>Morphing_morphing+ x ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>-6.02</td><td>6.22</td><td>2810.09</td><td>-0.97</td><td>0.33</td></tr><tr><td>Face_presence x ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>0.82</td><td>6.22</td><td>2804.91</td><td>0.13</td><td>0.89</td></tr><tr><td>Morphing_morphing+ x Face_presence x ParticipantSex_female x ActorSex_female x Prosody_neutral</td><td>2.07</td><td>8.80</td><td>2851.53</td><td>0.24</td><td>0.81</td></tr></table>
1035 **Appendix IV.** Dendrogram (bottom) and gap statistic plot (bottom) based on valence

1036 judgments of neutral prosody.

1037

<figure><img src="image_9.png" /><figcaption>Cluster Dendrogram</figcaption></figure>

d
hc|ust (", "ward.D")

1038

<figure><img src="image_10.png" /></figure>

1039
1040 **Appendix IV.** Output of the linear mixed models estimated for ratings of valence (a) and intensity (b), with PROSODY (neutral vs. angry), FACE (neutral vs. angry) and CLUSTER (cluster1 vs. cluster2) as fixed effects. Reference level (intercept): Prosody: angry; Face: angry; Cluster: cluster1. (n= 1206 for valence and n= 1206 for intensity). The output is reported only one model (i.e., the one with no reveling of the intercept). Models on the same data sets with releval of the intercept are not reported for the sake of brevity. Significance codes: p < 0.001 (***), p < 0.01 (**), p < 0.05 (*). Cut-off point for significance: p = 0.016. [p = 0.05 divided by the number of models (3) run].

1048 (a) Valence

<table><thead><tr><th>Effect</th><th>β</th><th>SE</th><th>df</th><th>t</th><th>p</th><th></th></tr></thead><tbody><tr><td>(Intercept)</td><td>64.97</td><td>3.53</td><td>26.42</td><td>18.36</td><td>&lt;2e-16</td><td>***</td></tr><tr><td>Face_neutral</td><td>-20.67</td><td>2.99</td><td>112.46</td><td>-6.91</td><td>3.01e-10</td><td>***</td></tr><tr><td>Prosody_neutral</td><td>-35.51</td><td>2.93</td><td>1124.98</td><td>-12.1</td><td>&lt;2e-16</td><td>***</td></tr><tr><td>Cluster_cluster2</td><td>-17.04</td><td>3.53</td><td>219.97</td><td>-4.81</td><td>2.71e-06</td><td>***</td></tr><tr><td>Face_neutral:Prosody_neutral</td><td>46.16</td><td>4.16</td><td>1125.88</td><td>11.09</td><td>&lt;2e-16</td><td>***</td></tr><tr><td>Face_neutral:Cluster_cluster2</td><td>13.07</td><td>3.99</td><td>1121.21</td><td>3.27</td><td>0.001</td><td>**</td></tr><tr><td>Prosody_neutral:Cluster_cluster2</td><td>0.82</td><td>3.99</td><td>1122.29</td><td>0.20</td><td>0.83</td><td></td></tr><tr><td>Face_neutral:Prosody_neutral:Cluster_cluster2</td><td>-35.44</td><td>5.66</td><td>1125.23</td><td>-6.25</td><td>5.64e-10</td><td>***</td></tr></tbody></table>

1049

1050 (b) Intensity

<table><thead><tr><th>Effect</th><th>β</th><th>SE</th><th>df</th><th>t</th><th>p</th><th></th></tr></thead><tbody><tr><td>(Intercept)</td><td>62.44</td><td>3.19</td><td>39.48</td><td>19.53</td><td>&lt;2e-16</td><td>***</td></tr><tr><td>Face_neutral</td><td>-10.59</td><td>2.83</td><td>45.03</td><td>-3.73</td><td>0.0005</td><td>***</td></tr><tr><td>Prosody_neutral</td><td>-34.79</td><td>2.47</td><td>1123.01</td><td>-14.06</td><td>&lt;2e-16</td><td>***</td></tr><tr><td>Cluster_cluster2</td><td>-11.29</td><td>3.49</td><td>147.33</td><td>-3.22</td><td>0.0015</td><td>**</td></tr><tr><td>Face_neutral:Prosody_neutral</td><td>20.73</td><td>3.51</td><td>1122.44</td><td>5.90</td><td>4.59e-09</td><td>***</td></tr></tbody></table>
<table><tr><td>Face_neutral:Cluster_cluster2</td><td>7.80</td><td>3.372</td><td>1119.35</td><td>2.31</td><td>0.02</td><td>*</td></tr><tr><td>Prosody_neutral:Cluster_cluster2</td><td>3.13</td><td>3.372</td><td>1120.4</td><td>0.92</td><td>0.35</td><td></td></tr><tr><td>Face_neutral:Prosody_neutral:Cluster_cluster2</td><td>-17.11</td><td>4.782</td><td>1121.8</td><td>-3.57</td><td>0.0003</td><td>***</td></tr></table>