# Atypical Audio-Visual Neural Synchrony and Speech Processing in children with Autism Spectrum Disorder

Xiaoyue Wang, Sophie Bouton, Nada Kojovic, Anne-Lise Giraud, Marie

Schaer

► To cite this version:

Xiaoyue Wang, Sophie Bouton, Nada Kojovic, Anne-Lise Giraud, Marie Schaer. Atypical Audio-Visual Neural Synchrony and Speech Processing in children with Autism Spectrum Disorder. 2024. pasteur-04627417

HAL Id: pasteur-04627417

https://pasteur.hal.science/pasteur-04627417

Preprint submitted on 27 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Atypical audio-visual neural synchrony and speech processing in early autism

Xiaoyue Wang, Sophie Bouton, Nada Kojovic, Anne-Lise Giraud, Marie Schaer

► To cite this version:

Xiaoyue Wang, Sophie Bouton, Nada Kojovic, Anne-Lise Giraud, Marie Schaer. Atypical audio-visual neural synchrony and speech processing in early autism. Journal of Neurodevelopmental Disorders, 2025, 17 (1), pp.9. 10.1186/s11689-025-09593-w. pasteur-04627417v2

HAL Id: pasteur-04627417

https://pasteur.hal.science/pasteur-04627417v2

Submitted on 24 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Atypical Audio-Visual Neural Synchrony and Speech Processing in children with Autism Spectrum Disorder

Xiaoyue WANG<sup>1,2*</sup>, Sophie BOUTON<sup>2</sup>, Nada KOJOVIC<sup>3</sup>, Anne-Lise GIRAUD<sup>1,2†</sup>, Marie SCHAER<sup>3†</sup>

1 Auditory Language Group, Department of Basic Neuroscience, University of Geneva, Geneva, Switzerland

2 Institut Pasteur, Université Paris Cité, Hearing Institute, Paris, France

3 Autism Brain & Behavior Lab, Department of Psychiatry, University of Geneva, Geneva, Switzerland

†Co-last authors

*Correspondence: xiaoyue.wang@pasteur.fr; wangxiaoyue43@gmail.com

Running title: Atypical audio-visual synchrony in ASD

## Acknowledgment

**Author Contributions:** Data acquisition and clinical resources: M.S., N. K.; Study design: X.W., N. K., M.S. and A-L.G.; Data analysis: X.W., S.B.; Writing-original draft: X.W.; Writing-review-editing: X.W., S.B., N. K., A-L.G. and M.S.; Supervision: A-L.G., and M.S.

**Funding:** This work is supported by grants from the Swiss National Science Foundation (#163859, #190084, #202235 & #212653 to M.S), by the National Centres of Competence in Research (NCCR) Synapsy (Grant No. 51NF40-185897 to M.S.) and Evolving Language (Grant No. 51NF40_180888 to A-L.G), by grant from Fondation pour l'Audition (FPA IDA11 to A-L.G) as well as by support from the Fondation Privée des Hôpitaux Universitaires de Genève (https://www.fondationhug.org), and the Fondation Pôle Autisme (https://www.pole-autisme.ch). Conflict of interest: The authors declare no conflict of interest.

**Data and materials availability:** The unprocessed datasets for this manuscript are not publicly available yet, because analysis of these data is ongoing as part of a longitudinal study, and results are expected to be published in the future. When all data has been published, requests to access the datasets should be directed to Dr Marie Schaer, marie.schaer@unige.ch, the custom MATLAB analysis scripts will be made available upon request to the Lead Contact, Xiaoyue Wang (xiaoyue.wang@pasteur.fr).
RESEARCH

Open Access

<figure><img src="image_2.png" /></figure>

Check for
updates

# Atypical audio-visual neural synchrony and speech processing in early autism

Xiaoyue Wang<sup>1,2,4*</sup>, Sophie Bouton<sup>2</sup>, Nada Kojovic<sup>3</sup>, Anne-Lise Giraud<sup>1,2†</sup> and Marie Schaer<sup>3†</sup>

**Abstract**

**Background** Children with Autism Spectrum disorder (ASD) often exhibit communication difficulties that may stem from basic auditory temporal integration impairment but also be aggravated by an audio-visual integration deficit, resulting in a lack of interest in face-to-face communication. This study addresses whether speech processing anomalies in young autistic children (mean age 3.09-year-old) are associated with alterations of audio-visual temporal integration.

**Methods** We used high-density electroencephalography (HD-EEG) and eye tracking to record brain activity and gaze patterns in 31 children with ASD (6 females) and 33 typically developing (TD) children (11 females), while they watched cartoon videos. Neural responses to temporal audio-visual stimuli were analyzed using Temporal Response Functions model and phase analyses for audiovisual temporal coordination.

**Results** The reconstructability of speech signals from auditory responses was reduced in children with ASD compared to TD, but despite more restricted gaze patterns in ASD it was similar for visual responses in both groups. Speech reception was most strongly affected when visual speech information was also present, an interference that was not seen in TD children. These differences were associated with a broader phase angle distribution (exceeding pi/2) in the EEG theta range in children with ASD, signaling reduced reliability of audio-visual temporal alignment.

**Conclusion** These findings show that speech processing anomalies in ASD do not stand alone and that they are associated already at a very early development stage with audio-visual imbalance with poor auditory response encoding and disrupted audio-visual temporal coordination.

**Keywords** Autism spectrum disorders (ASD), Gaze direction, Speech envelope, Visual motion, Audio-visual, Oscillation phase entrainment

<footnote>†Anne-Lise Giraud and Marie Schaer are co-last authors.</footnote>

<footnote>*Correspondence:</footnote>

<footnote>Xiaoyue Wang
wangxiaoyuepdf@gmail.com</footnote>

<footnote n="1">¹Auditory Language Group, Department of Basic Neuroscience, University of Geneva, Geneva, Switzerland</footnote>

<footnote n="2">²Institut Pasteur, Université Paris Cité, Hearing Institute, Paris, France</footnote>

<footnote n="3">³Autism Brain & Behavior Lab, Department of Psychiatry, University of Geneva, Geneva, Switzerland</footnote>

<footnote n="4">⁴Department of Medical Psychology and Ethics, School of Basic Medical Sciences, Cheeloo College of Medicine, Shandong University, Jinan, Shandong, China</footnote>
1 Abstract (240 words)

2

3 Background: Children with Autism Spectrum Disorders (ASD) often exhibit communication difficulties that may stem from basic auditory temporal integration impairment but also be aggravated by an audio-visual integration deficit, resulting in a lack of interest in face-to-face communication. This study addresses whether speech processing anomalies in young (mean age 3.09-year-old) children with ASD are associated with alterations of audio-visual temporal integration.

8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Methods: We used high-density electroencephalography (HD-EEG) and eye tracking to record brain activity and gaze patterns in 31 children (6 females) with ASD and 33 typically developing (TD) children (11 females), while they watched cartoon videos. Neural responses to temporal audio-visual stimuli were analyzed using Temporal Response Functions model and phase analyses for audiovisual temporal coordination.

Results: The reconstructability of speech signals from auditory responses was reduced in children with ASD compared to controls, but despite more restricted gaze patterns in ASD it was similar for visual responses in both groups. Speech reception was most strongly affected when visual speech information was also present, an interference that was not seen in TD children. These differences were associated with a broader phase angle distribution (exceeding pi/2) in the EEG theta range in autistic children, signaling reduced reliability of audio-visual temporal alignment.

Conclusion: These findings show that speech processing anomalies in ASD do not stand alone and that they are associated already at a very early development stage with audio-visual imbalance with lousier auditory response encoding and disrupted audio-visual temporal coordination.

23

24 **Keywords (6 keywords):** Autism, Gaze direction, Speech envelope, Visual motion,  
25 Audio-visual, Oscillation Phase entrainment

26

27

28
## Background

Newborns are immediately attracted to the human voice. In-utero exposure to speech sounds enables them to accurately discriminate speech sounds at birth [1-3]. Since vision develops with a delay relative to hearing, babies only progressively discover that vocal stimuli are related to facial movements. Unlike typically developing (TD) children, children with ASD do not show this primary interest in speech [4-7]. Instead, they tend to engage in slow and repetitive visual exploration of their environment, eventually leading to atypical interests [8-13]. This focus on visual aspects of their surroundings allows ASD children to explore the world at their own pace and avoid highly dynamic stimuli such as speech and biological motion [14-16], which are often perceived as overwhelming [17, 18].

A basic auditory dysfunction in ASD might lead to speech-processing anomalies that in turn cascade into a decreased interest in speech [19-24]. Atypical speech processing becomes apparent early in development, and the fact that the associated neural anomalies in delta, theta, and gamma oscillations, accurately predict the severity of future language deficits suggests that they are causal to difficulties in language comprehension and production [25-28]. The tendency of children with ASD to prefer static or slow visual processing [29-32] possibly exacerbates speech reception challenges by counteracting dynamic audio-visual interaction, a crucial process for speech reception in ecological (e.g., noisy) environments [33-36]. Accordingly, excessively long integration time windows for audio and visual stimuli have been reported in children with ASD [31, 32], implying disturbed audio-visual integration.

Two essential mechanisms participate in audio-visual integration. The first one is the relative timing of auditory and visual stimuli: when these stimuli fall about 250 ms apart, they are often perceived as a single event, potentially influencing each other (e.g. the McGurk effect [37, 38]). The second mechanism is the re-synchronization provoked by the stimulus in one sensory modality on the neural responses to the other one [39-44]. Orofacial visual movements typically precede speech onset and lead to an auditory re-synchronization that sharpens responses to speech [39, 45]. Independent from the integration/fusion of the exact visual and speech content, visual re-synchronization enhances speech processing by boosting the tracking of the speech's syllabic structure. While anomalies of audio-visual (AV) binding (AV vs. A + V in ERP study) which traditionally studied over short time windows in ASD are well documented [31, 32, 46-48], how audio and visual signals dynamically synchronize, involving the rhythmic synchronization of auditory and visual signals over longer periods, such as during natural speech exchanges, or when watching

movies, remain hypothetical. Dynamic synchronization reflects the capacity for rapid re-synchronization with zero lag across an extended time course.

Auditory and visual sensory processing both operate rhythmically [49-53]. Visual speech information is characterized by a dominant 2-7 Hz rhythm (theta band [54]) and these quasiperiodic visual cues influence speech perception by modulating auditory neuronal oscillations within the same theta range at about 5 Hz [55-60], corresponding to the typical AV integration temporal time around 250ms [39, 61-64]. The resetting of auditory neural oscillations triggered by visual input [65] rhythmically enhances auditory processing [66], a phenomenon that is already observable in typical children [67]. Despite the documented presence of auditory processing anomalies in ASD at around 3 years [24], we still do not know whether they are associated with dynamic audio-visual synchrony anomalies.

This study aims to fill this gap by investigating with high-density EEG the dynamics of auditory and visual processing in young children with and without ASD, aged 1.13 to 5.56 years old, under naturalistic audio-visual conditions, i.e., when children are watching a popular cartoon adapted to their age. The goal is to compare the quality of the neural encoding/decoding of dynamic auditory and visual stimuli and audio-visual temporal coordination across groups.

## Methods

### Participants

Participants were selected from the Geneva Autism Cohort, a longitudinal study that aims to better understand the developmental trajectories in young children with ASD. This cohort's protocol has been detailed in previous studies [24, 68, 69]. In this study, we used clinical and behavioral assessments, as well as the electroencephalogram (EEG) recorded simultaneously with eye-tracking while children were watching popular cartoon videos.

The sample comprised 31 children with ASD (6 females, mean age = 3.09 years, SD = 0.91, age range: 1.74-5.14) and 32 TD peers (11 females, mean age = 2.95 years, SD = 1.31, age range: 1.31-5.56). Selection criteria for all participants included: age below 6 years, data collected during the participant's initial visit (i.e. at autism diagnosis for the autistic group), clear and accurate markers associated with movie onset, usable raw data for four different movies, and focus on the screen throughout all recordings. The age difference between the two groups was not significant (Bayesian independent samples t-test, BF10 = 0.287).

Autism clinical diagnoses were meticulously confirmed using standardized tools: either the Autism Diagnosis Observation Schedule-Generic (ADOS-G) [70] or the
# Introduction

Newborns are immediately attracted to the human voice. In utero exposure to speech sounds enables them to accurately discriminate speech sounds at birth (1–3). Since vision develops with a delay relative to hearing, babies only progressively discover that vocal stimuli are related to facial movements. Unlike typically developing (TD) children, children with Autism Spectrum Disorders (ASD) do not show this primary interest in speech (4–7). Instead, they tend to engage in slow and repetitive visual exploration of their environment, which has been suggested to lead to atypical interests over time (8–13). Focusing on visual aspects of their surroundings allows children with ASD to explore the world at their own pace, keeping them away from highly dynamic stimuli such as speech and biological motion (14–16), which are often perceived as overwhelming (17,18).

A basic auditory dysfunction in ASD might lead to speech-processing anomalies that in turn cascade into a decreased interest in speech (19–24). Atypical speech processing becomes apparent very early in development, and the fact that early neural anomalies, such as delta, theta, and gamma oscillations, accurately predict the severity of future language deficits could suggest that they are causal to later difficulties in language comprehension and production (25–28). The tendency of children with ASD to prefer static or slow visual processing (29–32) possibly exacerbates speech reception challenges by counteracting dynamic audio-visual interaction, a crucial process for speech reception in ecological (e.g., noisy) environments (33–36). Accordingly, exceedingly long integration time windows for audio and visual stimuli have been reported in children with ASD (31,32), implying disturbed integration of audio and visual stimuli in ASD.

Two essential mechanisms participate in audio-visual integration. The first one is the relative timing of auditory and visual stimuli: when falling within approximately 250 ms of each other, they are often perceived as a single event, potentially influencing each other (e.g. the McGurk effect (37,38)). The second mechanism is the resynchronization provoked by the stimulus in one sensory modality affecting neural responses in the other one (39–44). Orofacial visual movements typically precede the onset of speech, leading to a resynchronization that sharpens the auditory speech response (39,45). And independent from the integration/fusion of the exact visual and speech content, visual resynchronization enhances speech processing by boosting the tracking of the speech’s syllabic structure. While audio-visual temporal integration anomalies in ASD are well documented (31,32,46–48), audio-visual dynamic synchronization anomalies remain hypothetical.

Auditory and visual sensory processing both operate rhythmically (49–53). Visual speech information (lip movements) is characterized by a dominant 2-7 Hz rhythm (theta band (54)) and these quasiperiodic visual cues influence speech perception by modulating auditory neuronal oscillations within the same theta range at about 5 Hz (53–58), corresponding to the typical audio-visual (AV) integration temporal time around 250ms (39,61–64). The reset of auditory neural oscillations triggered by visual input (65) rhythmically enhances auditory processing (66), a
Autism Diagnosis Observation Schedule, Second Edition (ADOS-2) [71]. Recruitment of participants occurred through specialized clinical centers and community-wide announcements. For the TD group, exclusion criteria included any suspicion of atypical psychomotor development, a history of neurological or psychological disorder, or having a first-degree relative with an autism diagnosis. Table 1 summarizes the clinical characteristics of the ASD and TD samples.

## Stimuli and procedure

To explore cortical processing of audio-visual stimuli, we employed a passive and naturalistic task suitable for young children. This task involved viewing an age-appropriate French cartoon “TROTRO” [72–75] (example: http://www.youtube.com/watch?v=jT9C9WCQr8%26t=81s). The selection of “TROTRO” was based on its cognitive accessibility and appeal to the target age group, including kids with ASD. The main character’s verbal interactions enabled isolation of speech and visual motion for brain response analysis. Participants watched four 2.5-minute episodes in a consistent order. Visual engagement was monitored using Tobii Studio, an embedded application of Tobii TX300 eye tracking system. The videos were displayed on a screen with dimensions of 1200 pixels in height (29°38' visual angle) and 1920 pixels in width (45°53') with a refresh rate of 60 Hz, optimized for children’s viewing comfort, with participants seated approximately 60 cm from the screen (Fig. 1A).

## Eye-tracking acquisition and analysis

Gaze data were collected using the Tobii TX300 eye-tracking system (https://www.tobiipro.com), which operates at a sampling rate of 300 Hz. The cartoon was displayed in a frame that provided a visual angle of 26°47'(height) × 45°53'(width). Calibration was performed using a child-friendly procedure integrated into the Tobii system. To maintain consistency and reliability in data quality, we ensured constant lighting conditions in the testing room throughout all sessions. Special consideration was given to the youngest participants, who were seated on their parent’s lap when they felt it more comfortable, a strategy that effectively minimized head and body movements that could have interfered with accurate data collection. We used the Tobii IV-T Fixation

filter [76] to extract fixation data, offering precise measures of visual attention and gaze patterns. Inspired by previous findings of altered gaze distribution in autistic children [13, 69], this study used retinal stimuli around the gaze-fixation point as the source of visual information for subsequent analyses.

## Audio and visual stimuli

We edited the movie soundtrack using Audacity v.2.2.1 to isolate speech excerpts, removing background noise like birdsong and music. Speech envelope data were extracted using the absolute value of the analytic signal [77], downsampled to 1000 Hz, and filtered with a 40 Hz zero-phase Butterworth filter. Visual motion data were tied to participants’ gaze, focusing on stimuli within an 8-degree diameter [78, 79] around the retinal fixation point (318 × 318 pixels) (Fig. 1A1 & A2). The region was converted to grayscale, and luminance differences between successive frames exceeding a threshold of 10 were averaged to represent visual motion [80]. Visual motion was upsampled to match the EEG sampling rate (1000 Hz). Speech envelopes and visual motion were aligned, providing individualized stimulus data based on participants’ gaze patterns for further analysis.

## Stimulus features analysis

In order to assess shared information between speech envelope and visual motion, in autism and TD groups, we calculated mutual information (MI) scores, a dynamic metric, expressed in bits, which quantifies the reduction in uncertainty of one variable when another is observed [77, 78]. We calculated MI using the *quickMI* function from the Neuroscience Information Theory Toolbox [79]. The parameters for this calculation were set to 4 bins, no delay, and a p-value threshold of 0.001 [79]. For generating the MI scores, we concatenated all kept excerpts in the same sequence across subjects, separately for each stimulus feature and each group (ASD and TD). This process was followed by a comparative analysis of MI values between groups.

We only included stimuli corresponding to time periods with usable EEG signals. This resulted in slight variations in the stimulus duration for the ASD and TD groups, which were controlled for. No significant disparities in MI scores emerged between the two groups ($t(1,$

<table><caption>**Table 1** Participants’ demographic information and group comparison of behavioral tests</caption><thead><tr><th rowspan="3"></th><th rowspan="3"></th><th colspan="2">Group</th><th colspan="2">TD (N=32, 11 females)</th><th colspan="2">Bayesian independent samples t-test</th></tr><tr><th colspan="2">ASD (N=31, 6 females)</th><th colspan="2">ASD (N=31, 6 females)</th><th rowspan="2">BF10</th><th rowspan="2">error%</th></tr><tr><th>Mean</th><th>SD</th><th>Mean</th><th>SD</th></tr></thead><tbody><tr><td>Age(in years)</td><td></td><td>3.092</td><td>0.913</td><td>2.947</td><td>1.308</td><td>0.287</td><td>0.011</td></tr><tr><td>ADOS</td><td>Range</td><td>1.74–5.154</td><td></td><td>1.31–5.56</td><td></td><td></td><td></td></tr><tr><td>ADOS</td><td>Range</td><td>7.742</td><td>1.879</td><td>1.031</td><td>0.177</td><td>1.283E25</td><td>4.105E-28</td></tr><tr><td>ADOS</td><td>Range</td><td>4–10</td><td></td><td>1–2</td><td></td><td></td><td></td></tr></tbody></table>
phenomenon that is already observable in typical children (67). Despite the documented presence
of auditory processing anomalies in ASD around 3-year-old (24), we still ignore whether they are
associated with dynamic audio-visual synchrony anomalies.

This study fills this gap by investigating with high-density EEG the dynamics of auditory and
visual processing in young children with and without ASD, aged 1.13 to 5.56 years old, under
naturalistic audio-visual conditions, i.e. when children are watching a popular cartoon adapted to
their age. The goal is to compare the quality of the neural encoding/decoding of dynamic auditory
and visual stimuli and audio-visual temporal coordination across groups.

# Methods

## 1 Participants

Participants were selected from the Geneva Autism Cohort, a longitudinal study that aims at better
understanding the developmental trajectories in young children with ASD. This cohort's protocol
has been detailed in previous studies (22,62,63). In this study, we used clinical and behavioral
assessments, as well as the electroencephalogram (EEG) recorded simultaneously with eye-
tracking when children were watching popular cartoon videos.

The sample comprised 31 children diagnosed with ASD (6 females, mean age = 3.09 years, SD =
0.91, age range: 1.74 - 5.14) and 32 TD peers (11 females, mean age = 2.95 years, SD = 1.31, age
range: 1.31 - 5.56). Selection criteria for all participants included: age below 6 years, data collected
during the participant's initial visit (i.e. at autism diagnosis for the autistic group), clear and
accurate markers associated with movie onset, usable raw data for four different movies, and focus
on the screen throughout all recordings. The age difference between the two groups was not
significant (Kolmogorov-Smirnov D = 0.28, p = 0.17).

ASD clinical diagnoses were meticulously confirmed using standardized tools: either the Autism
Diagnosis Observation Schedule-Generic (ADOS-G) (70) or the Autism Diagnosis Observation
Schedule, Second Edition (ADOS-2) (71). Recruitment of participants occurred through
specialized clinical centers and community-wide announcements. For the TD group, exclusion
criteria included any suspicion of atypical psychomotor development, a history of neurological or
psychological disorder, or having a first-degree relative with an autism diagnosis.

Informed consent was obtained from the parents of all participants prior to inclusion in the study.
The research was conducted with the ethical standards set forth by the Ethics Committee of the
Faculty of Medicine at the University of Geneva Hospital and adhered to the principles outlined
in the Declaration of Helsinki.

## 2 Stimuli and Procedure
<figure><img src="image_3.png" /><figcaption>**Fig. 1** Overview of Experimental Procedures and Features of Interest. **A** Experimental procedures **B** Gaze fixation. Example of individual gaze fixation points (green dots) on a black and white image; **C** Example of gaze-captured screen areas. Depiction of screen areas captured by the gaze of participants in ASD and TD (Typically Developing) groups. **D** Example of a stimulus speech envelope from the cartoon soundtrack. **E** Visual motion corresponds to the same stimulus in each group. **F** Comparison of speech envelope and visual motion. Mutual Information (MI) between ASD and TD groups (ns. *p* > 0.05)</figcaption></figure>

61) = 1.250, *p* = 0.216, Cohen's *d* = 0.315, Fig. 1C), indicating that these minor differences did not lead to notable group differences in the shared information between the speech envelope and visual motion.

## EEG acquisition and pre-processing

The EEG data were acquired using a 129-electrode (Hydrogel Geodesic Sensor Net (HCGSN) system (Electrical Geodesics, USA) at a sampling rate of 1000 Hz. During recording, the signals were subjected to real-time 0–100 Hz band-pass filtering. The reference electrode was positioned at the vertex (Cz). Data pre-processing was conducted using the EEGLAB v2019 toolbox within the MATLAB environment [80] and Cartool (https://site.s.google.com/site/cartoolcommunity/). One hundred and ten channels were kept, excluding the cheek and neck

electrodes to prevent contamination by muscle artifacts. EEG signals were filtered using a zero-phase fourth-order Butterworth bandpass (0.1–70 Hz) and a 50 Hz notch filter to eliminate power line noise. EEG data were visually inspected to remove movement artifact-contaminated periods. Bad channels were identified and excluded for exhibiting excessive signal amplitude. Eye blinks, sacades, electrical noise, and heartbeat artifacts were removed using independent component analysis (ICA). A spherical spline interpolation was used to interpolate the channels contaminated by noise using the ICA-corrected data. Finally, a common average reference was recalculated on the cleaned data, with an additional step of applying a 30 Hz low-pass filtering [81]. To ensure that all the EEG signals and stimulus features were on a similar scale and thus comparable, we normalized both the
To explore cortical processing of audio-visual stimuli, we employed a passive and naturalistic task suitable for young children. This task involved viewing an age-appropriate French cartoon "TROTRO" (72–75) (example: https://www.youtube.com/watch?v=jT9C9WCIOr8&t=81s). The selection of “TROTRO” was based on its cognitive accessibility and appeal to the target age group. Importantly, TROTRO, the main character, speaks and interacts verbally with other characters, which allows us to isolate the speech soundtrack and associated visual motion and to probe related brain responses. Participants watched four Trotro episodes, each lasting approximately 2.5 minutes. The videos were presented in a consistent, predetermined order to all participants. To monitor the participant’s visual engagement with the stimulus, Tobii Studio (Tobii® Technology, Sweden) was used. The screen for the video display was configured with dimensions of 1200 pixels in height (29°38′ visual angle) and 1920 pixels in width (45°53′), with a refresh rate of 60 Hz. This setup was optimized for clear and comfortable viewing in children. Participants were seated at an optimal distance of approximately 60 cm from the screen (Figure 1A).

## 2.1 Eye-tracking acquisition and analysis

Gaze data were collected using the Tobii TX300 eye-tracking system (https://www.tobiipro.com), which operates at a sampling rate of 300 Hz. This high-frequency data collection was instrumental in assessing participants’ visual exploration patterns during the cartoon viewing. The cartoon was displayed in a frame that provided a visual angle of 26°47′(height) × 45°53′(width). Calibration was performed using a child-friendly procedure integrated into the Tobii system, specifically designed to engage young participants. This calibration was critical for accurate gaze position tracking and was repeated as necessary, particularly in instances where the eye-tracking device showed any discrepancies in detecting the participant’s gaze. To maintain consistency and reliability in data quality, we ensure constant lighting conditions in the testing room throughout all sessions. Special consideration was given to the youngest participants, who were seated on their parent’s lap when they felt more comfortable in this setting, a strategy that effectively minimized potential head and body movements that could interfere with accurate data collection. For data analysis, we employed the Tobii IV-T Fixation filter (13,76). This tool is specifically designed to extract fixation data, providing us with precise and reliable measures of visual attention and engagement.

## 2.2 Audio and visual stimuli

We edited the original movie soundtrack using Audacity v.2.2.1 (Audacity Team, 2021) to isolate speech excerpts while removing extraneous background noise such as birds' singing and musical interludes.

Having extracted the video speech-track, we explored corresponding visual dynamics. Within the video excerpts that contained speech, we distinguished two key components: speech envelope (Figure 1B1) and visual motion (Figure 1B2). To extract the speech envelope, we used the absolute value of the analytic signal (77). The obtained speech envelope was then down-sampled to 1000
EEG signals and stimulus features (i.e. speech envelope and visual motion) using the nt_normcol function (Noisetools: http://audition.ens.fr/adc/NoiseTools/).

## Temporal response functions (TRF)

To quantify how well EEG in ASD and TD children linearly varied with the stimulus features, we performed regularized regression (with ridge parameter $\lambda$) as implemented in the mTRF toolbox [82]. The TRF models the strength and direction of the brain's response to stimulus features, such as speech envelope or visual motion, at specific time lags. For the TRF modeling, we downsampled all signals to a rate of 100 Hz to accelerate computation.

## Estimation of TRF using forward encoding models

We used a forward encoding model to predict EEG responses over time lags from 300 ms before to 300 ms after the stimulus. Separate univariate models were constructed for auditory (speech envelope, A-only) and visual (visual motion, V-only) stimuli. Additionally, a multivariate model (AV-joint) integrated both regressors, using trade-off weights to balance auditory and visual contributions, ensuring balanced representation of multisensory integration.

We compared the AV-joint model with A-only and V-only models to assess the benefits of audio-visual integration over unimodal processing. Using an n-fold leave-one-out cross-validation strategy, “generic” models were created to predict individual EEG data from TRFs derived from other participants. Model performance was optimized through a parameter search for the regularization parameter $\lambda$ (see supplementary methods for $\lambda$ selection). Pearson's correlation coefficient (r) between EEG signals with TRF-predicted signals was used to quantify model prediction accuracy per electrode. Then, correlations were averaged across participants to create a scalp-wide map of prediction accuracy. Finally, prediction accuracy was converted into Z-scores for statistical comparison by subtracting the surrogate data mean and dividing by its standard deviation. Surrogate distributions were generated by randomly shifting testing EEG segments, maintaining temporal structure. This analysis evaluated how accurately stimulus features were predicted from EEG data for each participant.

To quantitatively compare the accuracy between the ASD and TD groups, we used a cluster-based permutation test with 1000 randomization iterations, following the approach of Maris and Oostenveld [83]. Clusters were defined by considering both time and spatial electrode configurations, requiring each cluster to include at least two adjacent electrodes. A pivotal aspect of this approach was to ensure that the cluster-level type-I-error probability remained below the 0.05 threshold. This strategy was

effective in controlling the family-wise error rate, maintaining it within the 5% type-I-error rate boundary.

## Stimulus reconstruction using decoding models

We trained EEG decoders within a -300 to 300 ms post stimulus onset temporal window, using leave-one-out cross-validation and optimization to assess the accuracy of stimulus reconstruction (, i.e., speech envelope and visual motion). This approach allows us to identify the most informative segments for decoding, i.e. the time-lags with the highest EEG-stimulus synchronization. To determine the accuracy of stimulus reconstruction and select the best time lags, we used the Kruskal-Wallis test with Dunn's multiple comparisons test [84]. This non-parametric statistical method was chosen for its capability to handle variations in group means and variances across different conditions.

## Low-frequency tracking of audio-visual signal

To explore whether the combined processing of auditory and visual stimuli relies on the tracking of audio-visual signals by low-frequency brain activity, we used a coherence-based and phase-based analytical framework [85]. This approach probed the interplay between neural responses and stimulus features by comparing their magnitude spectra and the phase relationship. Our analyses are centered on the delta and theta frequency bands, which are critical for effective integration of multimodal information [86].

## Coherence analysis

We assessed individual responses to speech envelope and visual motion by computing magnitude-squared coherence for each trial and electrode using the *mscohere* function in Matlab, applying Welch's averaged modified periodogram method. The analysis spanned a frequency range from 0.1 to 30 Hz, in 0.33 Hz steps [87].

The analysis targeted delta (δ, ~4 Hz) and theta (θ, 4–8 Hz) frequency bands, identifying frequencies where coherence peaked most prominently for each stimulus condition. Statistical comparisons across groups and stimuli were conducted using the clusters identified through the method outlined in Sect. 4, combined with a nonparametric test and Dunn's multiple comparisons test [84]. Statistical significance was established using a surrogate-corrected coherence approach. Surrogate distributions were generated by randomly shifting the neural time course relative to the stimulus feature time courses, preserving their original temporal structure. This process was repeated 50 times for each stimulus condition to generate a robust surrogate distribution. The resulting coherence values were then standardized (Z-scored) against this distribution.
1 Hz and filtered using a zero-phase, fourth-order Butterworth filter with a 40 Hz cutoff. The visual

2 motion was restricted to the participant's gaze-attended zone. We considered individual gaze-

3 fixation positions and the size of retinal stimuli around the gaze-fixation point. Our analysis took

4 into account the decline in visual acuity and the crowding effects of parafoveal vision (2–5 degrees

5 from the fixation point) (78). The region for capturing visual stimuli through eye gaze was defined

6 as a square with sides the length of an 8-degree diameter centered on the fixation point, aligning

7 with findings on the effective visual span guiding saccades (79). This corresponded to

8 approximately 318 × 318 pixels (Figure 1 A1 & A2). The gaze-capture zone was converted to

9 greyscale, and luminance differences between successive frames were computed following

10 established methodologies (80). The extraction of corresponding visual stimuli involved

11 converting the region of interest of each frame to a grayscale and computing the luminance

12 difference between successive frames. Pixels with a luminance change greater than 10 (a threshold

13 chosen to mitigate video recording noise) were selected, and the average luminance change

14 constructed the visual motion component.

15

16 Visual motion data was upsampled to match the 1000 Hz EEG sampling frequency. Speech

17 envelopes and visual motion corresponding to the same speech-track were aligned and prepared

18 for further analysis. With this method, visual motion is inherently individual-specific, as it relies

19 on the visual exploration of each participant.

20

21 **2.3 Stimulus features analysis**

22 In order to assess shared information between speech envelope and visual motion, in ASD and TD

23 groups, we calculated mutual information (MI) scores, a dynamic metric, expressed in bits, which

24 quantifies the reduction in uncertainty of one variable when another is observed (81,82). We

25 calculated MI using the *quickMI* function from the Neuroscience Information Theory Toolbox

26 (83). The parameters for this calculation were set to 4 bins, no delay, and a p-value threshold of

27 0.001 (83). For generating the MI scores, we concatenated all kept excerpts in the same sequence

28 across subjects, separately for each stimulus feature and each group (ASD and TD). This process

29 was followed by a comparative analysis of MI values between groups.

30

31 We only included stimuli corresponding to time periods with usable EEG signals. This resulted in

32 slight variations in the stimulus duration for the ASD and TD groups, for which we controlled. No

33 significant disparities in MI scores emerged between the two groups ($t(1, 61)= 1.250, p = 0.216,$

34 Cohen's *d* = 0.315, Figure 1C), indicating that these minor differences did not lead to notable

35 group differences in the shared information between the speech envelope and visual motion.

36

37 **3 EEG acquisition and pre-processing**

38 The EEG data were acquired through a 129-electrode (Hydrocel Geodesic Sensor Net (HCGSN))

39 system (Electrical Geodesics, USA) at a sampling rate of 1000 Hz. During recording, the signals

40 were subjected to real-time 0-100 Hz band-pass filtering. The reference electrode was positioned
## Phase analysis

We also performed a phase analysis by calculating the cross-power spectral density (CPSD) phase for each stimulus, electrode, and trial. This was done using the cpsd function in Matlab, employing parameters consistent with the coherence analyses. Phase values were determined based on the peak frequency identified in the coherence analysis. Group comparisons were conducted using Matlab's Circular Statistics Toolbox [88]. A two-way parametric ANOVA for circular data was performed to facilitate a nuanced comparison between pairs of conditions and groups, followed by post-hoc comparisons using the Watson-Williams multi-sample test [88]. Meanwhile, the Rayleigh test was used to investigate whether phase distribution was unimodal. Our focus was primarily on electrodes identified through TRF estimation outcomes.

## Results

### Atypical speech envelope processing in autistic children

We found distinct neural tracking patterns for the auditory and visual parts of the stimuli. Different scalp distribution patterns were observed between the two groups (Fig. 2) with cluster-correction *p* < 0.05. The ASD group had reduced neural response to the speech envelope relative to the TD group (Fig. 2A, top row). In contrast, there

was no between-group difference in visual motion processing (Fig. 2A middle row).

A univariate stimulus reconstruction accuracy measure was compared between the ASD and TD groups, and aligned with the neural tracking findings, suggesting that speech processing is primarily impaired in autistic children. While reconstruction accuracy was comparable for speech envelope and visual motion (*p* > 0.9999, Fig. 2B) in the TD group, it was lower for speech than for visual motion (*p* < 0.0001, Fig. 2B) in the ASD group. These results suggest intact visual processing dynamic communicative stimuli in young children with ASD but atypical auditory processing.

### Audiovisual integration anomaly in autism disrupts visual enhancement of auditory processing

We then explored whether speech anomalies in autism are limited to auditory processing difficulties or associated with audiovisual (AV) processing anomalies. First, the joint model suggested weaker AV representation in the ASD than in the TD group (Fig. 2A bottom row) and we found the expected stronger neural representation of the combined AV stimulus compared to individual single stimuli in both groups.

Distinct audiovisual integration patterns in ASD and TD children (Table 2; Fig. 3) were found by comparing

<figure><img src="image_4.png" /><figcaption>**Fig. 2** Comparison of audio, visual, and AV models. **A.** Neural representations in ASD (left) and TD (middle) groups, for each model (A-speech envelope, V-visual motion, and AV joint) across all scalp electrodes. The right column shows EEG channels where significant group differences are observed using cluster-based nonparametric statistics (*p* < 0.05; with a positive t-value indicating greater predictability in the ASD group compared to the TD group). **B.** Stimulus reconstruction accuracy for speech envelope and visual motion in both groups. Error bars indicate the standard error of the mean. Significance levels are indicated as follows: ‘ns’ for *p* > 0.05 (not significant), * for *p* < 0.05, ** for *p* < 0.01, *** for *p* < 0.001, and **** for *p* < 0.0001</figcaption></figure>
1 at the vertex (Cz). Data pre-processing was conducted using the EEGLAB v2019 toolbox within
2 the MATLAB environment (84) and Cartool (https://sites.google.com/site/cartoolcommunity/).

3 One hundred and ten channels were kept excluding the cheek and neck to prevent contamination
4 by muscle artifacts. EEG signals were filtered using a zero-phase fourth-order Butterworth
5 bandpass (0.1-70 Hz) and a 50 Hz notch filter to eliminate power line noise. EEG data were
6 visually inspected to remove movement artifact-contaminated periods. Bad channels were first
7 identified and excluded for excessive signal amplitude. Eye blinks, saccades, electrical noise, and
8 heartbeat artifacts were excluded using independent component analysis (ICA). A spherical spline
9 interpolation was used to interpolate the channels contaminated by noise using the ICA-corrected
10 data. Finally, a common average reference was recalculated on the cleaned data, with an additional
11 step of applying a 30Hz low-pass filtering (80). To ensure that all the EEG signals and stimulus
12 features were on a similar scale and thus comparable, we normalized both the EEG signals and
13 stimulus features (i.e. speech envelope and visual motion) using the nt_normcol function
14 (Noisetools: http://audition.ens.fr/adc/NoiseTools/).

15

4 Temporal Response Functions (TRF)

17 To quantify how well EEG in ASD and TD children linearly varied with the stimulus features, we
18 performed regularized regression (with ridge parameter λ) as implemented in the mTRF toolbox
19 (85). The Temporal Response Function (TRF) captures how the brain’s EEG activity correlates
20 with and responds to changes in the stimulus over time, providing a dynamic mapping of the neural
21 processing of the stimulus features. More precisely, the TRF accounts for the fact that the brain’s
22 response to a stimulus occurs with a certain delay. Specifically, the TRF analysis models the
23 relationship between each stimulus feature (i.e. speech envelope or visual motion) and the brain’s
24 response, particularly the time-lagged aspects of this relationship. The TRF includes a coefficient
25 for each time lag that quantifies the strength and direction (positive or negative) of the brain’s
26 response to the stimulus at that specific time delay. The lags in a TRF are represented as a series
27 of time-points or intervals, typically in milliseconds. For the TRF estimation, we downsampled all
28 signals to a rate of 100 Hz to speed computation.
29

29

4.1. Estimation of TRF using forward encoding models

30 Specifically, we used a forward encoding model approach. Since changes in the EEG signal are
31 expected with an unknown time lag after the stimulus, predictions were computed over a range of
32 time lags between 300 ms earlier than the stimulus and 300 ms later than the stimulus. To single
33 out brain signals involved in auditory and visual dynamics, we constructed two distinct univariate
34 encoding models using the speech envelope and visual motion as independent regressors
35 respectively (labeled A-only and V-only). Further advancing our investigation, we developed a
36 multivariate encoding model, labeled ‘AV-joint’, which integrates both the speech envelope and
37 visual motion as regressors. The integration within this model is operationalized through the
38 assignment of trade-off weights to the AV regressor, which are calibrated to reflect the respective
39 contributions of auditory and visual stimuli. These trade-off weights ensure a balanced
40
<table><caption>**Table 2** The statistical difference among joint (AV) model and single models (A- &V-) for speech envelope and visual motion</caption><thead><tr><th></th><th>Dunn's multiple comparisons test</th><th>Mean rank diff.</th><th>Significant?</th><th>Adjusted P Value</th></tr></thead><tbody><tr><td rowspan="2">ASD</td><td>speech envelope</td><td>-60.48</td><td>Yes</td><td>0.0304</td></tr><tr><td>visual motion</td><td>-137.9</td><td>Yes</td><td>&lt; 0.0001</td></tr><tr><td rowspan="2">TD</td><td>speech envelope</td><td>-51.09</td><td>No</td><td>0.1414</td></tr><tr><td>visual motion</td><td>-58.47</td><td>Yes</td><td>0.0373</td></tr></tbody></table>

the decoding accuracies for speech envelope and visual motion across univariate (A-only and V-only) and multivariate (AV-joint) models. Notably, in the TD group, the accuracy of speech envelope reconstruction in the AV-joint model (concurrent speech envelope and visual motion) did not significantly differ from the A-only model ($p = 0.1414$, Fig. 3A). Conversely, in the ASD group, the speech envelope was less accurately decoded in the AV-joint model than in the A-only model ($p = 0.0304$, Fig. 3B), indicating a disruptive effect of AV integration on auditory processing specific to this group.

<table><caption>**Table 3** The statistical difference of the decoding accuracy evolution for speech envelope and visual motion</caption><thead><tr><th></th><th>Dunn's multiple comparisons test</th><th>Mean rank diff.</th><th>Significant?</th><th>Adjusted P Value</th></tr></thead><tbody><tr><td>Visual motion</td><td>ASD v.s.TD</td><td>-37.55</td><td>Yes</td><td>0.0003</td></tr><tr><td>Speech envelope</td><td>ASD v.s.TD</td><td>-13.32</td><td>No</td><td>0.8866</td></tr><tr><td>TD</td><td>V vs. A</td><td>-6.094</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td>ASD</td><td>V vs. A</td><td>-30.32</td><td>Yes</td><td>0.0065</td></tr></tbody></table>

A: speech envelope

V: visual motion

A decrease in visual motion reconstruction accuracy in the AV-joint model compared to the V-only model was observed in both groups (ASD group: $p < 0.0001$, TD group: $p = 0.0373$, Fig. 3), with a more pronounced decrement observed in the ASD group ($p = 0.0003$, Table 3). These findings suggest that integrating AV speech signals has a cost on visual processing, and that this cost is higher in children with ASD.

<figure><img src="image_5.png" /><figcaption>**Fig. 3** Evaluation of decoding accuracy in TD A and ASD B. Stimulus reconstruction accuracy: speech envelope(A-) and visual motion(V-) in both the single-stimulus model (A-only = A-A and V-only = V-V) and the AV-joint model (A-AV, and V-AV)). Significance levels are indicated as follows: ‘ns’ for $p > 0.05$ (not significant), * for $p < 0.05$, ** for $p < 0.01$, *** for $p < 0.001$, **** for $p < 0.0001$. For additional details, see Supplemental Figs. 3–1</figcaption></figure>
1 representation within the model, enabling for an accurate representation of the brain’s concurrent
2 processing of both modalities. This balanced approach aims to embody effective multisensory
3 integration, preventing the overshadowing of one sensory modality by another.

4
5 The comparative analysis of the AV-joint model against the A-only and V-only models is essential
6 to elucidate the incremental benefits of simultaneous audio-visual integration over unimodal
7 processing. By evaluating the performance and predictive accuracy of the multivariate model
8 relative to its univariate counterparts (one focused only on auditory processing and the other on
9 visual processing), we aim to substantiate the hypothesis that the synergistic consideration of
10 auditory and visual stimuli offers a more comprehensive understanding of sensory processing in
11 naturalistic listening environments.
12

13 For each group, we created "generic" models that predict the EEG data of an individual participant
14 (nth participant) using a TRF derived from the EEG data of the other participants (the remaining
15 n-1 participants). We thus implemented an *n*-fold leave-one-out cross-validation strategy and
16 optimized the model through a parameter search for the regularization parameter λ.

17
18
## 4.2 Selection of optimal regularization parameter

19 To mitigate the risk of data overfitting in forward encoding models, we integrated an optimized
20 regularization parameter λ, determined for each stimulus feature. To do so, we trained multiple
21 model iterations on subsets of data comprising n-1 participants. During these iterations, λ was
22 systematically adjusted within a predefined range from 10⁰ to 10⁵, with increments in the exponent
23 of 0.5. The criterion for selecting the optimal λ was maximal predictive accuracy. This was
24 quantitatively assessed by calculating Pearson's correlation coefficient (r) between the predicted
25 EEG signals and the observed signals, for each electrode and each participant. Once optimal λ was
26 identified, the refined model was used to predict the EEG responses of the nth participant, using
27 an n-fold leave-one-out cross-validation paradigm. This methodology was applied consistently
28 across both ASD and TD groups, ensuring the robustness and reliability of the predictive models
29 within each group.
30

31
32
33
34
35
36
37
38
39
40
# 4.3. Pearson's correlation coefficient (r)

Pearson's correlation coefficient (r) was computed to quantify the model's prediction accuracy. For each electrode, the correlation between the EEG signals and the TRF-predicted signals was calculated across all time points. This process was repeated for each participant for the validation set. The correlation coefficients were then averaged across participants to obtain a mean Pearson's r-value per electrode. This allowed us to create a scalp-wide map of the model's prediction accuracy, highlighting areas with the strongest correlation between the predicted and observed EEG responses. The accuracy of EEG predictions derived from the optimized model was then converted into Z-scores for further statistical group comparisons. The Z-score computation involved subtracting the mean of surrogate data and dividing it by the standard deviation of
Surprisingly, we found distinct time-lags in auditory and visual decoding accuracies between ASD and TD when analyzing the temporal dynamics of audiovisual integration. In the TD group (Fig. 4A), auditory decoding reached significance at ~200 ms, while visual decoding took ~50 ms. In contrast, the ASD group showed the opposite pattern: ~200 ms for visual decoding and ~50 ms for auditory decoding (Fig. 4B). This reveals a visual lead in TD children, aligning with visual cues typically preceding sounds, but an auditory lead in autistic children, indicating a fundamental shift in sensory processing order.

## Theta-range desynchronization of audio-visual responses in autism

To determine whether speech tracking anomalies in ASD stem from general stimulus/brain synchronization deficits or audiovisual integration issues, we analyzed the stimulus-response relationships in the delta (1–4 Hz) and theta (4–8 Hz) bands. Both groups showed higher stimulus/brain coherence in the delta than the theta band, with remarkably similar average coherence values and distribution patterns across groups (Fig. 5; Table 4). This indicates that the capacity of neural synchronization to auditory and visual stimuli is consistent in ASD and TD groups.

Given the preserved synchrony between brain activity and external stimuli in each modality, we then sought whether audio-visual integration anomalies, notably the inverted AV temporal patterns, are associated with a

phase desynchronization of auditory and visual processing. In the delta band, we observed similar phase angles for both groups ($F(1,62) = 0.494, p < 0.470$), indicating comparable phase locking at this frequency, with small angles indicating the absence of delta band phase-shift between modalities. Yet, a significant group effect was observed in the theta band. In children with ASD the phase-shift amounted to 180 degrees and the group difference was significant ($F(1,62) = 12.05, p < 0.001$) (Fig. 6). The observed 180-degree phase shift in autism could suggest that auditory and visual information is out of sync: when one sensory modality is at its peak processing efficiency, the other is at its lowest, potentially leading to disjointed, even conflicting sensory processes.

## AV phase-shift is related to auditory encoding accuracy in TD and visual encoding accuracy in autism

Finally, we explored the relationship between the AV phase-shift in the theta band and the accuracy of auditory and visual information reconstruction within an unimodal framework (Fig. 7). As expected, in TD children the AV phase-shift did not influence visual reconstruction accuracy ($r = 0.033, p = 0.858$), but there was a weak negative correlation between the phase-shift extent and speech reconstruction accuracy ($r = -0.272, p = 0.132$): when the AV phase-shift increased speech reconstruction accuracy decreased, which given the visual lead previously observed could suggest a causal effect. A different pattern was seen in children with ASD, with no relation between the phase shift extent and speech reconstruction

<figure><img src="image_6.png" /><figcaption>**Fig. 4** Optimal EEG-stimuli time lag for ASD (red) and TD (black) groups. A depicts the optimal time-lag observed in the reconstruction of stimulus features in AV-joint model, specifically speech envelope (A-) and visual motion(V-); Positive values represent stimulus lead EEG signal. B illustrates the A-V time lag AV-joint model. Positive values represent V leads A. Significance levels are indicated as follows: ns > 0.05, \*p < 0.05, \*\*p < 0.01, \*\*\*p < 0.001, \*\*\*\*p < 0.0001</figcaption></figure>
surrogate data. Surrogate distributions were generated by randomly shifting (50 times) the orders of the testing EEG segments, preserving their original temporal structure. In summary, this analysis indicated the accuracy with which stimulus features are predicted from the EEG data for each participant.

To quantitatively compare the accuracy between the ASD and TD groups, we used a cluster-based permutation test with 1000 randomization iterations, following the approach of Maris and Oostenveld (86). Clusters were defined by considering both time and spatial electrode configurations, requiring each cluster to include at least two adjacent electrodes. A pivotal aspect of this approach was ensuring that the cluster-level type-I-error probability remained below the 0.05 threshold. This strategy was effective in controlling the family-wise error rate, maintaining it within the 5% type-I-error rate boundary.

## 4.4. Stimulus reconstruction using decoding models

We trained decoders using EEG data across a wide temporal range, from -300 to 300 ms relative to the stimulus, aiming for optimal stimulus reconstruction. This analysis involved an iterative process of leave-one-out cross-validation and optimization techniques to refine our decoding accuracy, assessing how effectively the EEG signals could predict the stimulus features (i.e., visual motion and speech envelope) for each group.

To enhance our understanding of the temporal alignment between EEG signals and the stimuli, our methodology evolved to focus on discrete, predefined time-lag intervals for decoder training, rather than a continuous range. By pinpointing discrete intervals and evaluating their decoding success, we identified the most informative time-lag segment that yielded the highest fidelity in stimulus reconstruction. Determining this optimal time-lag interval informs us about the specific moments when the EEG data are most synchronized with the stimulus features, thus shedding light on the precise neural timing critical for effective sensory processing and integration.

For the decoding models, we determined the accuracy of stimulus reconstruction accuracy and the identity of the best time lag using the Kruskal-Wallis test with Dunn's multiple comparisons test (87). This non-parametric statistical method was chosen for its capability to handle variations in group means and variances across different conditions.

## 5. Low-frequency tracking of audio-visual signal

To explore whether the combined processing of auditory and visual stimuli can be explained by the tracking of audio-visual signals by low-frequency brain activity, we used a coherence-based and phase-based analytical framework (88). This approach probed the interplay between neural responses and stimulus features by comparing their magnitude spectra and the phase relationship. Our focus centered on the delta and theta frequency bands, which are critical for the temporal coordination necessary for effective integration of multimodal information (89).
<figure><img src="image_7.png" /><figcaption>**Fig. 5** Stimulus-response coherence in Theta and Delta Bands for ASD (red) and TD (black) groups. The plot displays the coherence between stimulus and response for Speech Envelope and Visual Motion. Error bars represent the standard error of the mean. The coherence levels are compared within the specific frequency bands of interest, highlighting potential group differences in sensory processing. Significance levels are indicated as follows: ns > 0.05, \*p < 0.05, \*\*p < 0.01, \*\*\*p < 0.001, \*\*\*\*p < 0.0001</figcaption></figure>

<table><caption>**Table 4** The statistical difference across groups and frequency bands in stimulus-response coherence</caption><thead><tr><th></th><th></th><th>Dunn's<br>multiple com-<br>parisons test</th><th>Mean rank<br>diff.</th><th>Significant?</th><th>Ad-<br>justed<br>P Value</th></tr></thead><tbody><tr><td rowspan="2">ASD</td><td rowspan="2">A</td><td>delta vs. theta</td><td>162.7</td><td>Yes</td><td>&lt; 0.0001</td></tr><tr><td>delta vs. theta</td><td>191.5</td><td>Yes</td><td>&lt; 0.0001</td></tr><tr><td rowspan="2">ASD</td><td rowspan="2">delta</td><td>A vs. V</td><td>-7.774</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td>A vs. V</td><td>20.94</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td rowspan="2">TD</td><td rowspan="2">A</td><td>delta vs. theta</td><td>178.6</td><td>Yes</td><td>&lt; 0.0001</td></tr><tr><td>delta vs. theta</td><td>176.4</td><td>Yes</td><td>&lt; 0.0001</td></tr><tr><td rowspan="2">TD</td><td rowspan="2">V</td><td>A vs. V</td><td>-0.6563</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td>A vs. V</td><td>-2.844</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td rowspan="2">A</td><td rowspan="2">delta</td><td>ASD vs. TD</td><td>12.3</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td>ASD vs. TD</td><td>28.19</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td rowspan="2">V</td><td rowspan="2">delta</td><td>ASD vs. TD</td><td>19.42</td><td>No</td><td>&gt; 0.9999</td></tr><tr><td>ASD vs. TD</td><td>4.406</td><td>No</td><td>&gt; 0.9999</td></tr></tbody></table>

A: speech envelope

V: visual motion

accuracy ($r=0.197, p=0.288$; group\*phase-shift $t = -1.835, p=0.072$), but a weak negative correlation between the phase-shift extent and the accuracy of visual information reconstruction ($r = -0.325, p=0.074$; group\*phase-shift $t=1.062, p=0.293$), with larger AV

phase-shifts linked to poorer visual reconstruction accuracy. Similarly, given the auditory lead observed in the group with ASD, this could suggest a causal effect.

## Discussion

Using several analyses of the EEG recorded in very young children with and without ASD while they were watching a short animated movie, we confirmed previous results showing profound anomalies in the capacity to follow speech rhythms [24, 89], an essential prerequisite to speech comprehension. The present study goes beyond this observation by showing that children with ASD did not exhibit the natural dominance of auditory processing when exposed to natural audio-visual speech conditions. Instead, audio-visual processing was impacted by a temporal misalignment of these sensory inputs, which disrupted the predictive processing typically at play when perceiving speech.

## Audio-visual integration anomalies interfere with sensory encoding in ASD

The synchronization of the two sensory modalities plays a pivotal role in understanding the communicative challenges observed in ASD. Our study reveals that speech
## 1. Coherence Analysis

We assessed individual responses to speech envelope and visual motion by computing magnitude-squared coherence for each trial and electrode using the *mscohere* function in Matlab, applying Welch's averaged modified periodogram method. The analysis spanned a frequency range from 0.1 to 30 Hz, in increments of 0.33 Hz (90).

The analysis targeted delta (δ, ~4 Hz) and theta (θ, 4-8 Hz) frequency bands, identifying frequencies where coherence peaked most prominently for each stimulus condition. Statistical comparisons across groups and stimuli were conducted using the clusters identified through the method outlined in Section 4, combined with a nonparametric test and Dunn's multiple comparisons test (87). Significance of these findings was established using a surrogate-corrected coherence approach. Surrogate distributions were generated by randomly shifting the neural time course relative to the stimulus feature time courses, preserving their original temporal structure. This process was repeated 50 times for each stimulus condition to generate a robust surrogate distribution. The resulting coherence values were then standardized (Z-scored) against this distribution.

## 2. Phase Analysis

We also performed a phase analysis by calculating the cross-power spectral density (CPSD) phase for each stimulus, electrode, and trial. This was done using the *cpsd* function in Matlab, employing parameters consistent with the coherence analyses. Phase values were determined based on the peak frequency identified in the coherence analysis. Group comparisons were conducted using Matlab's Circular Statistics Toolbox (91). A two-way parametric ANOVA for circular data was performed to facilitate a nuanced comparison between pairs of conditions and groups, followed by post-hoc comparisons using the Watson-Williams multi-sample test (91). Meanwhile, the Rayleigh test was performed to investigate whether phase distribution is under unimodal distribution. Our focus was primarily on electrodes identified through TRF estimation outcomes.
<figure><img src="image_8.png" /><figcaption>**Fig. 6** Phase-shift distribution between speech envelope and visual motion. This figure shows the phase-shift distribution between the brain processes of speech envelope and visual motion stimuli for each group. The circular mean of the phase-shift across all subjects is indicated by colored lines: red for the ASD group and black for the TD group. Corresponding polar histograms in red (ASD) and black (TD) visually represent the distribution of phase-shifts for each group. Both groups were tested against the hypothetical uniform distribution of delta (rayleigh test, ASD: *p* < 0.001, rayleigh *r* = 0.98, TD: *p* < 0.001, rayleigh *r* = 0.98) and theta phase (rayleigh test, ASD: *p* < 0.001, rayleigh *r* = 0.95, TD: *p* < 0.001, rayleigh *r* = 0.96)</figcaption></figure>

processing anomalies in ASD from an early developmental stage [19–24] are not merely isolated auditory deficits but are deeply connected to the integration of auditory and visual information, a process critical for effective communication, particularly in dynamic or complex listening environments [90, 91].

Our findings reveal a specific disruption in audio-visual integration among children with ASD, manifesting in visual dominance and temporal disorganization in auditory and visual processing. This disruption sharply contrasts with the expected auditory processing dominance [92] and might significantly contribute to the language development difficulties encountered by these children. In TD, the precedence of orofacial visual cues during speech facilitates auditory comprehension through predictive processing, optimizing the brain's synchronization to incoming speech signals [93]. In ASD, extended integration time windows and the lack of effective synchronization of auditory responses by visual signals (as evidenced by the atypical theta band phase-shifts) suggest they do not use visual cues to facilitate auditory speech processing, and that on the contrary auditory cues disrupt the visual processing in communicative situations.

## Repercussions of disrupted audio-visual processing on speech tracking

Children with ASD exhibit visual motion tracking and processing capabilities comparable to their TD peers

despite different scene analysis patterns, as previously observed [13, 94]. Within their preferred exploration zones, children with ASD process visual motion similarly to TD children [94] with an equivalent level of bottom-up excitability to visual stimuli [95, 96]. The univariate encoding results indeed suggest that the neural activity responsible for visual motion tracking operates similarly in both ASD and TD groups.

However, when visual processing co-occurs with speech processing, difficulties appear. Our multivariate modeling indicates that the neural encoding of audio-visual percepts in ASD children is less efficient, confirming that audiovisual contexts can disrupt brain responses to speech in this population [97]. Our study reinforces this crucial finding by showing that while autistic children encode single visual streams relatively well (visual motion tracking in univariate model), they struggle to concurrently encode auditory and visual streams (multivariate model).

Our study uncovers the potential underpinnings of the audiovisual integration difficulties observed in autism. The decoding results further indicate that while audio-visual integration interferes with visual processing in both groups, and that its impact on speech processing is particularly detrimental in the ASD group. Thus, the impairments in AV integration we observe are not merely additive but exacerbate sensory processing challenges in ASD. This framework explains that even though 12-month-old infants at risk for ASD explore faces and
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Overview of Experimental Procedures and Features of Interest. (A) Experimental procedures (B) Gaze fixation. Example of individual gaze fixation points (green dots) on a black and white image; (C) Example of gaze-captured screen areas. Depiction of screen areas captured by the gaze of participants in ASD (Autism Spectrum Disorder) and TD (Typically Developing) groups. (D) Example of a stimulus speech envelope from the cartoon soundtrack. (E) Visual motion corresponds to the same stimulus in each group. (F) Comparison of speech envelope and visual motion. Mutual Information (MI) between ASD and TD groups (ns. *p*>0.05).</figcaption></figure>

# Results

## 1.1 Atypical neural tracking of speech envelope in ASD children

We found distinct neural tracking for the auditory and visual parts of the stimuli. Different scalp distribution patterns were observed between the two groups (Figure 2). The ASD group had reduced neural response to the speech envelope relative to the TD group (Figure 2A, top row). In contrast, there was no difference in visual motion processing between the two groups (Figure 2A middle row). These results suggest atypical auditory but not visual processing of dynamic communicative stimuli in young children diagnosed with ASD.
<figure><img src="image_9.png" /><figcaption>**Fig. 7** The relationship between theta phase-shift and reconstruction accuracy of speech envelope (left) and visual motion (right) in ASD and TD. ASD group suggests a greater phase shift between speech envelope and visual motion positively correlates with speech reconstruction accuracy but negatively correlates with visual reconstruction, while reversely in TD group</figcaption></figure>

mouths similarly to infants with no family history of autism [98], they cannot leverage audiovisual cues for language acquisition as do typical children.

## Audio-visual temporal integration underlies speech impairment in autism

Audiovisual integration relies on the temporal alignment of sensory events, with visual cues enhancing auditory clarity, especially in noisy or ambiguous conditions [99–101]. Our findings confirm in TD children a visual lead (~50 ms) within a temporal window that is conducive to effective interaction and coordination between auditory and visual cues [102]. This window reasonably aligns with established models, positing a 200 ms integration period [39, 61–63], ranging from a 30 ms visual lag to a 170 ms of visual lead [61].

The precise timing of auditory and visual sequences is fundamental to audio-visual integration via predictive processing, whereby the brain leverages visual cues to anticipate and decode forthcoming auditory information. Here, phase-locking analyses in TD children show that the neural responses associated with auditory and

visual processing exhibit a 90-degree phase shift, indicating that such a phase relationship optimizes a dynamic balance between the sensory streams, facilitating integration and enhancing perception and communication [101, 103]. The pivotal role of the theta frequency band in orchestrating audio-visual speech processing is robustly supported in the literature [56, 58, 59]. A pi/2 visual lead results in aligning visual information processing with the auditory inputs. This reliable phase alignment observed in TD children sharply contrasts with the broad phase distribution observed in the ASD group, signalling inconsistent audio-visual integration. In logic, reconstruction accuracy is a proxy of sensory encoding accuracy. Thus, in TD children, the relationship between phase-shift and reconstruction accuracy confirms the known reliance on visual cues to enhance auditory processing, with any misalignments adversely affecting speech information integration. Conversely, in ASD children, while speech encoding is weaker and overall less dependent on visual-auditory phase congruency, visual processing is vulnerable to strong AV resynchronization.
## 1.2 Univariate decoding models confirm atypical speech envelope processing in children with ASD

Stimulus reconstruction accuracy was different in the ASD and TD groups. While reconstruction accuracy was equivalent for speech envelope and visual motion (p>0.9999, Figure 2B) in the TD group, it was lower for speech than for visual motion (p<0.0001, Figure 2B) in the ASD group. Group comparison shows that speech envelope reconstruction was weaker in the ASD than in the TD group (p <0.0001, Figure 2B), without significant group difference for visual motion reconstruction (Figure 2B). These results align with neural tracking results to suggest that it is mostly speech processing that is impaired in autistic children.

## 2. Multivariate encoding models indicate atypical audiovisual processing in ASD children

Further, we explored whether speech anomalies in ASD are limited to auditory processing difficulties or associated with audiovisual (AV) processing anomalies. We found stronger neural representation of the combined AV stimulus compared to individual single stimuli in both groups, an expected result as multivariate models provide in general more accurate prediction of neural responses by combining information from multiple sources. More interestingly, the joint model suggested weaker AV representation in the ASD than in the TD group (Figure 2A bottom row).

<figure><img src="image_3.png" /></figure>

**Figure 2.** Comparison of audio, visual, and AV models. (A.) Neural representations in ASD (left) and TD (middle) groups, for each model (A-speech envelope, V-visual motion, and AV joint) across all scalp electrodes. The right column shows EEG channels where significant group differences are observed using cluster-based nonparametric statistics (p < 0.05; with a positive t-value indicating greater predictability in the ASD group compared to the TD group). (B.) Stimulus reconstruction accuracy for speech envelope and visual motion in both groups. Error bars indicate the standard error of the mean. Significance levels are indicated as follows: ‘ns’ for p>0.05 (not significant), * for p <0.05, ** for p<0.01, *** for p<0.001, and **** for p<0.0001.
The atypical auditory lead (~50 ms) observed in ASD indicates that audio-visual integration is jeopardized and that the conventional sequence where visual information typically precedes auditory is inverted. Furthermore, the 180-degree phase-shift in the neural activities associated with each stream reflects a profound disruption in temporal coordination, potentially leading to confusion or interpretation errors. Such a discrepancy underscores a critical deficiency in predictive processing in ASD, where, rather than synergistically enhancing each other, auditory and visual cues conflict, undermining the synthesis of coherent audio-visual perception [31, 32].

Our results are consistent with the notion that the phase of low-frequency neural oscillations is crucial for the temporal parsing in speech [104]. The anomaly in temporal encoding mechanisms described in our experiment is constrained by the temporal features provided by external stimulation to build a temporal reference frame. While delta oscillations have previously been linked to temporal predictability [102, 104], we observed here that sensory integration is affected by AV misalignment in the theta range, which is associated with atypical speech perception in ASD. AV integration primarily occurs at the syllable level with a typical tolerance to AV asynchrony around 250ms, which corresponds to the theta range [39, 61–64].

## Conclusion

Our results show marked anomalies in audio-visual integration in young children with ASD that provide specific underpinnings for previous findings depicting disrupted speech rhythm tracking. They further reveal that disruption in audio-visual integration, manifesting as temporal desynchronization, impacts speech processing and contributes to the communicative challenges in autism. Our results also highlight the critical role of temporal processing in audio-visual integration and underscore the importance of characterizing these mechanisms in ASD. Moving forward, these insights could inform the development of targeted interventions aimed at regulating temporal speech processing and AV synchronization to improve communication in ASD children.

## Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s11689-025-09593-w.

Supplementary Material 1

Supplementary Material 2

## Acknowledgements

Not applicable.

## Author contributions

Data acquisition and clinical resources: M.S., N. K.; Study design: X.W., N. K., M.S. and A-L.G.; Data analysis: X.W., S.B.; Writing—original draft: X.W.; Writing—review—editing: X.W., S.B., N. K., A-L.G. and M.S.; Supervision: A-L.G., and M.S.

## Funding

This work is supported by grants from the Swiss National Science Foundation (#163859, #190084, #202235, & #212653 to M.S.), by the National Centres of Competence in Research (NCCR Synapsy) (Grant No. 51NF40–185897 to M.S.), by the Agence Nationale de la Recherche (ANR-21-CE28-0028 to S.B.), and by Evolving Language (Grant No. 51NF40_180888 to A-L.G.). Additional support is provided by a grant from the Fondation pour l'Audition (FPA IDA11 to A-L.G.), as well as by the Fondation Privée des Hôpitaux Universitaires de Genève (https://www.fondationhug.org) and the Fondation Pôle Autisme (https://www.pole-autisme.ch).

## Data availability

The unprocessed datasets for this manuscript are not publicly available yet due to ongoing analysis as part of a longitudinal study. The results are expected to be published in the future. Once all data has been published, requests to access the datasets should be directed to Dr. Marie Schaer at marie.schaer@unige.ch. The custom MATLAB analysis scripts will be made available upon request to the xiaoyue.wang@pasteur.fr.

## Declarations

### Competing interests

The authors declare no competing interests.

### Ethics approval and consent to participate

Informed consent was obtained from the parents of all participants prior to inclusion in the study. The research was conducted with the ethical standards set forth by the Ethics Committee of the Faculty of Medicine at the University of Geneva Hospital and adhered to the principles outlined in the Declaration of Helsinki.

### Competing interests

The authors declare no conflict of interest.

### Preprint servers

The manuscript was deposited as a preprint in bioRxiv with the license CC BY-NC-ND 4.0.

Received: 17 August 2024 / Accepted: 24 January 2025

Published online: 18 February 2025

## References

1. Alegria J, Noirit E. Neonate orientation behaviour towards human voice. Int J Behav Dev. 1978;1:291–312.

2. Jusczyk PW, Bertoncini J. Viewing the development of speech perception as an innately guided learning process. Lang Speech. 1988;31:217–38.

3. Lecanuet J, Granier-Deferre C, Decasper A, Maugeais R, Andrieu A, Busnel M. Fetal perception and discrimination of speech stimuli: demonstration by cardiac reactivity; preliminary results. Comptes Rendus Académie Sci Sér III Sci Vie. 1987;305:161–4.

4. Williams JHG, Massaro DW, Peel NJ, Bosseler A, Suddendorf T. Visual-audi-tory integration during speech imitation in autism. Res Dev Disabil. 2004;25:559–75.

5. Iarocci G, Rombough A, Yager J, Weeks DJ, Chua R. Visual influences on speech perception in children with autism. Autism. 2010;14:305–20.

6. Feng S, Wang Q, Hu Y, Lu H, Li T, Song C, et al. Increasing audiovisual speech integration in autism through enhanced attention to mouth. Dev Sci. 2023;26:e13348.

7. Shic F, Wang Q, Macari SL, Chawarska K. The role of limited salience of speech in selective attention to faces in toddlers with autism spectrum disorders. J Child Psychol Psychiatry. 2020;61:459–69.

8. Klin A. Young autistic children's listening preferences in regard to speech: a possible characterization of the symptom of social withdrawal. J Autism Dev Disord. 1991;21:29–42.
3. Compromised audiovisual integration in ASD disrupts visual enhancement of auditory processing

By comparing the decoding accuracies for speech envelope and visual motion across univariate (A-only and V-only) and multivariate (AV-joint) models, we found distinct patterns of audiovisual integration in ASD and TD children (Table 1, Figure 3). Notably, in the TD group, the accuracy of speech envelope reconstruction in the AV-joint model (concurrent speech envelope and visual motion) did not significantly differ from the A-only model (p = 0.1414, Figure 3A). Conversely, in the ASD group, the speech envelope was decoded with significantly less accuracy in the AV-joint model than in the A-only model (p=0.0304, Figure 3B), indicating a disruptive effect of AV integration on auditory processing specific to this group. Despite these differences, both groups exhibited decreased visual motion reconstruction accuracy in the AV-joint model compared to the V-only model (ASD group: p < 0.0001, TD group: p = 0.0373, Figure 3), with a more pronounced decrement observed in the ASD group (p = 0.0003, Table 2). These findings suggest that while AV integration generally impacts visual processing across both groups, it adversely affects auditory processing primarily in the ASD group.

<figure><img src="image_4.png" /><figcaption>Figure 3. Evaluation of decoding accuracy in ASD (A) and TD (B). Stimulus reconstruction accuracy: speech</figcaption></figure>
9. Klin A, Lin DJ, Gorrindo P, Ramsay G, Jones W. Two-year-olds with autism orient to non-social contingencies rather than biological motion. Nature. 2009;459:257–61.

10. Klin A. Listening preferences in regard to speech in four children with developmental disabilities. J Child Psychol Psychiatry. 1992;33:763–9.

11. Dawson G, Meltzoff AN, Osterling J, Rinaldi J, Brown E. Children with autism fail to orient to naturally occurring social stimuli. J Autism Dev Disord. 1998;28:479–85.

12. Kuhl PK, Coffey-Corina S, Padden D, Dawson G. Links between social and linguistic processing of speech in preschool children with autism: behavioral and electrophysiological measures. Dev Sci. 2005;8:F1–12.

13. Kojovic N, Cekic S, Castaón SH, Franchini M, Sperdin HF, Sandini C et al. Unraveling the developmental dynamic of visual exploration of social interactions in autism. Büchel C, editor. eLife. 2024;13:e85623.

14. Federici A, Parma V, Vicovaro M, Radassao L, Casartelli L, Ronconi L. Anomalous perception of Biological Motion in Autism: a conceptual review and Meta-analysis. Sci Rep. 2020;10:4576.

15. Knight EJ, Krakowski AI, Freedman EG, Butler JS, Molholm S, Foxe JJ. Attentional influences on neural processing of biological motion in typically developing children and those on the autism spectrum. Mol Autism. 2022;13:33.

16. Todorova GK, Hatton REM, Pollick FE. Biological motion perception in autism spectrum disorder: a meta-analysis. Mol Autism. 2019;10:49.

17. Marco EJ, Hinkley LB, Hill SS, Nagarajan SS. Sensory processing in autism: a review of neurophysiologic findings. Pediatr Res. 2011;69:R48–54.

18. Todorova GK, Pollick FE, Muckli L. Special treatment of prediction errors in autism spectrum disorder. Neuropsychologia. 2021;163:108070.

19. Collet L, Roge B, Descouens D, Moron P, Duverdy F, Urgell H. Objective auditory dysfunction in infantile autism. Lancet. 1993;342:923–4.

20. Haesen B, Boets B, Wagemans J. A review of behavioural and electrophysiological studies on auditory processing and speech perception in autism spectrum disorders. Res Autism Spectr Disord. 2011;5:701–14.

21. Edgar JC, Fisk IV CL, Berman JI, Chudnovskaya D, Liu S, Pandey J, et al. Auditory encoding abnormalities in children with autism spectrum disorder suggest delayed development of auditory cortex. Mol Autism. 2015;6:69.

22. Foss-Feig JH, Schauder KB, Key AP, Wallace MT, Stone WL. Audition-specific temporal processing deficits associated with language function in children with autism spectrum disorder. Autism Res Off J Int Soc Autism Res. 2017;10:1845–56.

23. Wang X, Wang S, Fan Y, Huang D, Zhang Y. Speech-specific categorical perception deficit in autism: an event-related potential study of lexical tone processing in Mandarin-speaking children. Sci Rep. 2017.

24. Wang X, Delgado J, Marchesotti S, Kojovic N, Sperdin HF, Rihs TA, et al. Speech reception in Young Children with Autism is selectively indexed by a neural oscillation coupling anomaly. J Neurosci. 2023;43:6779–95.

25. Benasich AA, Gou Z, Choudhury N, Harris KD. Early cognitive and language skills are linked to resting frontal gamma power across the first 3 years. Behav Brain Res. 2008;195:215–22.

26. Benitez-Burraco A, Murphy E. The Oscilpathetic Nature of Language deficits in Autism: from genes to Language Evolution. Front Hum Neurosci. 2016;10:120.

27. Cermak CA, Arshinoff S, Ribeiro de Oliveira L, Tendera A, Beal DS, Brian J, et al. Brain and Language associations in Autism Spectrum disorder: a scoping review. J Autism Dev Disord. 2022;52:725–37.

28. Morrel J, Singapuri K, Landa RJ, Reetzke R. Neural correlates and predictors of speech and language development in infants at elevated likelihood for autism: a systematic review. 2023 [cited 2023 Aug 28]; Available from: https://www.proquest.com/docview/2851820899/abstract/57E7B032F53541B0PQ/1

29. Crosse MJ, Foxe JJ, Tarrit K, Freedman EG, Molholm S. Resolution of impaired multisensory processing in autism and the cost of switching sensory modality. Commun Biol. 2022;5:1–17.

30. Jao Keehn RJ, Sanchez SS, Stewart CR, Zhao W, Grenesko-Stevens EL, Keehn B, et al. Impaired downregulation of visual cortex during auditory processing is associated with autism symptomatology in children and adolescents with autism spectrum disorder. Autism Res Off J Int Soc Autism Res. 2017;10:130–43.

31. Stevenson RA, Siemann JK, Schneider BC, Eberly HE, Woynaroski TG, Camarata SM, et al. Multisensory temporal integration in autism spectrum disorders. J Neurosci. 2014;34:691–7.

32. Stevenson RA, Segers M, Ferber S, Barense MD, Wallace MT. The impact of multisensory integration deficits on speech perception in children with autism spectrum disorders. Front Psychol. 2014;5:379.

33. Alm M, Behne DM, Wang Y, Eg R. Audio-visual identification of place of articulation and voicing in white and babble noise). J Acoust Soc Am. 2009;126:377–87.

34. Bertels J, Niesen M, Destoky F, Coolen T, Vander Ghinst M, Wens V, et al. Neurodevelopmental oscillatory basis of speech processing in noise. Dev Cogn Neurosci. 2023;59:101181.

35. Fleming JT, Maddox RK, Shinn-Cunningham BG. Spatial alignment between faces and voices improves selective attention to audio-visual speech. J Acoust Soc Am. 2021;150:3085–100.

36. Yuan Y, Lleo Y, Daniel R, White A, Oh Y. The Impact of Temporally Coherent Visual Cues on Speech Perception in Complex Auditory Environments. Front Neurosci [Internet]. 2021 [cited 2024 Mar 11];15. Available from: https://www.frontiersin.org/journals/neuroscience/articles/https://doi.org/10.3389/fnins.2021.678029/full

37. Guiraud JA, Tomalski P, Kushnerenko E, Ribeiro H, Davies K, Charman T, et al. Atypical audiovisual speech integration in infants at risk for autism. PLoS ONE. 2012;7:e36428.

38. Lindborg A, Baart M, Stekelenburg JJ, Vroomen J, Andersen TS. Speech-specific audiovisual integration modulates induced theta-band oscillations. PLoS ONE. 2019;14:e0219744.

39. Munhall KG, Gribble P, Sacco L, Ward M. Temporal constraints on the McGurk effect. Percept Psychophys. 1996;58:351–62.

40. van Wassenhove V. Speech through ears and eyes: interfacing the senses with the supramodal brain. Front Psychol [Internet]. 2013 [cited 2023 Aug 28];4. Available from: https://doi.org/10.3389/fpsyg.2013.00388

41. Choi I, Lee JY, Lee SH. Bottom-up and top-down modulation of multisensory integration. Curr Opin Neurobiol. 2018;52:115–22.

42. van Wassenhove V, Grant KW, Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci U A. 2005;102:11811–6.

43. Kushnerenko E, Teinonen T, Volein A, Csibra G. Electrophysiological evidence of illusory audiovisual speech percept in human infants. Proc Natl Acad Sci. 2008;105:11442–5.

44. Musacchia G, Schroeder CE. Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res. 2009;258:72–9.

45. Murray MM, Wallace MT, editors. The neural bases of multisensory processes. Boca Raton: CRC; 2011.

46. Baum SH, Stevenson RA, Wallace MT. Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. Prog Neurobiol. 2015;134:140–60.

47. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadariv H, Stone WL, et al. An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res. 2010;203:381–9.

48. Kwakye LD, Foss-Feig JH, Cascio CJ, Stone WL, Wallace MT. Altered auditory and multisensory temporal processing in autism spectrum disorders. Front Integr Neurosci. 2011;4:129.

49. Gao M, Lim S, Chubykin AA. Visual familiarity induced 5 Hz oscillations and improved orientation and direction selectivities in V1. J Neurosci. 2021.

50. Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE. An Oscillatory Hierarchy Controlling neuronal excitability and stimulus Processing in the auditory cortex. J Neurophysiol. 2005;94:1904–11.

51. Romei V, Brodbeck V, Michel C, Amedi A, Pascual-Leone A, Thut G. Spontaneous fluctuations in posterior α-Band EEG activity reflect variability in excitability of human visual areas. Cereb Cortex. 2008;18:2010–8.

52. Leszczynski M, Schroeder CE. The Role of Neuronal Oscillations in Visual Active Sensing. Front Integr Neurosci [Internet]. 2019 [cited 2024 Mar 11];13. Available from: https://doi.org/10.3389/fnint.2019.00032

53. Poeppel D, Assaneo MF. Speech rhythms and their neural foundations. Nat Rev Neurosci. 2020;21:322–34.

54. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA. The Natural Statistics of Audiovisual Speech. PLOS Comput Biol. 2009;5:e1000436.

55. Arnal LH, Wyart V, Giraud AL. Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nat Neurosci. 2011;14:797–801.

56. Young AW, Frühholz S, Schweinberger SR. Face and Voice Perception: understanding commonalities and differences. Trends Cogn Sci. 2020;24:398–410.

57. Aller M, Økland HS, MacGregor LJ, Blank H, Davis MH. Differential Auditory and Visual Phase-Locking are observed during Audio-Visual Benefit and Silent Lip-Reading for Speech Perception. J Neurosci. 2022;42:6108–20.

58. Hagan CC, Woods W, Johnson S, Calder AJ, Green GGR, Young AW. MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus. Proc Natl Acad Sci. 2009;106:20010–5.
envelope(A-) and visual motion(V-) in both the single-stimulus model (A-only = A-A and V-only = V-V) and the AV-joint model (A-AV, and V-AV)). Significance levels are indicated as follows: ‘ns’ for p>0.05 (not significant), * for p<0.05, ** for p<0.01, *** for p<0.001, **** for p<0.0001. For additional details, see Supplemental Figure 3-1.

## 4. No visual precedence in audiovisual processing in ASD

When studying the temporal dynamics of auditory and visual processing to assess audiovisual integration, we found distinct time-lags in auditory and visual decoding accuracies in ASD and TD children. In the TD group, it took 200~ms to reach significant decoding accuracy for auditory responses but only ~50ms for visual ones. The exact opposite pattern was found in the ASD group with a ~200 ms time-lag got visual decoding versus ~50 ms for auditory decoding (Figure 4A). This analysis revealed a visual lead in the TD consistent with the fact that speech sources are usually ahead of sounds, but an auditory lead in the ASD group (Figure 4B). This temporal inversion indicates a fundamental alteration in the sensory processing sequence for children with ASD.

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Optimal EEG-stimuli time lag for ASD (red) and TD (black) groups. (A) depicts the optimal time-lag observed in the reconstruction of stimulus features in AV-joint model, specifically speech envelope (A-) and visual motion(V-); Positive values represent stimulus lead EEG signal. (B) illustrates the A-V time lag AV-joint model. Positive values represent V leads A. Significance levels are indicated as follows: ns>0.05, \*p <0.05, \*\*p<0.01, \*\*\*p<0.001, \*\*\*\*p<0.0001.</figcaption></figure>

## 5. Intact capacity to synchronize neural activity to the stimulus input in ASD

To ascertain whether speech tracking anomalies in ASD are primarily attributable to a general defect in stimulus/brain synchronization or rather results from audiovisual integration deficits, we investigated the intricate temporal dynamics underlying these processes. Specifically, we explored the coherence of stimulus-response relationships within delta (1–4 Hz) and theta (4–8 Hz) frequency bands, typically associated with syllable-and phrase-level speech processing. We observed a higher stimulus/brain coherence in the delta band than the theta band, yet when
59. Hagan CC, Woods W, Johnson S, Green GGR, Young AW. Involvement of right STS in Audio-Visual Integration for Affective Speech demonstrated using MEG. PLoS ONE. 2013;8:e70648.

60. Plöchl M, Fiebelkorn I, Kastner S, Obleser J. Attentional sampling of visual and auditory objects is captured by theta-modulated neural activity. Eur J Neurosci. 2022;55:3067–82.

61. van Wassenhove V, Grant KW, Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia. 2007;45:598–607.

62. Stevenson RA, Wallace MT. Multisensory temporal integration: task and stimulus dependencies. Exp Brain Res. 2013;227:249–61.

63. Massaro DW, Cohen MM. The paradigm and the fuzzy logical model of perception are alive and well. J Exp Psychol Gen. 1993;122:115–24.

64. Guilleminet P, Graef C, Butters E, Reichenbach T. Audiotactile stimulation can improve Syllable discrimination through multisensory integration in the Theta frequency Band. J Cogn Neurosci. 2023;35:1760–72.

65. Power A, Mead N, Barnes L, Goswami U. Neural Entrainment to Rhythmically Presented Auditory, Visual, and Audio-Visual Speech in Children. Front Psychol [Internet]. 2012 [cited 2023 Mar 3];3. Available from: https://doi.org/10.3389/fpsyg.2012.00216

66. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A. Neuronal oscillations and visual amplification of speech. Trends Cogn Sci. 2008;12:106–13.

67. Keshavarzi M, Mandke K, Macfarlane A, Parvez L, Gabrielczyk F, Wilson A, et al. Atypical delta-band phase consistency and atypical preferred phase in children with dyslexia during neural entrainment to rhythmic audio-visual speech. Neurolmage Clin. 2022;35:103054.

68. Franchini M, Wood de Wilde H, Glaser B, Gentaz E, Eliez S, Schaer M. Brief report: a preference for Biological Motion predicts a reduction in Symptom Severity 1 Year later in preschoolers with Autism Spectrum disorders. Front Psychiatry. 2016;7:1434.

69. Franchini M, Zoller D, Gentaz E, Glaser B, Wood de Wilde H, Kojovic N, et al. Early adaptive functioning trajectories in Preschoolers with Autism Spectrum disorders. J Pediatr Psychol. 2018;43:800–13.

70. Lord C, Risi S, Lambrecht L, Cook EH, Leventhal BL, DiLavore PC, et al. The Autism Diagnostic Observation Schedule—Generic: a standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Disord. 2000;30:205–23.

71. Lord C, Rutter M, DiLavore P, Risi S, Gotham K, Bishop S. Autism diagnostic observation schedule: ADOS-2. West Psychol. J Psychoeduc Assess. 2012;32:88–92.

72. Trotro. l'anniversaire de nana. Storimages; 2013.

73. Trotro part en vacance. Storimages; 2013.

74. Trotro. et La boite a secrets. Storimages; 2013.

75. Trotro es tres amoureux. Storimages; 2013.

76. Olsen A. The Tobii I-VT fixation filter. Tobii Technol. 2012;21:4–19.

77. Weineck K, Wen OX, Henry MJ. Neural synchronization is strongest to the spectral flux of slow music and depends on familiarity and beat salience. Jensen O, Shinn-Cunningham BG, Zoefel B, editors. eLife. 2022;11:e75515.

78. Cover TM, Thomas JA. Entropy. Relative Entropy, and Mutual Information. Elem Inf Theory [Internet]. John Wiley & Sons, Ltd; 2005 [cited 2024 Mar 11]. pp. 13–55. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/047174882X.ch2

79. Timme NM, Lapish C. A Tutorial for Information Theory in Neuroscience. eNeuro [Internet]. 2018 [cited 2024 Mar 11];5. Available from: https://www.euro.org/content/5/3/ENEURO.0052-18.2018.

80. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods. 2004;134:9–21.

81. Jessen S, Fiedler L, Münte TF, Obleser J. Quantifying the individual auditory and visual brain response in 7- month-old infants watching a brief cartoon movie. Neurolmage. 2019.

82. Crosse MJ, Di Liberto GM, Bednar A, Lalor EC. The multivariate temporal response function (mTRF) toolbox: a MATLAB Toolbox for relating neural signals to continuous stimuli. Front Hum Neurosci. 2016;10:604.

83. Maris E, Oostenveld R. Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods. 2007;164:177–90.

84. Dinno A. Nonparametric pairwise multiple comparisons in Independent groups using Dunn's test. Stata J. 2015;15:292–300.

85. Nozaradan S, Peretz I, Missal M, Mouraux A. Tagging the neuronal entrainment to beat and meter. J Neurosci. 2011;31:10234–40.

86. Clouter A, Shapiro KL, Hanslmayr S. Theta Phase synchronization is the glue that binds human associative memory. Curr Biol. 2017;27:3143–e31486.

87. Pefkou M, Arnal LH, Fontolan L, Giraud AL. theta-Band and beta-Band neural activity reflects independent Syllable Tracking and Comprehension of Time-compressed Speech. J Neurosci. 2017;37:7930–8.

88. Berens P, CircStat. A MATLAB Toolbox for Circular Statistics. J Stat Softw [Internet]. 2009 [cited 2023 Jun 22];31. Available from: http://www.jstatsoft.org/v31/i10/

89. Jochaut D, Lehongre K, Saitovitch A, Devauchelle AD, Olasagasti I, Chabane N, et al. Atypical coordination of cortical oscillations in response to speech in autism. Front Hum Neurosci. 2015;9:171.

90. Puschmann S, Daeglau M, Stropahl M, Mirkovic B, Rosemann S, Thiel CM, et al. Hearing-impaired listeners show increased audiovisual benefit when listening to speech in noise. Neurolmage. 2019;196:261–8.

91. Haider CL, Suess N, Hauswald A, Park H, Weisz N. Masking of the mouth area impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker. Neurolmage. 2022;252:119044.

92. O'CONNOR N, Hermelin B. Sensory dominance: in autistic imbecile children and controls. Arch Gen Psychiatry. 1965;12:99–103.

93. Arnal LH, Morillon B, Kell CA, Giraud A-L. Dual neural routing of Visual Facilitation in Speech Processing. J Neurosci. 2009;29:13445–53.

94. Liu W, Li M, Yi L. Identifying children with autism spectrum disorder based on their face processing abnormality: a machine learning framework. Autism Res. 2016;9:888–98.

95. Sinnett S, Soto-Faraco S, Spence C. The co-occurrence of multisensory competition and facilitation. Acta Psychol (Amst). 2008;128:153–61.

96. Cuppini C, Ursino M, Magosso E, Rowland BA, Stein BE. An emergent model of multisensory integration in superior colliculus neurons. Front Integr Neurosci [Internet]. 2010 [cited 2024 Mar 11];4. Available from: https://doi.org/10.3389/fnint.2010.00006

97. Irwin J, Harwood V, Kleinman D, Baron A, Avery T, Turcios J, et al. Neural and Behavioral Differences in Speech Perception for Children with Autism Spectrum Disorders within an Audiovisual Context. J Speech Lang Hear Res. 2023;66:2390–403.

98. Zoefel B, Archer-Boyd A, Davis MH. Phase entrainment of brain oscillations causally modulates neural responses to Intelligible Speech. Curr Biol. 2018;28:401–8. e5.

99. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze H-J, et al. Audiovisual temporal correspondence modulates Human Multisensory Superior Temporal Sulcus Plus primary sensory cortices. J Neurosci. 2007;27:11431–41.

100. Stacey PC, Kitterick PT, Morris SD, Sumner CJ. The contribution of visual information to the perception of speech in noise with and without informative temporal fine structure. Hear Res. 2016;336:17–28.

101. Han S, Chen Y-C, Maurer D, Shore DJ, Lewis TL, Stanley BM, et al. The development of audio-visual temporal precision precedes its rapid recalibration. Sci Rep. 2022;12:21591.

102. Herbst SK, Stefanics G, Obleser J. Endogenous modulation of delta phase by expectation—A replication of Stefanics et al., 2010. Cortex. 2022;149:226–45.

103. Henry MJ, Obleser J. Frequency modulation entrains slow neural oscillations and optimizes human listening behavior. Proc Natl Acad Sci. 2012;109:20095–100.

104. Giraud AL, Poeppel D. Cortical oscillations and speech processing: emerging computational principles and operations. Nat Neurosci. 2012;15:511–7.

105. Stefanics G, Hangya B, Hernádi I, Winkler I, Lakatos P, Ulbert I. Phase entrainment of Human Delta oscillations can mediate the effects of Expectation on reaction speed. J Neurosci. 2010;30:13578–85.

**Publisher's note**

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1 comparing across groups, both the average coherence values and their distribution patterns showed
2 remarkable similarity (Figure 5, Table 3). This uniformity shows that the intrinsic capacity of
3 neural activity to synchronize with auditory and visual stimuli is consistent between groups (ASD
4 v.s. TD).

5

<figure><img src="image_6.png" /></figure>

6
7
8
9
10
11

<caption>**Figure 5.** Stimulus-response coherence in Theta and Delta Bands for ASD (red) and TD (black) groups. The plot displays the coherence between stimulus and response for Speech Envelope and Visual Motion. Error bars represent the standard error of the mean. The coherence levels are compared within the specific frequency bands of interest, highlighting potential group differences in sensory processing. Significance levels are indicated as follows: ns>0.05, *p* <0.05, **p**<0.01, ***p**<0.001, ****p<0.0001.</caption>

12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

**6. Theta-range desynchronization of audio-visual responses in ASD**

Given the preserved synchronization capacity between brain activity and external stimuli in each modality, we then sought whether audio-visual integration anomalies, notably the inverted AV temporal patterns, are associated with a phase desynchronization of auditory and visual processing. In the delta band, we observed similar phase angles for both groups (F(1,62) = 0.494, p < 0.470), indicating comparable phase locking at this slower frequency, suggesting that temporal alignment in this low-frequency range does not differentiate between groups. Moreover, the small angle indicates the absence of delta band phase-shift between modalities. Yet, a significant group difference was observed in the theta band. The TD group showed a phase-shift of approximately 90 degrees, signaling effective sequential integration with one modality leading the other by a consistent temporal offset, that optimizes audio-visual integration at the syllable level. In ASD children the phase-shift amounted to 180 degrees and the group difference was significant (F(1,62) = 12.05, p < 0.001) (Figure 6). The observed 180-degree phase shift in ASD could suggest that auditory and visual information is out of sync: when one sensory modality is at its peak processing efficiency, the other is at its lowest, potentially leading to disjointed even conflicting sensory processes.
<figure><img src="image_7.png" /><figcaption>**Figure 6.** Phase-shift distribution between speech envelope and visual motion. This figure shows the phase-shift distribution between the brain processes of speech envelope and visual motion stimuli for each group. The circular mean of the phase-shift across all subjects is indicated by colored lines: red for the ASD group and black for the TD group. Corresponding polar histograms in red (ASD) and black (TD) visually represent the distribution of phase-shifts for each group. Both groups were tested against the hypothetical uniform distribution of delta (rayleigh test, ASD: *p* <0.001, *rayleigh r* = 0.98, TD: *p* <0.001, *rayleigh r* = 0.98) and theta phase (rayleigh test, ASD: *p* <0.001, *rayleigh r* = 0.95, TD: *p* <0.001, *rayleigh r* = 0.96).</figcaption></figure>

## 7. AV phase-shift is related to auditory encoding accuracy in TD and visual encoding accuracy in ASD

Finally, we sought to understand the relationship between the theta band AV phase-shift and the accuracy of auditory and visual information reconstruction within an unimodal framework (Figure 7). As could be expected, in TD children the AV phase-shift did not influence visual reconstruction accuracy (*r* = 0.033, *p* = 0.858), but there was a weak negative correlation between the phase-shift extent and speech reconstruction accuracy (*r* = -0.272, *p* = 0.132): when the AV phase-shift increased speech reconstruction accuracy decreased, which given the visual lead previously observed could suggest a causal effect. A different pattern was seen in ASD children, with no relation between the phase shift extent and speech reconstruction accuracy (*r* = 0.197, *p* = 0.288), but a weak negative correlation between the phase-shift extent and the accuracy of visual information reconstruction (*r* = -0.325, *p* = 0.074), with larger AV phase-shifts linked to poorer visual reconstruction accuracy. Likewise, given the auditory lead observed in children with ASD, this could suggest a causal effect.

Contrasting patterns in TD and ASD children underscore distinct audiovisual integration mechanisms. In logic, reconstruction accuracy is a proxy of sensory encoding accuracy. Thus, in TD children, the data confirm the known reliance on visual cues to enhance auditory processing, with any misalignments adversely affecting speech information integration. Conversely, in ASD
1 children, while speech encoding is weaker and overall less dependent on visual-auditory phase

2 congruency, visual processing is vulnerable to strong AV desynchronization.

3

<figure><img src="image_8.png" /><figcaption>**Figure 7.** *The relationship between theta phase-shift and reconstruction accuracy of speech envelope (left) and visual motion (right) in ASD and TD. ASD group suggests a greater phase shift between speech envelope and visual motion positively correlates with speech reconstruction accuracy but negatively correlates with visual reconstruction, while reversely in TD group.*</figcaption></figure>

4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Discussion

Using several analyses of the EEG recorded in very young children with and without ASD while they were watching a short animated movie, we confirmed previous results showing profound anomalies of the capacity to follow speech rhythms (24,92), an essential prerequisite to speech comprehension. The present study goes beyond this observation by showing that children with ASD did not exhibit the natural dominance of auditory processing when exposed to natural audio-visual speech conditions. Instead, their processing of audio-visual stimuli was impacted by a temporal misalignment of these sensory inputs, which disturbs the predictive processing typically at play when perceiving speech.

Audio-visual integration anomalies interfere with sensory encoding in ASD

21
22
23
24
25

Audio-visual processing, especially the synchronization of the two sensory modalities, plays a pivotal role in understanding the communicative challenges observed in ASD. Previous studies have established that basic auditory dysfunctions and atypical speech processing are characteristic of ASD from an early developmental stage (19–24). Our study reveals that such anomalies are not merely isolated auditory deficits but are deeply connected to the integration of auditory and visual
1 information, a process critical for effective communication, particularly in dynamic or complex
2 listening environments (93,94).

3
4 Our findings reveal a specific disruption in audio-visual integration among children with ASD,
5 manifesting in visual dominance and temporal disorganization in auditory and visual processing.
6 This disruption sharply contrasts with the expected auditory processing dominance (95) and might
7 significantly contribute to the language development anomalies encountered by these children. In
8 typical development, the precedence of orofacial visual cues during speech facilitates auditory
9 comprehension through predictive processing, optimizing the brain’s synchronization to incoming
10 speech signals (96). In ASD, extended integration time windows and the lack of effective
11 synchronization of auditory responses by visual signals (as evidenced by the atypical theta band
12 phase-shifts) suggest they cannot use visual cues to facilitate auditory speech processing. On the
13 contrary, our findings show that auditory cues perturb visual processing of communicative
14 situations.

15
16 Specific repercussions of disrupted audio-visual processing on speech tracking

17 Children with ASD demonstrate effective visual motion tracking and processing capabilities,
18 comparable to their TD peers, despite distinctive scene analysis patterns as previously observed
19 (13,97). Within their preferred exploration zones, children with ASD process visual motion
20 similarly to their TD peers (97), exhibiting a level of bottom-up excitability to visual stimuli akin
21 to TD children (98,99). Our univariate encoding results suggest that the neural activity responsible
22 for visual motion tracking operates similarly in both ASD and TD groups.

23
24 However, when visual processing co occurs with speech processing, some difficulties appear. Our
25 multivariate modeling indicates that the neural encoding of audio-visual percepts in ASD children
26 is less efficient, confirming that audiovisual contexts can disrupt brain responses to speech in this
27 population (100). In the same vein, Chic et al. (2020) underscore such AV integration difficulties,
28 noting that children with ASD tend to attend less to faces and mouths in general and more
29 specifically when they produce speech (7). Our study reinforces this crucial finding by showing
30 that while children with ASD encode single visual streams relatively well (visual motion tracking
31 in univariate modeling), they struggle with the concurrent encoding of both auditory and visual
32 streams (multivariate modeling).

33
34 Building upon this framework, the research conducted by Chawarska et al. (2022) reveals that 12-
35 month-old infants at risk for ASD, even though they explore faces and mouths similarly to infants
36 with no family history of autism (101), cannot leverage audiovisual cues for language acquisition
37 as do typical children. Our study uncovers the potential underpinnings of the audiovisual
38 integration difficulties observed in ASD. The decoding results indicate that while audio-visual
39 integration interferes with visual processing in both ASD and TD groups, its impact on speech
40 processing is particularly detrimental in the ASD group. Thus, the impairments in AV integration
1 we observe are not merely additive but they interactively exacerbate sensory processing challenges
2 much more adversely in children with ASD.

3

4 Audio-visual temporal integration underlies speech impairment in ASD

5 Audiovisual integration capitalizes on the temporal alignment of sensory events, with visual
6 information often enhancing the auditory signal’s clarity and precision, especially when auditory
7 cues are poor, noisy, or ambiguous (102–104). Visual cues related to speech are typically
8 processed faster than auditory cues, allowing visual information to facilitate synchronizing
9 subsequent auditory processing (105). Our findings confirm in TD children a visual lead (~50 ms)
10 within a temporal window is conducive to effective interaction and coordination between auditory
11 and visual cues. This window reasonably aligns with established models, positing a 200 ms
12 integration period (39,61–63), ranging from a 30 ms visual lag to a 170 ms of visual lead (61).
13

13

14 The precise timing of audio-visual sequences is fundamental to audio-visual integration via
15 predictive processing, whereby the brain leverages visual cues to anticipate and decode
16 forthcoming auditory information. Here, phase-locking analyses in TD children show that the
17 neural responses associated with auditory and visual processing exhibit a 90-degree phase shift.
18 This observation indicates that the brain orchestrates visual and auditory information in a
19 synergistic but temporally distinct manner. Such a phase relationship is instrumental in achieving
20 a dynamic balance between the sensory streams, facilitating an integration that enhances
21 perception and communication (106,107). The pivotal role of the theta frequency band in
22 orchestrating audio-visual speech processing is robustly supported in the literature (56,58,59). A
23 pi/2 phase shift during an audio-visual speech event might fine-tune the phase alignment to a
24 timing that is congruent with the auditory inputs. This meticulous phase alignment contrasts
25 sharply with the broad phase distribution observed in the ASD group, hinting at a pronounced
26 disparity in how auditory and visual cues are integrated.

27

28 Crucially, in ASD children, the integration of auditory and visual streams is jeopardized, as
29 evidenced by our observation of an atypical auditory lead (~50 ms), which disrupts the
30 conventional sequence where visual information typically precedes auditory. This inversion
31 undermines the usual enhancement provided by visual signals to auditory processing, highlighting
32 a marked alteration or impairment in multisensory integration. Furthermore, we noted a 180-
33 degree phase-shift in the neural activities associated with processing these 2 streams. This
34 substantial phase misalignment reflects a profound disruption in temporal coordination, potentially
35 leading to confusion or interpretation errors. Such a discrepancy underscores a critical deficiency
36 in predictive processing in ASD, where, rather than synergistically enhancing each other, auditory
37 and visual cues conflict, undermining the synthesis of coherent audio-visual perception. This
38 misalignment is also reflected in the broader phase distribution seen in children with ASD,
39 suggesting that they might require an extended temporal window to reach effective processing
40 (31,32).

40
Our results thus confirm that the phase of low-frequency neural oscillations is crucial for the encoding of order - for instance with the implication of the theta band in working memory (108) or for temporal parsing in speech (109). The anomaly in temporal encoding mechanisms described in our experiment is constrained by the temporal features provided by external stimulation to build a temporal reference frame. While delta oscillations have previously been linked to temporal predictability (110,111), we observed here that sensory integration is affected by AV misalignment in the theta range, which is associated with atypical speech perception in ASD. The AV integration primarily occurs at the syllable level with a typical tolerance of AV asynchrony at 250ms, which corresponds to the theta range (39,61–64).

## Conclusion

We show remarkable anomalies in audio-visual integration in children with ASD. We confirm previous findings of disrupted speech rhythm tracking and further reveal a specific disruption in audio-visual integration, manifesting as temporal desynchronization. This disruption significantly impacts speech processing, contributing to the communicative challenges faced by children with ASD. Our results highlight the critical role of temporal processing in audio-visual integration and underscore the importance of characterizing these mechanisms in ASD. Moving forward, these insights could inform the development of targeted interventions aiming at regulating temporal speech processing and AV synchronization to improve communication in children with ASD.

## References

1. Alegria J, Noirot E (1978): Neonate orientation behaviour towards human voice. *Int J Behav Dev* 1: 291–312.

2. Jusczyk PW, Bertoncini J (1988): Viewing the development of speech perception as an innately guided learning process. *Lang Speech* 31: 217–238.

3. Lecanuet J, Granier-Deferre C, Decasper A, Maugeais R, Andrieu A, Busnel M (1987): Fetal perception and discrimination of speech stimuli; demonstration by cardiac reactivity; preliminary results. *Comptes Rendus Académie Sci Sér III Sci Vie* 305: 161–4.

4. Williams JHG, Massaro DW, Peel NJ, Bosseler A, Suddendorf T (2004): Visual–auditory integration during speech imitation in autism. *Res Dev Disabil* 25: 559–575.

5. Iarocci G, Rombough A, Yager J, Weeks DJ, Chua R (2010): Visual influences on speech perception in children with autism. *Autism* 14: 305–20.

6. Feng S, Wang Q, Hu Y, Lu H, Li T, Song C, et al. (2023): Increasing
1

audiovisual speech integration in autism through enhanced attention to
mouth. *Dev Sci* 26: e13348.

2

3

7. Shic F, Wang Q, Macari SL, Chawarska K (2020): The role of limited salience of speech in selective attention to faces in toddlers with autism spectrum disorders. *J Child Psychol Psychiatry* 61: 459–469.

4

5

6

8. Klin A (1991): Young autistic children's listening preferences in regard to speech: a possible characterization of the symptom of social withdrawal. *J Autism Dev Disord* 21: 29–42.

7

8

9

9. Klin A, Lin DJ, Gorrindo P, Ramsay G, Jones W (2009): Two-year-olds with autism orient to non-social contingencies rather than biological motion [no. 7244]. *Nature* 459: 257–261.

10

11

12

10. Klin A (1992): Listening preferences in regard to speech in four children with developmental disabilities. *J Child Psychol Psychiatry* 33: 763–9.

13

14

11. Dawson G, Meltzoff AN, Osterling J, Rinaldi J, Brown E (1998): Children with autism fail to orient to naturally occurring social stimuli. *J Autism Dev Disord* 28: 479–85.

15

16

17

12. Kuhl PK, Coffey-Corina S, Padden D, Dawson G (2005): Links between social and linguistic processing of speech in preschool children with autism: behavioral and electrophysiological measures. *Dev Sci* 8: F1–F12.

18

19

20

13. Kojovic N, Cekic S, Castañón SH, Franchini M, Sperdin HF, Sandini C, et al. (2024): Unraveling the developmental dynamic of visual exploration of social interactions in autism ((C. Büchel, editor)). *eLife* 13: e85623.

21

22

23

14. Federici A, Parma V, Vicovaro M, Radassao L, Casartelli L, Ronconi L (2020): Anomalous Perception of Biological Motion in Autism: A Conceptual Review and Meta-Analysis [no. 1]. *Sci Rep* 10: 4576.

24

25

26

15. Knight EJ, Krakowski AI, Freedman EG, Butler JS, Molholm S, Foxe JJ (2022): Attentional influences on neural processing of biological motion in typically developing children and those on the autism spectrum. *Mol Autism* 13: 33.

27

28

29

30

16. Todorova GK, Hatton REM, Pollick FE (2019): Biological motion perception in autism spectrum disorder: a meta-analysis. *Mol Autism* 10: 49.

31

32

17. Marco EJ, Hinkley LB, Hill SS, Nagarajan SS (2011): Sensory processing in autism: a review of neurophysiologic findings. *Pediatr Res* 69: 48R-54R.

33

34

18. Todorova GK, Pollick FE, Muckli L (2021): Special treatment of prediction errors in autism spectrum disorder. *Neuropsychologia* 163: 108070.

35
1

19. Collet L, Roge B, Descouens D, Moron P, Duverdy F, Urgell H (1993):

2

Objective auditory dysfunction in infantile autism. *The Lancet* 342: 923–

3

924.

4

20. Haesen B, Boets B, Wagemans J (2011): A review of behavioural and

5

electrophysiological studies on auditory processing and speech perception in

6

autism spectrum disorders. *Res Autism Spectr Disord* 5: 701–714.

7

21. Edgar JC, Fisk Iv CL, Berman JI, Chudnovskaya D, Liu S, Pandey J, et al.

8

(2015): Auditory encoding abnormalities in children with autism spectrum

9

disorder suggest delayed development of auditory cortex. *Mol Autism* 6: 69.

10

22. Foss-Feig JH, Schauder KB, Key AP, Wallace MT, Stone WL (2017):

11

Audition-specific temporal processing deficits associated with language

12

function in children with autism spectrum disorder. *Autism Res Off J Int Soc*

13

*Autism Res* 10: 1845–1856.

14

23. Wang X, Wang S, Fan Y, Huang D, Zhang Y (2017): Speech-specific

15

categorical perception deficit in autism: An Event-Related Potential study of

16

lexical tone processing in Mandarin-speaking children. *Sci Rep*.

17

https://doi.org/10.1038/srep43254

18

24. Wang X, Delgado J, Marchesotti S, Kojovic N, Sperdin HF, Rihs TA, et al.

19

(2023): Speech Reception in Young Children with Autism Is Selectively

20

Indexed by a Neural Oscillation Coupling Anomaly. *J Neurosci* 43: 6779–

21

6795.

22

25. Benasich AA, Gou Z, Choudhury N, Harris KD (2008): Early cognitive and

23

language skills are linked to resting frontal gamma power across the first 3

24

years. *Behav Brain Res* 195: 215–22.

25

26. Benitez-Burraco A, Murphy E (2016): The Oscillopathic Nature of Language

26

Deficits in Autism: From Genes to Language Evolution. *Front Hum*

27

*Neurosci* 10: 120.

28

27. Cermak CA, Arshinoff S, Ribeiro de Oliveira L, Tendera A, Beal DS, Brian J,

29

et al. (2022): Brain and Language Associations in Autism Spectrum

30

Disorder: A Scoping Review. *J Autism Dev Disord* 52: 725–737.

31

28. Morrel J, Singapuri K, Landa RJ, Reetzke R (2023): Neural correlates and

32

predictors of speech and language development in infants at elevated

33

likelihood for autism: a systematic review.

34

https://doi.org/10.3389/fnhum.2023.1211676

35

29. Crosse MJ, Foxe JJ, Tarrit K, Freedman EG, Molholm S (2022): Resolution of
1 impaired multisensory processing in autism and the cost of switching
2 sensory modality [no. 1]. *Commun Biol* 5: 1–17.

30. Jao Keehn RJ, Sanchez SS, Stewart CR, Zhao W, Grenesko-Stevens EL, Keehn B, Müller R-A (2017): Impaired downregulation of visual cortex during auditory processing is associated with autism symptomatology in children and adolescents with autism spectrum disorder. *Autism Res Off J Int Soc Autism Res* 10: 130–143.

31. Stevenson RA, Siemann JK, Schneider BC, Eberly HE, Woynaroski TG, Camarata SM, Wallace MT (2014): Multisensory temporal integration in autism spectrum disorders. *J Neurosci* 34: 691–7.

32. Stevenson RA, Segers M, Ferber S, Barense MD, Wallace MT (2014): The impact of multisensory integration deficits on speech perception in children with autism spectrum disorders. *Front Psychol* 5: 379.

33. Alm M, Behne DM, Wang Y, Eg R (2009): Audio-visual identification of place of articulation and voicing in white and babble noisea). *J Acoust Soc Am* 126: 377–387.

34. Bertels J, Niesen M, Destoky F, Coolen T, Vander Ghinst M, Wens V, et al. (2023): Neurodevelopmental oscillatory basis of speech processing in noise. *Dev Cogn Neurosci* 59: 101181.

35. Fleming JT, Maddox RK, Shinn-Cunningham BG (2021): Spatial alignment between faces and voices improves selective attention to audio-visual speech. *J Acoust Soc Am* 150: 3085–3100.

36. Yuan Y, Lleo Y, Daniel R, White A, Oh Y (2021): The Impact of Temporally Coherent Visual Cues on Speech Perception in Complex Auditory Environments. *Front Neurosci* 15. https://doi.org/10.3389/fnins.2021.678029

37. Guiraud JA, Tomalski P, Kushnerenko E, Ribeiro H, Davies K, Charman T, et al. (2012): Atypical audiovisual speech integration in infants at risk for autism. *PLoS One* 7: e36428.

38. Lindborg A, Baart M, Stekelenburg JJ, Vroomen J, Andersen TS (2019): Speech-specific audiovisual integration modulates induced theta-band oscillations. *PLoS One* 14: e0219744.

39. Munhall KG, Gribble P, Sacco L, Ward M (1996): Temporal constraints on the McGurk effect. *Percept Psychophys* 58: 351–362.

40. van Wassenhove V (2013): Speech through ears and eyes: interfacing the
1 senses with the supramodal brain. *Front Psychol* 4. Retrieved August 28,
2 2023, from https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00388

3 41. Choi I, Lee JY, Lee SH (2018): Bottom-up and top-down modulation of multisensory integration. *Curr Opin Neurobiol* 52: 115–122.

5 42. van Wassenhove V, Grant KW, Poeppel D (2005): Visual speech speeds up the neural processing of auditory speech. *Proc Natl Acad Sci U A* 102: 1181–6.

7 43. Kushnerenko E, Teinonen T, Volein A, Csibra G (2008): Electrophysiological evidence of illusory audiovisual speech percept in human infants. *Proc Natl Acad Sci* 105: 11442–11445.

10 44. Musacchia G, Schroeder CE (2009): Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. *Hear Res* 258: 72–79.

13 45. Murray MM, Wallace MT (Eds.) (2011): *The Neural Bases of Multisensory Processes*. Boca Raton: CRC Press. https://doi.org/10.1201/9781439812174

15 46. Baum SH, Stevenson RA, Wallace MT (2015): Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. *Prog Neurobiol* 134: 140–60.

18 47. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010): An extended multisensory temporal binding window in autism spectrum disorders. *Exp Brain Res* 203: 381–9.

21 48. Kwakye LD, Foss-Feig JH, Cascio CJ, Stone WL, Wallace MT (2011): Altered auditory and multisensory temporal processing in autism spectrum disorders. *Front Integr Neurosci* 4: 129.

24 49. Gao M, Lim S, Chubykin AA (2021): Visual familiarity induced 5 Hz oscillations and improved orientation and direction selectivities in V1. *J Neurosci*. https://doi.org/10.1523/JNEUROSCI.1337-20.2021

27 50. Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE (2005): An Oscillatory Hierarchy Controlling Neuronal Excitability and Stimulus Processing in the Auditory Cortex. *J Neurophysiol* 94: 1904–1911.

30 51. Romei V, Brodbeck V, Michel C, Amedi A, Pascual-Leone A, Thut G (2008): Spontaneous Fluctuations in Posterior α-Band EEG Activity Reflect Variability in Excitability of Human Visual Areas. *Cereb Cortex* 18: 2010–2018.

34 52. Leszczynski M, Schroeder CE (2019): The Role of Neuronal Oscillations in Visual Active Sensing. *Front Integr Neurosci* 13.
1 https://doi.org/10.3389/fnint.2019.00032

2 53. Poeppel D, Assaneo MF (2020): Speech rhythms and their neural foundations [no. 6]. *Nat Rev Neurosci* 21: 322–334.

3 54. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA (2009): The Natural Statistics of Audiovisual Speech. *PLOS Comput Biol* 5: e1000436.

7 55. Arnal LH, Wyart V, Giraud AL (2011): Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. *Nat Neurosci* 14: 797–801.

10 56. Young AW, Frühholz S, Schweinberger SR (2020): Face and Voice Perception: Understanding Commonalities and Differences. *Trends Cogn Sci* 24: 398–410.

13 57. Aller M, Økland HS, MacGregor LJ, Blank H, Davis MH (2022): Differential Auditory and Visual Phase-Locking Are Observed during Audio-Visual Benefit and Silent Lip-Reading for Speech Perception. *J Neurosci* 42: 6108–6120.

17 58. Hagan CC, Woods W, Johnson S, Calder AJ, Green GGR, Young AW (2009): MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus. *Proc Natl Acad Sci* 106: 20010–20015.

20 59. Hagan CC, Woods W, Johnson S, Green GGR, Young AW (2013): Involvement of Right STS in Audio-Visual Integration for Affective Speech Demonstrated Using MEG. *PLOS ONE* 8: e70648.

23 60. Plöchl M, Fiebelkorn I, Kastner S, Obleser J (2022): Attentional sampling of visual and auditory objects is captured by theta-modulated neural activity. *Eur J Neurosci* 55: 3067–3082.

26 61. van Wassenhove V, Grant KW, Poeppel D (2007): Temporal window of integration in auditory-visual speech perception. *Neuropsychologia* 45: 598–607.

29 62. Stevenson RA, Wallace MT (2013): Multisensory temporal integration: task and stimulus dependencies. *Exp Brain Res* 227: 249–61.

31 63. Massaro DW, Cohen MM (1993): The paradigm and the fuzzy logical model of perception are alive and well. *J Exp Psychol Gen* 122: 115–124.

33 64. Guilleminot P, Graef C, Butters E, Reichenbach T (2023): Audiotactile Stimulation Can Improve Syllable Discrimination through Multisensory Integration in the Theta Frequency Band. *J Cogn Neurosci* 35: 1760–1772.
1 65. Power AJ, Mead N, Barnes L, Goswami U (2012): Neural entrainment to rhythmically presented auditory, visual, and audio-visual speech in children.

2
3 *Front Psychol* 3: 216.

4 66. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008): Neuronal oscillations and visual amplification of speech. *Trends Cogn Sci* 12: 106–13.

6 67. Keshavarzi M, Mandke K, Macfarlane A, Parvez L, Gabrielczyk F, Wilson A, Goswami U (2022): Atypical delta-band phase consistency and atypical preferred phase in children with dyslexia during neural entrainment to rhythmic audio-visual speech. *NeuroImage Clin* 35: 103054.

10 68. Franchini M, Wood de Wilde H, Glaser B, Gentaz E, Eliez S, Schaer M (2016): Brief Report: A Preference for Biological Motion Predicts a Reduction in Symptom Severity 1 Year Later in Preschoolers with Autism Spectrum Disorders. *Front Psychiatry* 7: 143.

14 69. Franchini M, Zoller D, Gentaz E, Glaser B, Wood de Wilde H, Kojovic N, et al. (2018): Early Adaptive Functioning Trajectories in Preschoolers With Autism Spectrum Disorders. *J Pediatr Psychol* 43: 800–813.

17 70. Lord C, Risi S, Lambrecht L, Cook EH, Leventhal BL, DiLavore PC, et al. (2000): The Autism Diagnostic Observation Schedule—Generic: A standard measure of social and communication deficits associated with the spectrum of autism. *J Autism Dev Disord* 30: 205–223.

21 71. Lord C, Rutter M, DiLavore P, Risi S, Gotham K, Bishop S (2012): Autism diagnostic observation schedule: ADOS-2. West Psychol. J Psychoeduc Assess 32: 88–92.

24 72. *Trotro I'anniversaire de nana* (2013): Storimages.

25 73. *Trotro part en vacance* (2013): Storimages.

26 74. *Trotro et la boite a secrets* (2013): Storimages.

27 75. *Trotro es tres amoureux* (2013): Storimages.

28 76. Olsen A (2012): The Tobii I-VT fixation filter. *Tobii Technol* 21: 4–19.

29 77. Boashash B (2015): *Time-Frequency Signal Analysis and Processing: A Comprehensive Reference*. Academic Press.

31 78. Pelli DG, Tillman KA (2008): The uncrowded window of object recognition. *Nat Neurosci* 11: 1129–35.

33 79. Nuthmann A (2013): On the visual span during object search in real-world scenes. *Vis Cogn* 21: 803–837.

35 80. Jessen S, Fiedler L, Münte TF, Obleser J (2019): Quantifying the individual
1 auditory and visual brain response in 7-month-old infants watching a brief

2 cartoon movie. *NeuroImage*.

3 https://doi.org/10.1016/j.neuroimage.2019.116060

4 81. Weineck K, Wen OX, Henry MJ (2022): Neural synchronization is strongest to

5 the spectral flux of slow music and depends on familiarity and beat salience

6 ((O. Jensen, B. G. Shinn-Cunningham, & B. Zoefel, editors)). *eLife* 11:

7 e75515.

8 82. Cover TM, Thomas JA (2005): Entropy, Relative Entropy, and Mutual

9 Information. *Elements of Information Theory*. John Wiley & Sons, Ltd, pp

10 13–55.

11 83. Timme NM, Lapish C (2018): A Tutorial for Information Theory in

12 Neuroscience. *eNeuro* 5. https://doi.org/10.1523/ENEURO.0052-18.2018

13 84. Delorme A, Makeig S (2004): EEGLAB: an open source toolbox for analysis

14 of single-trial EEG dynamics including independent component analysis. *J Neurosci Methods* 134: 9–21.

15

16 85. Crosse MJ, Di Liberto GM, Bednar A, Lalor EC (2016): The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for

17 Relating Neural Signals to Continuous Stimuli. *Front Hum Neurosci* 10:

18 604.

20 86. Maris E, Oostenveld R (2007): Nonparametric statistical testing of EEG- and

21 MEG-data. *J Neurosci Methods* 164: 177–90.

22 87. Dinno A (2015): Nonparametric Pairwise Multiple Comparisons in

23 Independent Groups using Dunn's Test. *Stata J* 15: 292–300.

24 88. Nozaradan S, Peretz I, Missal M, Mouraux A (2011): Tagging the neuronal

25 entrainment to beat and meter. *J Neurosci* 31: 10234–40.

26 89. Clouter A, Shapiro KL, Hanslmayr S (2017): Theta Phase Synchronization Is

27 the Glue that Binds Human Associative Memory. *Curr Biol* 27: 3143–

28 3148.e6.

29 90. Pefkou M, Arnal LH, Fontolan L, Giraud AL (2017): theta-Band and beta-

30 Band Neural Activity Reflects Independent Syllable Tracking and

31 Comprehension of Time-Compressed Speech. *J Neurosci* 37: 7930–7938.

32 91. Berens P (2009): **CircStat**: A MATLAB Toolbox for Circular Statistics. *J Stat*

33 Softw 31. https://doi.org/10.18637/jss.v031.i10

34 92. Jochaut D, Lehongre K, Saitovitch A, Devauchelle AD, Olasagasti I, Chabane

35 N, et al. (2015): Atypical coordination of cortical oscillations in response to
1 speech in autism. *Front Hum Neurosci* 9: 171.

2 93. Puschmann S, Daeglau M, Stropahl M, Mirkovic B, Rosemann S, Thiel CM,
3 Debener S (2019): Hearing-impaired listeners show increased audiovisual
4 benefit when listening to speech in noise. *NeuroImage* 196: 261–268.

5 94. Haider CL, Suess N, Hauswald A, Park H, Weisz N (2022): Masking of the
6 mouth area impairs reconstruction of acoustic speech features and higher-
7 level segmentational features in the presence of a distractor speaker.
8 *NeuroImage* 252: 119044.

9 95. O'CONNOR N, Hermelin B (1965): Sensory dominance: In autistic imbecile
10 children and controls. *Arch Gen Psychiatry* 12: 99–103.

11 96. Arnal LH, Morillon B, Kell CA, Giraud A-L (2009): Dual Neural Routing of
12 Visual Facilitation in Speech Processing. *J Neurosci* 29: 13445–13453.

13 97. Liu W, Li M, Yi L (2016): Identifying children with autism spectrum disorder
14 based on their face processing abnormality: A machine learning framework.
15 *Autism Res* 9: 888–98.

16 98. Sinnett S, Soto-Faraco S, Spence C (2008): The co-occurrence of multisensory
17 competition and facilitation. *Acta Psychol (Amst)* 128: 153–161.

18 99. Cuppini C, Ursino M, Magosso E, Rowland BA, Stein BE (2010): An
19 emergent model of multisensory integration in superior colliculus neurons.
20 *Front Integr Neurosci* 4. https://doi.org/10.3389/fnint.2010.00006

21 100. Irwin J, Harwood V, Kleinman D, Baron A, Avery T, Turcios J, Landi N
22 (2023): Neural and Behavioral Differences in Speech Perception for
23 Children With Autism Spectrum Disorders Within an Audiovisual Context.
24 *J Speech Lang Hear Res* 66: 2390–2403.

25 101. Chawarska K, Lewkowicz D, Feiner H, Macari S, Vernetti A (2022):
26 Attention to audiovisual speech does not facilitate language acquisition in
27 infants with familial history of autism. *J Child Psychol Psychiatry* 63: 1466–
28 1476.

29 102. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze H-
30 J, Driver J (2007): Audiovisual Temporal Correspondence Modulates
31 Human Multisensory Superior Temporal Sulcus Plus Primary Sensory
32 Cortices. *J Neurosci* 27: 11431–11441.

33 103. Stacey PC, Kitterick PT, Morris SD, Sumner CJ (2016): The contribution of
34 visual information to the perception of speech in noise with and without
35 informative temporal fine structure. *Hear Res* 336: 17–28.
1

104. Han S, Chen Y-C, Maurer D, Shore DI, Lewis TL, Stanley BM, Alais D

2

(2022): The development of audio-visual temporal precision precedes its

3

rapid recalibration. *Sci Rep* 12: 21591.

4

105. Venezia JH, Thurman SM, Matchin W, George SE, Hickok G (2016): Timing

5

in audiovisual speech perception: A mini review and new psychophysical

6

data. *Atten Percept Psychophys* 78: 583–601.

7

106. Zoefel B, Archer-Boyd A, Davis MH (2018): Phase Entrainment of Brain

8

Oscillations Causally Modulates Neural Responses to Intelligible Speech.

9

*Curr Biol* 28: 401-408 e5.

10

107. Henry MJ, Obleser J (2012): Frequency modulation entrains slow neural

11

oscillations and optimizes human listening behavior. *Proc Natl Acad Sci*

12

109: 20095–20100.

13

108. Lisman JE, Jensen O (2013): The theta-gamma neural code. *Neuron* 77:

14

1002–16.

15

109. Giraud AL, Poeppel D (2012): Cortical oscillations and speech processing:

16

emerging computational principles and operations. *Nat Neurosci* 15: 511–7.

17

110. Stefanics G, Hangya B, Hernádi I, Winkler I, Lakatos P, Ulbert I (2010):

18

Phase Entrainment of Human Delta Oscillations Can Mediate the Effects of

19

Expectation on Reaction Speed. *J Neurosci* 30: 13578–13585.

20

111. Herbst SK, Stefanics G, Obleser J (2022): Endogenous modulation of delta

21

phase by expectation–A replication of Stefanics et al., 2010. *Cortex* 149:

22

226–245.

23

24
1 Tables

2

<table><caption>Table 1 The statistical difference among joint model and single models for speech envelope and visual motion</caption><thead><tr><th></th><th>Dunn's multiple comparisons test</th><th>Mean rank diff.</th><th>Significant?</th><th>Adjusted P Value</th></tr></thead><tbody><tr><td rowspan="2">ASD</td><td>speech envelope</td><td>-60.48</td><td>Yes</td><td>0.0304</td></tr><tr><td>visual motion</td><td>-137.9</td><td>Yes</td><td>&lt;0.0001</td></tr><tr><td rowspan="2">TD</td><td>speech envelope</td><td>-51.09</td><td>No</td><td>0.1414</td></tr><tr><td>visual motion</td><td>-58.47</td><td>Yes</td><td>0.0373</td></tr></tbody></table>

3

4

<table><caption>Table 2 The statistical difference of the decoding accuracy evolution for speech envelope and visual motion</caption><thead><tr><th></th><th>Dunn's multiple comparisons test</th><th>Mean rank diff.</th><th>Significant?</th><th>Adjusted P Value</th></tr></thead><tbody><tr><td>Visual motion</td><td>ASD v.s.TD</td><td>-37.55</td><td>Yes</td><td>0.0003</td></tr><tr><td>Speech envelope</td><td>ASD v.s.TD</td><td>-13.32</td><td>No</td><td>0.8866</td></tr><tr><td>TD</td><td>V v.s. A</td><td>-6.094</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>ASD</td><td>V v.s. A</td><td>-30.32</td><td>Yes</td><td>0.0065</td></tr></tbody></table>

A: speech envelope

V: visual motion

5

6

7

<table><caption>Table 3 The statistical difference across groups and frequency bands in stimulus-response coherence</caption><thead><tr><th></th><th></th><th>Dunn's multiple comparisons test</th><th>Mean rank diff.</th><th>Significant?</th><th>Adjusted P Value</th></tr></thead><tbody><tr><td>ASD</td><td>A</td><td>delta v.s. theta</td><td>162.7</td><td>Yes</td><td>&lt;0.0001</td></tr><tr><td>ASD</td><td>V</td><td>delta v.s. theta</td><td>191.5</td><td>Yes</td><td>&lt;0.0001</td></tr><tr><td>ASD</td><td>delta</td><td>A v.s. V</td><td>-7.774</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>ASD</td><td>theta</td><td>A v.s. V</td><td>20.94</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>TD</td><td>A</td><td>delta v.s. theta</td><td>178.6</td><td>Yes</td><td>&lt;0.0001</td></tr><tr><td>TD</td><td>V</td><td>delta v.s. theta</td><td>176.4</td><td>Yes</td><td>&lt;0.0001</td></tr><tr><td>TD</td><td>delta</td><td>A v.s. V</td><td>-0.6563</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>TD</td><td>theta</td><td>A v.s. V</td><td>-2.844</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>A</td><td>delta</td><td>ASD v.s. TD</td><td>12.3</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>A</td><td>theta</td><td>ASD v.s. TD</td><td>28.19</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>V</td><td>delta</td><td>ASD v.s. TD</td><td>19.42</td><td>No</td><td>&gt;0.9999</td></tr><tr><td>V</td><td>theta</td><td>ASD v.s. TD</td><td>4.406</td><td>No</td><td>&gt;0.9999</td></tr></tbody></table>

A: speech envelope

V: visual motion

8