SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes
Authors: Tony Alex
Sara Atito
Armin Mustafa
Muhammad Awais
Philip J B Jackson
Venue: ICLR 2025 Poster (2025)
Keywords: audio self-supervised learning
audio representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Audio self-supervised learning approach that  not only improves model performance on polyphonic audio, but also maintains or exceed performance on standard audio SSL benchmarks.

Abstract: Self-supervised pre-trained audio networks have seen widespread adoption in real-world systems, particularly in multi-modal large language models. These networks are often employed in a frozen state, under the assumption that the self-supervised pre-training has sufficiently equipped them to handle real-world audio. However, a critical question remains: how well do these models actually perform in real-world conditions, where audio is typically polyphonic and complex, involving multiple overlapping sound sources? Current audio self-supervised learning (SSL) methods are often benchmarked on datasets predominantly featuring monophonic audio, such as environmental sounds, and speech. As a result, the ability of SSL models to generalize to polyphonic audio, a common characteristic in natural scenarios, remains underexplored. This limitation raises concerns about the practical robustness of SSL models in more realistic audio settings. To address this gap, we introduce Self-Supervised Learning from Audio Mixtures (SSLAM), a novel direction in audio SSL research,  designed to improve the model’s ability to learn from polyphonic data while maintaining strong performance on monophonic data. We thoroughly evaluate SSLAM on standard audio SSL benchmark datasets which are predominantly monophonic and conduct a comprehensive comparative analysis against state-of-the-art (SOTA) methods using a range of high-quality, publicly available polyphonic datasets. SSLAM not only improves model performance on polyphonic audio, but also maintains or exceeds performance on standard audio SSL benchmarks. Notably, it achieves up to a 3.9% improvement on the AudioSet-2M(AS-2M), reaching a mean average precision (mAP) of 50.2. For polyphonic datasets, SSLAM sets new SOTA in both linear evaluation and fine-tuning regimes with performance improvements of up to 9.1%(mAP). These results demonstrate SSLAM's effectiveness in both polyphonic and monophonic soundscapes, significantly enhancing the performance of audio SSL models. Code and pre-trained models are available at https://github.com/ta012/SSLAM.

Published as a conference paper at ICLR 2025
SSLAM: ENHANCING SELF-SUPERVISED MODELS WITH AUDIO MIXTURES FOR POLYPHONIC SOUND- SCAPES
Tony Alex, Sara Ahmed, Armin Mustafa, Muhammad Awais", Philip JB Jackson"
Surrey Institute for Ceople-Centree AI, University of Surrey, Guildford, GU2 7XH, UK Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey (t.alexx sara at i to, rm musta fa, muhammad. awai i 8, p. [ackson)@surrae ac..u "Equal senior contributions.
ABSTRACT
Self-supervised pre-trained audio networks have seen widespread adoption in real- world systems, particularly in multi-modal large language models. These net- works are often employed in a frozen state, under the assumption that the self- supervised pre- pre-training has sufficiently equipped them to handle real-world audio. However, a critical question remains: how well do these models actually perform in real-world conditions, where audio is typically polyphonic and complex, in- volving multiple overlapping sound sources? Current audio self-supervised learn- ing (SSL) methods are often benchmarked on datasets predominantly featuring monophonic audio, such as environmental sounds, and speech. As a result, the ability of SSL models to generalize to polyphonic audio, common character- istic in natural scenarios, remains underexplored. This limitation raises con- cerns about the practical robustness of SSL models in more realistic audio set- tings. To address this gap, we introduce Self-Supervised Learning from Audio Mixtures (SSLAM), novel direction in audio SSL research, designed to improve the model's ability to learn from polyphonic data while maintaining strong per- formance on monophonic data. We thoroughly evaluate SSLAM on standard audio SSI. benchmark datasets which are predominantly monophonic and con- duct a comprehensive comparative analysis against state-of-the-ard (SOTA) meth- ods using range of high-quality, publicly available polyphonic datasets. SS- LAM not only improves model performance on polyphonic audio, but also main- tains exceeds performance
standard audio SSL benchmarks. Notably, it achieves up to a 3.9% improvement on the AudioSet-2M(AS-2MMM reaching a average precision (mAP) of 50.2. For polyphonic datasets, SSLAM sets new SOTA in both linear evaluation and fine-tuning regimes with performance improvements of up 9.1 (mA)). These results demonstrate SSLAM's effec- tiveness in both polyphonic and monophonic soundscapes, significantly enbancing the performance of audio SSL models. Code and pre-trained models are available at http:/thhhwccccmt///o///oto
www.
INTRODUCTION
Self-supervised learning (SSL) has significantly advanced various domains by leveraging large vol- umes of unlabeled data to pre-train models effectively. In the audio domain, SSL has enabled models to achieve state-of-ther -fe-th-rrt (SOTA) performance (Gong al. 2022 Chong al., 2023 Huang al,, 2022a Chen et al., 2022 Ahmed et al.] [222]] Chen et al. 2022), particularly with the integration of transformer architectures (Dosovitskiy et al.) 2020). Pre-trained audio models are extensively used in real-world applications, such audio-visual segmentation (Liu et al.) 2023 Park et al. 2024 Labb et al., (0044). audio captioning (Labb et al. 2024) etc. Recently, they have also been increasingly integrated into multi-modal models, particularly curllly in multi-modal large language models (LLMs) (Zhang et al. 2023 Tang et al. 2023 Zhao et al. 2023 Panagopoulou et al. (023). These models often utilized in frozen state, where projection layer is added on top of the tre-trained

---

Published as a conference paper at ICLR 2025
backbone to interface with systems like LLMs. In this setup, only the projector is trained, based on the assumption that the pre-trained audio models are capable handling real-worke polyphonic audio scenarios. However, the audio SSL methods are rarely evaluated for polyphonic scenarios.
Unlike monophonic audio, which features a single sound source, polyphonic audio involves audi- tory environments with overlapping sounds, such musical ensembles, multi-speaaker settings, or diverse natural soundscapes, like traffic and park noises. Although the term "polyphony" is often associated with music, we use here in broader sense describe any audio scene with multi- ple concurrent sources. Given that much real-world audio is inherently polyphonic, it is essential for audio encoder representation learning to account for this complexity. While one might argue that pre- pre-training with AudioSet (Gemmeke et al. 2017), dataset containing multi-label samples should adequate, it is crucial to recognize that many the audio files in AudioSet are not truly polyphonic (refer to detailed analysis in Appendix B. Instead, these files often carry multiple labels that describe different facets of a single sound event with only proportion of dataset being actually polyphonic. For example, recording labeled as 'Carnatic music', 'Music', 'Musical in- strument', and "Classical music' all refer to the same audio event. This labeling approach does not fully encompass the complexity and richness of authentically polyphonic audio.
Given the intuitive nature of audio mixing, it would seem natural for it to serve as a standard pre- text task in SSL for audio. However, this approach remains underexplored for self-supervised pre- training. Beyond its potential as a pretext task, incorporating audio mixtures into SSL offers several key advantages. First, it establishes flexible framework for learning invariant representations that are robust to noise and generalizable across a wide range of tasks, whether involving monophonic or polyphonic audio data. Additionally, mixing audio allows for the creation of synthetic data from existing datasets, significantly increasing the diversity of training examples. This enhanced va- riety fosters richer learning signals, enabling the development stronger and more semantically meaningful representations. Despite these strengths, audio mixing in SSL is still an underutilized approach, leaving substantial room for further exploration.
(A)
Mask
Mask
S
SelEDisiilation
Audio
SelfDisuiilation
Audio
EIMA
Mix
EMA
Audio
Downstream Task
(D)
Mix
Mask
Audio 2 Audio 1
Cur Horn
Self-Distillttinn
Ambulance Siren
Bicycle
AVE
Motorcycle Engine
Bus Brake
Fire Screectinng
Figure 1: Overview of the components in our proposed audio SSL pre-training on unlabeled data and the audio event tagging downstream task (T: Teacher encoder, Student encoder). (A): Masked Latent Bootstrapping (self-distillation where the teacher is the exponentially moving average of the student) with unmixed audio (baseline). (B): Masked Latent Bootstrapping with mixed audio. (C): Source retention loss to preserve the distinct characteristics of individual audio sources. (D): Overview of the audio event tagging downstream task.
To address this limitation, we introduce SSLAM, a novel self-supervised pre-training strategy de- signed to enhance the ability of transformer-based models to learn from polyphonic data. In our approach, the student model receives mixed audio inputs, where multiple audio signals are ran- domly combined form polyphonic signal. Since this is self-supervised learning method, do not know in advance whether the selected audios are monophonic or polyphonic, meaning that the resulting audio be either mixture
mixture of mixtures. Concurrently, the teacher
2

---

Published as a conference paper at ICLR 2025
model processes the same audio sources separately, averaging their features at the output of the teacher model. The student model's output is then compared to the teacher's aggregated features, enabling a robust learning mechanism that better adapts to polyphonic scenarios (refer Figure
We evaluate the proposed SSLAM on standard audio SSL benchmark datasets, including event tagging tasks covering a range of sounds, e.g. environmental sound and speech, achieving state-of- the-art performance across all categories. Furthermore, we integrate publicly available polyphonic datasets into the evaluation pipeline assess the model's robustness in handling complex, real-world audio scenarios. We demonstrate substantial improvements in handling polyphonic audio, with our method consistently outperforming prior approaches in real-world polyphonic audio scenarios.
In summary, our contributions include:
1. The introduction of audio mixtures for the self superviee pre-training, enabling the network to better adapt to real-world polyphonic audio environments.
2. A novel source retention loss, which explicitly preserves the individual characteristics of each audio source within the mixture. By encouraging the network to recognize and retain the distinct features of each input, this loss function ensures the integrity of each source, even when multiple sources are combined.
A comprehensive evaluation of our model on major audio SSL benchmark datasets, demonstrating SOTA performance across general audio and speech tasks compared to prior approaches.
4. Extensive evaluation of the model's polyphonic capabilities using several polyphonic datasets, demonstrating substantial improvements in handling real-world polyphonic audio compared to ex- isting methods.
RELATED WORK
Masked latent bootstrapping. Several works (Baade et al. [022 Chong et al.] 2023 Niizumi
al., 2022, Baevski et al. [0222 Gong al. [022] Ahmed al., 2024) have explored various strategies for masking specific regions of the spectrogram, with pre-training primarily focused on reconstructing the masked regions within the spectrogram space. Although this approach has shown promise, other studies have questioned whether spectrogram reconstruction is the most effective method. For instance, Chen et al. (2022) introduced the prediction of patch-level discrete labels generated by acoustic tokenizers instead of reconstructing the spectrogram. Another approach is predicting target masked latent features that capture higher- semantically meaningful infor- mation about the audio rather than focusing low-level spectrogram reconstruction. Bootstrap- ping via EMA teacher-based self-distillation (Grill et al., 20200) Niizumi et al. 202T) is an effective exponential moving average (EMA) approach for generating target representations. Works such Fei et al. (2024); Chen et al. (2024) are based on the concept of masked latent bootstrapping. In partieular, Baevski et al. (2022) 2023); Chen et al. (2024) gen- erate target representations by averaging outputs from multiple layers of the teacher model within an EMA-based deff-distillatio framework. This approach captures both high-level and low-level information, rather than relying solely on the final layer, leading to richer representations.
as Baevski et al. 2002) 2023);;
Polyphonic data modeling. Abeßer et al. (2023) analyzed supervised audio networks such as PANN Kong et al. (2020) and P Koutini et al. (202T) on low-degrees polyphony (2 events) created using ESC-50 (Piczak] 2015). Salamon et al. (2017) introduced the Scaper library to create polyphonic datasets. with URBAN-SED (Salamon et al. 2017) demonstrating its poten- tial. The IDMT-DESED-FL (Johnson al. 2021) dataset was created using DESED (Turpault) et al.) 2019) using Scaper. SPASS is a relatively high-quality dataset created using tools such as RAVEN (Schröder 2011). using monophonic source datasets like ESC-50 and UrbanSound8K (Ar- nault et al. 2020). Networks trained with SPASS outperformed those trained on AudioSet, raising the question of whether addressing polyphony requires solely the creation of more high-qualit, data, or if model development should also be considered. However, to the best of our knowledge, none of the widely used audio SSL models incorporate specific design choices or training objectives aimed at addressing the challenges posed by polyphonic data.
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
In this section, we outline the core components of our framework, starting with the baseline model in Section 3.1 Following, we describe the integration of audio mixtures into our SSL framework in Section 3.2 beginning with explanation of the audio mixing strategies used to generate mixtures and pre-training the baseline model with these mixtures in Section 3.2.1 This is followed by the discussion of our proposed source retention loss, Section [.2.2] which preserves individual source characteristics within the mixtures. Finally, we introduce our unified framework Section which efficiently integrates all components into cohesive model, enabling robust learning in polyphonic as well as monophonic audio scenarios.
3.3
3.1
BASELINE MODEL: MASKED LATENT BOOTSTRAPPING
Our baseline leverages masked latent bootstrapping, an SSL paradigm that combines the concept of bootstrapping (Nizuumi al. 2021 Ahmed al. 2024 spectrogram masking (Huang al. 20226 Baade et al.) 2022), and the prediction of masked tokens in the feature space (Fei ct al. 2024
Baevski al.] 2022 2023). Approaches like Baevski et 2022 2023); Chen al. (2024) effec- tively combine these techniques, enabling the capture of rich contextual representations of audio data. We adopt this paradigm as foundation our method, aiming to enhance performance in complex, polyphonic environment. Below, we outline the key steps of our baseline framework.
First, the input spectrogram S undergoes patchification (Gong al., [021), where S is divided into fixed-size, non-overlapping fieeunrryy patches. To introduce self supervised objective, employ inverse block multi-masking (Baevski et al.] 20233 Chen et al. 2024) with a non- non-masking block size of (5 5), where multiple masked versions of the spectrogram are generated by randomly dropping 80% of the patches from each version. A classification token (CLS) is then appended to each masked spectrogram, which is then passed through the student encoder. The student encoder processes these masked spectrograms and outputs encoded representation Z, with the CLS to- ken's representation denoted ZCLS, Following, random tokens are inserted in place of the masked tokens in the encoded representation and fed to a light-weight CNN-based decoder to predict the representations of the masked tokens, producing the student's patch-level outputs yrpatch.
To guide the student model during training, we employ a momentum -based teacher encoder that processes the original, unmasked spectrogram. The teacher encoder's outputs are averaged across all layers to generate target representations for the student encoder.
The training objective consists of two loss functions. First, global loss ensures that the student model captures the overall structure of the audio signal as its mean square error (MSE):
B nuC ΣΣ j=1
Gglobal
2 (5,)
-2---211 2
(I)
B X TIMC
where B is the batch size, aMe is the number of multi-mask copies, and ZCLS is the spatially pooled output of Z. Second, local loss facilitates fine-grained understanding, as below:
B NJC Σ
Dioaa
Σ
Patch (F..k)
-(1,)))
B x NMC X IM
REM
where M is the set of masked tokens.
3.2 SSLAM: SELF-SUPERVISED LEARNING FROM AUDIO MIXTURES
In this work, we introduce two key innovations to enhance self-supervised learning for audio. First, the introduction of audio mixtures in the SSL setting, enabling the network to better adapt to real- world polyphonic audio environments (Section 3.2.1) To complement this, propose novel source retention loss that explicitly preserves the individual characteristics of each audio source within the mixture. By encouraging the network to recognize and retain the distinet features of
4

---

Published as a conference paper at ICLR 2025
Partial Mixing
Multi Inverse Masking
Decoder
Mixed Spectrogram
Student ViT
Insert random tokens
(mizet))
Linear Projection + Position Embedding
inplace masked tokens
(S (mizze)
Purtial Mix
EMA
mirdd
Spatial
Pool
Time Chank Dropping
Drop Unmixed
Teacher ViT
Layer Avg.
8
Llogglomised
SRLL
Region
(5) (S)
Patch Avg.
SG
Spoctrugram (5t)
Specingram (5p)
ff SG
Figure 2:
Illustraioon of novel contributions in SSLAM. The left side demonstrates the partial mix-
ing of two audio log-mel spectrograms, while the right side visualizes the proposed novel training objectives. SG denotes the stop gradient operation applied during training.
each input, this loss function ensures the integrity of each source, even when multiple sources combined (Section 3.2.2
3.2. MASKED LATENT BOOTSTRAPPING USING AUDIO MIXTURES
Building on the same architecture our baseline model, we introduce key modification: instead of feeding the student and teacher networks spectrogram derived from a single audio source, provide an audio mixture spectrogram. This adjustment introduces additional complexity to learning process.
Audio mixing can be performed either in the waveform or the spectrogram domain. Through our experiments, we found that performing the mix in the spectrogram domain yielded superior per- formance (refer to Appendix E..O.T. Specifically, we apply an element-wise max operation to the log-mel spectrograms of two audio signals create audio mixture. This technique is in- spired by principles from Computational Auditory Scene Analysis (CASA), particularly the Ideal Binary Mask (IBM) (Wang) 2005), which retains the most dominant time-frequency components of overlapping signals. By using the element-wise max operation, we ensure that the most promi- nent frequency features from each source are preserved, similar to how the IBM prioritizes target-dominant regions to enhance separation and intelligibility. To formalize this, let S₁ (f. r) and (2().r) represent the bins and denotes time frames. Thee mixed log-mel spectrogram (f. r) is then computed using the element- wise maximum operation at each time-frequency bin, as follows:
two audio signals, where f denotes mel frequency
Smixed (J, max (S, S2(S,T)), WS5
We include sample visualizations of this approach in Appendix E.0.3 Further, additional experi- mental results and analysis related this approach are provided in the Appendix
Mixing vs Partial Mixing, To achieve a balance between introducing novel audio events and pre- serving the main audible concepts within the original signal, we implement partial audio mixing. Instead of applying mixing across the entire audio clip, only fraction of the audio duration is mixed. For an audio clip of length i, mixing applied distinct regions, covering total duration of t/2. while the original audio is preserved in the remaining 2 x t/4 duration (refer Figure 2 left). By limiting the extent of mixing, we avoid overwhelming the original content, thus preserving key audio characteristics while still benefiting from the additional variability introduced by the mix- ing. This strategy maintains a balance between retaining essential audio features and introducing new information, ultimately leading to enhanced performance in polyphonic audio tasks.
As for the loss functions, a key modification we made when incorporating partially mixed audios into the baseline model involves the selection of teacher layers for the target representation. In the
5

---

Published as a conference paper at ICLR 2025
baseline, the global loss function Lglobal and the local loss funetion Liocat. use the average of all 12 teacher encoder layers to form the target representation. However, with the added complexity from mixing multiple audio sources, averaging over the 12 layers then spatial pooling can potentially lead to excessive information compression, limiting the model's ability to capture meaningful global representations. To address this, used only the final layer's output for the global loss, denoted as Ggoobat, mixred (refer to 6 For local loss Llocal, mized we used all 12 layers, It is worth to note, the optimal selection of teacher layers (top-k) may vary across modalities, as observed in Baevski et al. 2022)
3.2.2 SOURCE RETENTION LOsS (SRL)
To improve the model's 's ability to learn and retain distinct concepts from mixed audios, we introduce an additional loss function, termed Source Retention Loss (SRL). This loss encourages the model to capture the characteristics from each audio source within mixture.
The process begins by feeding the student model with the mixed audio Smizeds as previously de- scribed in Section 3.2.1 The student model generated representations for the mixed audio patches, which are then fed the decoder obtain yrpatch, mixed
Next, the teacher model separately processes Audio spectrogram S1 and Audio 2 spectrogram S2- Note that, discard the tokens from S1 that correspond to the unmixed regions in the partially mixed audio before passing it through the teacher model, ensuring that the teacher processes only the relevant mixed segments.
The target for the student model created by averaging the representations produced by the teacher for S2 and the processed portions of Sj. Finally, we employ MSE the loss function, defined as follows:
NMC Σ ΣΣ
(
/ + (1.b) (i.k)
))
2
LSRL. B x B X M
patch, mixed (b...))
(4)
REM
This method encourages the network to learn multiple concepts at granular level, thereby improv- ing its performance real-world polyphonic scenarios.
3.3 UNIFIED LEARNING FRAMEWORK
In this section, we detail how we efficiently integrate the training objectives discussed in the previous sections to construct the SSLAM framework. Our approach ensures that each objective contributes to the overall learning process without introducing unnecessary computational overhead.
To recap, the methodology consists of five objectives, The global loss for unmixed audio LgloballUnmmived captures high-level representation, while the local loss for unmixed audio Leecaa.Unmixed focuses on fine-grained details. For mixed audio, the global loss Legobbal.mieed helps the model to understand the overall structures in combined sources, and the local loss Llocal.mixod understands overlapping sounds granular level. Finally, the source retention loss LSRL ensures that distinct characteristics of each source are preserved in the mixed audio. For clarity, Figure 2higghighht only the novel contributions of our SSLAM framework.
To train the model, we first preitrain it using only unmixed audio using the Lglobul, Lglobal,Unmixed Unmixed and Uaocal.Unmised loss functions. This step allows the network to learn foundational representations of distinct audio events and establish robust feature extraction capabilities without the added complex- ity of mixed signals. We refer to this as "Stage 1".
After this, in "Stage 2", the model is trained on combination of partially mixed audio (half the batch) and unmixed audio (the other half), utilizing all five losses. This two-stage training approach allows the model to progressively develop the capacity to handle more complex polyphonic audio. The inclusion of all five training objectives in "Stage 2" is motivated as follows: Using unmixed audio based training objectives enhances the robustness of approach monophonic datasets (refer to Appendix D) while enabling the network to learn foundational representations enabling robust feature extraction capability without the added complexity of mixed signals. Incorporat-

---

Published as a conference paper at ICLR 2025
ing mixed audio based training objectives exposes the model to diverse polyphonic data, thereby improving its performance on polyphonic tasks. Additionally, the source retention loss (SRL), fectively computed by leveraging the two mixed and unmixed halves of the batch, explicitly ensures that the representation of mixed audio stays true to its source components. This further enhances the network's s ability understand polyphonic audio.
Finally, these objectives are efficienlly incorporated into the SSLAM framework as follows:
Algorithm Efficient Incorporation of Training Objectives
1: Input: A batch of log-mel spectrograms B
2: Step I: Create a partially mixed batch Bm by rolling and mixing along the batch dimension.
3: Step 2: Concatenate B and Bm to form combined batch 2B.
4: Step 3: Forward 2 through the student and teacher networks, reducing the number multitask
clones from 16 to 8 for consistency with the baseline.
5: Step 4: For SRI., mask and drop unmixed regions in B post-positional embedding and forward
the result to the teacher.
6: Step 5: Compute the five training objectives using the relevant parts of the batches.
This process enables the seamless integration of both unmixed and partially mixed audio, along with their respective training objectives, ensuring SSLAM's adaptability and effectiveness across both monophonic and polyphonic audio datasets.
4 EXPERIMENTS
4.1 DATASETS
For pre-training, utilized the AS-2M dataset without any label information. For downstream evaluation, we employed various audio SSL benchmark datasets, including AS-2M, AS-20K, ESC- 50, KSI, and KS2, well as polyphonic datasets such SPASS, IDMT- and URBAN- SED. More information about these datasets can be found in Appendix
4.2 IMPLEMENTATION DETAILS
Patchification and Positional Encoding. In the SSLAM framework, input spectrograms are di- vided into non-overlapping patches using CNN layer with kernel size of (16,16) and stride of 16, followed by the addition of positional encoding to retain information about the order of the patches.
Encoder. We used ViT-Base model (Dosovitskiy et al. 2020) for both the student and teacher encoders. The teacher model is an Exponential Moving Average (EMA) version of the student model. The teacher's parameters, Oz, are updated according to the EMA
0₂ 70, + 1),
(5)
where A₂ represents the student model's parameters. Initially, the momentum term low to allow greater flexibility during early training. As training progresses, т gradually increases toward 1, ensuring more stable updates and convergence between the student and teacher models.
Decoder. To decode the masked patches, we employ lightweight 6-layer network comprising 2D CNN layers, LayerNorm, and GELU activation.
The total number of parameters used was 93M during pre-trae training and 88M during fine-tuning.
4.3 PRE-TNAINING DETAILS
Pre-Training, For pre-training used only the AS-2M dataset. Input waveforms uniformly resampled to 16kHz and transformed into 128-dimensional mel-frequency bands using a 25ms Han- ning window and 10ms hop size.

---

Published as a conference paper at ICLR 2025
Stage 1: The model was initialized from scratch and trained for 10 epochs using training objectives Egloball Unmixedd and Uumcxe_Uumixed as described in Section 3.3
Stage 2: The model was initialized with the pre-trained weights from Stage and further trained for 5 epochs using all the training objectives outlined in Section 3.3
All pre-training experiments were conducted on 4x Nvidia 3090 GPUs, with each epoch taking 7 hours in Stage and 7.5 hours in Stage 2.
Component-wisgh analysis of SSLAM framework. To better understand the contribution of each component in helping the model in understanding polyphonic audio, we developed four variants of our approach, incrementally building the SSLAM framework during Stage of pre-training. These variants are as follows: M.Maseed latent bootstrapping using unmixed audio (MB-UA): uses only Unmixed and Llocal, 2.Masked latent bootstrapping using partially mixed audio (MB- PMA): uses Lelobal mixed and Liocal.mixed- 3.Masked latent bootstrapping using unmixed and partially mixed audio IMB-UA-PMA. Retention Loss + Masked latent bootstrapping using unmixed and partially mixed audio (SSLAM): This final variant integrates the source retention loss with the other training objectives.
uses Unmobll. U
and Liocalaixedd 4. Source
For fair comparison, we pre-trained all four models on the AS-2M dataset with batch size of 48 for 5 epochs as part of Stage pre-training (refer to the Appendix/AJfor further details). We evaluate how performance varies across various polyphonic soundscape datasets (refer to Table including AS-20K, and across different degrees of polyphony (refer to Table 3
Downstream Task Training. We conducted a comparative analysis of our approach standard au- dio SSL benchmark datasets, alongside prior methods (refer to Table [ To further assess polyphony handling, we evaluated the model different polyphonic datasets discussed before (refer to Ta- ble 2) All downstream tasks, except for AS-2M, were trained using 1x Nvidia 3090 GPU, while AS-2M used 1 Nvidia A100 GPU (refer Appendix A for further details).
4.4 EVALUATION CRITERION
Fine-tuning, We assess the effectiveness of our proposed approach by fine-tuning, where the entire network is trained on downstream tasks with labeled data.
Linear evaluation. We use linear evaluation (also referred to as linear probing) to better assess the quality of the representations learned through self-supervised learning. In this method, a lin- ear classifier is trained on top of the frozen pre-trained representations, preventing the rest of the network from updating. This provides clearer measure of the intrinsic quality of the learned fea- tures, without task-specific adaptation, making it a more reliable evaluation of SSL performance than fine-tuning. Moreover, since pre-trained networks are often used in a frozen state in real-world ap- plications, linear evaluation offers more practical and insightful assessment of the model's general utility across diverse tasks.
5 PERFORMANCE DISCUSSION
Audio SSL benchmark datasets. Our approach demonstrated significant performance improve- ments across most audio SSL benchmark datasets (refer to Table [ with a notable gain of 3.9% relative improvement over the previous state-of-the-art on AS-2M, reaching a mAP of 50.2. Among the purely monophonic datasets, such as ESC-50, KS2, and KS1, our model demonstrated compa- rable performance to the SOTA, which varies across different datasets. This underscores the univer- sality and robustness of our approach, as it performs consistently across a variety of monophonic and general audio datasets.
Polyphonic audio datasets Overall, both Table 2 and Table demonstrate that incorporating par- tially mixed data (MB-PMA) consistently improves performance compared to the baseline MB-UA. These results highlight that our mixing strategy, by exposing the model to mixed audio, increases its access to diverse polyphonic data that AudioSet alone cannot sufficiently provide. The slight perfor- mance decrease seen in MB-UA-PMM data to accommodate the unmixed data. However, this design is crucial for handling both types of data within the framework and for the efficient integration of the SRL., which ultimately enables
is expected, as only half the batch is used for partially mixed
8

---

Published as a conference paper at ICLR 2025
Table I: Evaluation on audio SSL benchmark datasets compared to previous methods. The pre- training datasets include AudioSet (AS), and LibriSpeech (LS). To enhance clarity, methods utilizing additional supervised training on external datasets are grayed out. For AS-2M and AS-20K mean average precision (mAP) is used as the evaluation metric, while classification accuracy is reported all other datasets. For more details about the datasets refer to Appendix B.0.1
Pre-training
Audio
Speech
Model
#Param
Data
AS-2M
AS-20K
ESC-50
KS2
KS1
SS-AST GGog ct al. 2022)
89M
AS+LS
31.0
88.8
98.0 96.0
MAE-AST Baade el al.
2022
86M
AS+LS
30.6
90.0
97.9 95.8
MaskSpec ( Chong
2023
86M
AS
47.1
32.3
89.6
97.7
MSM-MAE Niizum
2022
86M
85.6
87.3
data2vee (Baevski cl al.
2022
94M
AS
34.5
Audio-MAB (Huang et al.
2022a
86M
AS
47.3
37.1
94.1
98.3 96.9
BEATs (Chen
90M
AS
48.0
38.3
95.6
98.3 97.7
BEAT Siteril
90M
AS
48.6
38.9
98.1
98.1 98.1
ASiT (Ahmed 2024
2024
86M
48.0
38.6
95.3
98.9 98.2
A-JEPA (Fei et al
86M
AS
48.6
38.4
96.3
98.5 97.7
EAT (Chen et al.
2024
88M
AS
48.6
40.2
95.9
98.3
SSLAM (Ours)
88M
AS
50.2
40.9
96.2
98.1 98.8
Table 2: Impact of individual novel contributions evaluated across various polyphonic datasets. All performances are reported in mAP. For more details about the datasets refer to Appendix B.0.2)
SPASS
IDMT
URBAN
AS-20K
Model
Square Park Waterfront Street Market DESED
SED
Linear Evaluation
MB-UA
60.1
59.7
55.2
63.7
62.8
75.8
71.3
13.9
MB-PMA (Ours)
63.1
63.5
58.5
66.5
67.4
78.4
70.9
16.1
MB-UA-PMA (Ours)
62.7
63.5
58.2
66.6
66.6
77.7
70.9
15.2
SSLAM (Ours)
64.2
64.2
59.5
67.4
68.5
77.8
71.4
16.9
Fine-tuning
MB-UA
84.4
78.4
80.1
81.4
89.7
94.4
90.9
40.4
MB-PMA (Ours)
85..1
80.0
82.0
82.2
90.8
94.4
90.9
40.6
MB-UA-PMA (Ours)
85.0
79.7
82.0
82.2
90.5
94.4
90.9
40.7
SSLAM (Ours)
85.6
80.5
82.6
82.2
90.2
94.5
90.9
40.9
SSLAM to surpass all other models, achieving performance improvement of up to 9.1 on SPASS (Market). The performance gap particularly evident in the linear evaluation setting, suggesting that contributions have enabled SSLAMM to learn representations that generalize more effectively to polyphonic data compared to other approaches. Upon examining the tables individually, Table shows that the proposed approach consistently outperforms the baseline across all datasets, achiev- ing improvements of up to 9.1 %, with a significant, jump of 21.6% in AS-20K. In Table although performance slightly decreased for lower polyphony level ([2.3)) in the linear evaluation setting, and the fine-tuning improvements were marginal, the performance gap widened as we moved to medium and higher polyphony levels, reaching up to 9.7% ((8,9)).
Additional ablations. As we have already analyzed the relevance of each of the components of the approach previous section, here we discuss additional ablations related to our approach. All the experiments discussed in this section are evaluated on downstream task AS-20K in the fine-tuning regime. The following are our observations. We observed that top k for teacher layer averaging is for global loss and 12 for local loss (refer to Table in regard to the extent of spectrogram mixing, partial mixing was found to be better than full mixing (refer to Table 4 as local loss contributed the foundation of the mask latent boostrapping approach, with unmixed data, mixed data and SRL. Our experiment showed that everywhere except SRL, the global loss showed performance improvement (refer to Table
investigated the effect of global loss

---

Published as a conference paper at ICLR 2025
Table Evaluation on the Degrees of polyphony dataset: Assessing the impact of various individual contributions across different polyphony levels. {a.b) denotes data subset where audio files contain b distinct sound events. All performances are reported mAP. For more details about the datasets refer to Appendix B.0.2
Number of Distinct sound events
Model
Unmixed Data
Partial Mixed Data
[2.3]
{4.5} (6.7) (8.9) (10,11) {12,13) {14+)
Linear Evaluation
MB-UA
61.5
69.4
45.8
53.5
58.3
61.6
66.7
MB BE-PMA (Ours)
x
58.6
70.0
50.7
57.2
61.3
64.8
67.6
MB-UA-PF S0--U-PPMA (Ours)
58.2
70.0
49.8
56.9
61.1
64.7
67.9
SSLAM (Ours)
60.6
53.2
58.7
63.0
69.7
Fine-tuning
MB-UA
x
87.3
86.5
69.5
81.5
82.5
80.7
78.1
MB-PMA (Ours)
X
87.3
86.9
71.4
83.0
83.4
82.0
79.3
AB-UAA-MMA (Ours)
87.2
86.4
70.3
82.7
83.4
81.8
78.8
SSLAM (Ours)
87.7
86.9
71.9
83.3
83.8
82.2
79.4
Table 4: Comparison of partial vs. full mixing of mel-spectrograms via element wise max operation, evaluated across Stage and Stage 2 of our curriculum. All models are exclusively trained with the specified data, e.g. models with tag "mixed audio" were trained solely on batches of mixed audio.
Model
mAP
Finetuning
Stage with unmixed audio
40.2
Stage with mixed audio
39.0
Stage with mixed audio 75% of updates
39.4
Stage with partial mixed audio
39.9
Stage with unmixed audio
40.4
Stage with mixed audio 75% of updates
40.4
Stage with partial mixed audio
40.6
Table 6: Top k layers
Table 5: Effect of global loss
Model
mAP
Local loss
Global loss UMMM
UM
PM|RL
AS-20K
Fine-tuning
Global:12. Local:12
40.5
/
x
X
x
40.4
Global:1, Local:12 40.6
X
x
40.7
Global:3, Local:3
39.3
x
40.9
Global:1, 1, Local:3
39.4
40.6
Global: Local:1
38.8
CONCLUSION
Despite the rapid advancements in audio self-supervised learning (SSL) over recent years, the abil- ity of audio SSL models to effectively handle polyphonic audio remains underexplored. This is significant concern, most real-world sounds inherently polyphonic. In this work, proposed the SSLAM framework, an audio SSL pre- pre-training approach aimed at enhancing the model's ability to handle polyphonic audio, while maintaining strong performance monophonic data. The frame- work enables the model to learn from audio mixtures employing novel training objectives, including the Source Retention Loss. Additionally, have predominantly focused on monophonic datasets, by incorporating diverse polyphonic datasets. Our proposed approach demonstrated state-of-the-art performance both traditional audio SSL benchmark datasets and the newly included polyphonic datasets.
expanded the existing audio SSL benchmarks, which
ACKNOWLEDGMENTS
This research was supported by the EPSRC-BBC Prosperity Partnership 'AI4ME: Future Person- alised Object-Based Media Experiences Delivered Scale Anywhere' (EP/V038087/1). We also
10

---

Published as a conference paper at ICLR 2025
acknowledge the of computational resources the Tier 2 HPC facility JADE2, funded by EP- SRC (EP/T022205/I). Additionally, we thank Srinivasa Nandam for the valuable discussions on self-supervised learning.
ETHICS STATEMENT
We relied on publicly available datasets for all of our experiments. While the proposed approach holds promise for variety of beneficial applications, there is possibility it could be misused for purposes such as surveillance, We are mindful of these risks and will ensure that the distribution of code and models is handled with caution and ethical consideration.
REPRODUCIBILITY STATEMENT
We document all implementation details in Section 4.2, pre-training details in 4.3 and Appendix A Code and pre-trained models available tttt /github..t.....gg com SSLAM
REFERENCES
Jakob Abeßer, Sascha Grollmisch, and Meinard Müller. How robust are audio embeddings for poly- phonic sound event tagging? IEEE/ACM Transactions on Audio, Speech, and Language Process- ing. 31:2658-2667, 2023.
www.ntemiiiio. URL https: //api/app..ppp/// .wt.t//........a.ii...aa..aa iommanticscol. cscholaraa..... Www..org//ooppuuee
259882249
Sara Atito Ali Ahmed, Muhammad Awais, Wenwu Wang, Mark D. Plumbley, and Josef Kittler. Asit: Local-global audio spectrogram vision transformer for event classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3684-3693, 2024. ISSN 2329- 9304. doi: 10.1 10011099911p22222444888880 URL http: dx doi wtg/01..o...oooooooooooooo.. 1t1.///.0//0...1/TALLLL 1109/TASLP. 2024 3428908
Augustin Arnault, Baptiste Hanssens, and Nicolas Riche. Urban sound classification: striving to- wards a fair comparison. arXiv preprint 0rXiv:2010.118000 2020.
Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: Masked autoencoding audio spectro- gram Transformer. arXiv preprint arXiv:2203. 2...:2223..661,, 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for supervupe learning in speech, vision and language. In Proc. ICML, 2022.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Con- ference on Machine Learning, pp. 1416-1429. PMLR, 2023.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins. Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. Proc. ICML, 2022.
Wenxi Chen, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, and Xie Chen. Eat: Self-supervised pre- training with efficient audio transformer. arXiv preprint arXiv:2401.03497, 2024.
Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng. Masked spectrogram prediction for self-supervised audio trainiann In Proc. ICASSP. IEEE, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, et al. An im- age is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929,, 2020.
Zhengcong Fei, Mingyuan Fan, and Junshi Huang. A jepa: Joint- embedding predictive architecture can listen, 2024. URL. htttt::t///aaatt org/ abs 22 15830
11

---

Published as a conference paper at ICLR 2025
Jort F. Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labelle dataset for audio events. In ICASSP, 2017.
Yuan Gong, Yu-An Chung, and James Glass, AST: Audio spectrogram Transformer. arXiv preprint arXiv:2104.01778, 2021.
Yuan Gong, Cheng Lai, Yu-An Chung, and James Glass. SSAST: Self-supervised audio spectrogram Transformer. In Proc. AAAI, 2022.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proc. CVPR, 2022.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Proc. ECCV. Springer, 2016.
Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, al. Masked autoencoders that listen. In Proc. NeurlPS, 2022a.
Po- Yao Huang, Hu Xu, Juncheng Li, et al. Masked autoencoders that listen. arXiv preprint arXhv:2207.06400 2022b.
David S Johnson, Wolfgang Lorenz, Michael Taenzer, Stylianos Mimilakis, Sascha Grollmisch, Jakob Abeßer, and Hanna Lukashevich. Desed-fl and urban-fl: Federated learning datasets for sound event detection. In 202/ 29th European signal processing conference (EUSIPCO), pp- 556-560. IEEE, 2021.
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark Plumbley, PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.
Khaled Koutini, Jan Schlüter, Hamid Eghhal-Zadeh, and Gerhard Widmer. Efficient training of audio Transformers with patchout. arXiv preprint (rXiv:2110.05069, 20211
Etienne Labb, Thomas Pellegrini, Julien Pinquier, et al. Conette: An efficient audio captioning system leveraging multiple datasets with task embedding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Chen Liu, Peike Patrick Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu, Audio- visual segmentation by exploring cooss-modal mutual semantics. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 7590-7598, 2023.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv:1608. 03983, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0511,, 2017.
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for
audio: Seff-supervised learning for general-purpose audio representation, 2021. URL https:
/ //arxxiv hatp://bb/////a///
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoencoders for learning general-purposs audio represen- tation. In HEAR: Holistic Evaluation of Audio Representations, pp. 1-24. PMIR, 2022.
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty. Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent tross-modal reasoning. arXiv preprint arXiv:2311.18799 2023.
12

---

Published as a conference paper at ICLR 2025
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.085
2019.
Sooyoung Park, Arda Senocak, and Joon Son Chung. Can clip help sound source localization? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5711- -- 5720, 2024.
Karol Piezak. ESC: Dataset for environmental sound classification. Proc. ACM MM, 2015.
Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello. Scaper: A library for soundscape synthesis and augmentation. In 2017 IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 344-348, IEEE, 2017.
Dirk Schröder. Physically based real-time auralization of interactive virtual environments, vol- ume 11. Logos Verlag Berlin GmbH, 2011.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ily Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15())11999-1988 2014.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310. 13289, 2023.
Nicolas Turpault, Romain Serizel, Ankit Parag Shah, and Justin Salamon. Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. In Workshop on Detection and Classification of Acoustic Scenes and Events, 2019.
DeLiang Wang. On ideal binary mask the computational goal of auditory scene analysis. In Speech separation by humans and machines, pp. 181-197. Springer, 2005.
Pete Warden. Speech commands: dataset for vinabud-occabularyy speech recognition. arXiv preprint 1800:0/004 2018
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsupervised sound separation using mixture invariant training. Advances in neural information processing systems, 33:3846-388 2020.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.000 02858, 2023.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lazez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint 20X.v:2305. 2023.
13

---

Published as a conference paper at ICLR 2025
A TRAINING HYPER-PARAMETER
Additional parem--asamettrs used in pre-training using AS-2M and fine-tuning of standard audio SSL benchmark datasets are listed in Table and that for evaluation of polyphonic datasets are listed Table g
Table 7: SSLAM pre- pre-training and audio SSL benchmark dataset fine- fine-tuning ayper-parameters.
Pre- Training
Fine- Tuning AS-20K
Hyperparameters
AS-2M
AS-2M
ESC-50
KS1
KS2
Stage
Stage
Optimizer
AdamW Loshchilov & Hutter 2017
Optimizer Momentum
0.9, 32 0.95
Weight Decay
0.05
Learning Rate Schedule
Cosine Loshchilov & Hutter 2016
Peak Learning Rate
0.0005 0.0005 0.00005
0.00005
0.01XK05
0.0101
0.0002
0.0002
Minimum Learning Rate
0.000001
Steps
400K] 200K
400K
40K
8K
60K
60K
Warm-up steps
53K 25K
40K
4K
800
6K
6K
Batch size
12
96
256
256
Clone batch
16/8
N/A
Number of GPUs
1
Dropout Srivastava al. 2014
0.0
0.0
0.0
0.0
0.0
0.0
Drop path HHuang
2016)
0.0
0.1
0.1
0.1
0.1
0.1
Weighted Sampling
False
True
False
False
False
False
Weighted Sampling size
N/A
200K
N/A
N/A
N/A
N/A
Roll Augmentation
False
True
True
True
False
False
Noise Augmentation
False
False
False
False
True
True
SpecAug (Park 2019
N/A
0.2
0.2
0.2
0.1
0.1
Mixup Zhang et al.
2017
0.8
0.8
0.8
Multilabel
N/A
True
True
False
False
False
Loss Function
MSE
BCE
BCE
CR
BCE
BCE
Dataset Mean for Normalization
.42668
-4.268
-4.268
-6,627
-6,846
-6.846
Dataset Std for Normalization
4.569
4.569
4.569
5.359
5,565
5.565
Table 8: SSLAM polyphonic datasets linear evaluation and fine-tuning pyper-parameters.
Linear Evaluation Fine- Tuning
Fine-Tuning
Hyperparameters
SPASS SUBSETS
IDMT
URBAN
Degrees
AS-20K
SED
Polyphony Dataset
Optimizer
AdamW
Loshchilov & Hutter 2017
Optimizer Momentum
0.9,
U.95
Weight Decay
0.05
Learning Rate Schedule
Cosine Loshchilov & Hutter
2016
Peak Learning Rate
0.001
Minimum Learning Rate
0.000001
Epochs
50
50
50
50
94
Warm-up epochs
5
10
Batch size
48
48
48
48
48
GPUs
1
1
1
1
Dropout (Srivastava et al. 2014
0
0
0
0
Drop path (Huang et al. 2016
0.1
0.1
Roll Augmentation
True
True
True
True
True
SpecAug (Park et al. 2019
0.2
0.2
0.2
0.2
0.2
Mixup Zhang et al.
2017
0.8
0.8
0.8
0.8
0.8
Multilabel
True
True
True
True
True
Loss Function
BCE
BCE
BCE
BCE
BCE
Dataset Mean for Normalization
-..75
-5.444
-5.611
-5.216)-5.659
4.268
Dataset Std for Normalization
3.268
3.380
2.699
3.376|2.620
4.569
14

---

Published as a conference paper at ICLR 2025
B DATASET DETAILS
B.0.1 COMMONLY USED AUDIO SSL BENCHMARK DATASETS
AudioSet (Gemmeke et al. 2017) is large-scale dataset containing over million 10-second audio clips sourced from YouTube, annotated with 527 audio event classes. The dataset is divided into three subsets: a balanced set (~200k clips), an unbalanced set (around 2M files), and an evaluation set (roughly 20k files). Since the dataset is dynamically sourced from YouTube,, its availability may decrease over time as videos are removed or taken down. Our downloaded and processed copy of AudioSet includes 1.91M files in unbalanced, 21K in balanced, and 19K in the evaluation set. For pre-training, we use the full dataset by combining the unbalanced and balanced sets, referred to as AudioSet-2M (AS-2M AM) in this paper, while the balanced set alone is referred to AS-20K. No label information used during pre-training
Environmental Sound Classification (ESC-50) (Piczak 2015) is collection of 2000, 5-second environmental sound recordings across 50 classes. Each recording is annotated with a single class. Following previous works Chen et al. (2024 2022]; |H al. (2022), employ 5-fold cross- validation setting and report the classification accuracy as the evaluation metric.
Speech Commands (KS1, KS2) (Warden 2018) are datasets designed for keyword spotting tasks. KS2 consists of 105,829 I-second recordings across 35 distinct speech commands, with the dataset split into 84,843 samples for training, 9,981 samples for validation, and 11,005 samples for testing. KSI, an earlier version of the dataset, contains 12 classes: 10 specific keyword classes, silence class, and unknown class that includes samples from 20 additional speech commands not explicitly covered by the keyword set. In line with previous works, we train models on the training split, select the best-pert best-performing model based on validation, and report test results. The evaluation metric used is classification accuracy.
B.0.2 POLYPHONIC AUDIO DATASETS
SPASS is a high-quality synthetic polyphonic dataset designed for sound event detection (SED) with particular focus on spatiotemporal labeling of sound sources. The dataset was generated using acoustic virtual reality tools such as RAVEN (Schröder) 2011) and monophonic source datasets like ESC-50 and UrbanSound8K. simulates the following five distinct urban soundscapes square, park, waterfront, street, and market, ensuring that the dataset remains general and not confined to specific environment. Each soundscape contains approximately 3,750 samples in the training split and 1,250 samples in the evaluation split.
IDMI-DESED-FL (Johnson et al. 2021) dataset was created using scraper tool with data from DESED Turpault et al. 2019) for sound event detection. The training split contains 10,000 audio files, while the evaluation split includes 2,000 files.
URBAN-SED (Salamon et al. 2017) introduced the Scaper library to create polyphonic datasets, with URBAN-SEA SED demonstrating its potential. As we are using this dataset to evaluate polyphonic capability, only included audio files with more than one label. After applying this filter, the training set consists of 5,268 audio files, and the evaluation set contains 1,739 files.
Degrees of Polyphony We created dataset with varying degrees of polyphony, ranging from 2 to 14+, using audio files from SPASS and URBAN-SED, which we will be referring to as the De- grees of Polyphony dataset in this paper. This allows to evaluate the model's performance across different levels of polyphony.
B. Is THE AUDIOSET DATASET TRULY POLYPHONIC? AN ANALYSIS
AudioSet (Gemmeke et al. 2017) is one of the largest collections of multi-labeled audio files. While the term "multi-label" may suggest the presence of multiple distinct sound events within a single audio file, implying polyphony, this is not always the case. As discussed the introduction, labels such "Camatic music,' 'Music,' 'Musical instrument," and 'Classical music' often refer to the same underlying sound event. Despite these labels appearing together, the audio file may feature only single distinct sound, thereby giving misleading impression of polyphony.
15

---

Published as a conference paper at ICLR 2025
44
42.5
Percentage (%)
40
37.8
38
36.4
35.6
35.6
1.0
15
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Hierarchy Level (L)
Figure 3: Percentage of audio files AudioSet (Gemmeke et al. 2017) with at least 2 distinct sound events (y-axis) based on evaluation criteria, different values of hierarchy level (x-axis).
To assess the extent of this and to analyze the proportion of audio files with truly distinct sound events, we conducted an analysis using the ontology and label information provided with AudioSet.
The steps in the process are detailed below:
Algorithm AudioSet Identify Files with Distinct Sound Events
1: Input: Ontology file, AudioSet label data, hierarchy level
2: Step 1: Parse the ontology file create mappings for id_to_parent and id.to_chi Idren.
3: Step 2: For each audio file:
Retrieve all associated labels.
For each label, find all related labels (parents children) up the hierarchy level L.
Remove these related labels from the label retain only the distinct sound events.
4:
Output: Percentage of audio files with at least distinct sound events.
In our analysis, we found that even when considering only one level of the hierarchy, only 42.5% of audio files contain least two distinct sound events. This number decreases as we explore deeper levels of the hierarchy. The motivation for considering levels greater than one (L, > 1) is illustrated by examples such as "Wild animals" - "Roaring cats (lions, tigers)" + "Roar." In cases where labels like "Roar" and "Wild animals" appear together, they may still refer to the same sound event. Please refer to Figure 3 for the "percentage of audio files with distinct sound events" based on our evaluation criteria described above, at different values of L.
As the hierarchy level increases from to 4 the percentage of audio files containing at least two distinct sound events (labels) progressively decreases: from 42.5% level to 37.8% level 2, 36.4% at level 3, and finally 35.6% at level 4. This substantiates our argument that relying solely on AudioSet insufficient for developing models capable of handling polyphonic audio effectively.
16

---

Published as a conference paper at ICLR 2025
C REPRESENTATION LEARNING VIA CONCEPT SEPARATION WITH MIXTURE INVARIANT Loss
In the development of SSLAM, we investigated whether separating multiple concepts from mixed audio inputs to the student model could enhance polyphonic audio understanding. To achieve this, we employed the Mixture Invariant Training (MixIT) loss (Wisdom et al. 20200, originally proposed for audio source separation tasks. Specifically, extended the student encoder with an additional MLP block to project its output representations into distinct representation vectors.
The general overview of the process is as follows. Given mixed audio input Smixed provided to the student model, its output representation is further projected by an MLP into K distinct representation
vectors, collectively represented as Y mixed ( y mixed- -.. y mixed }. This projection aims to separate the mixed audio representation into its constituent representations or concepts. parallel, the teacher model processes individual spectrograms S1 and S2 corresponding to the constituent audio signals, producing outputs S and 82, respectively (refer to Figure 4).
Mask
Audio 1
Mix
S
MLP
EMA
Audio 2
MixIT
Loss
T
Figure 4: Overview of concept separation using mixture invariant training loss
To enforce concept separation, we use the MixIT loss, which defined as follows:
750 y missod) = min LMIxTT (,8,,750
L (78 AY AYmmood)) )) 1=1
(6)
where we employ Mean Squared Error (MSE) as C. The mixing matrix A 2 is binary matrix constrained such that each column sums to 1. This assigns each of the K representation vectors to either S or Sa, our initial experiments, we observed that this approach yielded worse performance compared to SSLAM on the AS-20K benchmark (39.9 mAP vs. 40.9 mAP).
One key drawback of this approach that mixture invariant training assumes the independence of individual sources, which is a reasonable assumption in the case of audio source separation task. However, in the representation (concept) space, this assumption less valid, as there is high like- lihood that representation overlaps with both individual audio inputs. This overlap undermines the independence constraint. Additionally, the number of possible partitions grows exponentially with K, leading to 2 potential loss terms. This exponential growth introduces significant compu- tational complexity, making the approach less practical for larger K.
Despite these challenges, we believe that concept separation remains promising avenue for poly- phonic audio understanding, warranting further studies to address these limitations and refine the approach.
17

---

Published as a conference paper at ICLR 2025
D REASONING BEHIND INCORPORATING UNMIXED DATA IN MB-UA-PMA VARIANT
In Tables and the MB-UA-PMA variant was developed by splitting the batch in MB-PMA into two halves: one for unmixed audio and the other for mixed audio, rather than using the entire batch for mixed audio. While this modification leads decrease in performance on polyphonic audio datasets (refer to Tables 2 and 3). it ensures that SSLAM achieves robustness across both monophonic and polyphonic audio datasets.
Empirical evidence supporting this design choice is provided in Table 9 where MB-PMA and MB- UA-PMA are evaluated on monophonic audio datasets, such as ESC-50 and KS2, as well as the AS-20K dataset. The improved performance of MB-UA-PMA with respect to MB-PMA in Table and the decreased performance in Tables 2 and highlight the trade-off between achieving higher performance polyphonic datasets and promoting better generalization across diverse audio data.
Additionally, the performance reduction observed for MB-UA-PMA in Tables 2 and 3 is mitigated in our final variant, SSLAM, through the introduction of the Source Retention Loss (SRL), which utilizes the unmixed half of the batch for its computation (refer to Algorithm
Table 9: Comaprision of MB-PMA and MB-UA-PMA monophonic and AS-20K datasets
Model
ESC-50 KS2
AS 20K
Fine-tuning
MB-PMA
95.5
97.9
40.6
MB-UA-PMA
96.2
98.0
40.7
E ADDITIONAL INFORMATION ON AUDIO MIXING
E.0.1 DIFFERENT INPUT AUDIO MIXING STRATEGIES
We explored several audio mixing strategies in the input space with SSLAM, including element- wise maximum, average in log- see-specto,,,am, and average in waveform. In experiments, element- wise max mixing in the log- mel-spectrogram domain performed best (refer to Table I0) Therefore, employed this approach for partially mixing audio in work.
Table 10: Comparison of Different Input Audio Mixing Strategies
Model
mAP
Finetuning
SSLAM with log- melestectrooram element- wise max 40.9
SSLAM with log-mel-spectrogram average
40.8
SSLAM with waveform average
40.4
E.0.2 DIFFERENT AGGREGATION STRATEGIES FEATURE SPACE FOR SRL
Table 11: Comparison of Different Feature Aggregation Strategies for SRL
Model
mAP
Finetuning
SRL with feature averaging
40.9
SRL with element-wist max of features 40.7
To generate target representations for SRL, it is necessary aggregate the features of individual audio clips feature space. We experimented with two aggregation strategies: averaging and dlement-wise maximum between the features of two audio clips. Our experiments showed that averaging provided better performance (refer to Table IT, Therefore, we adopted this approach for feature aggregation in work.
18

---

Published as a conference paper at ICLR 2025
E.0.3 VISUALISATION OF MIXING VIA FLEMENT-WISE MAXIMUM OF LOG-MEL SPECTROGRAMS
Example 1
Example
Figure 5: Visualisation of mixing using element wise maximum of log-mel spectrograms. Each example presents three -ggeees-peectooraam:: the first two are individual audio samples from Au- dioSet, and the third is their fully mixed version, created by applying the element-wise maximum of the two spectrograms.
F ROBUSTNESS ANALYSIS ACROSS MULTIPLE RUNS
To further validate the robustness of our approach, we conducted three independent runs with dif- ferent random seeds for the experiments reported in Table 2 and reported the mean and standard deviation in Table 12 These runs specifically aimed to compare our baseline MB-UA with the proposed SSLAM framework, assessing the consistency and reliability of their performance.
the observed trends remain consistent across multiple runs, confirming the robustness of SSLAM and its improvements over MB-UA
As shown Table I2
Table 12: Robustness analysis of SSLAM across three independent runs, comparing its performance with the baseline MB-UA. All performances reported in mAP. For more details about the datasets refer to Appendix B.0.2
SPASS Waterfront
Model
Square
Park
Street
Market
Linear Evaluation
MB-UA (Baseline)
60.15 0.01 59.72
62.69 0.11 0.08 55.23 0.01 63.50 0.14
SSLAM
64.25 0.07 64.25 0.07 59.51 0.09 67.46 0.09 68.55 0.06
Fine-tuning
MB-UA (Baseline)
84.40 + 0.09 78.38 + 0.29 79.81 + 0.54 81.47 0.17 89.91 + 0.23
SSLAM
85.82 0.17 80.45
0.16 82.31 + 0.26 82.35 + 0.19 90.51 0.44
19


Reviews:
[{"review_id": "TqoXn0FVEu", "reviewer": "Reviewer_rVH4", "summary": "This paper presents SSLAM, an SSL framework for audio that incorporates audio mixtures to better adapt to polyphonic environments. SSLAM leverages a two-stage training process: the first stage uses single-source audio, while the second incorporates partially mixed audio to simulate polyphonic scenarios. The paper introduces Source Retention Loss to preserve the distinct characteristics of each audio source in a mixture, making the model more robust in complex, multi-source audio settings. The authors evaluate SSLAM on a mix of monophonic and polyphonic datasets, reporting strong improvements over prior methods.", "questions": "Since AudioSet is not fully monophonic, what is the intuition behind the mixing technique’s consistent performance improvement in polyphonic tasks? For example, in URBAN-SED, this improvement is not always observed, so further insights into why mixing boosts performance in some cases but not others would be helpful.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper addresses a relevant challenge in audio SSL, especially as polyphonic environments are common in real-world audio. The two-stage training approach combined with the Source Retention Loss appears effective, allowing the model to learn from mixed audio in a way that preserves source integrity. The evaluations include a range of polyphonic datasets, and the method generally performs well across them. The component-wise analysis of the model’s objectives is also informative, showing the impact of each training objective.", "weaknesses": "The paper does not provide an analysis of SSLAM’s generalization to unseen audio domains, such as speech or music distribution. It would be great to also observe if this technique is applicable to other domains’ downstream applications such as multi-speaker recognition or instrument recognition, where audio types could differ significantly from pre-training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R1n8nMfU8U", "reviewer": "Reviewer_oY15", "summary": "This paper introduces **SSLAM (Self-Supervised Learning from Audio Mixtures)**, a novel self-supervised pre-training strategy for enhancing the performance of audio models on polyphonic soundscapes. Recognizing the limitation of current audio SSL methods, which are often benchmarked on predominantly monophonic datasets, SSLAM incorporates audio mixtures into the pre-training process. The approach utilizes a masked latent bootstrapping framework where a student model is trained on mixtures of audio spectrograms, created via an element-wise maximum operation inspired by the Ideal Binary Mask. Concurrently, a teacher model processes the individual audio sources separately, and its aggregated feature representations serve as targets for the student. A novel source retention loss is introduced to further encourage the model to learn and retain distinct features of each source within the mixture. Experiments on standard audio SSL benchmark datasets (AS-2M, AS-20K, ESC-50, KS1, KS2) demonstrate that SSLAM not only improves performance on polyphonic audio (achieving up to a 9.1% improvement on SPASS and setting a new SOTA mAP of 50.2 on AudioSet-2M) but also maintains or exceeds performance on monophonic datasets. Ablations study the contribution of individual components and the impact of mixing strategies and loss functions.", "questions": "The paper uses the term \"self-supervised,\" yet the approach relies on a pre-trained teacher model that presumably required supervision. How many labeled audio clips were used to train the teacher model? Can the authors clarify the extent to which their method truly qualifies as self-supervised, given this dependency on a pre-trained, supervised component?  Would \"teacher-student training\" or a similar term be more accurate?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles the under-explored issue of polyphonic sound processing in self-supervised audio learning. This is crucial because real-world audio scenes rarely consist of isolated sounds, and models trained primarily on monophonic data may struggle to generalize effectively in realistic scenarios.\n\n2. SSLAM introduces a novel training strategy by incorporating audio mixtures and a source retention loss, both well-motivated by principles of auditory scene analysis (specifically, the Ideal Binary Mask concept). The partial mixing strategy further demonstrates a nuanced understanding of the balance between introducing new information and preserving existing audio characteristics.\n\n3. The paper presents convincing empirical results, demonstrating state-of-the-art performance on both polyphonic datasets and standard (largely monophonic) audio SSL benchmarks. The ablation studies and analysis of mixing and aggregation strategies provide further insights into the effectiveness and robustness of the proposed method.", "weaknesses": "1. The core contribution of SSLAM, training with mixtures of mixtures, closely resembles the MixIT [A] approach for unsupervised sound separation. The novelty seemingly lies in applying this concept within a self-supervised representation learning framework. However, the paper does not sufficiently justify why this adaptation is novel or contributes significant new insights beyond the well-established principles of mixture invariant training.\n\n2. A major weakness is the absence of a comparison with a straightforward baseline: pre-processing the mixed datasets with an unsupervised sound separation model like MixIT [A] to obtain separated sources, and then applying the self-distillation method. This would directly address the polyphony challenge and provide a more meaningful assessment of the value added by the proposed synthetic mixing strategy.\n\n3. The reliance on a pre-trained model (here, an EMA teacher) trained on potentially millions of labeled audio clips undermines the claim of a fully self-supervised approach. While the student model is trained without labels, the teacher network embodies prior knowledge acquired through supervision. The paper should clearly acknowledge this dependency and avoid potentially misleading terminology. Please revise the manuscript accordingly to reflect that.\n\n4. Figure 4, intended to visualize the mixing process, is of extremely low quality and uses an unconventional orientation (high frequencies displayed at the bottom). This hinders understanding and should be replaced with a lossless image with a correctly oriented frequency axis (low frequencies at the bottom).\n\n5. Despite suggesting broader applicability, the evaluation primarily focuses on sound event detection. While performance gains are demonstrated on polyphonic SED datasets, the paper does not explore other downstream tasks like speech recognition or music analysis where polyphony is also prevalent. This limits the evidence for the generalizability of the learned representations.\n\n[A] Wisdom, S., Tzinis, E., Erdogan, H., Weiss, R., Wilson, K. and Hershey, J., 2020. Unsupervised sound separation using mixture invariant training. Advances in neural information processing systems, 33, pp.3846-3857.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9NHopMTgCB", "reviewer": "Reviewer_q4mh", "summary": "This paper proposes SSLAM, an approach to integrate audio mixing and self-supervised learning, an approach that stems from the fundamental concept that real-world audio data is, generally, polyphonic. The method claims that training on audio mixtures in masked latent bootstrapping framework improves performance for both monophonic and polyphonic soundscapes.", "questions": "1. Why does SSLAM require both unmixed and mixed versions of the objectives?\n2. Why is Table 1 missing the other variants you have evaluated: MB-UA (which I suppose corresponds to the \"baseline\" you talk about in Section 3.1), MB-PMA, MB-UA-PMA?\n3. In table 2, MB-PMA works the best for IDMT DESED in linear evaluation and on SPASS-Market when fine-tuning, but it's not the one highlighted.\n\n\nPOST REBUTTAL\n--------------------\n\nMost of my questions have been addressed, so I am raising my recommendation to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clarity: paper is well written and apart from some sections, quite easy to read.\n2. Soundness seems sufficient.", "weaknesses": "1. Small set of evaluated tasks: not enough diversity in evaluated downstream tasks. Only speech (keyword spotting) and in-domain audio classification tasks are evaluated. Instead of evaluating KS2 and KS1, either one would've sufficed. Dataset choice is also not motivated well enough.\n2. Evaluation itself: no mean/std or confidence intervals are reported, which would be even more useful for polyphonic evaluations. Are the downstream results reported from a single test run? \n3. The overall objective function in the final SSLAM model is a composite of 5 losses. However, the losses don't seem to be motivated well enough. For instance, the global loss for the mixed version uses only the final layer outputs, which is not the case for your baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YgwHbZiDqb", "reviewer": "Reviewer_dU3n", "summary": "In this paper, the authors propose a new self-supervised learning strategy for polyphonic soundscapes, called SSLAM. Unlike traditional masked latent bootstrapping, SSLAM leverages audio mixtures to enhance the self-supervised learning process at both global and local levels. The method demonstrates performance improvements on standard audio self-supervised learning benchmarks, as well as on polyphonic datasets.", "questions": "1. **Audio Mixing**: How does your method compare to commonly used data augmentation techniques, such as Mixup? In Table 4, the performance improvement of mixed audio on AS-20K appears limited, and the baseline results with unmixed audio in stage 1 are already equal or stronger than those of prior works, as shown in Table 1. Could you clarify the specific advantages of your approach in this context?\n2. **Layer Averaging**: In section 3.1.2, you mention that averaging over all 12 layers, followed by spatial pooling, could result in excessive information compression. This is why you chose to use only the output of the last layer for the global loss and all 12 layers for the local loss. Could you elaborate on how performance is affected when averaging all 12 layers, as is done in the baseline?\n3. **Necessity of Unmixed Objective**: In Tables 2 and 3, the MB-UA-PMA configuration does not show significant improvements over MB-PMA. What would be the impact of using only mixed objectives and SRL? Is it necessary to include the unmixed objectives in this setup?\n4. **Polyphonic Audio Datasets**: The datasets you selected for polyphonic scenario evaluation are primarily used for sound event detection, yet you treat them as multi-label classification tasks. Have you considered evaluating performance on sound event detection tasks with time-stamped data? For instance, how would the model perform on SED tasks if an RNN were added after the current model?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written, with clear explanations and logical flow.\n2. The paper addresses a fundamental challenge in audio classification—handling the polyphonic nature of real-world audio environments—and proposes a solution based on this insight.\n3. SSLAM demonstrates competitive performance on standard audio classification benchmarks.", "weaknesses": "1. **Confusing Figure Presentation**: In Figure 1(C), it appears that there are two teacher models, but the description in the text suggests that there is only one teacher model handling two audio inputs separately, which creates some confusion. Additionally, Figure 2 shows layer averaging after the teacher model, whereas in Section 3.1.2, it is stated that only the last layer or all 12 layers are used for global or local loss, which seems inconsistent.\n2. **Writing Errors**: There are some citation format issues in Tables 1, 7, and 8, as well as in several paragraphs. Additionally, a numerical error was found in Table 1, where the A-JEPA score for ESC-50 is listed as 06.3 instead of the correct value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)