CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing
Authors: Yen-Ju Lu
Jing Liu
Thomas Thebaud
Laureano Moro-Velazquez
Ariya Rastrow
Najim Dehak
Jesus Villalba
Venue: NeurIPS 2024 poster (2024)
Keywords: multilingual speech recognition
speaker verification
self-supervised learning representation
conditional adaptation
Primary area: speech_and_audio
TL;DR: We propose a condition-aware learning method to enhance the performance and generalization ability of self-supervised representations.

Abstract: We introduce Condition-Aware Self-Supervised Learning Representation (CA-SSLR), a generalist conditioning model broadly applicable to various speech-processing tasks. Compared to standard fine-tuning methods that optimize for downstream models, CA-SSLR integrates language and speaker embeddings from earlier layers, making the SSL model aware of the current language and speaker context.
This approach reduces the reliance on the input audio features while preserving the integrity of the base SSLR. CA-SSLR improves the model’s capabilities and demonstrates its generality on unseen tasks with minimal task-specific tuning. Our method employs linear modulation to dynamically adjust internal representations, enabling fine-grained adaptability without significantly altering the original model behavior. Experiments show that CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and excels in under-resourced and unseen tasks. Specifically, CA-SSLR achieves a 10\% relative reduction in LID errors, a 37\% improvement in ASR CER on the ML-SUPERB benchmark, and a 27\% decrease in SV EER on VoxCeleb-1, demonstrating its effectiveness.

CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing
Yen-Ju Lu', Jing Liu, Thomas Thebaud', Laureano Moro- Moro-Velazquezi! Ariya Rastrow, Najim Dehak Jesus Villalbat *Center for Language and Speech Processing, Johns Hopkins University (ylu125, tthebau1, laureano, ndehak3, jvillal7]@jhu. edu
Abstract
We introduce Condition-Aware Self- Self-Supervised Learning Representation (CA- SSLR), a generalist conditioning model broadly applicable to various speech- processing tasks. Compared to standard fine-tuning methods that optimize for downstream models, CA-SSLR integrates language and speaker embeddings from earlier layers, making the SSL model aware of the current language and speaker context. This approach reduces the reliance on the input audio features while preserving the integrity of the base SSLR. CA-SSLR improves the model's capa- bilities and demonstrates its generality on unseen tasks with minimal task-specific tuning. Our method employs linear modulation to dynamically adjust internal representations, enabling fine-grained adaptability without significantly altering the original model behavior. Experiments show that CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and excels in under-resources and unseen tasks. Specifically, CA-SSLR achieves a 10% relative reduction in LID errors, a 37% improvement in ASR CER on the ML-SUPERB benchmark, and a 27% decrease in SV EER on VoxCeleb-1, demonstrating effectiveness.
Introduction
The emergence of Self-Supervised Learning Representations (SSLRs) models has revolutionized speech processing, setting new standards in the field. Pioneering models like Wav2vec 2.0 [aevski al. 2020). HuBERT [Hsu al., 2021], and WavLM [Chen al. 2022a] leverage unlabeled audio data to learn rich representations of spoken language. These models are pivotal in a wide range of applications, including Speech Recognition (ASR) [Chang al.) 202T], Speaker Verification
(SV) [Chen et al. 2022b, Tak et al. 2022 Language Identification (I (LID) |Bartley et al., 2023 and
Speech Translation (ST) Tang et al. [022]. Benchmarks such as SUPERB |Yang et al. 202 and
ML-SUPERB (Shi et al. 2023a] have been crucial in evaluating SSL model performance, providing standardized tasks.
Although SSLR training approaches combine speech from various sources, these models learn representations solely from unpaired audio-only data. When extending SSLR features to multilingual scenarios and low-resource languages, unsupervised training limits the model's ability to distinguish between different languages, resulting in unified features for all languages. Additionally, labeling all SSL training data with language and speaker information requires significant human effort and is impractical. Thus, post- testttranning conditioning approach is more favorable. In other fields, methods like [Zhang et al. 2023) and IP-Adaptor Ye et al. 2023 in image processing, and CTRL. (Keskar al..|12.00 2019 NLP, have successfully integrated conditioning into pretrained models, demonstrating potential applications for speech processing.
In response these challenges, we propose Condition-Aware Self-Supervised Learning Representa- tion (CA-SSLR), a generalist conditioning model applicable to various speech-processee tasks such
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

as language identification, multilingual speech recognition, and speaker verification. Unlike standard adaptation methods that heavily revise the SSLR for downstream tasks, CA-SSLR minimally adjusts the pretrained model by integrating language and speaker embeddings from earlier layers, making the SSLR aware of the current language and speaker context. This technique enables the creation of models that perform multiple tasks with single adapted SSL encoder by strategically injecting conditional adapters into each encoder block while keeping the pretrained encoder weights frozen. CA-SSLA employs a hierarchical self-adaptation structure in which adapters at each layer leverage intermediate task-specific embeddings derived from lower layers. Through attention mechanisms and linear modulation, CA-SSLR dynamically adjusts scaling and biasing, effectively tailoring the model's 's response to language and speaker contexts. Our initialization strategy enables the condi- tioning module to perform identity transformations, preserving the original model behavior when introducing new conditions. By applying channel-wise linear modulation and time-wise scaling without mixing channel dimensions, CA-SSLR avoids introducing new cross-channel couplings, thereby mitigating overfitting and catastrophic forgetting. We demonstrate its versatility and efficiency on three widely studied multilingual speech processing tasks LID, and SV.
The key contribution of this work is a novel approach to condition SSLRs using limited supervised labels, resulting in generalized speech representations that maintain the underlying model's behavior while improving performance with minimal additional parameters. Specifically, CA-SSLR offers:
Hierarchical Dynamic Conditioning: We develop attention- -baeed conditional adapters that are integrated throughout the SSL model. By periodically leveraging language- and speaker-specific information extracted from previous layers, CA-SSLR dynamically tailors the model's behavior at each time step, offering robust adaptation to multilinguul tasks.
Preservation of Pre-trained Weights with Efficient Parameter Utilization: CA-SSLR leverages the pre- pre-trained encoder by introducing lightweight, channel- -wise scala y and bias 3 adjustments. Since these adjustments do not mix information across different channels, the original encoder's s behavior is minimally altered. This strategy ensures stable, parameter- efficient training while preserving the benefits of the pretrained model.
Harmonized Task Compatibility with Notable Performance Gains: Our experiments show that CA-SSLR effectively reduces trainable parameters, mitigates overfitting, and excels in under-resourced and unseen tasks. Notably,
achieves a 27% relative reduction in L.D error rates, a 37% improvement in ASR CER on the MI--SUPERB benchmark, and 27% decrease in SV EER on VoxCeleb-1. These results highlight CA- SSLR's effectiveness enhancing multilingual SSLRs while also minimizing computational overhead for multitask fine-tuning.
2 Related Work
Self-supervised learning representation. Self-Supervised Learning (SSL) models, such as Wav2Vec 2.0 [Baevski et al. 2020). HuBERT [Hsu et al., 2021), and WavLM [Chen et al. 2022a]]. leverage large-scale of unlabeled audio learn expressive speech representations. These representa- tions capture acoustic, phonetic, and semantic features, which can then be subsequently fine-tuned on smaller labeled datasets for tasks such as speech recognition [Yi et al., 20200 Zhho and Zhang 2022]. emotion recognition (Pepino et al. 2021]. vocal intensity classification [odali et al. 2023]. and speaker change detection [Zajíc and Kunešová, 2023). In multilingual settings, Wav2Vec 2.0- XLSR [Babu et al. (02T] extends pre-training to diverse languages for cross-lingual transfer, while mHuBERT [Lee et |l.[ (02T] expands HuBERT to effectively handle multiple languages. These multilingual extensions facilitate a broad range of tasks, including speech recognition [Chen et al.) 2023a], spoken language understanding [Hu et al. 2024], and speech generation tasks [Li et al. 2024], across various languages. Meanwhile, the scope of evaluation has broadened from monolingual benchmarks like SUPERB (Yang et al., (02TT to multilingual ones such as ML-SUPERB (Shi et al. 2023a reflecting growing demand for robust and scalable SSL representations.
Adaptation Methods. In many studies (Yang et al. 2021 Shi et al. 2023a Chen et al., 2023a], the SSL representations (SSLR) remains frozen while a decoder is trained for a specific task. Because the same encoder is shared across tasks, this approach allows multiple tasks to be evaluated on a single speech signal with only one encoder run. However, such systems often underperform compared to approaches that adapt the SSLR to the target task fine-tuning the entire SSL encoder Chen

---

Condiooned--ee aeliionne-aaat Awware
Speech Reprasentation
Gureralill
Enooder
Bias
Time-wisse
Altention
Xax
Projection
Condition Features
Scale
76
identificatin
Language
Condiioner
Copdiioner
Speakar Verification Decoder
Projection
Decoder
1x Nxx&cC
TxNx1
Channel-wise modulation
Encoded Speech
SSL Inyers
Attention Pathway
Tx Nxs
Modulation Pallway
(a) CA- CA-SSLR improves SSL features by integrat- ing intermediate LIDISV conditions, keeping pre- trained parameters frozen.
(b) The trainable time-chie -enenchaneel attention conditioner for integrating language and speaker prediction in CA-
It predicts bias / and scale using condition feature z.
CA-SSLR scheme and its timenchannel attention conditioner. Only the conditioner and linear projections for the decoders are trainable, and all other parameters are frozen during adaptation.
Figure 1:
ct al. 2022a]. fine-tuning only selected layers, or adding lightweight adapters [Chen et al. 2023b Unfortunately, these methods require separate encoder for each task, leading to significant increase in computational load that scales linearly with the number of tasks.
Conditioning Pre-trained Models. Image processing has successfully integrated conditioning into pretrained models using methods such as ControlNet (Zhang et al. 2023) and IP-Adaptor (Ye et al. 2023]. ControlNet enables precise control over generated images by incorporating additional inputs (e.g. edge maps or sketches), while IP-Adaptor utilizes lightweight adapter modules to modulate the model's behavior based specific conditions-witteeee parameters. These techniques have achieved significant success and offer insights for potential applications in speech processing. Similarly, in Natural Language Processing (NLP), the Conditional Transformer Language Model (CTRL) Keskar et al.) 2019] introduces conditioning via control codes, guiding text generation based on attributes like style domain and enabling efficient adaptation without extensive retraining. These successes image processing and NI.P demonstrate the potential for conditioning pre-trained SSLRs in speech applications.
modifying the pre- pre-trained model's
Hierarchical Conditioning. Hierarchical modeling has also been explored in previous research. For instance, [Sanabria and Metze 2018) propose multi-task ASR model that applies intermediate representations by performing connectionist temporal classification (CTC) at multiple layers, each targeting different granularities. Lower layers predict character tokens, whereas higher layers predict subword units with increasingly larger vocabularies- final layer. [Chen al. 2023a] extend this idea by incorporating hicrarchical conditional layers into the ASR decoder, using tokens predicted by earlier layers to guide the predictions of subsequent layers.
from 300 to 10k subword units in the
3 Methodology
We introduce Condition-Aware SSLR (CA-SSLR), universal encoder designed for multiple down- stream speech tasks. CA-SSLR enriches pre-trained SSL model by integrating intermediate LID and SV predictions to condition and adapt subsequent layers dynamically. This strategy allows the model capture key language and speaker attributes, progressively refining outputs and excelling in multilingual, multispeaker scenarios. Figure Ta illustrates the CA-SSLR architecture. It comprises a frozen SSL encoder, which is augmented with trainable conditioners and lightweight task apee-sicc decoders. The conditioner modulates the encoder's 's hidden representations based on conditioning features drawn from intermediate LID and SV embeddings. This hierarchical conditioning mech- anism enables dynamic adaptation to variety of input conditions while keeping the pre- pre-trained
3

---

Encoder Layer
Layer Group
Layer-Norm
FC
Time-Channel Allention Conditooners
Linear wi LayerNorm
7,8
Waighted Sum
Linear Projection
Shared LIDISV Decoder
LIDIVV Emboddings
Generalist
Enpoder
Attention
Representation Features
Layer-Norm
1
(a) Hierarchical conditioning with TCAC's to generate feature z and modulate layers (b) SSLR layer with with scale and bias 3.
conditioning integration,
transforming into S.
Figure 2: Architecture of the CA-SSLR model employing hierarchical self-conditioning with Time- Channel Attention Conditioners (TCACs).
parameters fixed. The subsections below describe (1) the conditioner module, (2) how it integrates into the overall architecture, and (3) the incremental training strategy for incorporating conditioning information without catastrophic forgetting.
3.1 Channel-wise and Time-wise Attention Conditioner
key component of CA-SSLR is the channel-wise conditioner (CC) or the time-channel attention conditioner (TCAC), which modulates the SSL encoder's hidden representations based on the conditioning features. As shown in Fig. 1b the TCAC receives latent representations 8()) E CxT from layer 1 of the SSL encoder and conditioning feature vector RR derived from intermediate LID or SV embeddings, The TCAC then produce modulated representations (()) by applying time- and channel-dependent scaling and bias:
t,e 5(0 = TCAC(S t.e: 7.0 (2, S + 300 (z,
(1)
where and index time and channel dimensions, respectively. The scales Ti. and biases 3(1) are products of time- and channel-dependeer components:
( Te. (z.S()) a. (1) a (z. (2. (z,S
(z)
(2 (()) at ( (z, (() x 300 (z)
(2)
The channel- channel-dependent scales (1)) C and biases ((l) RC are computed by linear transforma- tions of the conditioning feature, akin to feature-wise linear modulation [Perez et al. 2018)
(2) (z) wh z + bu) (0) w + bg
with
while the time-dependent scales (!) Rgd obtained via an additive attention mechanism:
S (1)
(2, - vT (WW(()))
+ b
(4)
where f(-) denotes ReLU nonlinearity, (! (C+())))) 6 RC and v. CR The conditioning feature z is produced by processing intermediate embeddings E R from the LID or SV decoders, as linear parameters. If sime-based modulation is not required, the simpler channel-wiss conditioner (CC) can be used by retaining only the channel-dependent components 7 and B. This flexibility allows the model to be adapted for tasks with varying complexity requirements. By integrating these conditioning methods, CA-SSLR dynamically adapts its internal representations according to language and speaker features, without altering the frozen pre- trained encoder's 's parameters.
LayerNorm( We b), where W E RR Rx and E R are shared

---

3.2 Hierarchical Self-Conditioning in CA-SSLR
CA-SSLR employs a hierarchical self-condi mechanism within the SSL encoder layers. building on the TCAC module. In Figure 2 the encoder is segmented into layer groups, with TCACs inserted after each attention module to modulate hidden representations using updated conditioning features. The model aggregates SSI. features through a weighted sum of outputs from all preceding layer groups, and this aggregate passed to the L.ID and SV decoders. The decoders extract and transform LID and SV embeddings via linear layer followed by layer normalization, forming the conditioning feature z used by the TCACs.
The conditioning feature z is re-estimated at intervals every three layers for LID and every six layers for SV -uiig aggregated SSL features from earlier groups. This hierarchical design progressively refines the model's 's representations, adapting to the input's 's language and speaker characteristics different depths of the network. For instance, the initial SSL layer group captures fundamental linguistic and speaker cues, producing embeddings that subsequently condition the next layer group via the TCAC modules. This ongoing refinement allows the model to adapt dynamically based on intermediate predictions, leading to context-aware, flexible representations.
Each layer group has distinct TCAC parameters, enabling fine-grained scaling and bias adjustments at different stages of the model. Notably, only the TCAC modules and the linear projections for the decoders are trainable, while all other SSL encoder parameters remain frozen throughout the conditioning process, This design mitigates overfitting risks and accelerates training by confining the number of learnable parameters. As a result, the hierarchical celf-conditioning mechanism equips the model to capture diverse aspects of input audio, making CA-SSLR a robust framework for comprehensive speech analysis.
3.3 Incremental Training Strategy
Introducing new components into pre-trained SSL encoder carries the risk of catastrophic forgetting. To address this, we use an incremental training strategy that gradually incorporates conditioning information. We initialize the TCAC parameters so that the initially modulated features match the original SSL outputs exactly. Specifically, we set a₃ = 1 for all t, % 1, and B = 0 for all c. According to Eq. (I) this ensures that S:d S at the outset, allowing seamless transition from the pre-trained model to the conditioned model. For multiple conditioning features (e.g. LID and SV), each task's parameters (OLID: "LD: (LID) and (asv, Ysv, (sv) are initialized in the same manner (i.e., a = 1, = 1, and B = 0). We then combine them as follows:
Oktoaal = OLID X asv,
Thotal = LLI X 7SV, Brotal = BLD + Asv
(5)
By initializing each set of parameters identity and then combining them, newly added conditioning tasks do not alter the features contributed by already integrated tasks. This allows the model to steadily integrate extra conditions without disrupting the knowledge acquired from previous tasks.
4
Experimental Setup
4.1 Datasets
We use the ML-SUPERE benchmark (Shi et al. 2023a for both LID and ASR experiments. ML- SUPERB provides two data configurations: (1) 10- O0-minutellanguage and (ii) 1-hour/languag, covering 123 well-represented languages (Normal). Both configurations also include five training utterances for each of 20 low-resource languages (Few-shot) [
For the Few-shol languages, we introduce an Extended Few-shol condition, which increases their data to match the Normal languages but provides only language labels without ASR transcripts. Our main goals are address the severe LID constraints imposed by just five training utterances and to leverage the comparative case of gathering LID-only data over fully transcribed data. For simplicity, we continue refer these extended data in subsequent LID experiments as Few-shot. As ML-SUPERB lacks speaker labels, we incorporate VoxCeleb2 (Ngranni et 2017 to train the models on the SV task. VoxCeleb2 contains 1,092 hours
'We found that some Lithuanian (lit) training and testing data are incorrectly replaced with Italian (it), so
L Lithuanian is excluded from the evaluation.
5

---

of speech from 5,994 speakers but lacks LID labels and ASR transcripts. SV performance is tested on the VoxCelebl original set. We augment the speech with Musan noise ( et al. [2015] and reverberation Ko et al. 2017] during SV training.
4.2 Model Architecture
SSLR Models. Our system leverages top multilingual SSL backbones from the ML-SUPERB
benchmark: (i) Wav2Vec2-XLSR (300M parameters) [Babu et al. 2021]. trained on 128 languages, and (ii) mHuBERT (100M parameter) (Lee al. 202T]. trained on English, Spanish, and French VoxPopuli [Wang et al. 2021] data. Both have demonstrated their efficacy in processing wide range of linguistic imputs and form the backbone of our system. We conduct experiments using S3PRL Yang ct al. 202T and ESPnet [Watamabe et al. 2018). Our training set comprises data with (1) ASR+LID labels, (2) LID only labels, or (3) SV only labels; we computed the task- spe-iicc loss only when corresponding labels are available. Detailed information on the remaining hyperparameters provided in the appendix Training a single model requires about one day on A100 GPUs.
Speaker and Language Decoders. The speaker and language decoders are based the BCAPA- TDNN architecture [Desplanques et al. 2020]. First, a convolutional layer projects the SSL repre- sentation to the decoder dimension (512 for LID, 1024 for SV). This is followed by one (for LID) or three (for SV) I-dimensional SE-Res Ret [Gao et al., 2021] layers. Next, thannel-wise attentive statistic pooling aggregates the frame-level features into an utterance-level vector, which is projected into lower-dimensional speaker embedding. We employ additive angular margin-softmax [Deng et al. 2019) with margin=0.3 for SV and margin=0.0 for LID. Large margin helps create highly compact speaker representations (Villalba et al.) 2022). while being detrimental in LID [Villalba al. 2023). Both the SV and LID decoders use a weighted average of all SSL layers for final predictions. In CA-SSLR, the intermediate embeddings that generate conditioning embeddings are similarly drawn from a weighted average of the SSL layers up that point. Because all SV and LID decoders share parameters, the total number of trainable parameters does not increase with more frequent recomputation of conditioning embeddings.
ASR decoder. The ASR decoder conforms to the framework set by the ML-SUPERB benchmark (Shi et al. 2023a]], facilitating comparable evaluations. A convolutional downsampling layer halves the SSL feature sequence duration. The resulting features are feed into a two-layer Transformer with 256-dims self-attention, eight heads, and 1024-dims feed-forward layer. A linear output layer applies connectionist temporal classification (CTC) to predict multilingual character-level tokens.
5
Experiments and Results
5.1 Generalization Ability on Unseen Tasks
Experiment Setting. We conducted experiments to evaluate the generalization capabilities of the adapted SSLR models on LID, ASR, and SV tasks. The SSLR models are adapted for one task (either LID or ASR) and then evaluated on both the adapted task and an unseen task. For LID adaptation, the SSLR is trained exclusively with LID labels. We compared three setups: full fine-tuning (LID-FT), Houlsby adaptors Hou et al. 2019 (LID-Houlsby), and our proposed condition- approach In this setup, we employed an additional LID decoder using the pre-trained SSLR to pre-generate language embeddings, which are then used to condition the SSLR model for a second inference pass. For ASR adaptation, the models are trained with ASR loss using three setups: full fine-tuning (ASR-FT), Houlsby adaptors (ASR- conditioning method with TCAC layers integrated into the SSLR model with single inference (ASR- CA-XI CA-XLSR-). During ASR adaptation, the LID decoder is integrated into the SSLR model to provide conditioning features, but SV information is not included during training.
and our proposed hierarchical
Results. In LID adaptation (Table Tap, both LID-FT and LID-Houlsby improved LID performance compared to the pre-trained SSL baseline. However, on the unseen ASR task, the fully fine-tuned SSLR encoder improved ASR CER by only 2%, while LID-Houlsby showed limited generalization, with CER improvements of 5.4% and 3.9% for normal and few-shot languages, respectively. Our
"Code, pre-trained models, and reproduction instructions for the experiments are available at https:// github. Stthub.com//nillluu///--ss

---

Table Evaluation adapted XLSR models on the 10-min ML-SUPERB and VoxCeleb dataset for LID, ASR, and SV tasks. These evaluations test the encoder's generalizability across different tasks, demonstrating effectiveness without further task-specific tuning.
(a) LID LID-adapted XLSR models evaluated on LID and ASR tasks.
LID Adapted.
Bottleneck Dims.
LID Acc
ASR CER +
Normal
Few-shots
Normal
Few-shots
XLSR
89.1
83.9
29.0
39.0
LID-FT
90.1
84.7
27.0
37.0
LID-Houlsby
256
90.1
85.3
23.6
35.1
LID-CA-XLSRI. danl (ours)
256
90.2
85.8
21.7
33.4
(b) ASR-adapted XLSR models evaluated on ASR and SV tasks.
ASR
Bottleneck Dims.
ASR CER
SV
Adapted.
Normal
Few-shots
EER \
DCF 4
XLSR
29.0
39.0
1.29
0.093
+ ASR-FT
17.1
32.2
1.29
0.095
+ ASR-Houlshe
256
20.3
34.6
1.37
0.097
+ ASR-CA-XLSR- (ours)
256
18.6
31.6
1.15
0.088
LID-CA-XLSRA dual method achieved substantially better generalization, improving ASR CER by 7.3% and 6.6% for normal and few-shot languages. In ASR adaptation (Table (b), all models enhanced ASR performance, but ASR-Houlsby and full fine-tuning degraded SV performance relative to the baseline, highlighting their limited generalization. Our ASR-CA-XI CA--SR---RRR- approach not only preserved but improved SV performance, reducing EER by relative 10.9% and DCF by 5.4%, showcasing strong generalization to the unseen SV task. These results demonstrate that CA-SSLR notably outperforms full fine-tuning and standard adaptation methods in terms of generalization. By effectively leveraging conditioning information, CA-SSLR adapts across tasks while maintaining performance unseen ones. Our proposed conditioner offers both robust adaptations on training tasks and superior generalization, making CA-SSLR a versatile and effective solution for multilingual and multispeaker speech processing.
5.2 Condition-Aware SSLR Model
Experiment Setting. Table 2 investigates the CA-SSLR approach with hierarchical language conditioning. The first block of the table refers to the baseline where the foundational models are frozen, while the second block (CA-XLSR dun)) utilizes a separate task-specific LID model to pre-generate the language embedding. The third block presents proposed approach, where re-estimate the language embedding every fourth third layer (CA-XLSR" (4L, 3L)) within the XLSR model, not required a separate LID system. The experiments utilized two types of conditioners: TCAC, which incorporates attention, and variant without cnceetioonrefereee conditioners (CC)-M) where the same scale and bias are applied uniformly across all time frames. The real-time factors (RTF) as proc-time/signal-lene
channel-wise
are provided for assessing efficiency
Results. First, we observed that both CA-XLSR diml and CA-XLSR" systems with TCAC (with attention) generally performed better than the CC (w/o attention) counterparts, reaffirming the benefits of the time-wise attention design. the second block. CA- CA-XLSR slightly outperformed CA-XLSR in terms of CER for both the 10-minute and 1-hour datasets. However, its real-time factor (RTF) is akin to the combined RTFs of separate LID and ASR models since it runs Wav2Vec2 Twice-once for language embedding extraction and again for ASR caddiiiningg-psse
challenges for streaming applications. On the other hand, CA-XLSR (CCC 3L) excelled among the three approaches, achieving 35.9% and 19.0% relative improvement in Normal and few-shot languages,
'The RTFss computed NVIDIA T4 GPU.

---

Table 2: ASR (ER(%) and LID Acc (%) in ML-SUPERB 10min. and 1h. sets, comparing different layers to generate the language embedding to condition the following layers. We adapt the XLSR model for LID and ASR tasks.
10mins
1hr
REL. RTF,
SSL Model
RTF,
LID (ACC
ASR (CER 4)
LID (ACC f)
ASR (CER ()
Normal
Normal
Few-shots
Normal
Normal
Few-shots
XLSR (hi et al. 2023a
0.021
1.00
66.9
29.2
40.9
87.9
22.0
39.3
MMS-1o [Shi el 2023b
84.8
21.3
30.2
86.1
30.8
XI.RR (Ours)
0.021
1.00
89.0
29.0
39.0
90.9
22.7
36.9
CA-XLSR (CC))
0.037
1.75
89.0
32.2
90.9
31.5
CA-XLSR (TCAC)
0.037
1.75
89.0
17.8
31.8
90.9
13.5
31.4
(C- 4L)
0.024
1.17
31.7
89.6
32.2
CA-XLSR*
0.027
1.27
88.6
31.5
90.0
16.0
32.4
CA-XLSR'
0.027
1.27
88.6
18.6
31.6
93.4
15.1
29.6
Table 3: Experiments on LID and LID + SV Hierarchical Conditioning. We adapt the XLSR and mHuBERT models for LID and ASR tasks using CA-SSLR', and for SV tasks using CA-SSLR Results for Normal languages with 10-min and I-hour datasets alongside VoxCeleb SV results.
10min. MI-SUPERB
VoxCeleb
1h.
MI.-SUPERB VoxCeleb
REL. RTF,
SSL Model
RTF
LID
ASR
SV
LID
ASR
SV
ACC1
CER
EER
DCF
ACC1
CERI
EER
DCF
mHHuBRRT
0.015
1.00
81.9
38.2
2.19
0.145
86.2
30.9
2.19
0.145
0.015
1.00
73.0
36.5
5.85
0.350
87.7
32.3
4.01
0.251
(C)
0.017
1.13
82.0
31.9
1.77
0.120
86.1
25.1
1.77
0.118
CA-MHUBERT (CC))
0.018
1.16
82.2
31.7
1.79
0.117
87.3
24.8
1.78
0.121
XLSR +FT
0.024
1.00
89.0
29.0
1.29
0.093
90.9
22.7
1.29
0.093
0.024
1.00
81.5
35.6
7.23
0.353
83.2
28.7
6.72
0.330
(C-
0.029
1.23
88.6
19.4
0.076
90.0
16.0
1.02
0.078
CA-XLSR'(ICAC)
0.029
1.23
88.6
18.6
1.15
0.088
93.4
15.1
1.06
0.077
(.XXSRR-S(CO)
0.032
1.34
89.1
18.8
1.04
0.075
88.1
15.0
0.94
0.073
" (TCAC)
0.032
1.34
18.3
0.086
93.5
1.01
0.077
respectively, compared the baseline in the 10-minute setup, and 33.5% and 19.8% in the 1-hour setup. LID accuracy remained comparable among the various CA-XLSR models, with a notable performance improvement from 90.9% to 93.4% in 1-hour setup.
5.3 Generalist Condition-Aware SSLR Model
Experiment Setting. Table presents results for general CA-SSLR models that combine multi- lingual ASR, LID, and SV tasks. The table compares the baselines, with frozen and fine-tuned SSL models, two differentt CA-SSLR Hicrarchical models (CA-SSLR" and CA-SSLR 4.S). We further include another kell- multilingual SSLR model, mHuBERT, for a comprehensive comparison. The LID conditioning systems (CA-SSLR*) are the same as from the previous section, conditioning the SSL model only on LID embeddings, with the SV decoder added on top of SSL features without 4,5, combines both LID and SV
further adaptation. The LID SV conditioning system (CA-SSLR 4.55 embeddings and is jointly trained on ASR, SV, and LID losses. The intermediate LID embeddings recomputed every three layers as the best configuration in Table 2 and SV embeddings recomputed every six SSL layers. Apart from ASR CER and LID Ace on ML-SUPERB, we present SV equal error rates (EER) and detection cost function (DCF), measured at target prior probability p = 0.05 [Sajjadi et al.) 2022], on VoxCelebI. SV performance varied depending on whether we trained the model combining 10min ML-SUPERB + VoxCeleb2 1h ML-SUPERB VoxCeleb2.
Fine-tuning Baseline. In the fully fine-tuning experiment, we initialized the model with pretrained ASR, LID, and SV decoders and fine-tuned for a few epochs. However, this approach resulted in suboptimal performance compared to the frozen SSLR baseline. The "FT"" experiments showed degraded performance, with LID accuracy decreasing by 5.7%, ASR CER increasing by 3. 3.1% and SV EER worsening by 4.2 in absolute values on average across the four settings. This decline unexpected, as fine-tuning typically improves performance. This suggests that simultancous

---

Table 4: Ablation study of condition-aware settings for ASR-adapted XLSR models on 10-min ML-SUPERB dataset, using CC or TCAC. Conditioning is based on predicted language labels LID embeddings, cept in the ground truth (G. T.) experiment.
XLSR Normal
Fowshots
LORA Normal
Character Error (96
LoRA Fewshots
Fewshats
Houlsby Nermal
Heulsby Fawshets
CCATAAC (ours]
CC-CAC (ours) Feushots
ASR
Normal CER 4
Few-shots CER +
Adapted.
XLSR
29.0
39.0
+.TT. CC
17.2
27.9
Hard CC
19.8
28.6
Trainable Parameterr
Soft CC
19.3
32.5
CER versus trainable parameters on XLSR model for Normal and Few-shots languages, demon- strating the adaptation ability for CC and TCAC.
Figure 3:
Embed CC
18.6
32.2
Embed TCAC
17.8
31.8
adaptation of the SSL layers multiple tasks causes conflicting adjustments, reducing the model's robustness. Consequently, catastrophic forgetting led to worse performance compared to the baseline. Conversely, the condition-aware SSLR models exhibited superior performance comparing with the frozen baseline. indicating that training the inserted condition layers does not alter the model's behavior for downstream tasks but improves its ability represent the input speech data.
Language Conditioned SSLR. CA-SSLR"(CE (CC) notably enhanced SV performance w.r.t. the baseline, despite its encoder being solely tuned for ASR and LID tasks. For XLSR, the EER improved by 14% and 20% relative for the 10-min. and 1-h. sets, respectively, while DCF improved by 16-18%. Similarly, for mHuBERT, we observed comparable enhancements, with the EER improving by 17% both sets and the DCF improving by 17-18%. This demonstrates that the CA-SSLR approach offers superior generalization capabilities compared to the original pre-trained SSL encoder, delivering improved performance. CA-SSLR' (TCAC) performance in SV is comparable to its non-attention counterpart with better performance in LID and ASR as discussed Sec 5.2
Language and Speaker Conditioned SSLR. Adding a speaker conditioner CA-SSLR fur- ther improved its performance. In ASR tasks, incorporating the speaker conditioner CA-XLSR reduced CER by 3.1% for the 10-min. set and 6.2% for the 1-hr set, relative to CA-XLSR For LID task, CA-SSLR L,S shows similar performance to other models with relative differences below 3%. For SV, CA-XLSR r., using channel-wise conditioner (CC) reduced EER by 19.4-27.1 14,, out- performing CA-XLSR2 Switching from CC to TCAC yielded additional gains in ASR, adding relative improvement of 2.7-4.0% In contrast, its impact on SV is more modest, with improvements in EER by 14.0-21.7% Overall, TCAC demonstrated better adaptation ability, while CC excelled in generalization.
ASR and RTF Discussion. Generally, we observed the largest improvement for ASR when in- cluding the language conditioner, as it enables the system to adapt to produce output tokens in the correct language. Conversely, adapting the model to the input speaker provided fewer ASR gains. The XLSR model benefitted from our approach better than mHuBERT, possibly because mHuBERT is 3x smaller than XLSR, but more importantly, because mHuBERT is trained on just four languages compared to 128 in XLSR. Therefore, the pre-trained mHuBERT has not encountered enough diversity in terms of languages and speakers, thereby limiting its performance in multilingual ASR and SV. In terms RTF, while the conditioned models are 13-34% slower compared to sharing the pre-trained SSL encoder for the three tasks, both CA-SSLR" and CA-SSLR offer superior performance while being much faster than running task-specific models separately, indicating more efficient use of computational resources while running the generalist model.
5.4 Analysis of the TCA Conditioner
Ablation study of Conditioning Approach. Table 4 conducts ablation study for different A conditioning methods with CA-XLSRR settings within the MI-SUPERB 10min dataset regarding ASR CER. First, we used conditioners without attention (CC) the ground truth LID predictions

---

(G.T.,, serving as the upper bound for the performance of our proposed approach. This improved the Normal languages from 29.0% to 17.2%. and Few-shor languages from 39.0% to 27. 27.9%, w.r.t. the pre-trained XI.SR model. This showcases the potential of the condition-awant SSLR. Following, we compared conditioning on hard-predicted language labels (Hard), soft-predicted language labels (Soft), and language embeddings from the LII) decoder bottleneck layer (Embed) for comparison. Conditioning on hard LID labels improved the most in Few-shot languages, improving by 26% relative to the baseline. On the other hand, the Embed case outperformed the Soft case and provided balanced performance for both Normals and Few-shots languages. Additionally, we compared CC to TCAC. The TCAC provided the best overall results, improving Normals and Few-shots by 38.6% and 18.5%, respectively, w.r.t. baseline.
Parameter Efficiency in CER Reduction. Figure3 compares CER versus the number of trainable parameters for different adaptation methods, including our proposed Channel-wise Conditioner and Time-Channel Attention Conditioner (CC-TCAC), the Houlsby adapter, LoRA (Hu et al., 2021], full fine-tuning (FT), and the baseline XLSR model. The Houlsby adapters, with hidden dimensions of 256 and 512, have 18.4M and 30.9M trainable parameters. In comparison, the CC-TCAC approach, conditioned on precomputed LID embeddings with 256 dimensions (18.7M for CC and 22.6M for TCAC), achieves lower CERs with similar parameter counts. LoRA provided only marginal gains over the baseline, aligning with findings from Chen et al. [2023b] In contrast, FT required fine-tuning 16-24 layers (200-300M parameters) to achieve comparable CER reductions, making CC-TCAC about ten times more efficient. As discussed in Sec CC-TCAC's key contribution is its superior generalization ability. While the Houlsby adapter enhances task-specific adaptation, it falls short in generalizing to unseen tasks. contrast, CC-TCAC achieves both effective adaptation and robust generalization, making it a versatile solution for diverse applications.
Conclusion
This paper introduces the CA-SSLR framework, innovative approach that integrates conditioning into pre-trained Self-Supervised Learning (SSL) models by adapting only the trainable conditioner. Through hierarchical conficonnii mechanism, where intermediate language and speaker features condition the upper layers of the SSL model, CA-SSLR achieves a 33% reduction in ASR CER compared the pre-trained baseline, matching the performance of single-task fully fine- tuned models. Additionally, it improves Speaker Verification EER by 27% and reduces Language Identification error rates by relative 10% in average. The results indicate that condition-aware SSLR models enhance the model's interpretation of input speech data, providing superior performance compared to traditional fine-tuning methods. This improvement is achieved by dynamically tailoring the model's 's response to the input language and speaker characteristics, ensuring robust generalization across various tasks. In summary, CA-SSLR offers a versatile and efficient approach to integrating conditioning information into pre-trained models. This method not only enhances performance across multiple tasks but also ensures efficient parameter utilization, supported by an improved RTF that facilitates its application in real-world scenarios.
Broader Impact and Limitations The CA-SSLR methodology improves the conditioning of pre-trained SSL models for speech processing, improving performance with minimal fine-tuning and reducing computational resource requirements. This advancement facilitates the deployment of robust models in resource-constrained environments, promoting broader access to advanced speech technology. Nevertheless, the methodology carries potential risks, as the conditioning mechanisms might amplify biases in the training data, leading to unfair outcomes, particularly for underrepresented languages and speaker groups. Ensuring diverse and balanced datasets, along with continuous monitoring, is crucial to mitigate biases and avoid reinforcing existing inequities.
References
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. XIs-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.099 2021.
10

---

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 53:122443 1t:11244-12466 2020.
Travis M Bartley, Fei Jia, Krishna C Puvvada, Samuel Kriman, and Boris Ginsburg. Accidental learners: Spoken language identification in multilingual self-supervised models. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Xuankai Chang, Takashi Mackaku, Pengcheng Guo, Jing Shi, Yen-Ju Lu, Aswin Shanmugam Subramanian, Tianzi Wang. Shu-wen Yang. Yu Tsao, Hung-yi Lee, et al. An exploration of self- supervised pretrained representations for end-to-end speech recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 228-235, IEEE, 2021.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6): 1505-1518, 2022a.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Zhuo Chen, Peidong Wang, Gang Liu, Jinyu Li, Jian Wu, Xiangzhan Yu, et al. Why does supervie learning for speech recognition benefit speaker recognition? arXiv preprint arXiv:2204. 12765, 2022b.
William Chen, Brian Yan, Jiatong Shi, Yifan Peng, Soumi Maiti, and Shinji Watanabe. Improving massively multilingual asr with auxiliary etc objectives. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023a.
Zih-Ching Chen, Chin-Lun Fu, Chih- Chih-Ying Liu, Shang- Wen Daniel Li, and Hung-yi Lee. Exploring efficient-tuning methods in self-supervised speech models. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1120-1127. IEEE, 2023b.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeirio.. Areface: Additive angular margin loss for deep face recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685-4694, 2019. doi: 10. (0.1009/CVRR.201900044
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized chan- nel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint 20X1::2005.07143. 2020.
Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin- Yu Zhang, Ming- Hsuan Yang, and Philip Torr. Res2Net: A New Multi-Scale Backbone Architecture. IEEE Transactions Pattern Analysis and Machine Intelligence, 43(2):652-662, Feb 2021. ISSN 1939-3539.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan. and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-344 13450, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint D:Xiv:2/06.09685 2021.
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng. Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.0080 000600000066,... 2024.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 19X9v:1909.05858 2019.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael I. Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 20/7 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5220-5224. IEEE, 2017.
11

---

Manila Kodali, Sudarsana Kadiri, and Paavo Alku. Classification of vocal intensity category from speech using the wav2vec2 and whisper embeddings. In Interspeech, pages 4134 4138. Interna- tional Speech Communication Association (ISCA). 2023.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, et al. Textless speech-to-speech translation on real data. arXiv preprint arXiv:21/2.08.00 2021.
Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36, 2024.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identifica- tion dataset. arXiv preprint 170.000170....11, 2017.
Leonardo Pepino, Pablo Riera, and Luciana Ferrer. Emotion recognition from speech using wav2vec 2.0 embeddings. arXiv preprint TrXiv:2104.03502, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Seyed Omid Sadjadi, Craig Greenberg, Elliot Singer, Lisa Mason, and Douglas Reynolds. The 2021 NIST Speaker Recognition Evaluation. In Proc. The Speaker and Language Recognition Workshop (Odyssey 2022), pages 322-329, 2022. doi: 10.2 (0.21437/0dyssey222222
Ramon Sanabria and Florian Metze. Hierarchical multitask learning with ctc. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 485-490, IEEE, 2018.
Jiatong Shi, Dan Berrebbi, William Chen, Ho-Lam Chung, En-Pei Hu, Wei Ping Huang, Xuankai Chang. Shang-Warr Wen Li, Abdelrahman Mohamed, Hung-yi Lee, et al. MI-superb: Multilingual speech universal performance benchmark. arXiv preprint arXiv:2305. 106/5 2023a.
Jiatong Shi, William Chen, Dan Berrebbi, Hsiu-Hsuan Wang, Wei-Ping Huang, En-Pei Hu, Ho-Lam Chuang, Xuankai Chang, Yuxun Tang, Shang-Wen Li, et al. Findings of the 2023 ml-superb challenge: Pre-training and evaluation over more languages and beyond. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023b.
David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint 1tt:::15110004444 2015.
Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee-weon Jung, Junichi Yamagishi, and Nicholas Evans. Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation. arXiv preprint arXiv:220..1223:: 2022.
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, et al. Unified speech-text pre-training for speech translation and recognition. arXiv preprint 20Xh::2204.05409, 2022.
Jesús Villalba, Bengt J Borgstromm Saurabh Kataria, Magdalena Rybicka, Carlos D Castillo, Jaejin Cho, L. Paola Perera, Pedro A. Torres- Carrss-arrasquilllo, and Najim Dehak. Advances in cross- lingual and cross- audio-visual speaker recognition: The jhu-mit system for nist sre21. pages 213-220. ISCA, 6 2022. doi: 10. 200447////dyysee2222-300 URL (ttthhttt:///e/ sssca-ppeech wtg/aarchvve//tyssyy.222//1
.ttl
Jesús Villalba, Jonas Borgstrom, Maliha Jahan, Saurabh Kataria, Leibny Paola Garcia, Pedro Torres- Carrasquillo, and Najim Dehak. Advances in Language Recognition in Low Resource African Languages: The JHU-MIT Submission for NIST LRE22. In Proc. INTERSPEECH 2023, pages 521-525, 2023. doi: 10.21457/lmcxppecch.033. 10994.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390 2021.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.00015, 2018
12

---

Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan- Guan-Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2/05.01051, 2021.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0677/, 2023.
Cheng Yi, Jianzhong Wang. Ning Cheng, Shiyu Zhou, and Bo Xu. Applying wav2vec2. 0 to speech recognition in various low-resource languages. arXiv preprint arXiv:2012.1222, 2020.
Zbyněk Zajíc and Marie Kunešová. Comparison wav2vec 2.0 transformer models for speaker change detection. In Proceedings of the 6th International Conference Natural Language and Speech Processing (ICNLSP 2023), pages 233-238,, 2023.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
Jing Zhao and Wei-Qiang Zhang. Improving automatic speech recognition performance for low- resource languages with self-supervised models. IEEE Journal of Selected Topics in Signal Processing, 16(6):1227-1241, 2022.
13

---

A Model/Dataset Details and Training Hyper-parameters
This appendix provides detailed configurations and experiments, including ASR, LID, SV decoders, and the CA-SSLR models. The rationale behind specific
for the decoder models used in
choices and architectural details are also discussed to offer insights into the experimental setup and model optimization strategies.
A.1 Decoder Models for ASR, LID, and SV
The ASR, LID, and SV decoder models are optimized for their respective tasks through careful selection of hyperr-aaamettes and architectural configurations. The ASR model directly follows setting in ML-SUPERB benchmark (Shi et al. 2023a for comparison. Table SableS5aa summarizes these configurations. The "full" means one epoch trained by passing all the training data.
Table 5: Hyper-parameters used for training ASR, LID, and SV decoder models.
ASR
LID
SV
Feature Projection
80
80
80
Decoder Layers
3
Hidden Channels
256
512
1024
Dropout Rate
0.1
0.3
0.0
Loss
CTC
CE
CE
Learning Rate
0.0001
0.0001
0.001
Warmup Steps
1000
Effective Batch Size
32
128
512
Iterations per Epoch
5000
5000
full
Epochs
20
10
20
Table 6: Training prree-pprameters for CA-SSIR models. The superscripts "Dec" and "Feat" represent the decoder and the feature projection layer, respectively.
CA-SSLR"
CA-SSLR ML-SUPERB + VoxCeleb 256 (L) + 256 (S) 0.5
Training Data
ML-SUPERB
VoxCeleb
Condition Embedding Condition Dropout Rate
256 (L)
0.5
ASR Dec LID
Initialization
and SyDe
CA-SSLR ASR LID
ASR LID
Trainable modules
SVDec
and Adapters
SVFeat and Adapters
Loss
ASR + LID
SV
ASR + LID + SV 0.0001
Learning Rate Effective Batch Size
0.0001
0.001
32
512
32 5000
Iterations per Epoch Epochs
5000
full
20
20
20
A.2 CA-SSLR Hierarchical Models
provides detailed configurations for the CA-SSLR model. In the CA-SSLR L.S setup, two 256- dimensional embeddings are used to encapsulate language (L) and speaker (S) information, which then determine the parameters (aL, 7L, (LL) and (as, 7s. fs) following the procedure outlined in Eq. 5 The training adopts a stepwise approach, using initial parameters from an earlier phase to set up the next. The pretrained ASR, LID, and SV decoders serve as the foundation for initializing CA-SSLR 4. the SV decoder is fine-tuned further on top of CA-SSLR'; and both CA-SSLR" and fine-tuned SV decoder initialize CA-SSIRL-. In the table's "Trainable modules" row, the notations L.D and Seat indicate that the feature projection layers of the LID and SV decoders are adjustable during
14

---

the training process. We conduct the experiments in Table 6 and Figure B multiple times, and the variations are all within 0.2% CERs range.
A.3
Dataset License and Details
A.3.1 MI-SUPERB Dataset
The ML-SUPERB dataset is assembled from wide collection of multilingual speech corpora, with each contributing corpus being governed by one of a variety of open-source licenses, such as Creative Commons, MIT, GNU, or Free-BSD. These licensing agreements guarantee that the dataset is openly available and can be used freely for both commercial and scholarly research purposes. The 10-minute training set encompasses 37.4 hours of data, and the l-hour dataset increases the total 222.4 hours of data. Additionally, the dataset includes development and testing sets, containing 41.8 hours and 45.0 hours of data, respectively. This dataset designed for multilingual speech recognition and language identification, as used in our work.
In the original ML-SUPERB settings, there are two types of languages:
Normal Languages: Each has 10 minutes or hour of data per language, used for both LID and ASR training with transcriptions.
Few-Shot Languages: Each has only 5 utterances. In the original settings, these languages are not presented in the results for LID training and used for ASR training with available
transcriptions.
For the extended few-shot condition, incorporate the language labels from these few-shot data for LID training but continue using only 5 utterances with transcriptions for ASR training. Since language labels more accessible than transcriptions, especially in low-ret oow-resource scenarios. Table summarizes the data configurations for the original and extended few-shot conditions.
Table 7: Data configurations for the original and extended few-shot conditions in ML-SUPERB.
Data Per Language
Language Type
LID Training
ASR Training
Normal
10 min
10 min -
Original Settings
Few-Shot
Not presented in result
5 utts
Extended Few-Shot
Few-Shot
10 min
1 hr (language labels only)
5 utts
A.3.2 VoxCeleb Dataset
The VoxCeleb dataset is available under the Creative Commons Attribution 4.0 International license and encompasses comprehensive training, development, and testing data collection. Specifically, contains 1092 hours of audio from 5,994 speakers for training, 110 hours from 4,933 speakers for development, and 20 hours from 40 speakers designated for testing. Designed to facilitate speaker verification and identification tasks, aligns with our usage in the speaker verification task. To ensure privacy, speaker names within the dataset are anonymized and represented through unique speaker IDs.
B CER vs. Trainable Parameters
Table X compares the mHuBERT model's ASR performance against the number of trainable param- eters, where the XLSR counterpart is shown in Fig. 3 Both CA-mHubert and CA-mHubert L.S with 256 condition feature dimensions. Notably, the CA-mHubertfiu. model excels in few-shots L.S scenarios, while the CA-mHubertt... dmal yields CERs for normal languages comparable to fully fine- tuned 12-layer mHuBERT model using only 15.9M parameters. This efficiency demonstrates the TCAC's capability in the CA-SSLR framework to deliver fine-tuned levels of ASR accuracy with notably reduced parameter count, providing an optimal balance for practical ASR applications.
15

---

Table Comparison of trainable parameters and CERs on ML-SUPERB 10min dataset, including
fine-tuning top layers, LoRA, and dual-inference condition aware mHuBERT model.
Trainable
Normal
Few-shots
Approach
Params.
CER
CER
mHuBERT
5.7M
38.2
42.9
LoRA
15.2M
38.3
42.7
FT (2L)
19.9M
36.0
42.3
FT (4L)
34.0M
35.0
42.3
FT (6L)
48.2M
33.1
41.4
FT (8L.
62.6M
32.0
40.5
FT (12L)
90.8M
30.8
40.5
CA-mHuberth
10.8M
31.6
40.0
Jual
CA-mHubert,
15.9M
30.9
40.4
dual
C Training Efficiency and Resource Usage
We compare the training speed and resource consumption of different adaptation methods, including Houlsby Adapters, CA-SSLR, and full fine-tuning (FT). Table summarizes the bottleneck dimen- sions, training times, and peak memory usage for each method. We evaluate the training speech for 10k iterations with batch size 8. We found that the CA-SSLR approach ranks second compared to the Houlsby Adapter and fully fine-tuning approach in speed and memory usage. However, it is important to note that CA-SSLR surpasses the Houlsby Adapter in adaptation effectiveness and generalization ability, as demonstrated in Table [ These results indicate that although CA-SSLR incurs a moderate increase in training resources, it provides benefits in performance and generaliza- tion. We acknowledge that the current implementation and memory efficiency. Future work will focus on optimizing the model to reduce training time and memory consumption without compromising performance.
CA-SSLR is not yet optimized for speed
Table 9: Comparison of adaptation methods in terms of bottleneck dimensions, training speed, and peak memory usage.
Method
Bottleneck Dims.
Training Speed
Peak Memory Usage
Houlsby Adapter
256
76 mins
58 GB
CA-SSLR" (3L)
256
120 mins
68 GB
Vine-Tuning (FT)
135 mins
79 GB
D RTF Analysis
Tables T0 and [T present the real-time factor (RTF) for each individual component, as well as for the combined systems. In Table [I] the term "separated tasks" refers to duplicating and fine-tuning the SSLR for each task individually, along with the corresponding total RTF.
E Few-shots Results
Within the ML-SUPERB dataset's s 20 few-shots languages, we examine the performance CA-SSLR against the established SSL baselines, XLSR and mHuBERT, on models trained in the 10-minute ML-SUPERB set. The LID results indicate close match between CA-SSLR and the baseline, with approximately half of the few-shot languages exhibiting improvements or matching their baseline performance. Section reveels that SSL-based LID models are inherently effective, and extending full fine-tuning does not necessarily enhance results. This observation aligns with the outcomes of other classification tasks adeptly handled by SSL models, as documented (Chen et al., 2023b). Furthermore, the CA-SSLR framework demonstrates subtle enhancements for the Normal languages in the 10-minute set Table 3 indicating that the LID performance remains robust despite the encoder's s additional modifications.
16

---

Modules
RTF
ASR Decoder
0.004
LID Decoder
0.001
SPK Decoder
0.003
XLSR SSL
0.016
CA-XLSR (61.)
+ 0.003
CA-XLSR" (4L)
+ 0.004
CA-XLSR (3L)
+ 0.006
mHubert SSL
0.007
CA-mHubert (6L)
+ 0.001
CA-mHubert (3L)
0.002
Table 10: RTF for different components.
XLSR Approaches
RTF
mHubert Approaches
RTF
ASR + LID (Table 2)
XLSR ASR +SR+LID LID CA-XLSR (4L) CA-XLSR (3L) Separated 2 tasks (+XLSR)
0.021
mHubert ASR + LID
0.013
0,024
0.027
CA-mHubert (3L)
0.015
0.037
Separated tasks (+mHubert)
0.020
ASR LID + SV (Table 3)
XLSR + ASR + LID + SV
0.024
mHubert ASR + LID + SV
0.015
CA-XLSR"
0.029
"A-mHubert"
0.017
CA-XLSR
0,032
CA-mHubertt..
0.018
Separated 3 tasks (+2"XLSR)
0.055
Separated tasks (+ 2*MHubrrt)
0.030
Table 11: Total RTFs for combined systems.
Table 12: Evaluation of LID and ASR performance in terms of Accuracy (Acc) and Character Error Rates (CERs) for few-shot learning in low-resource languages using the ML-SUPERB 10-minute set, comparing XLSR and mHuBERT models.
Lang.
XLSR
CA-XLSRL.S
mHuBERT
CA-MHuBERT
Acc 1
CER
Ace
CER
Acc 1
CER
Acc
CER |
bos
82.0
21.3
70.0
11.7
30.0
29.0
28.0
26.0
ceb
97.6
20.5
97.6
12.4
92.9
27.2
97.6
25.4
dan
89.5
44.7
76.5
37.0
80.1
49.1
80.4
47.5
epo
81.7
15.3
76.9
14.5
46.2
24.8
52.9
25.4
fir
87.5
33.6
89.3
29.9
67.9
40.4
63.4
37.7
ful
55.0
27.1
67.5
28.2
37.5
34.7
62.5
32.0
kaz
98.0
32.6
99.3
21.5
91.4
37.8
88.7
36.0
kea
84.1
28.9
90.9
27.6
65.9
35.2
75.0
33.1
lit
87.3
52.0
87.7
45.2
79.3
52.4
82.5
49.3
luo
100.0
29.4
95.1
24.4
92.7
29.4
92.7
30.1
srp
64.8
57.4
50.9
48.1
53.5
56.7
45.7
56.2
93.5
26.4
94.4
19.1
94.4
32.7
93.5
26.2
98.5
15.4
98.5
13.1
98.5
23.2
98.5
19.2
tos
100.0
49.1
99.4
44.7
99.4
53.1
99.4
48.9
tso
84.0
25.4
81.3
21.7
83.3
29.4
81.3
26.0
tsn
87.1
22.7
85.7
17.3
83.6
27.3
84.3
23.9
tur
82.4
60.0
79.1
37.0
62.6
65.1
57.7
61.7
umb
64.0
24.6
40.0
23.3
44,0
29.8
36.1
30.1
vie
94.7
88.4
92.9
83.1
85.8
80.1
74.2
80.3
zul
60.6
20.4
62.3
14.4
53.7
24.3
52.0
20.4
17

---

Regarding the ASR results, most languages achieve significant CER reductions, ranging from modest few percent to over 30%, when compared with SSL baselines. Notably. the Bosnian (bos) language experiences an impressive 45.1% relative improvement in CER, while Cebuano (ceb) improved by 39.5% with the XI.SR model. With the mHuBERT model, the most substantial gains are observed in Sundanese (sun) and Toki Pona (took), with 19.9% and 17.2% CER relative improvements, respectively, These results underscore the CA-SSLR framework's 's profound effect in bolstering ASR performance for few-shot languages, especially demonstrating more pronounced improvements with the XLSR model.
When examining the correlation between LID accuracy and ASR performance, it is apparent that a lower CER does not necessarily align with high LID accuracy. For instance, Serbian (srp) on the XLSR model, despite having a modest LID accuracy of 50.9%, shows a CER improvement from 57.4% to 48. I 1%. Conversely, Fulah (ful), the sole language to exhibit CER increase in the XISR model, presents a higher L.ID accuracy of 67.5%. This indicates that the CA-SSLR framework's 's efficacy is not solely contingent on high LID prediction accuracy. CA-SSLR's reliance on embeddings instead of one-hot hard labels for predictions enables the model to maintain or improve ASR performance despite suboptimal LID scores. This approach allows the model utilize embeddings to distinguish between easily confused languages, enabling the ASR model to predict the correct language accurately.
F Decode Examples
Table 13 visualizes ASR outcomes for the XLSR and CA-XLSR-.S models on the MI-SUPERB 10-minute dataset, covering both few-shot and standard language scenarios. It highlights CA-SSLR's superior language recognition capabilities and success in rectifying the misclassifications encountered with XLSR, often resulting in completely incorrect transcriptions. This is evident in languages such as Lithuanian and Turkish, categorized as Few-shot, and Bulgarian, which is better resourced (Normal). These findings demonstrate the TCA conditioner's 's effectiveness in accurately managing LID embedding features and distinguishing between languages for downstream tasks. Moreover, the results from other samples suggest that CA-SSLK achieves better outcomes during training due its incorporation of language information, even when the XLSR model correctly predicts the language. This underscores the efficacy of the TCA conditioner in exploiting lanuuage-specinn data, thereby enabling CA-SSLR to achieve heightened accuracy across a diverse range of languages.
G Ethical Statement
We affirm our commitment to ethical research practices, including respect for privacy and the responsible use of data. The proposed CA-SSLR model improves multilingual ASR in 143 languages, including 20 low-resource ones with just five training utterances- thus broadening access to speech technology for previously underserved communities. Furthermore, the model's focus reducing computational costs at inference time helps lower both financial barriers and the environmental footprint of large-scale speech applications. Our experiments rely on publicly available datasets (ML-SUPERB and VoxCeleb), chosen for their moderate size to limit computational demands. We also employ widely used pre-trained models (XLSR and mHuBERT) in line with their intended research purposes, thereby adhering to community standards for reproducible and transparent research. Nevertheless, the capability perform speech and speaker recognition raises concerns over potential misuse, such as the covert monitoring of private conversations by unauthorized entities or oppressive regimes intent on identifying dissidents. We therefore underscore the importance of public awareness and transparent dialogue about the implications of automated speech analysis, and the need for responsible governance of emerging technologies.
18

---

Language Esperanto (epo)
Group
Ground Truth
XLSR
CA-XLSR
Few-shots
LI STUDVOIAGIS
LESSTOS
LI STUD VOJAGIS
AL ITALIO HIS-
VOLJAGIS
AL
AL ITALIO
HIS-
PANIO KAJ FRAN-
LITALIO
HIS- CKAI
PANIO
KAJ
CIO
PANIO
FRANZCIO
FRANCIO
Lithuanian (lit)
Few-shots
KARALIUS NEISDRISO KALD-
KAPAJIICCS
KARALIUS
HE ККРИИС
NE LZDR
INTI VARINIU
КАЛIННЕ
SO KALNEN
MONETU KAR-
TE ВОАPИНУ
TIE VORINIU
ALIŠKOIOJE
MOHETY
MONETU
MONETQ KALYK- LOJE
KAPAJлОПИТO JOAE B MOHATY
KARALJUŠKON
JE MONETU
KALIKLOJE
Serbian (srp)
Few-shots
OVO OTKRIĆE
OVO ODKRUĆIE
OVO OD KRIĆĆE
TAKODE PRUŽA UVID U EVOLU- CIUU PERA KOD PTICA
TAKO CE PRUŽA UVID UJEVOLUC
TAKO DE PRUZA UVID U EVOLUCI
I JU PERA KOB
JU PERA KOD
PTICT
PTICT
Northern Frisian (frr)
Few-shots
MEI IK TAKOM
EK TAKGKOMME WIGGE DAT
MEA
MAA
WIKE DAT BOEK
TAKOMME WIKE DAT BOEK VAN LIENE
FAN DY LIENE
BOEK VAN DIE IE
LIÉÈNE
Turkish (tur)
Few-shots
TÜM BUNLAR ILGIMIZI ÇEKSE
TYM ВУНЛAР ИЛГИИМЕЕ YAK CE ДЕ У3YH
TÜM BUNLAR İLGiMiZi KSES DE UZUN KALA- MAZDIK
DE UZUN KALA-
MAZDIK
КАЛДАААдЛИ
Belarusian (bel)
Normal
HA ПЕРПIТМ
HA ПЕРППИМ
HA ПЕРПНТI
LLAHH
HLAHH
KAPIIHНI
KAPПИНE
KAPПЦНН
НАMАЛИBAНЫЫЙ
gOCHI
TOCTE
HA
TOCII
НАКРЫТТИЯ
KPВITИE TECKКEIE PУЖ
НАКРЫТТИЯ
ПЯЛЕСТКAМI
П.ЛΣCTAAMI
PУЖ
PУЖ
Bulgarian (bul)
Normal
CJET MАJKO ВАСИЛЕННА IAAK ИЗЛТЕЗE
SLED MAUKU VASILENA PAKI 1IZLEZE
CTET MAJKO BACEJIНA IAA ИЗЛЛЕЗЕ
Basque (eus)
Normal
KORRONTE ETIKO HORREN HELBURUA ZO-
KORRONTE
KORRONTE
ETIKO HORREN HELBURUA ZO- RIOANTASUNA
ETIKO HORREN HELBURUA ZO-
RIONTASUNA
RIONTASUNA
LORTZEA DA
LORTZEA-DA
LORTZEA DA
Ndebele (nbl)
Normal
UMNQOPHO WOMSEBENZI LO
UMNCOPHO
UMNCOPHO WOMSEBENZI LO
CROMSEVENDIA LOU
Table 13: The ground truth, predictions from XLSR and CA-SSLR models. Deletions are shown with red strikethrough text, insertions are underlined in blue, and substitutions are marked with yellow highlighting.
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We discuss the generalization ability for the condition-awar SSLR in Sec. 3 and it has been thoroughly evaluating Sec. 5 We compare the generalization ability across different tasks, which is unseen for the SSLR model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
We discussion the limitation in Sec 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail the dataset settings in Sec. 4.1 and Sec. 4.2 describes the model architecture. The detail parameters settings is described in Sec. A
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides open access to the code and pre-trained models via the CA-SSLR GitHub repository /https://gith:i
com/neilluu23 SSLR). The repository includes integration with ESPnet and S3PRL, along with detailed instructions help users reproduce the main experimental results faithfully.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cc.
ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.cc/publl.
ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail the dataset settings in Sec. 4.1 and Sec. 4.2 describes the model architecture. The detail parameters settings is described in Sec. A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Appendix A.2 we report the error hars results while running the experiments different times. Also, we conduct different experiments with slightly different settings, and provide consistant ASR and SV improvements, where the variation among these experiments is much smaller than the improvement comparing with the baseline.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
23

---

Justification: We discuss the computation resource for training in 4.2 and RTF in See 5.2 and Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We include the code of ethics in Appendix
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussion the broader impacts in Sec
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
24

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our model has been designed for ASR, SV, and LID, which has been welly studied and have lower risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We mention the datasets and license in Appendix A.3
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprr thcode. com commdatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We detail the dataset settings in Sec. 4.1 and Sec. 4.2 describes the model
architecture. The detail parameters settings described Sec. The model will be released to public after the review process.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
25

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do research with human subject in our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjeets
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do research with human subject in our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "LMS8svYF8i", "reviewer": "Reviewer_Jsg6", "summary": "This paper introduces conditioning into self-supervised learning of speech representations. In particular, a hierarchical self-conditioning mechanism is introduced where intermediate language and speaker embeddings are used to condition upper layers. The proposed approach is used together with XSLR and mHubert models and several experiments are conducted using the SUPERB benchmark.", "questions": "Please see above.", "limitations": "Limitations have been addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Multiple experiments are conducted demonstrating the benefits of the proposed method.\n\n- A novel conditioning method is introduced for learning speech representations in a self-supervised way.", "weaknesses": "I think the main weakness of the paper is that it is too dense. Too much information is presented which makes several sections of the paper hard to follow. For example, it's not very easy to follow section 3. It would help if additional equations are added or a more detailed figure is included. Fig. 1 is too high-level and it's not very informative. \n\nIn addition, some sentences from sections 1 and 2 are literally copied and pasted in section 3. For example, the following sentence can be found both in sections 2 and 3 \"Unfortunately, this results in employing a distinct encoder per task, leading to a large increase in computational load that scales linearly with the number of tasks to be assessed.\". It would good if the authors rephrase such sentences. There are a few more such examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comments. We have included additional equations and figures to improve the clarity and address the writing in Sec. 3.\n\nRegarding additional figures, we add **Figure A in the rebuttal PDF** to clarify our system as the elaboration for the original Figure 1.\n\n\nRegarding additional equations, we further add **details equations to clarify the Time-Channel Attention Conditioner to form a separate subsection of Sec 3.2** to improve the readability:\n\n\"As depicted in Fig.1b, the TCAC module ingests the latent representations of the CA-SSLR at layer $l$, $\\mathbf{S}^{(l)}\\in\\mathbb{R}^{C\\times T}$, and the latest estimate of the conditioning features $\\mathbf{z}\\in\\mathbb{R}^R$ and generates modulated latent representations $\\mathbf{\\tilde{S}}^{(l)}$ as\n\n$$\\mathbf{\\tilde{S}} _{t,c}^{(l)} = \\text{TCAC}(S _{t,c}^{(l)}, \\mathbf{z}) = \\tilde{\\gamma} _{t,c}^{(l)}(\\mathbf{z}, S^{(l)})S _{t,c}^{(l)}+ \\tilde{\\beta} _{t,c}^{(l)} (\\mathbf{z}, S^{(l)})$$\n\nwhere $t$, $c$, and $l$ represent time, channel, and layer indices, respectively. Thus, the latent features are modulated by time-channel dependent scales $\\tilde{\\gamma} _{t,c}^{(l)}$ and biases $\\tilde{\\beta} _{t,c}^{(l)}$. These are obtained as the products:\n\n$\\tilde{\\gamma} _{t,c}^{(l)} (\\mathbf{z}, \\mathbf{S}^{(l)})=\\alpha _t^{(l)}(\\mathbf{z}, \\mathbf{S}^{(l)})\\times  \\gamma _c^{(l)}(\\mathbf{z})$ \n\n$\\tilde{\\beta}_{t,c}^{(l)} (\\mathbf{z}, \\mathbf{S}^{(l)})=\\alpha_t^{(l)} (\\mathbf{z}, \\mathbf{S}^{(l)}) \\times \\beta_c^{(l)}(\\mathbf{z})$\n\nwhere channel-dependent $\\gamma^{(l)},\\beta^{(l)}\\in \\mathbb{R}^C$ are obtained as\n\n$\\gamma^{(l)}(\\mathbf{z})=\\mathbf{W}_\\gamma^{(l)}\\mathbf{z}+\\mathbf{b} _\\gamma^{(l)}$\n\n$\\beta^{(l)}(\\mathbf{z}) = \\mathbf{W}_\\beta^{(l)} \\mathbf{z} + \\mathbf{b} _\\beta^{(l)}$\n\nThe time-dependent scales $\\alpha^{(l)} \\in \\mathbb{R}^T$ are obtained with an additive attention mechanism as\n\n$\\alpha^{(l)} _t(\\mathbf{z}, \\mathbf{S}^{(l)}) = \\mathbf{v}^T _\\alpha f(\\mathbf{W} _\\alpha^{(l)}  [\\mathbf{S} _t^{(l)T} \\mathbf{z}^T]^T$+ $\\mathbf{b} _a^{(l)})$\n\nwhere $f(.)$ is a ReLU non-linearity, $\\mathbf{W} _\\alpha^{(l)}\\in \\mathbb{R}^{C'\\times (C+R)}$, $\\mathbf{b} _\\alpha^{(l)}\\in\\mathbb{R}^{C'}$, and $\\mathbf{v _\\alpha}\\in\\mathbb{R}^{C'}$.\n\nWe obtained the conditioning features $\\mathbf{z}$ from the hard decisions or internal layer (embedding layer) $\\mathbf{e}\\in\\mathbb{R}^E$ of the intermediate decoders, by $\\mathbf{z} = \\mathrm{LayerNorm}(\\mathbf{W} \\mathbf{e} + \\mathbf{b})$, where the affine transform parameters $\\mathbf{W}$, $\\mathbf{b}$ are shared across TCAC layers.\n\nThus, TCAC enables the model to dynamically adjust its behavior to the input audio in response to the provided conditioning features.\"\n\nAlso, for the **Condition-Aware Learning Mechanism**, we remove Lines 121-131 to eliminate duplication and focus the section on our unique contribution. This duplication was inadvertently introduced by one of the authors who didn’t notice that this content had been moved to another section. Following are the revisions of the second and third paragraphs in Sec. 3.1:\n\n“The proposed CA-SSLR model improves the efficiency of evaluating multiple tasks by introducing a hierarchy of conditioners within the pre-trained SSL encoder layers. It utilizes intermediate predictions from the language identification (LID) and speaker verification (SV) decoders as conditions to recursively adapt subsequent layers, as shown in Figure A. This hierarchical approach structures the SSL encoder layers so that each layer refines its output based on the predictions of the preceding layers. Early layers produce embeddings that capture the essential language and speaker characteristics, which are then used to inform scaling and bias adjustments in later layers. \n\nWe propose a novel mechanism, the Time-Channel Attention Conditioner (TCAC), which modulates the encoder's hidden representations using time-channel-dependent scales and biases. This approach enables the SSL encoder to dynamically adjust to varying tasks and input conditions. The inputs to these conditioners are embeddings derived from intermediate evaluations of the LID and SV decoders.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yHM6I3gNK", "reviewer": "Reviewer_iiaB", "summary": "The paper introduces a method named CA-SSLR, a versatile model for various speech-processing tasks that integrates language and speaker embeddings from earlier layers to reduce reliance on input audio features while preserving the base model's integrity. More specifically, both LID and SID conditioning features are integrated as additional inputs for the SSL model. The proposed approach shows advantages in improving the downstream speech tasks.  ML-SUPERB dataset is used in the evaluation, which shows superior performance over the conventional SSL models.", "questions": "1. The first question relates to the second point in the weaknesses: when is attention necessary? Can we visualize the attention using one or two examples?\n2. Which layers are helpful in generating the embeddings? 3L and 4L are reported in the tables; are the shallow layers sufficient?", "limitations": "See the Weaknesses.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The topic is interesting, and the proposed idea is simple yet effective.\n- The experiments are thorough and convincing.", "weaknesses": "- There is no comparison between the proposed approach and the SOTA numbers on ML-SUPERB. Readers would likely be interested in understanding the gap between the proposed approach and the SOTA numbers.\n- It is somewhat unclear when to use the proposed TCAC, as sometimes using CC (without attention) seems sufficient, as shown in Table 2 and Table 4. More analysis should be provided to explain these results, rather than just mentioning them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments. Here are some clarifications.\n\nFirst, for **the generalist model without fine-tuning, the SOTA results** for pre-trained SSLR models can be found in the paper ML-SUPERB challenge [A], where the MMS-1b model performs the best with 1hr ASR WER 18.1% and LID accuracy 86.1% respectively (see the table below). However, we used XLS-R 300M to make our experiments feasible.\n\nSecond, **with fine-tuning, the single-task SOTA results** can be found in Table 1 and Figure 2, with 90.1% LID accuracy and 17.3% ASR CERs. \n\n\n\n| **SSL Model**    | Model size            | **10mins LID**          |  **10mins ASR**                |       **10mins ASR**           | **1hr LID**               |      **1hr ASR**            |    **1hr ASR**              |\n|-------------|-----------------|------------------------|-----------------|-----------------|----------------------|-----------------|-----------------|\n|               |                | Normal      | Normal | Few-shots   | Normal      | Normal | Few-shots   |\n| XLS-R~\\citep{shi2023ml} | 0.3B     | 66.9                   | 29.2            | 40.9            | 87.9                 | 22.0            | 39.3            |\n| MMS-1b ~\\citep{shi2023findings} | 1B | 84.8               | 21.3            | 30.2            | 86.1                 | 18.1            | 30.8            |\n| XLS-R (Ours)   | 0.3B              | 89.0                   | 29.0            | 39.0               | 90.9                 | 22.7            | 36.9               |\n| + Embed TCAC^{L}  | 0.3B            | 89.0                   | 17.8            | 31.8            | 90.9                 | 13.5            | 31.4            |\n| + CA-SSLR^{L} (TCAC, 3L) | 0.3B | 88.6                | 18.6            | 31.6            | 93.4                 | 15.1            | 29.6            |\n\n\nWe found that compared with the MMS-1b baseline, the proposed system achieves better results in Normal languages and comparable results in few-shots languages than the model with three times more parameters (300M vs. 1B). We will add these results in Table 3.\n\nRegarding the **concern about using shallow layers**, we would like to clarify that the CA-SSLR system recomputes the embeddings every 3 or 4 layers rather than predicting only once in the initial layers, as shown in Table A in the rebuttal PDF. From the SSL feature weights utilized by the LID and SV decoders, we observe that LID weights are evenly distributed, while SV weights are more concentrated in the earlier layers. Consequently, incorporating predictions from higher LID layers can enhance LID accuracy and improve conditioning features. Conversely, for SV, using only the first 6 or 12 layers of XLS-R might be sufficient. We will further explore this in future work.\n\nWhile CC applies the same transformation to all time frames, we added the attention module in **TCAC to be able to emphasize different time-frames** of the encoded features within the SSLR layers.  When deciding between TCAC and CC, our experiments indicate that TCAC provides better results for LID and ASR. However, as shown in Table 4, CC currently yields slightly better results for SV. Overall, relative improvements of TCAC in ASR and LID are larger than relative degradation in SV. We plan to conduct further investigations to enhance the SV performance of TCAC. In the updated version, we will provide visualizations of the attention mechanism for the TCAC module using one or two examples. We will further provide a more detailed analysis of our results in the revised version. \n\n[A] Shi, J., Chen, W., Berrebbi, D., Wang, H. H., Huang, W. P., Hu, E. P., ... & Watanabe, S. (2023, December). Findings of the 2023 ml-superb challenge: Pre-training and evaluation over more languages and beyond. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 1-8). IEEE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lqmKmt0Dfw", "reviewer": "Reviewer_mDeV", "summary": "This paper introduced a framework, CA-SSLR, that integrates conditioning into pre-trained Self-Supervised Learning (SSL) models by adapting only the trainable conditioner. Through a hierarchical self-conditioning mechanism, CA-SSLR can match or achieve better the performance of single-task fully fine-tuned models and benefit from more efficient model tuning. CA-SSLR offers a versatile and efficient approach to integrating conditioning information into pre-trained models.", "questions": "Here are some minor concerns to the paper:\n1. Is it necessary to provide the rel RTF in Table 3 and 4? I feel like the absolute value would be enough though.\n\n2. It is intuitive that the multilingual ASR model can be improved by using LID as the condition. Is there a cascaded baseline doing LID + ASR that the pretrained model is not fused? The results can help us understand the real effectiveness of the proposed framework.\n\n3. Since the proposed framework achieves better parameter efficient finetuning, is there a training speed or peak memory usage comparison  between the proposed method and full finetuning? How about its efficiency compared to other PEFT methods, like LoRA?", "limitations": "The advantage of the paper is to use less resources than full finetuning but achieve similar performance. However, the inference cost is increased. This can be problematic when the method is used in real applications. The inference cost might weigh more than the training cost when the training cost is affordable. Another limitation is that the framework cannot be used for streaming purposes as mentioned by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are:\n\n1. Good presentation of the proposed framework.\n2. Strong results on different tasks and different scenarios.", "weaknesses": "The weaknesses of the papers are that the overall system might be complicated because of performing multiple tasks and some tasks would be dependent on the output of other tasks. As a result, the inference cost would be increased for the subsequent tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comment. Here are some clarifications.\n\nRegarding **inference cost**, the encoder parameters are shared among all three tasks, which helps to minimize expenses. The LID decoder is lightweight, with an RTF of less than 0.001, so it doesn't significantly add to the computational load of the ASR task. While the ASR and SSL models are more complex, with single-pass RTFs of 0.004 and 0.016, respectively, they may need multiple inferences based on the decoding algorithm. As shown in Tables 9 and 10, the CA-SSLR system achieves 30 to 45% faster RTF than multi-task baselines across different components and scenarios.\n\nConcerning the **cascaded baseline**, we examined the cascading baseline for LID + ASR in Table 2 and Figure 2. In this method, we train the LID decoder with a pre-trained SSL model and use another SSL model with TCAC components to train the ASR decoder. Our findings indicate that the TCAC can achieve performance comparable to a fully fine-tuned approach for languages with abundant resources and surpasses the fully fine-tuned approach in few-shot ASR languages.\n\nTo assess **training speed and peak memory usage**, we compared the CA-SSLR approach with the additional baseline, Houlsby Adapter, and a fully fine-tuning approach (the results comparing the Houlsby Adapter and CA-SSLR$^{L}$ are detailed in Table A of the rebuttal PDF). We evaluate the training speech for 10k iterations with batch size 8, and the results are shown in the following table:\n\n\n| **Method**     | **Bottleneck Dims.**             | **Training Speed**                          | **Peak Memory Usage** |\n|---------------|------|---------------------------------------------|-----------------------|\n| Houslby Adapter | 256  | 76 mins          |            58B           |\n| CA-SSLR$^{L}$(3L) | 256 | 120 mins          |             68B          |\n| FT              | -  | 135 mins                            |         79B             |\n\nwe found that the CA-SSLR approach ranks second compared to the Houlsby Adapter and a fully fine-tuning approach in speed and memory usage. However, CA-SSLR surpasses the Houlsby Adapter in adaptation effectiveness, as shown in Table A(b) of the rebuttal PDF. It also demonstrates **the best generalization ability among the three methods, detailed in Tables 1, 4, and A**. Additionally, we acknowledge that the current implementation is not yet optimal and are committed to further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E4Z4b9aJzy", "reviewer": "Reviewer_4eyk", "summary": "This paper employs multi-task learning with hierarchical conditioning to adapt pre-trained speech SSL models. By utilizing lightweight task-related auxiliary decoders repeatedly at various positions, the method gradually tailors the SSL representations. A time-channel-dependent conditioner is introduced to facilitate the fusion process. By incorporating LID and Speaker SV decoders, the proposed approach achieves significant improvements in multi-lingual ASR, LID, and SV.", "questions": "(Line 144-145) It states \"... the TCAC, SV, and LID decoders are trained...\". But in Figure 1, the two decoders look frozen and the caption confirms this. Which is the actual setting?\n\n(Line 163) It states that the output projection layers of conditioners are all shared. But as stated at line 221, each auxiliary decoder, at different points, takes the previous SSL representations using a weight average. Are those weights and input projection layers partially shared across positions, or totally position-dependent?\n\n(Line 188) What augmentation is leveraged in the \"Extended Few-shot condition\"?\n\n(Line 236) Which model does \"the pre-trained and fixed SSLR model\" refer to? If it refers to \"XLS-R\" in Table 1, then it is a bit confusing because a prediction head is added and the SSLR model is not purely pre-trained.\n\n(Line 251) The conditioner takes $z \\in \\mathbb{R}^{C_E}$ as input, then how is it adpted to consume LID labels? Some illustrations here would avoid confusion.\n\n(Table 4) Why is $\\text{CA-SSLR}^{L,S}(\\text{TCAC})$ not compared? As Eq. (3) mentions the adaptation method of this \"full configuration\", it is surprising that it does not appear in the final comparison.\n\nSome personal suggestions/concerns regarding the presentation:\n\n- Line 145: \"ASR,a\" -> \"ASR,\"\n- Line 163 implies the re-estimation is conducted every three layers for both LID and SV, which conflicts with the actual setting.\n- Table 4: the results are derived by *jointly optimization* for all three tasks, including system \"**+ FT**\", but this operation is not explicitly mentioned in the experiment setting, which could be confusing at first.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The method is novel in reusing the decoders multiple times for incremental conditioning.\n- The framework is general for various sets of speech tasks. One could change the conditioning decoder(s) according to the target tasks.\n- Relatively strong results in adapting SSL models for multi-lingual ASR, which may inspire more SSL-based methods in multi-lingual scenarios.", "weaknesses": "Although the authors refer to the proposed model as \"generalist\", it still requires proper selection of conditioning/auxiliary decoders according to the target tasks. In the paper, the major task is multi-lingual ASR and the initial conditioning decoder is a language identifier, which are closely related. As illustrated in Table 4, further incorporating the SV conditioner leads to less performance gain.\n\nA terming issue: the \"Time-wise Attention\" seems *time-dependent scaling* rather than an \"attention mechanism\". In Eq. (2), $\\alpha_t^{(l)}(\\textbf{z}, \\textbf{S}^{(l)}) \\in \\mathbb{R}^T$ is computed frame-wise with a linear projection, and it scales the SSL features at different timesteps without interactions among frames/timesteps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comment. Here are some clarifications.\n\nRegarding the **generalist model concern**, we want to emphasize that CA-SSLR is considered a generalist model because it maintains the base model's integrity while improving performance on previously unseen tasks.  In Table 1, CA-SSLR conditioners/LID-decoder are trained on LID loss, then frozen, and the ASR decoder is trained on top. Thus, no component in the CA-SSLR encoder is trained for ASR but obtains improvements w.r.t. Baseline. In the same manner, CA-SSLR$^L$ models improve SV by 20% while the CA-SSLR$^L$ encoder conditioners were adapted on ASR and LID losses, not SV, showing that CA-SSLR$^L$ is a generalist encoder.\n\nWe acknowledge the concern about using **the term \"Time-wise Attention.\"** Although it seems like time-dependent scaling, it fits the concept of additive attention, without normalizing weights to sum up to one with softmax. We observed that removing the softmax provided better results. Our approach involves using a linear projection to compute $\\alpha _t^{(l)}(z, S^{(l)} _{t,c}) \\in \\mathbb{R}^T$  by concatenating a single feature vector $\\mathbf{z}$ with $S^{(l)} _{t,c}$ at each timestep, similar to the method used in [A]. This process assigns different weights to each timestep, allowing us to scale $S^{(l)} _{t,c}$ accordingly. Even though there are no direct interactions between timesteps, the dynamic assignment of weights based on features aligns with the principles of attention, as the approach applied in [B].\n\nIn Table 4, we have updated **CA-SSLR$^{L,S}$ TCAC results** for comparison with XLS-R, achieving the best results in LID and ASR and ranking second in SV, showing strong adaptation capabilities while maintaining competitive generalization ability.\n\n| Model                                    | LID (10min) | ASR | EER | CDF  | LID (1h) | ASR | EER | CDF \n|------------------------------------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| XLS-R                                    | 89.0     | 29.0     | 1.29     | 0.093    | 90.9     | 22.7     | 1.29     | 0.093    |\n| + CA-SSLR$^{L,S}$ (CC)             | **89.1** | 18.8     | **1.04** | **0.075**| 88.1     | 15.0     | **0.94** | **0.073**|\n| + CA-SSLR$^{L,S}$(TCAC)           | 89.0     | **18.3** | 1.11     | 0.086    | **93.5** | **14.4** | 1.01     | 0.077    |\n\nRegarding the **confusion about frozen and trainable parameters**, first, we train the LID decoder on the pre-trained SSL model. The decoder gets a weighted average of the SSL encoder layers. Afterward, this LID decoder is then frozen and used to get the conditioning features that are fed to the LID TCAC conditioners. In this case the LID decoder gets a weighted average of the CA-SSLR encoder layers computed up to that point, and the conditioning feature is recomputed every three CA-SSLR layers. Here, we only train the linear projection before the LID decoder and the LID TCAC parameters to the LID-conditioned CA-SSLR encoder, termed CA-SSLR$^L$. Similarly, the SV decoder is trained on top of the CA-SSLR$^L$ then frozen. Then, we just train the SV TCAC parameters to get the CA-SSLR$^{L,S}$. The adaptation training has been detailed in **Figure A of the rebuttal PDF** and will be added to Sec. 3. We also update lines 144 and 145 to avoid confusion.\n\nRegarding **parameter sharing**, the only layer-dependent features used for calculating the condition feature z are the linear projections for the decoders and the weights used for the weighted sum of the SSL layers before the linear predictions, as shown in Figure A of the rebuttal PDF. All other decoder parameters are shared.\n\nFor the **extended few-shot condition**, the original ML-SUPERB settings include \"normal languages\" with 10 minutes to 1 hour of data per language and \"few-shot languages\" with only 5 utterances each. In the \"extended few-shot condition,\" we incorporate the language labels from these few-shot data for LID training but continue using only 5 utterances with transcriptions for ASR training. This is because language labels are more accessible to obtain than transcriptions. We will add this table to the appendix to clarify these settings.\n\n| **Data Per Language**           | **Language Type**    | **LID Training**                      | **ASR Training**                         |\n|-------------------------|----------------------|---------------------------------------|------------------------------------------|\n| Original Settings       | Normal    |  10 min - 1 hr      | 10 min - 1 hr  with transcription          |\n|                         | Few-shot  |  Not used for LID                   | 5 utts with transcriptions     |\n| Extended Few-shot       | Extended Few-shot   | 10 min - 1 hr          | 5 utts  with transcriptions     |\n\n\nRegarding whether the **\"pre-trained and fixed SSLR model\" refers to XLS-R**, Yes, it does refer to the frozen pre-trained XLS-R model with a trained decoder head. We will update the sentence to \"frozen XLS-R model with ASR decoder\" for clarity.\n\nRegarding the **condition feature z**, we have updated the sentence in L163 for clarity: \"We re-estimate the condition feature 𝑧 from the updated language or speaker embedding every three layers using a linear projection and layer normalization, which are shared across layers.” This has been revised to “As shown in Figure A, 𝑧 is derived from the updated language or speaker embedding through a linear projection layer or an embedding layer for hard LID/SV labels.”\n\n\n[A] Luong, M. T., Pham, H., & Manning, C. D. (2015, September). Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 1412-1421).\n\n[B]Lin, Z., Feng, M., dos Santos, C. N., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2022, July). A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING. In International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9rBc0AVEm8", "reviewer": "Reviewer_xgMo", "summary": "This work proposes a method to automatically condition a (frozen) speech foundation model to a particular language and/or speaker. This method consists of 2 parts; they use an ECAPA-TDDN model to compute speaker or language embeddings from a set interval of intermediate layers of the SSL model, and a conditioning layer which uses those embeddings to modify subsequent layer outputs of the SSL model. The authors conduct experiments with the multilingual SUPERB benchmark (for language identification and ASR) and VoxCeleb1 dataset (for speaker recognition).\n\n\nEdit Rebuttal: Houlsby adapter was addressed, presentation style was addressed but cannot be judged. Updated score from reject to borderline reject.", "questions": "I do not understand the time-channel attention conditioner. Given formula 1, what are \\alpha, \\gamma, and \\beta? Are they a function? In line 164, they are defined as vectors? In line 166, you mention a linear layer to compute attention scores? How are these attention scores used? Do you not use a self-attention layer?   \n\nIt is unclear to me how exactly the time-channel attention conditioner is used within the SSL model. Given formula 1, I assume the conditioner is inserted between each wav2vec 2.0 transformer layer?\n\nIn section 5.1, does the LID-FT experiment imply 2 fine-tunings, first on LID (with what data?), and then on ASR (with what data?), where the first fine-tuning updates the SSL model, and the second only the ASR decoder?\n\nIn section 5.1, what hyperparameters were used for these experiments? Is there a more comprehensive learning rate scan for the LID-TCAC setting compared to the XLSR-R baseline and LID-FT setting? Are they conducted with the same amount of update steps?\n\nIn section 5.1, line 236, it states “only TCAC layers are updated”, while I assume the ECAPA-TDNDN language ID decoder was also trained?\n\nIn section 5.2, Table 2 states ASR adapted, but aren’t the models adapted to language ID? \n\nIn section 5.2, how do you condition using the ground-truth language ID labels? In the same context, how are hard-predicted and soft-predicted language labels used for conditioning, and where do they come from?\n\nIn section 5.2, second paragraph, are the results shown in Figure 2 all your experiments? Regardless, how did you ensure a fair comparison (learning rate scan, number of updates)? Why did you choose to compare with LoRA? You mention that Chen et. al. [2023b] shows marginal improvements for LoRA, but this paper shows that AdapterBias and Houlsby are more effective for ASR?", "limitations": "While the proposed method improves performance for the specific condition of the ML-SUPERB benchmark, it might not work as well with unbalanced datasets (e.g., 100 hours of english data language, 1 hour of spanish and mandarin data).", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "This work tackles the difficult problem of using a single model to perform ASR on 123 languages, with only 1 hour of training data for each language. They show that, during inference on the test set, making use of a ground-truth language label significantly improves the character error-rate, indicating that baseline models like wav2vec2-xlsr do not adequately activate the representations of a specific language. The authors propose a method where a model can learn to condition itself during inference on a specific target domain.", "weaknesses": "The presentation of this paper is, in my opinion, subpar to what is expected for NeurIPS. Most importantly, I found it difficult to grasp the proposed method from the methodology section. Line 123-131 are copied from the related work section. Line 133 introduces a “hierarchy” of conditioners, but hierarchy is never explained. Then line 140 assumes the reader is very familiar with TCAC while it has not been explained yet. Line 141 to 146 mention a lot of different configurations related to SV and LID decoders, this could be streamlined into a single “standard” setup, while the experimental section could ablate on different settings. Moreover, I think that the speaker decoder story only distracts from the interesting results of this paper (e.g., Table 1, Table 2, Figure 2 do not make use of the speaker decoder at all, so why make the method section hard to understand by including it?). Furthermore, I cannot implement the conditioner (TCAC) from the description in line 156-170. Also, line 221-222 should be in the method section.\n\nIn the experiment section, references to the appendix should be made to guide the reader to relevant information regarding experimental details. I cannot judge whether the current experimental setup is fair. \n\nFurthermore, I think some relevant literature and methodology is missing:\n\n[1] Houlsby, Neil, et al. \"Parameter-efficient transfer learning for NLP.\" International conference on machine learning. PMLR, 2019.     \n[2] Thomas, Bethan, Samuel Kessler, and Salah Karout. \"Efficient adapter transfer of self-supervised speech models for automatic speech recognition.\" ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.    \n[3] Otake, Shinta, Rei Kawakami, and Nakamasa Inoue. \"Parameter efficient transfer learning for various speech processing tasks.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.    \n[4] Peng, Junyi, et al. \"Parameter-efficient transfer learning of pre-trained transformer models for speaker verification using adapters.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.    \n\nI think a comparison with the standard adapter from [1] is required. To keep things as simple as possible, instead of the proposed TCAC, the adapter from [1] could potentially use the ECAPA-TDNN langauge/speaker embedding as a second input.  \n\nI’ll end with listing minor (formatting) issues:\n* Line 20: inconsistent citation style   \n* Line 34: word ‘ControlNet’ missing   \n* Line 37: this challenge instead of these challenges (labeling data is a solution, not a challenge)   \n* Line 44,52: CA-SSLR instead of CA-SSL   \n* Line 76,78,80,84: hyperbolic language   \n* Line 84-86: This sentence does not make sense to me  \n* Line 87: inconsistent citation style  \n* Figure 1: “Condition-aware” instead of “conditioned-aware\"\n* Multiple instances where an abbreviation is explained while it has been used previously, e.g. CER in line 249,      \n* Figure 2 is unreadable in black and white print, using a different marker for each hue could alleviate this problem.   \n* Line 199: inconsistent citation style    \n* Line 280: requires instead of required    \n* Line 280: Real-Time Factors    \n* Most justifications in the paper checklist contain a spelling mistake, are missing, or do not refer correctly. Also, the instruction block was not deleted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments, which have helped us improve our manuscript. Below are our clarifications:\n\n**Speaker Decoder and Generalization Ability:**\nWe respectfully disagree that the speaker decoder detracts from the results. We intend to show that integrating conditioning in the SSL improves unseen tasks. In Table 1, CA-SSLR conditioners/LID-decoder are trained on LID loss, then frozen, and the ASR decoder is trained on top. Thus, no component in the CA-SSLR encoder is trained for ASR but obtains improvements w.r.t. Baseline. In the same manner, CA-SSLR$^L$ models improve SV by 20% while the CA-SSLR$^L$ encoder conditioners were adapted on ASR and LID losses, not SV. Therefore, the speaker decoder is essential for demonstrating the generalization ability of our condition-aware approach and showing that CA-SSLR$^L$ is a generalist encoder.\n\n**Comparison with Houlsby Adapter ([1], Chen et al. [2023b]):**\nCA-SSLR not only offers parameter-efficient fine-tuning (PEFT), but its principal contribution lies in incorporating dynamic adaptation to the language and speaker of the input, thereby improving generalizability, as noted by Reviewers iiaB and Jsg6. We performed additional experiments using the Houlsby adapter, as shown in Table A of the rebuttal PDF. Our ASR-CA-SSLR approach outperforms the ASR-Houlsby adapter, achieving WER of 18.6% and 31.6% compared to 20.3% and 34.6%, respectively. In SV, CA-SSLR improves EER from 1.29 to 1.15, while the Houlsby adapter worsens it from 1.29 to 1.37 compared to the XLS-R baseline. These results suggest that standard adaptation methods lack the enhanced generalization capabilities inherent to CA-SSLR. We appreciate the suggestions and plan to explore integrating our conditioner into the Houlsby adapter in future work.\n\n**Unbalanced Dataset and Performance**\nIt's important to highlight that ML-SUPERB includes 20 few-shot languages with only five utterances each for ASR to evaluate performance on unbalanced datasets. Improvement in few-shot CERs is documented in Tables 1 to 3, Figure 2, Appendix Table 8, and additional Table A. These results demonstrate CA-SSLR's robustness in unbalanced scenarios.\n\n**Fair Comparison and Hyperparameter Search:**\nWe confirm that the model parameters, learning rates, and batch size for ASR training are consistent with those in the ML-SUPERB benchmark paper. Parameters specific to the LID and SV decoders, such as dropout rates and hidden channels, were set using frozen SSLR settings to ensure a fair experimental setup. We added a reference in Section 4.2 to Appendix A.1 for clarification.\n\n**Enhancements in Presentation:**\n- Removed the duplicated content in Lines 121-131. This duplication was inadvertently introduced by one of the authors who didn’t notice that this content had been moved to another section.\n- Revised the hierarchical writing with Figure A.\n- Moved TCAC explanation before line 140 and substituted text with precise equations to assure reproducibility.\n- Consolidated SV and LID configurations in lines 141-146 into a standard setup, relocated to the experimental section.\n- Moved the information in 221-222 to the methods section.\n- Included suggested parameter-efficient transfer learning references [1~4] in the related work and fixed the minor formatting issues. \n\n\nWe add the following equations for TCAC Implementation:  \n\nThe TCAC module process latent representations at layer $l$, $\\mathbf{S}^{(l)}\\in\\mathbb{R}^{C\\times T}$, and the latest estimate of the conditioning features $\\mathbf{z}\\in\\mathbb{R}^R$ and generates modulated latent representations $\\mathbf{\\tilde{S}}^{(l)}$ as\n\n$$\\mathbf{\\tilde{S}} _{t,c}^{(l)} = \\text{TCAC}(S _{t,c}^{(l)}, \\mathbf{z}) = \\tilde{\\gamma} _{t,c}^{(l)}(\\mathbf{z}, S^{(l)})S _{t,c}^{(l)}+ \\tilde{\\beta} _{t,c}^{(l)} (\\mathbf{z}, S^{(l)})$$\n\nThus, the latent features are modulated by time-channel dependent scales $\\tilde{\\gamma} _{t,c}^{(l)}$ and biases $\\tilde{\\beta} _{t,c}^{(l)}$, obtained by:\n\n\n$\\tilde{\\gamma}_{t,c}^{(l)} (\\mathbf{z}, \\mathbf{S}^{(l)})=\\alpha_t^{(l)}(\\mathbf{z}, \\mathbf{S}^{(l)})\\times \\gamma_c^{(l)}(\\mathbf{z})$ \n\n$\\tilde{\\beta}_{t,c}^{(l)} (\\mathbf{z}, \\mathbf{S}^{(l)})=\\alpha_t^{(l)} (\\mathbf{z}, \\mathbf{S}^{(l)}) \\times \\beta_c^{(l)}(\\mathbf{z})$\n\nwhere channel-dependent $\\gamma^{(l)},\\beta^{(l)}\\in \\mathbb{R}^C$ are obtained as\n\n$\\gamma^{(l)}(\\mathbf{z})=\\mathbf{W}_\\gamma^{(l)}\\mathbf{z}+\\mathbf{b} _\\gamma^{(l)}$\n\n$\\beta^{(l)}(\\mathbf{z}) = \\mathbf{W}_\\beta^{(l)} \\mathbf{z} + \\mathbf{b} _\\beta^{(l)}$\n\nThe time-dependent scales $\\alpha^{(l)} \\in \\mathbb{R}^T$ are obtained with an additive attention mechanism as\n\n$\\alpha^{(l)} _t(\\mathbf{z}, \\mathbf{S}^{(l)}) = \\mathbf{v}^T _\\alpha f(\\mathbf{W} _\\alpha^{(l)}  [\\mathbf{S} _t^{(l)T} \\mathbf{z}^T]^T$+ $\\mathbf{b} _a^{(l)})$\n\nwhere $f(.)$ is a ReLU non-linearity, $\\mathbf{W} _\\alpha^{(l)}\\in \\mathbb{R}^{C'\\times (C+R)}$, $\\mathbf{b} _\\alpha^{(l)}\\in\\mathbb{R}^{C'}$, and $\\mathbf{v _\\alpha}\\in\\mathbb{R}^{C'}$.\n\nWe obtained the conditioning features $\\mathbf{z}$ from the hard decisions or internal embedding layer $\\mathbf{e}\\in\\mathbb{R}^E$ of the intermediate decoders, by $\\mathbf{z} = \\mathrm{LayerNorm}(\\mathbf{W} \\mathbf{e} + \\mathbf{b})$, where the affine transform parameters $\\mathbf{W}$, $\\mathbf{b}$ are shared across TCAC layers.\n\n**Additional Clarifications:** \n- Alpha, beta, and gamma are functions that predict vectors, in parts of the text the dependency on (z,S) is dropped to keep the notation uncluttered.\n- The conditioning feature can be derived from the decoder's second last layer (embedding layer) or decoder output (either as hard or soft labels).\n- The LID decoder was updated with the pre-trained, frozen SSLR model but remained frozen during TCAC training, as shown in Figure A.\n- \"ASR-adapted\" refers to training CC/TCAC with ASR loss only.\n- TCAC is integrated within wav2vec 2.0 layers after the self-attention module. These details are included in the model architecture section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We thank the reviewers for their insightful and positive feedback! We are encouraged that they appreciate various aspects of CA-SSLR, including the novelty (Reviewers Jsg6, 4eyk), the clarity and presentation of our writing (Reviewers iiaB, mDev), and the impressive experimental results (Reviewers Jsg6, iiaB, mDev, 4eyk) that demonstrate the benefits and general applicability of our CA-SSLR across multiple tasks and scenarios.

Your time and effort dedicated to improving our work are truly appreciated. We have answered all your questions and addressed the issues in detail in our rebuttal and the latest revision.

These revisions include additional explanations, paragraphs, and equations to help readers understand the proposed method and additional experiments to highlight its advantages.

Most importantly, we have added new results from the Houlsby adapter experiments further to illustrate CA-SSLR's impact on adaptation and generalization abilities (Reviewer xgMo), as well as results for the MMS-1b SSL model to update the current SOTA baseline (Reviewer iiaB). This response offers a high-level overview of these revisions for the convenience of reviewers and future readers.

Major revisions include:

- New Experiment Results in Sec. 5.2 and Sec. 5.4: A new experimental study of the **Houlsby adapter model** (Reviewer xgMo).
- New Experiment Results in Sec. 5.3 and Sec. 5.4: A **new SOTA baseline** of the MMS-1b SSL model and CA-SSLRL,S$^{L,S}$(TCAC) (Reviewers iiaB, 4eyk).
- Improved Writing in Sec. 3: Added **system figure and equations for TCAC** and removed duplicate sentences (Reviewers Jsg6, xgMo).

Minor revisions include:
- A detailed table of the extended few-shot settings is included in the appendix (Reviewer 4eyk).
- An analysis of training speed and peak memory usage was added in the appendix (Reviewer mDeV).


We hope these revisions address your concerns and improve the overall quality of our paper.

Thank you again for your review!

Best regards,

Authors

Meta-review: The reviewers commend the paper for conducting multiple experiments that effectively demonstrate the benefits of the proposed method. They highlight the introduction of a novel conditioning technique for learning speech representations in a self-supervised manner. The paper is praised for being well-written, easy to follow, and presenting an interesting and effective idea. The experiments are described as thorough and convincing, with a well-presented framework. The paper shows relatively strong results in adapting self-supervised learning (SSL) models for multi-lingual automatic speech recognition (ASR), which could inspire more SSL-based methods in multi-lingual scenarios.
Decision: Accept (poster)