CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing

Authors: yen-ju  lu
jing  liu
thomas  thebaud
laureano  moro-velazquez
ariya  rastrow
najim  dehak
jesus  villalba
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 07:05:07

Abstract: We introduce Condition-Aware Self-Supervised Learning Representation (CA-SSLR), a generalist conditioning model broadly applicable to various speech-processing tasks. Compared to standard fine-tuning methods that optimize for downstream models, CA-SSLR integrates language and speaker embeddings from earlier layers, making the SSL model aware of the current language and speaker context.
This approach reduces the reliance on the input audio features while preserving the integrity of the base SSLR. CA-SSLR improves the model’s capabilities and demonstrates its generality on unseen tasks with minimal task-specific tuning. Our method employs linear modulation to dynamically adjust internal representations, enabling fine-grained adaptability without significantly altering the original model behavior. Experiments show that CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and excels in under-resourced and unseen tasks. Specifically, CA-SSLR achieves a 10\% relative reduction in LID errors, a 37\% improvement in ASR CER on the ML-SUPERB benchmark, and a 27\% decrease in SV EER on VoxCeleb-1, demonstrating its effectiveness.

Main paper: 
 introduction: 
 The emergence of Self-Supervised Learning Representations (SSLRs) models has revolutionized speech processing, setting new standards in the field. Pioneering models like Wav2vec 2.0 Baevski et al. [2020] , HuBERT [Hsu et al., 2021] , and WavLM [Chen et al., 2022a] leverage unlabeled audio data to learn rich representations of spoken language. These models are pivotal in a wide range of applications, including Speech Recognition (ASR) [Chang et al., 2021] , Speaker Verification (SV) [Chen et al., 2022b , Tak et al., 2022] , Language Identification (LID) [Bartley et al., 2023] , and Speech Translation (ST) [Tang et al., 2022] . Benchmarks such as SUPERB [Yang et al., 2021] and ML-SUPERB [Shi et al., 2023a] have been crucial in evaluating SSL model performance, providing standardized tasks.Although SSLR training approaches combine speech from various sources, these models learn representations solely from unpaired audio-only data. When extending SSLR features to multilingual scenarios and low-resource languages, unsupervised training limits the model's ability to distinguish between different languages, resulting in unified features for all languages. Additionally, labeling all SSL training data with language and speaker information requires significant human effort and is impractical. Thus, a post-training conditioning approach is more favorable. In other fields, methods like [Zhang et al., 2023] and IP-Adaptor [Ye et al., 2023] in image processing, and CTRL [Keskar et al., 2019] in NLP, have successfully integrated conditioning into pretrained models, demonstrating potential applications for speech processing.In response to these challenges, we propose Condition-Aware Self-Supervised Learning Representation (CA-SSLR), a generalist conditioning model applicable to various speech-processing tasks such as language identification, multilingual speech recognition, and speaker verification. Unlike standard adaptation methods that optimize the SSLR parameters for downstream models, CA-SSLR integrates language and speaker embeddings from earlier layers, making the SSLR aware of the current language and speaker context. This technique enables the creation of models that perform multiple tasks with a single adapted SSL encoder by strategically injecting conditional adapters into each encoder block while freezing the pretrained encoder weights. CA-SSLR follows a hierarchical self-adaptation structure, where adapters at each layer are conditioned on intermediate task-specific embeddings estimated from lower layers. Attention mechanisms and linear modulation dynamically adjust scaling and biasing, tailoring the model's response at each time step. The initialization techniques allow the conditioning module to perform identity transformations, ensuring the existing model behavior is maintained when incorporating new conditions. This approach reduces the number of trainable parameters, mitigates overfitting, and avoids catastrophic forgetting. We conduct experiments on three popular types of multilingual speech processing tasks-ASR, LID, and SV to demonstrate the versatility and efficiency of CA-SSLR. This work's main contribution is introducing a novel method for conditioning the SSLRs with limited supervised labels. This leads to generalized speech representation with improved performance using minimal trainable parameters and maintains the model's behavior. This includes:• Hierarchical Dynamic Conditioning: We design attention-based conditional adapters and integrate them into the SSL model. Our approach dynamically tailors the model's behavior to the input language and speaker characteristics at each time step, which are periodically estimated from previous layers. • Preservation of Pre-trained Weights with Efficient Parameter Utilization: The model capitalizes on the knowledge of the foundational model pre-trained weights and introduces lightweight adapters that modulate the encoder hidden representation by a scalar γ and bias β, significantly reducing the trainable parameters. This strategy ensures more stable and parameter-efficient training. • Harmonized Task Compatibility with Notable Performance Improvements: Our experiments show that CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and excels in under-resourced and unseen tasks. Specifically, CA-SSLR achieves an 27% relative reduction in LID errors, a 37% improvement in ASR CER on the ML-SUPERB benchmark, and a 27% decrease in SV EER on VoxCeleb-1. These results highlight CA-SSLR's effectiveness in enhancing multilingual SSLRs while also lowering computational costs for multitask fine-tuning. 
related work: 
 Self-supervised learning representation. Self-Supervised Learning (SSL) models, epitomized by Wav2Vec 2.0 [Baevski et al., 2020] , HuBERT [Hsu et al., 2021] , and WavLM [Chen et al., 2022a] , have significantly advanced speech processing by leveraging vast amounts of unlabeled audio data. These models excel in extracting rich speech representations, capturing its intricate acoustic, phonetic, and semantic nuances. These models are fine-tuned on smaller, labeled datasets to adapt the generic representations for specific tasks, achieving impressive results.In the realm of cross-lingual speech representation, Wav2Vec 2.0-XLSR (Cross-Lingual Speech Representation) [Babu et al., 2021] takes a significant leap forward. It builds on the robust architecture of Wav2Vec 2.0 but is pre-trained on a diverse, multilingual dataset, learning universal representations transferable across languages. Similarly, mHuBERT (Multilingual HuBERT) [Lee et al., 2021] extends the foundational HuBERT model to process multiple languages effectively. This makes them immensely powerful for multilingual speech recognition and understanding tasks. The benchmark for the SSL models also extends from monolingual SUPERB [Yang et al., 2021] to multilingual ML-SUPERB [Shi et al., 2023a] , building new standards for the SSLR models.Adaptation Methods. In many studies Yang et al. [2021] , Shi et al. [2023a] , Chen et al. [2023a] , the SSLR remains frozen while decoders are trained for a specific task. Since the encoder is shared across all tasks, this approach offers the advantage that it allows us to evaluate multiple tasks on a given speech signal with just one encoder run. However, systems of this kind often exhibit poorer performance when compared to those incorporating some degree of adaptation of the SSLR to the target task. The latter can involve fine-tuning the entire SSL encoder Chen et al. [2022a] , a subset of its layers, or introducing lightweight adapters [Chen et al., 2023b] within its layers. Unfortunately, this results in employing a distinct encoder per task, leading to a large increase in computational load that scales linearly with the number of tasks to be assessed.Conditioning Pre-trained Models. Image processing has successfully integrated conditioning into pretrained models using methods like ControlNet [Zhang et al., 2023] and IP-Adaptor [Ye et al., 2023] . ControlNet allows for precise control over generated images by incorporating additional conditions such as edge maps or sketches, while IP-Adaptor uses small-scale adapter modules to adjust the model's behavior based on specific conditions without altering the pre-trained model's parameters. These techniques have achieved significant success and offer insights for potential applications in speech processing. Similarly, in Natural Language Processing (NLP), models like the Conditional Transformer Language Model (CTRL) [Keskar et al., 2019] have introduced conditioning to improve model performance. CTRL uses control codes to guide text generation based on specified attributes like style or domain, allowing for efficient adaptation without extensive retraining. The successes in image processing and NLP highlight the potential for conditioning pre-trained SSLRs in speech processing.Hierarchical Conditioning. Hierarchical models have been used in previous speech models. [Sanabria and Metze, 2018] proposes a multi-task ASR model that improves intermediate representations by performing Connectionist Temporal Classification at different levels of the network with targets of different granularity. Essentially, representations in lower layers are used to predict character tokens, while higher layers predict subword units with growing vocabulary sizes-from 300 to 10k subword units in the last layer. [Chen et al., 2023a] further explored this by integrating hierarchical conditional layers within the ASR decoder, using ASR tokens predicted from preceding layers to inform subsequent layers. 
methodology: 
 We propose Condition-Aware SSLR (CA-SSLR), designed to serve as a universal encoder for multiple downstream speech tasks. CA-SSLR enhances pre-trained SSL models by integrating intermediate LID and SV predictions to condition and adapt subsequent layers dynamically. This approach allows the model to capture essential language and speaker characteristics, refining its outputs progressively and making it particularly effective in multilingual and multispeaker scenarios. modulates the hidden representations of the SSL encoder layers based on conditioning features derived from intermediate LID and SV embeddings. This hierarchical conditioning mechanism enables the model to adapt dynamically to different input conditions while keeping the majority of the pre-trained parameters fixed. In the following sections, we detail the components of CA-SSLR, starting with the conditioner module and then explaining how it integrates into the overall architecture. We also describe the incremental training strategy employed to incorporate conditioning information without catastrophic forgetting.A central component of CA-SSLR is the channel-wise conditioner (CC) or the time-channel attention conditioners (TCAC), which modulates the SSL encoder's hidden representations based on conditioning features. As depicted in Fig. 1b , the TCAC tasks the latent representations S (l) ∈ R C×T from layer l of the SSL encoder and a conditioning feature vector z ∈ R R , derived from intermediate LID or SV embeddings. The TCAC outputs modulated latent representations S(l) by applying time-channel-dependent scaling and bias:EQUATIONwhere t and c index time and channel dimensions, respectively. The scales γ(l) t,c and biases β(l) t,c are products of time-dependent and channel-dependent components:EQUATIONThe channel-dependent scales γ (l) ∈ R C and biases β (l) ∈ R C are computed via linear transformations of the conditioning feature, similar to feature-wise linear modulation [Perez et al., 2018] :γ (l) (z) = W (l) γ z + b (l) γ β (l) (z) = W (l) β z + b (l) βwith .(3)The time-dependent scales α (l) ∈ R T are obtained with an additive attention mechanism asEQUATIONwhere f (.) is a ReLU non-linearity, WEQUATIONThe conditioning feature z is obtained by processing the intermediate embeddings e ∈ R E from the LID or SV decoders, as z = LayerNorm(We + b), where W ∈ R R×E and b ∈ R R are shared linear transformation parameters, and LayerNorm(•) denotes layer normalization. In scenarios where time-based modulation is unnecessary, the model can switch to the simpler Channel-wise Conditioner (CC) by using only the channel-dependent components γ and β. This flexibility in conditioning design enables the model to be tailored to various speech tasks with differing complexity requirements. By integrating these conditioning methods, CA-SSLR dynamically adapts its internal representations based on language and speaker characteristics. This mechanism enables the integration of conditioning into the model's latent representations without altering the pre-trained encoder's original parameters.3.2 Hierarchical Self-Conditioning in CA-SSLR Building upon the TCAC module, CA-SSLR employs a hierarchical self-conditioning mechanism within the SSL encoder layers. As shown in Figure 2 , the SSL encoder is partitioned into layer groups, with TCACs inserted after the attention module in each layer to modulate hidden representations based on updated conditioning features. The model aggregates SSL features through a weighted sum, combining outputs from all preceding layer groups. These aggregated features are then provided to the LID and SV decoders, where LID and SV embeddings are extracted and processed through a linear layer followed by layer normalization to create the conditioning feature z for the TCACs.The conditioning feature z is re-estimated at intervals-every three layers for LID and every six layers for SV-using the aggregated SSL features from previous groups. This hierarchical design progressively refines the model's representations, adapting to the input's language and speaker characteristics at different depths of the network. For example, the initial SSL layer group captures basic language and speaker characteristics, generating embeddings that condition the next group of layers via TCACs. This ongoing refinement allows the model to dynamically adapt based on intermediate predictions, resulting in a context-aware and dynamic representation.Each layer group uses distinct TCAC parameters, enabling tailored scaling and bias adjustments at different stages of the model. Notably, only the TCACs and the linear projections for the decoders are trainable, while all other SSL encoder parameters remain fixed during the conditioning insertion. This design minimizes overfitting and accelerates training due to the smaller number of trainable parameters. This hierarchical self-conditioning mechanism enables the model to dynamically capture diverse aspects of input audio, making it a robust tool for comprehensive speech analysis.Incorporating new components into a pre-trained SSL encoder poses the risk of catastrophic forgetting, where the model loses previously acquired knowledge. To mitigate this, we adopt an incremental training strategy that gradually integrates the conditioning information. We initialize the TCAC parameters to ensure that the initial modulated features are identical to the original SSL features. Specifically, we set the initial values such that α t = 1 for all t, γ c = 1, and β c = 0 for all c. According to Eq. (1), this initialization means that S(l) t,c = S (l)t,c at the start of training, allowing a smooth transition from the pre-trained model to the conditioned model.When multiple conditioning features are involved, such as both LID and SV, we compute separate scaling and bias parameters for each:α total = α LID × α SV , γ total = γ LID × γ SV , β total = β LID + β SV .(5)This approach allows the model to incrementally incorporate additional conditioning tasks without disrupting the knowledge acquired from previous tasks.4 Experimental Setup 
datasets: 
 For the LID and ASR tasks, we utilized the ML-SUPERB benchmark [Shi et al., 2023a] . This corpus comprises two distinct data configurations: 10-minute/language and 1-hour/language for each of the 123 well-represented languages (Normal). Additionally, both configurations include five training utterances for each of 20 selected low-resource languages (Few-shot) 1 . For the Few-shot languages, we also considered an Extended Few-shot condition in which we augmented the amount of data to match that of the Normal languages. However, the Extended Few-shots only included language labels but not ASR transcripts. This aims to analyze the behavior of few-shot languages with improved language ID accuracy, as achieving satisfactory language accuracy with just five utterances is challenging. This is also a reasonable assumption since obtaining data with language labels is easier and cheaper than obtaining transcribed data. The moderate size of this dataset was ideal for testing our approach, as it allowed us to conduct multiple ablation experiments with limited computing resources. As ML-SUPERB lacks speaker labels, we combined it with VoxCeleb2 [Nagrani et al., 2017] for training models incorporating the SV task. VoxCeleb2 contains 1,092 speech hours from 5,994 speakers, although it lacks LID labels and ASR transcripts. The SV task was tested on the VoxCeleb1 original set. The speech was augmented with Musan noise Snyder et al. [2015] and reverberation Ko et al. [2017] during SV training.SSLR Models. In our system, we employed the best multilingual SSL back-bones in the ML-SUPERB benchmark: Wav2Vec2-XLSR with 300M parameters, trained on 128 languages2 , and the 100M parameter multilingual Hubert (mHuBERT) model [Lee et al., 2021] , trained on English, Spanish, French data from VoxPopuli [Wang et al., 2021] unlabeled speech as our foundational acoustic encoders. These models have demonstrated their efficacy in processing a wide range of linguistic inputs and form the backbone of our system. We experimented with the S3PRL [Yang et al., 2021] and ESPnet [Watanabe et al., 2018] toolkits. Our training dataset combined data labeled for ASR+LID labels, LID only, or SV only. Hence, we computed the losses only for the available tasks for each sample. Detailed information on the remaining training hyperparameters is provided in the appendix, and the code will be made available for reproducibility. A model's training takes about one day using 2 A100 GPUs.Speaker and Language Decoders. The speaker and language decoders are based on the ECAPA-TDNN architecture [Desplanques et al., 2020] . Initially, a convolutional layer projects SSL representation to the decoder dimension (512 for LID and 1024 for SV). This is followed by a sequence of 1-dimensional SE-Res2Net [Gao et al., 2021] layers (one for LID and three for SV). Next, channelwise attentive statistic pooling aggregates the frame-level features into a single utterance-level vector, which is projected into lower-dimensional speaker embedding. The training loss was Additive Angular margin-softmax [Deng et al., 2019] with margin=0.3 for SV and margin=0.0 for LID. Large margin helps to create highly compact speaker representations [Villalba et al., 2022] , while being detrimental in LID [Villalba et al., 2023] . The SV and LID decoders producing the final result consume a weighted average of all SSL layers. Meanwhile, the ones estimating the conditioning embeddings use a weighted average of the SSL layers evaluated up to that point in the chain. Note that all SV and LID decoders share parameters, so the number of trainable parameters remains independent of the frequency with which we re-compute the conditioning embeddings.ASR decoder. The ASR decoder conforms to the framework set by the ML-SUPERB benchmark [Shi et al., 2023a] , facilitating comparable evaluations. A convolutional downsampling layer halves the SSL feature sequence duration. These features are channeled into a two-layer Transformer with 256-dim self-attention, eight attention heads, and 1024-dim feed-forward layers. A linear output layer with connectionist temporal classification (CTC) loss predicts multi-lingual character-level tokens.5 Experiments and Results 
generalization ability on unseen tasks: 
 Experiment Setting. We conducted experiments to evaluate the generalization capabilities of the adapted SSLR models on LID, ASR, and SV tasks. The SSLR models were adapted for one task (either LID or ASR) and then evaluated on both the adapted task and an unseen task. For LID adaptation, the SSLR was trained exclusively with LID labels. We compared three setups: full fine-tuning (LID-FT), Houlsby adaptors Houlsby et al. [2019] (LID-Houlsby), and our proposed condition-aware approach (LID-CA-XLSR L dual ). In this setup, we employed an additional LID decoder using the pre-trained SSLR to pre-generate language embeddings, which were then used to condition the SSLR model for a second inference pass. For ASR adaptation, the models were trained with ASR loss using three setups: full fine-tuning (ASR-FT), Houlsby adaptors (ASR-Houlsby), and our Table 1 : Evaluation of adapted XLSR models on the 10-min ML-SUPERB and VoxCeleb dataset for LID, ASR, and SV tasks. These evaluations test the encoder's generalizability across different tasks, demonstrating effectiveness without further task-specific tuning.(a) LID-adapted XLSR models evaluated on LID and ASR tasks.Adapted.Dims. Results. In LID adaptation (Table 1a ), both LID-FT and LID-Houlsby improved LID performance compared to the pre-trained SSL baseline. However, on the unseen ASR task, the fully fine-tuned SSLR encoder improved ASR CER by only 2%, while LID-Houlsby showed limited generalization, with CER improvements of 5.4% and 3.9% for normal and few-shot languages, respectively. Our LID-CA-XLSR L dual method achieved significantly better generalization, improving ASR CER by 7.3% and 6.6% for normal and few-shot languages. In ASR adaptation (Table 1b ), all models enhanced ASR performance, but ASR-Houlsby and full fine-tuning degraded SV performance relative to the baseline, highlighting their limited generalization. Our ASR-CA-XLSR L approach not only preserved but improved SV performance, reducing EER by relative 10.9% and DCF by 5.4%, showcasing strong generalization to the unseen SV task. These results demonstrate that CA-SSLR significantly outperforms full fine-tuning and standard adaptation methods in terms of generalization. By effectively leveraging conditioning information, CA-SSLR adapts across tasks while maintaining performance on unseen ones. Our proposed conditioner offers both robust adaptations on training tasks and superior generalization, making CA-SSLR a versatile and effective solution for multilingual and multispeaker speech processing.Experiment Setting. Table 2 investigates the CA-SSLR approach with hierarchical language conditioning. The first block of the table refers to the baseline where the foundational models are frozen, while the second block (CA-XLSR L dual ) utilizes a separate task-specific LID model to pre-generate the language embedding. The third block presents our proposed approach, where we re-estimate the language embedding every fourth or third layer (CA-XLSR L (4L, 3L)) within the XLSR model, not required a separate LID system. The experiments utilized two types of conditioners: TCAC, which incorporates attention, and a variant without attention-referred to as Channel-wise Conditioners (CC)-where the same scale and bias are applied uniformly across all time frames. The real-time factors (RTF) as proc-time/signal-length are provided for assessing efficiency3 . Results. First, we observed that both CA-XLSR L dual and CA-XLSR L systems with TCAC (with attention) generally performed better than the CC (w/o attention) counterparts, reaffirming the benefits of the time-wise attention design. In the second block, CA-XLSR L dual slightly outperformed CA-XLSR L in terms of CER for both the 10-minute and 1-hour datasets. However, its real-time factor (RTF) is akin to the combined RTFs of separate LID and ASR models since it runs Wav2Vec2 twice-once for language embedding extraction and again for ASR conditioning-posing challenges for streaming applications. On the other hand, CA-XLSR L (CC, 3L) excelled among the three approaches, achieving a 35.9% and 19.0% relative improvement in Normal and few-shot languages, respectively, compared to the baseline in the 10-minute setup, and 33.5% and 19.8% in the 1-hour setup. LID accuracy remained comparable among the various CA-XLSR models, with a notable performance improvement from 90.9% to 93.4% in 1-hour setup.Experiment Setting. Table 3 presents results for general CA-SSLR models that combine Multilingual ASR, LID, and SV tasks. The table compares the baselines, with frozen and fine-tuned SSL models, to two different CA-SSLR Hierarchical models (CA-SSLR L and CA-SSLR L,S ). We further include another well-known multilingual SSLR model, mHuBERT, for a comprehensive comparison. The LID conditioning systems (CA-SSLR L ) are the same as from the previous section, conditioning the SSL model only on LID embeddings, with the SV decoder added on top of SSL features without further adaptation. The LID + SV conditioning system (CA-SSLR L,S ) combines both LID and SV embeddings and is jointly trained on ASR, SV, and LID losses. The intermediate LID embeddings were recomputed every three layers as the best configuration in Table 2 , and SV embeddings were recomputed every six SSL layers. Apart from ASR CER and LID Acc on ML-SUPERB, we present SV equal error rates (EER) and detection cost function (DCF), measured at target prior probability p = 0.05 [Sadjadi et al., 2022] , on VoxCeleb1. SV performance varied depending on whether we trained the model combining 10min ML-SUPERB + VoxCeleb2 or 1h ML-SUPERB + VoxCeleb2.Fine-tuning Baseline. In the fully fine-tuning experiment, we initialized the model with pretrained ASR, LID, and SV decoders and fine-tuned for a few epochs. However, this approach resulted in suboptimal performance compared to the frozen SSLR baseline. The "FT" experiments showed degraded performance, with LID accuracy decreasing by 5.7%, ASR CER increasing by 3.1%, and SV EER worsening by 4.2 in absolute values on average across the four settings. This decline is unexpected, as fine-tuning typically improves performance. This suggests that simultaneous adaptation of the SSL layers to multiple tasks causes conflicting adjustments, reducing the model's robustness. Consequently, catastrophic forgetting led to worse performance compared to the baseline. Conversely, the condition-aware SSLR models exhibited superior performance comparing with the frozen baseline, indicating that training the inserted condition layers does not alter the model's behavior for downstream tasks but improves its ability to represent the input speech data.Language Conditioned SSLR. CA-SSLR L (CC) notably enhanced SV performance w.r.t. the baseline, despite its encoder being solely tuned for ASR and LID tasks. For XLSR, the EER improved by 14% and 20% relative for the 10-min. and 1-h. sets, respectively, while DCF improved by 16-18%. Similarly, for mHuBERT, we observed comparable enhancements, with the EER improving by 17% in both sets and the DCF improving by 17-18%. This demonstrates that the CA-SSLR approach offers superior generalization capabilities compared to the original pre-trained SSL encoder, delivering improved performance. CA-SSLR L (TCAC) performance in SV is comparable to its non-attention counterpart with better performance in LID and ASR as discussed in Sec.5.2.Language and Speaker Conditioned SSLR. Adding a speaker conditioner to CA-SSLR L,S further improved its performance. In ASR tasks, incorporating the speaker conditioner to CA-XLSR L,S reduced CER by 3.1% for the 10-min. set and 6.2% for the 1-hr set, relative to CA-XLSR L . For LID task, CA-SSLR L,S shows similar performance to other models with relative differences below 3%. For SV, CA-XLSR L,S using channel-wise conditioner (CC) reduced EER by 19.4-27.1%, outperforming CA-XLSR L . Switching from CC to TCAC yielded additional gains in ASR, adding a relative improvement of 2.7-4.0%. In contrast, its impact on SV was more modest, with improvements in EER by 14.0-21.7%. Overall, TCAC demonstrated better adaptation ability, while CC excelled in generalization.ASR and RTF Discussion. Generally, we observed the largest improvement for ASR when including the language conditioner, as it enables the system to adapt to produce output tokens in the correct language. Conversely, adapting the model to the input speaker provided fewer ASR gains. The XLSR model benefitted from our approach better than mHuBERT, possibly because mHuBERT is 3× smaller than XLSR, but more importantly, because mHuBERT was trained on just four languages compared to 128 in XLSR. Therefore, the pre-trained mHuBERT has not encountered enough diversity in terms of languages and speakers, thereby limiting its performance in multi-lingual ASR and SV. In terms of RTF, while the conditioned models are 13-34% slower compared to sharing the pre-trained SSL encoder for the three tasks, both CA-SSLR L and CA-SSLR L,S offer superior performance while being much faster than running task-specific models separately, indicating a more efficient use of computational resources while running the generalist model.Ablation study of Conditioning Approach. Table 4 conducts an ablation study for different conditioning methods with CA-XLSR L dual settings within the ML-SUPERB 10min dataset regarding ASR CER. First, we used conditioners without attention (CC) on the ground truth LID predictions (G.T.), serving as the upper bound for the performance of our proposed approach. This improved the Normal languages from 29.0% to 17.2%, and Few-shot languages from 39.0% to 27.9%, w.r.t. the pre-trained XLSR model. This showcases the potential of the condition-aware SSLR. Following, we compared conditioning on hard-predicted language labels (Hard), soft-predicted language labels (Soft), and language embeddings from the LID decoder bottleneck layer (Embed) for comparison. Conditioning on Hard LID labels improved the most in Few-shot languages, improving by 26% relative to the baseline. On the other hand, the Embed case outperformed the Soft case and provided balanced performance for both Normals and Few-shots languages. Additionally, we compared CC to Table 4 : Ablation study of condition-aware settings for ASR-adapted XLSR models on 10-min ML-SUPERB dataset, using CC or TCAC. Conditioning is based on predicted language labels or LID embeddings, except in the ground truth (G.T.) experiment. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Since the experiments are conducted with ESPnet and S3Prl, it is harder to submit the whole package for the review, but we promise to release our branch after the review process.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 This paper introduces the CA-SSLR framework, an innovative approach that integrates conditioning into pre-trained Self-Supervised Learning (SSL) models by adapting only the trainable conditioner. Through a hierarchical self-conditioning mechanism, where intermediate language and speaker features condition the upper layers of the SSL model, CA-SSLR achieves a 33% reduction in Character Error Rate compared to the pre-trained baseline, matching the performance of single-task fully fine-tuned models. Additionally, it improves Speaker Verification EER by 27% and reduce Language Identification errors by relative 10% in average. The results indicate that condition-aware SSLR models enhance the model's interpretation of input speech data, providing superior performance compared to traditional fine-tuning methods. This improvement is achieved by dynamically tailoring the model's response to the input language and speaker characteristics, ensuring robust generalization across various tasks. In summary, CA-SSLR offers a versatile and efficient approach to integrating conditioning information into pre-trained models. This method not only enhances performance across multiple tasks but also ensures efficient parameter utilization, supported by an improved RTF that facilitates its application in real-world scenarios.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We detail the dataset settings in Sec. 4.1, and Sec. 4.2 describes the model architecture. The detail parameters settings is described in Sec. A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: In Appendix A.2, we report the error bars results while running the experiments different times. Also, we conduct different experiments with slightly different settings, and provide consistant ASR and SV improvements, where the variation among these experiments is much smaller than the improvement comparing with the baseline.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We discuss the computation resource for training in 4.2, and RTF in Sec.5.2 and Sec.5.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We include the code of ethics in Appendix G.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discussion the broader impacts in Sec.6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The CA-SSLR methodology improves the conditioning of pretrained Self-Supervised Learning (SSL) models for speech processing, improving performance with minimal fine-tuning and reducing computational resource requirements. This advancement facilitates the deployment of robust models in resource-constrained environments, promoting broader access to advanced speech technology. However, there are potential risks. The conditioning mechanisms might amplify biases in the training data, leading to unfair outcomes, particularly for underrepresented languages and speaker groups. Ensuring diverse and balanced datasets, along with continuous monitoring, is crucial to mitigate these risks and prevent perpetuating existing inequities.This appendix provides detailed configurations and hyper-parameters for the decoder models used in our experiments, including ASR, LID, SV decoders, and the CA-SSLR models. The rationale behind specific hyper-parameter choices and architectural details are also discussed to offer insights into the experimental setup and model optimization strategies.A.1 Decoder Models for ASR, LID, and SVThe ASR, LID, and SV decoder models were optimized for their respective tasks through careful selection of hyper-parameters and architectural configurations. The ASR model directly follows the setting in ML-SUPERB benchmark [Shi et al., 2023a] for comparison. Table 6 provides detailed configurations for the CA-SSLR model. In the CA-SSLR L,S setup, two 256dimensional embeddings are used to encapsulate language (L) and speaker (S) information, which then determine the parameters (α L , γ L , β L ) and (α S , γ S , β S ) following the procedure outlined in Eq. 5.The training adopts a stepwise approach, using initial parameters from an earlier phase to set up the next. The pretrained ASR, LID, and SV decoders serve as the foundation for initializing CA-SSLR L ; the SV decoder is fine-tuned further on top of CA-SSLR L ; and both CA-SSLR L and fine-tuned SV decoder initialize CA-SSLR L,S . In the table's "Trainable modules" row, the notations LID Feat and SV Feat indicate that the feature projection layers of the LID and SV decoders are adjustable during the training process. We conduct the in Table 6 and Figure 3 multiple times, and the variation are all within 0.2% CERs range.The ML-SUPERB dataset is assembled from a wide collection of multilingual speech corpora, with each contributing corpus being governed by one of a variety of open-source licenses, such as Creative Commons, MIT, GNU, or Free-BSD. These licensing agreements guarantee that the dataset is openly available and can be used freely for both commercial and scholarly research purposes. The 10-minute training set encompasses 37.4 hours of data, and the 1-hour dataset increases the total to 222.4 hours of data. Additionally, the dataset includes development and testing sets, containing 41.8 hours and 45.0 hours of data, respectively. This dataset is designed for multilingual speech recognition and language identification, as used in our work.In the original ML-SUPERB settings, there are two types of languages:• Normal Languages: Each has 10 minutes or 1 hour of data per language, used for both LID and ASR training with transcriptions.• Few-Shot Languages: Each has only 5 utterances. In the original settings, these languages are not presented in the results for LID training and are used for ASR training with available transcriptions.For the extended few-shot condition, we incorporate the language labels from these few-shot data for LID training but continue using only 5 utterances with transcriptions for ASR training. Since language labels are more accessible than transcriptions, especially in low-resource scenarios. Table 7 summarizes the data configurations for the original and extended few-shot conditions. Table 8 compares the mHuBERT model's ASR performance against the number of trainable parameters, where the XLSR counterpart is shown in Fig. 3 . Both CA-mHubert L dual and CA-mHubert L,S dual are with 256 condition feature dimensions. Notably, the CA-mHubert L dual model excels in few-shots scenarios, while the CA-mHubert L,S dual yields CERs for normal languages comparable to a fully fine-tuned 12-layer mHuBERT model using only 15.9M parameters. This efficiency demonstrates the TCA conditioner's capability in the CA-SSLR framework to deliver fine-tuned levels of ASR accuracy with a significantly reduced parameter count, providing an optimal balance for practical ASR applications. We compare the training speed and resource consumption of different adaptation methods, including Houlsby Adapters, CA-SSLR, and full fine-tuning (FT). Table 9 summarizes the bottleneck dimensions, training times, and peak memory usage for each method. We evaluate the training speech for 10k iterations with batch size 8. We found that the CA-SSLR approach ranks second compared to the Houlsby Adapter and a fully fine-tuning approach in speed and memory usage. However, it is important to note that CA-SSLR surpasses the Houlsby Adapter in adaptation effectiveness and generalization ability, as demonstrated in Table 1 . These results indicate that although CA-SSLR incurs a moderate increase in training resources, it provides benefits in performance and generalization. We acknowledge that the current implementation of CA-SSLR is not yet optimized for speed and memory efficiency. Future work will focus on optimizing the model to reduce training time and memory consumption without compromising performance. Tables 10 and 11 present the real-time factor (RTF) for each individual component, as well as for the combined systems. In Table 11 , the term "separated tasks" refers to duplicating and fine-tuning the SSLR for each task individually, along with the corresponding total RTF.Within the ML-SUPERB dataset's 20 few-shots languages, we examined the performance of CA-SSLR against the established SSL baselines, XLSR and mHuBERT, on models trained in the 10-minute ML-SUPERB set. The LID results indicate a close match between CA-SSLR and the baseline, with approximately half of the few-shot languages exhibiting improvements or matching their baseline performance. Section 5.4 reveals that SSL-based LID models are inherently effective, and extending full fine-tuning does not necessarily enhance results. This observation aligns with the outcomes of other classification tasks adeptly handled by SSL models, as documented in [Chen et al., 2023b] . Furthermore, the CA-SSLR framework demonstrates subtle enhancements for the Normal languages in the 10-minute set in Regarding the ASR results, most languages achieved significant CER reductions, ranging from a modest few percent to over 30%, when compared with SSL baselines. Notably, the Bosnian (bos) language experienced an impressive 45.1% relative improvement in CER, while Cebuano (ceb) improved by 39.5% with the XLSR model. With the mHuBERT model, the most substantial gains were observed in Sundanese (sun) and Toki Pona (took), with 19.9% and 17.2% CER relative improvements, respectively. These results underscore the CA-SSLR framework's profound effect in bolstering ASR performance for few-shot languages, especially demonstrating more pronounced improvements with the XLSR model.When examining the correlation between LID accuracy and ASR performance, it is apparent that a lower CER does not necessarily align with high LID accuracy. For instance, Serbian (srp) on the XLSR model, despite having a modest LID accuracy of 50.9%, shows a CER improvement from 57.4% to 48.1%. Conversely, Fulah (ful), the sole language to exhibit a CER increase in the XLSR model, presents a higher LID accuracy of 67.5%. This indicates that the CA-SSLR framework's efficacy is not solely contingent on high LID prediction accuracy. CA-SSLR's reliance on embeddings instead of one-hot hard labels for predictions enables the model to maintain or improve ASR performance despite suboptimal LID scores. This approach allows the model to utilize embeddings to distinguish between easily confused languages, enabling the ASR model to predict the correct language accurately.Table 13 visualizes ASR outcomes for the XLSR and CA-SSLR L,S models on the ML-SUPERB 10-minute dataset, covering both few-shot and standard language scenarios. It highlights CA-SSLR's superior language recognition capabilities and success in rectifying the misclassifications encountered with XLSR, often resulting in completely incorrect transcriptions. This is evident in languages such as Lithuanian and Turkish, categorized as few-shot, and Bulgarian, which is better resourced (normal). These findings demonstrate the TCA conditioner's effectiveness in accurately managing LID embedding features and distinguishing between languages for downstream tasks.Moreover, the results from other samples suggest that CA-SSLR can achieve better outcomes during training due to its incorporation of language information, even when the XLSR model correctly predicts the language. This underscores the efficacy of the TCA conditioner in exploiting languagespecific data, thereby enabling CA-SSLR to achieve heightened accuracy across a diverse range of languages.We affirm our commitment to ethical research practices, including respect for privacy and the responsible use of data. The proposed CA-SSLR model improves multi-lingual ASR in 143 languages, including 20 low-resource ones with just five training utterances. In this manner, CA-SSLR contributes to the democratization of speech technology, fostering inclusivity for previously underserved communities. Furthermore, CA-SSLR prioritizes the reduction of computational costs at evaluation time, thereby aiming to mitigate the environmental impact associated with speech applications. We utilized publicly available datasets, namely ML-SUPERB and VoxCeleb, chosen for their moderate size to minimize computing requirements. Our utilization of pre-trained models, specifically XLSR and mHuBERT, aligns with their intended research purposes, as they are widely used within the speech research community.However, the capacity for conducting speech and speaker recognition in human conversations poses a notable ethical concern linked to covert eavesdropping by nefarious entities. This capability could be exploited by authoritarian government bodies seeking to suppress free speech and identify dissidents. Therefore, it is imperative to promote public awareness and comprehension regarding the automated analysis of spontaneous speech and its ramifications. • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.We discovered that a portion of the few-shot Lithuanian (lit) training and testing data was erroneously substituted with Italian (it), leading us to omit the Lithuanian outcomes from the evaluation.https://huggingface.co/facebook/wav2vec2-xls-r-300mThe RTFs are computed on an NVIDIA T4 GPU. 


Decision: The reviewers commend the paper for conducting multiple experiments that effectively demonstrate the benefits of the proposed method. They highlight the introduction of a novel conditioning technique for learning speech representations in a self-supervised manner. The paper is praised for being well-written, easy to follow, and presenting an interesting and effective idea. The experiments are described as thorough and convincing, with a well-presented framework. The paper shows relatively strong results in adapting self-supervised learning (SSL) models for multi-lingual automatic speech recognition (ASR), which could inspire more SSL-based methods in multi-lingual scenarios.

Reviews: review: paper_summary: paper_summary: This paper introduces conditioning into self-supervised learning of speech representations. In particular, a hierarchical self-conditioning mechanism is introduced where intermediate language and speaker embeddings are used to condition upper layers. The proposed approach is used together with XSLR and mHubert models and several experiments are conducted using the SUPERB benchmark.
strength_weakness: strength_weakness: I think the main weakness of the paper is that it is too dense. Too much information is presented which makes several sections of the paper hard to follow. For example, it's not very easy to follow section 3. It would help if additional equations are added or a more detailed figure is included. Fig. 1 is too high-level and it's not very informative. 

In addition, some sentences from sections 1 and 2 are literally copied and pasted in section 3. For example, the following sentence can be found both in sections 2 and 3 "Unfortunately, this results in employing a distinct encoder per task, leading to a large increase in computational load that scales linearly with the number of tasks to be assessed.". It would good if the authors rephrase such sentences. There are a few more such examples.
questions: questions: Please see above.
limitations: limitations: Limitations have been addressed.
paper_summary: paper_summary: The paper introduces a method named CA-SSLR, a versatile model for various speech-processing tasks that integrates language and speaker embeddings from earlier layers to reduce reliance on input audio features while preserving the base model's integrity. More specifically, both LID and SID conditioning features are integrated as additional inputs for the SSL model. The proposed approach shows advantages in improving the downstream speech tasks.  ML-SUPERB dataset is used in the evaluation, which shows superior performance over the conventional SSL models.
strength_weakness: strength_weakness: - There is no comparison between the proposed approach and the SOTA numbers on ML-SUPERB. Readers would likely be interested in understanding the gap between the proposed approach and the SOTA numbers.
- It is somewhat unclear when to use the proposed TCAC, as sometimes using CC (without attention) seems sufficient, as shown in Table 2 and Table 4. More analysis should be provided to explain these results, rather than just mentioning them.
questions: questions: 1. The first question relates to the second point in the weaknesses: when is attention necessary? Can we visualize the attention using one or two examples?
2. Which layers are helpful in generating the embeddings? 3L and 4L are reported in the tables; are the shallow layers sufficient?
limitations: limitations: See the Weaknesses.
paper_summary: paper_summary: This paper introduced a framework, CA-SSLR, that integrates conditioning into pre-trained Self-Supervised Learning (SSL) models by adapting only the trainable conditioner. Through a hierarchical self-conditioning mechanism, CA-SSLR can match or achieve better the performance of single-task fully fine-tuned models and benefit from more efficient model tuning. CA-SSLR offers a versatile and efficient approach to integrating conditioning information into pre-trained models.
strength_weakness: strength_weakness: The weaknesses of the papers are that the overall system might be complicated because of performing multiple tasks and some tasks would be dependent on the output of other tasks. As a result, the inference cost would be increased for the subsequent tasks.
questions: questions: Here are some minor concerns to the paper:
1. Is it necessary to provide the rel RTF in Table 3 and 4? I feel like the absolute value would be enough though.

2. It is intuitive that the multilingual ASR model can be improved by using LID as the condition. Is there a cascaded baseline doing LID + ASR that the pretrained model is not fused? The results can help us understand the real effectiveness of the proposed framework.

3. Since the proposed framework achieves better parameter efficient finetuning, is there a training speed or peak memory usage comparison  between the proposed method and full finetuning? How about its efficiency compared to other PEFT methods, like LoRA?
limitations: limitations: The advantage of the paper is to use less resources than full finetuning but achieve similar performance. However, the inference cost is increased. This can be problematic when the method is used in real applications. The inference cost might weigh more than the training cost when the training cost is affordable. Another limitation is that the framework cannot be used for streaming purposes as mentioned by the authors.
paper_summary: paper_summary: This paper employs multi-task learning with hierarchical conditioning to adapt pre-trained speech SSL models. By utilizing lightweight task-related auxiliary decoders repeatedly at various positions, the method gradually tailors the SSL representations. A time-channel-dependent conditioner is introduced to facilitate the fusion process. By incorporating LID and Speaker SV decoders, the proposed approach achieves significant improvements in multi-lingual ASR, LID, and SV.
strength_weakness: strength_weakness: Although the authors refer to the proposed model as "generalist", it still requires proper selection of conditioning/auxiliary decoders according to the target tasks. In the paper, the major task is multi-lingual ASR and the initial conditioning decoder is a language identifier, which are closely related. As illustrated in Table 4, further incorporating the SV conditioner leads to less performance gain.

A terming issue: the "Time-wise Attention" seems *time-dependent scaling* rather than an "attention mechanism". In Eq. (2), $\alpha_t^{(l)}(\textbf{z}, \textbf{S}^{(l)}) \in \mathbb{R}^T$ is computed frame-wise with a linear projection, and it scales the SSL features at different timesteps without interactions among frames/timesteps.
questions: questions: (Line 144-145) It states "... the TCAC, SV, and LID decoders are trained...". But in Figure 1, the two decoders look frozen and the caption confirms this. Which is the actual setting?

(Line 163) It states that the output projection layers of conditioners are all shared. But as stated at line 221, each auxiliary decoder, at different points, takes the previous SSL representations using a weight average. Are those weights and input projection layers partially shared across positions, or totally position-dependent?

(Line 188) What augmentation is leveraged in the "Extended Few-shot condition"?

(Line 236) Which model does "the pre-trained and fixed SSLR model" refer to? If it refers to "XLS-R" in Table 1, then it is a bit confusing because a prediction head is added and the SSLR model is not purely pre-trained.

(Line 251) The conditioner takes $z \in \mathbb{R}^{C_E}$ as input, then how is it adpted to consume LID labels? Some illustrations here would avoid confusion.

(Table 4) Why is $\text{CA-SSLR}^{L,S}(\text{TCAC})$ not compared? As Eq. (3) mentions the adaptation method of this "full configuration", it is surprising that it does not appear in the final comparison.

Some personal suggestions/concerns regarding the presentation:

- Line 145: "ASR,a" -> "ASR,"
- Line 163 implies the re-estimation is conducted every three layers for both LID and SV, which conflicts with the actual setting.
- Table 4: the results are derived by *jointly optimization* for all three tasks, including system "**+ FT**", but this operation is not explicitly mentioned in the experiment setting, which could be confusing at first.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This work proposes a method to automatically condition a (frozen) speech foundation model to a particular language and/or speaker. This method consists of 2 parts; they use an ECAPA-TDDN model to compute speaker or language embeddings from a set interval of intermediate layers of the SSL model, and a conditioning layer which uses those embeddings to modify subsequent layer outputs of the SSL model. The authors conduct experiments with the multilingual SUPERB benchmark (for language identification and ASR) and VoxCeleb1 dataset (for speaker recognition).


Edit Rebuttal: Houlsby adapter was addressed, presentation style was addressed but cannot be judged. Updated score from reject to borderline reject.
strength_weakness: strength_weakness: The presentation of this paper is, in my opinion, subpar to what is expected for NeurIPS. Most importantly, I found it difficult to grasp the proposed method from the methodology section. Line 123-131 are copied from the related work section. Line 133 introduces a “hierarchy” of conditioners, but hierarchy is never explained. Then line 140 assumes the reader is very familiar with TCAC while it has not been explained yet. Line 141 to 146 mention a lot of different configurations related to SV and LID decoders, this could be streamlined into a single “standard” setup, while the experimental section could ablate on different settings. Moreover, I think that the speaker decoder story only distracts from the interesting results of this paper (e.g., Table 1, Table 2, Figure 2 do not make use of the speaker decoder at all, so why make the method section hard to understand by including it?). Furthermore, I cannot implement the conditioner (TCAC) from the description in line 156-170. Also, line 221-222 should be in the method section.

In the experiment section, references to the appendix should be made to guide the reader to relevant information regarding experimental details. I cannot judge whether the current experimental setup is fair. 

Furthermore, I think some relevant literature and methodology is missing:

[1] Houlsby, Neil, et al. "Parameter-efficient transfer learning for NLP." International conference on machine learning. PMLR, 2019.     
[2] Thomas, Bethan, Samuel Kessler, and Salah Karout. "Efficient adapter transfer of self-supervised speech models for automatic speech recognition." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.    
[3] Otake, Shinta, Rei Kawakami, and Nakamasa Inoue. "Parameter efficient transfer learning for various speech processing tasks." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.    
[4] Peng, Junyi, et al. "Parameter-efficient transfer learning of pre-trained transformer models for speaker verification using adapters." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.    

I think a comparison with the standard adapter from [1] is required. To keep things as simple as possible, instead of the proposed TCAC, the adapter from [1] could potentially use the ECAPA-TDNN langauge/speaker embedding as a second input.  

I’ll end with listing minor (formatting) issues:
* Line 20: inconsistent citation style   
* Line 34: word ‘ControlNet’ missing   
* Line 37: this challenge instead of these challenges (labeling data is a solution, not a challenge)   
* Line 44,52: CA-SSLR instead of CA-SSL   
* Line 76,78,80,84: hyperbolic language   
* Line 84-86: This sentence does not make sense to me  
* Line 87: inconsistent citation style  
* Figure 1: “Condition-aware” instead of “conditioned-aware"
* Multiple instances where an abbreviation is explained while it has been used previously, e.g. CER in line 249,      
* Figure 2 is unreadable in black and white print, using a different marker for each hue could alleviate this problem.   
* Line 199: inconsistent citation style    
* Line 280: requires instead of required    
* Line 280: Real-Time Factors    
* Most justifications in the paper checklist contain a spelling mistake, are missing, or do not refer correctly. Also, the instruction block was not deleted.
questions: questions: I do not understand the time-channel attention conditioner. Given formula 1, what are \alpha, \gamma, and \beta? Are they a function? In line 164, they are defined as vectors? In line 166, you mention a linear layer to compute attention scores? How are these attention scores used? Do you not use a self-attention layer?   

It is unclear to me how exactly the time-channel attention conditioner is used within the SSL model. Given formula 1, I assume the conditioner is inserted between each wav2vec 2.0 transformer layer?

In section 5.1, does the LID-FT experiment imply 2 fine-tunings, first on LID (with what data?), and then on ASR (with what data?), where the first fine-tuning updates the SSL model, and the second only the ASR decoder?

In section 5.1, what hyperparameters were used for these experiments? Is there a more comprehensive learning rate scan for the LID-TCAC setting compared to the XLSR-R baseline and LID-FT setting? Are they conducted with the same amount of update steps?

In section 5.1, line 236, it states “only TCAC layers are updated”, while I assume the ECAPA-TDNDN language ID decoder was also trained?

In section 5.2, Table 2 states ASR adapted, but aren’t the models adapted to language ID? 

In section 5.2, how do you condition using the ground-truth language ID labels? In the same context, how are hard-predicted and soft-predicted language labels used for conditioning, and where do they come from?

In section 5.2, second paragraph, are the results shown in Figure 2 all your experiments? Regardless, how did you ensure a fair comparison (learning rate scan, number of updates)? Why did you choose to compare with LoRA? You mention that Chen et. al. [2023b] shows marginal improvements for LoRA, but this paper shows that AdapterBias and Houlsby are more effective for ASR?
limitations: limitations: While the proposed method improves performance for the specific condition of the ML-SUPERB benchmark, it might not work as well with unbalanced datasets (e.g., 100 hours of english data language, 1 hour of spanish and mandarin data).

Comments: title: Reply to the rebuttal
comment: Thank you for reviewing our rebuttal and for increasing your score. We appreciate your feedback on the writing and will continue to refine it to ensure clarity and quality.
Thank you for your feedback. We’re glad to hear that your concerns have been addressed. We will definitely include the explanations in the revised version.
Thank you for considering our clarifications and new experiments. We appreciate your adjusted score and will work to further improve the clarity of our presentation.
I have read the rebuttal and based on it and I have increased my score from 5 to 6.
Dear Reviewer 4eyk,

As the discussion between the reviewers and authors is coming to an end, could you please respond to the authors to confirm whether your concerns have been addressed?

Thanks!

AC
Dear Reviewer Jsg6,

As the discussion between the reviewers and authors is coming to an end, could you please respond to the authors to confirm whether your concerns have been addressed?

Thanks!

AC
Thanks for the rebuttal. My concerns have been addressed. If the explanations can be included in the revised version, I am willing to raise my score to 6.
Thanks for these clarifications and new experiments. I've raised my score, incorporating the new experiments comparing to Houlsby, and the inability to grok the updated presentation style fully (due to no fault of your own).
Thank you for your positive feedback and suggestion. We will add the SOTA results along with descriptions to the paper.
Thanks for the answers. I'm suggesting putting the previous SOTA results (with some descriptions) in the paper as a reference for the readers. I'll keep my rating of the paper.