Context-aware Dynamic Pruning for Speech Foundation Models
Authors: Masao Someki
Yifan Peng
Siddhant Arora
Markus Müller
Athanasios Mouchtaris
Grant Strimel
Jing Liu
Shinji Watanabe
Venue: ICLR 2025 Poster (2025)
Keywords: Pruning
Speech Foundation Model
Automatic Speech Recognition
Speech Translation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Foundation models, such as large language models, have achieved remarkable success in natural language processing and are evolving into models capable of handling multiple modalities.
Listening ability, in particular, is crucial for many applications, leading to research on building speech foundation models. However, the high computational cost of these large models presents a significant challenge for real-world applications. Although substantial efforts have been made to reduce computational costs, such as through pruning techniques, the majority of these approaches are applied primarily during the training phase for specific downstream tasks. In this study, we hypothesize that optimal pruned networks may vary based on contextual factors such as speaker characteristics, languages, and tasks. To address this, we propose a dynamic pruning technique that adapts to these contexts during inference without altering the underlying model. We demonstrated that we could successfully reduce inference time by approximately 30\% while maintaining accuracy in multilingual/multi-task scenarios. We also found that the obtained pruned structure offers meaningful interpretations based on the context, e.g., task-related information emerging as the dominant factor for efficient pruning.

Published as a conference paper at ICLR 2025
CONTEXT-AWARE DYNAMIC PRUNING FOR SPEECH FOUNDATION MODELS
Masao Someki, Yifan Peng, Siddhant Arora, Shinji Watanabe Language Technologies Institute Carnegie Mellon University Pinsburgh, PA 15213, USA (msomeki, yi Fanpen, siddhanaa cmu, edu, shinjiwlieee.org
Markus Müller, Thanasis Mouchtaris, Grant Strimel, Jing Liu
Neural Efficiency Science Amazon Artificial General Intelligence
Pittsburgh. PA 15203, USA
{mumarkus, mouchta, gastrime, j j1mk
amazon. amazznn com
ABSTRACT
Foundation models, such as large language models, have achieved remarkable success in natural language processing and are evolving into models capable of handling multiple modalities. Listening ability, in particular, is crucial for many applications, leading research on building speech foundation models. However, the high computational cost of these large models presents significant challenge for real-world applications. Although substantial efforts have been made reduce computational costs, such as through pruning techniques, the majority of these ap- proaches are applied primarily during the training phase for specific downstream tasks. In this study, hypothesize that optimal pruned networks may vary based on contextual factors such as speaker characteristics, languages, and tasks. To ad- dress this, we propose dynamic pruning technique that adapts these contexts during inference without altering the underlying model. We demonstrated that we could successfully reduce inference time by approximately 30% while maintain- ing accuracy in multilingual/multi-tat pruned structure offers meaningful interpretations based the context, e.g., task- related information emerging as the dominant factor for efficient pruning.
scenarios. We also found that the obtained
INTRODUCTION
In recent years, foundation models have achieved remarkable success across various tasks in natural
language processing (OpenAl 2022
2023 Google [203] Metal 2024 Anthropic 2024 Amazon
2024 DeepSeek-Al (025). These Large Language Models (LLMs) have been particularly effective
as multi-modal systems, incorporating modalities such as images and videos (Google 2024 An-
thropic 2024 Amazon 2024). The integration of voice as a modality for communication between
humans and LLMs has also gained traction, leading applications that facilitate interactive conver- sations with LLMs (OpenAl 2024 Défossez et al. 2024). Many studies have explored features to integrate the hearing ability into LLMs employing methods such connecting massive audio en-
coders to L.MSs (Changli al.) 2024 Yuan et al. 2024 Chu et al. 2023 HU et al. 2024 Défosse7)
jet al. 2024) and utilizing large speech-to-text foundation models with powerful multilingual and
multi-task capabilities (Radford al. 2023 Peng al. 2023d Puvvada et al. 2024).
However, this broad support necessitates the training of large-scale models with billions of param- eters, introducing new challenges such increased inference costs. In speech processing, where input sequences tend to be longer than those in language processing, computationally intensive mod- els significantly prolong inference times. To address these challenges, various methodologies have been proposed, including bifocal networks (Macoskey et al. 202Ia), dual-attention architec-
tures (Sahai al. 2023), amortized networks (Macoskey al.
2021b: Xie et al. 2022 Strimel

---

Published as a conference paper at ICLR 2025
et al., 2023), pruning (Fu al.) 2022 Lai et al., 2021 Peng al., 2023a Wang et al. 2023 Ding
et al., 2024). quantization (Nguyen et al. 2020 Ding et al., 2024). distillation (Liu et al., 2021
Chang et al. 2022] 2024 Gandhi et al.) 2023). and combinations of those methods (Peng et al.
2023c). However, these approaches primarily focus on reducing the model size during training.
While these models handle various tasks, it raises fundamental question: is single model optimal structured for all tasks and languages? Different languages and tasks may require unique pruning strategies for effective processing. For example, Chen et al. (2022) highlights that different layers contribute differently depending on the task, indicating that the optimal model structure might vary across tasks. They also showed excellent performance in ASR using the WavLM encoder with only a single linear layer as the decoder, which raises questions about the need for a large-scale decoder ASR systems. Conversely, Peng et al. (2024a) highlight the importance of the decoder network in ST, suggesting that the decoder may play a more critical role in ST compared to ASR. Therefore, we hypothesize that there might be an optimal model structure depending on each task and language combination so that each subnetwork has the potential to perform with comparable accuracy with less inference complexity.
Given this hypothesis, we propose method for dynumically pruning pre-trained foundation model based on the context information, including speech features, language, and task characteristics, en- abling the construction of optimal model architecture tailored to the contextual requirements during inference. Specifically, we train a model that computes module-level masks for each layer in the encoder and decoder networks based the provided context while simultaneously fine-tuning the foundation model. The predicted mask is utilized to determine which modules to activate or skip while maintaining accuracy. By analyzing the pruned network, we offer an interpretation of the importance of the optimal subnetwork within the given contexts.
This paper makes the following key contributions:
1. We propose to apply a novel context-aware pruming technique to each module in a speech foundation model dynamically within multilingual and multi-task scenarios.
2. We were able to reduce inference time by 34.3% without degrading the BLEU scores for the ST task and 28.6% with only 2.8% WER degradation on the ASR task.
3. We conducted a detailed comparative analysis and found that the obtained pruned struc- ture offers meaningful interpretations based on the context, e.g., task-related information emerging as the dominant factor for efficient pruning.
2 RELATED WORK
Pruning techniques are mainly classified into unstructured and structured approaches. The former is technique for deleting individual weights in network (LeCun et al., 1989 Hassibi et al.) 1993
Han et al. 2016b); however, it has a problem of low compatibility with hardware accelerators (Han al. 2016a Liu et al. 2024). This method is further investigated in Appendix[D On the other hand, structured pruning has a more direct benefit in reducing the complexity, which performs pruning on
layer module basis, including filters/layers in CNNs (Wen al. 2016) Li et al. 2017 Alvarez
& Salzmann] 2016 Han et al. 2017) or layer-wise pruning in models (Fan et al.) 2020)
Lee et al.
202 Chen Zhao 2019). Thus, our paper employs structured pruning.
Methods for determining pruning targets include gradient-based techniques GGoo et al. 2016 He
al., 2020 Fu al. 2022 Wen al. 2016) as well magnitude-based approaches for mod-
ules Li et al. 2017 2022). However, these methods typically use a fixed architecture during infer- ence. Addressing this limitation, recent research has focused on implementing efficient inference by dynamically adjusting the computational load during the inference process (Bengio et al. 2016]] Jernite et al. [2017] Bolukbasi al. 2017 Graves 2016). Notably, in the speech domain, numerous studies have explored streaming models to achieve dynamic model structures aimed at speedup (Ma- coskey et al.] 202 b Strimel et al. 2023 Xie et al. 2022 Xu et al. 2023). In this context, Peng al. (2023b); Bittar al. (2024) extended this concept to large-scale Transformer-basee
models,
exploring dynamic layer-wiss wise structural changes to enhance efficiency. In our study, we extend the work of Peng al. (2023b) by utilizing the model structure of a speech foundation model to address multilingual and multi-task scenarios. This extension explores how a large-scale speech foundation model adapts its structure based on context and input condition (Lu et al. 2024), providing insights
2

---

Published as a conference paper at ICLR 2025
Feed-Forward Network
Feed-Forward Network
Source-
Merge
attention
Global Gate
Predictor
glob-ATT
ogMLP
self attention
Global Predictor
Globel
Foed-Forward Network
French : [1, 0, 0]
Encoder
German : [0, 1, 0]
Italian: : [0, 0, 0]
Mean
Mean
Language
Task
Downsammled
Language
Task
Downeampled
<fra><st_den [1, 0, 0, 1,
<fra>>cstita>> [1, 0, 0, 1]
audio feature
audio feature
(a) E-Branchformer
(b) TransformerDecoder
(c) Language/Task token
Figure The sparse E-Branchformer and Transformer architectures in the experiment, and the method for emhedding the language/task information. The audio information and the language/task information are concatenated, and the Gate Predictor calculates the gate probability for each module in each layer.
into more efficient and context-aware (Chang et al. 2021 Sathyendra et al. 2022 speech processing systems.
METHODS
3.1 OPEN WHISPER- STYLE SPBECH MODELS
In this study, utilized Open Whisper-Style Speech Models (OWSM) (Peng et al. 2023d) as the foundation for our speech model experiments. OWSM an open-source reproduction of Ope- nAI's Whisper model (Radford al. 2023). Among the available versions, selected OWSM- v3.1 (Peng et al.) 2024b) as the speech foundation model for our experiments. The key rationale for selecting SWSM models lies in its fully open training data, processes, and configuration com- pared to Whisper. We ensure that the data used for validation in our experiments is not part of the pre-training corpus. This transparency is critical. If validation data was included in the pre-training data, it could inflate post-pruning accuracy, hindering a proper evaluation of the pruned network's performance.
Additionally, the E-Branchformer architecture integrated into OWSM-v3.1 offers a more flexible and generalized structure compared the conformer, due its parallel design. The model employs dual-branch structure: one branch extracts global context using self-attention based module (glob- ATT), while the other captures local context using a convolution-based module (cgMLP) Sakuma et al. (2022). These branches are merged through convolution-basee merging layer, and are en- closed between two feed-forward networks (FFNI and FFN2). Through the climination of particular modules in the E-Branchformer layers, enable us to thoroughly evaluate the Transformer's 's efficiency and effectiveness as an architectural design.
create model resembling conformer. This will
3.2 MODULE-LEVEL PRUNING
We implement module-level pruning in our study, targeting essential components within foundation models like self-attention (ATT) and feed-forward networks (FFN). For example, in the Transformer architecture, prune the self-ATT, muurceattenntinn (src-ATT), and FFN modules. In the of the E- E-Branchformer, pruning modules include the FFN1, glob-ATT, cgMLP, and the FFN2.
The motivation for adopting module-wise pruming is assumption that the model's architecture should remain flexible and adaptable during inference. Pruning techniques that operate at a finer granularity, such kemel pruning layer pruning, which removes individual kernels or layers
3

---

Published as a conference paper at ICLR 2025
from convolution components, would disrupt the model's structural integrity and limit its ability to dynamically adapt to different audio inputs. While layer-wise pruming aligns with our goal of simpli- fying the model, it oversimplifies the pruned model and prevents us from observing the importance of individual modules. For these reasons, we decided to employ module-wise pruning techniques to balance between structural flexibility and model interpretability. Further considerations layer- skip approaches have been included in the Appendix C for reference.
3.3 PRUNING
Given the dynamic nature of the imput speech, it is necessary to generate a mask for each module based on audio input, language, and task information. To achieve this, we employ neural network model to estimate binary mask that determines whether to use a module. We frame the pruning problem as an LO regularization task [ouizoo et al.) (2018), optimizing the expected value of the binary mask to achieve the desired sparsity. While Louizos et al. (2018) uses a Sigmoid-based approach, we follow Peng et al. (2023b) for implementation efficiency and treat the mask estimation as a two-class classification problem using Gumbel-Softme Jang et al. (2017) for implementation efficiency.
In these works, pruning masks were learned using the sigmoid function or Gumbel-Softmax. How- ever, the masks used during training were continuous values between 0 and 1, rather than strict binary values. As a result, modules that should have been completely skipped during inference were still partially utilized during training. During fine-tuning the OWSM model, we observed that even with a very low temperature for the softmax operation for probabilities, the gate probabilities often remained in the range between 0.4 and 0.6. This led to discrepancy where the output of module was scaled by factor of 0.4 during training, while the same module was entirely skipped during in- ference because the probability fell below the threshold value, such as 0.5. To address this issue, we employed the Straight-through Gumbel-Softmax Estimator (SGSE) et al. (2017) to ensure that the output of the gate predictor was strictly binary. With SGSE, the forward pass computations are performed using binary values, while the backward pass estimates gradients with continuous values, allowing the model to be trained effectively. The detailed formulation is provided in Appendix A
In our work, inspired by Peng et al. (003bb) and Wang et al. 2020 we define the sparsity loss function, Capansity as follows: Expariity a Surgel (g Surget) ))
where g is the average of the gate probabilities for all modules, refers the weight for Lspariiyy and Starget is the desired sparsity ratio for the model. Since we use Gumbel-Softmal to binarize all gate probabilities, g represents the proportion of modules that are activated in the entire model, i.e., the model's 's sparsity ratio. For a detailed derivation of the loss function, please refer to the Appendix E
Additionally, when both the Encoder and Decoder are pruned, we calculate the Csparsity based on the gate probabilities from both components to bring the overall model sparsity closer to Stagett Let Jen: be the gate probability for any module in the encoder, and Jdee be the gate probability for any module in the decoder. Then, the Isparitty is calculated for three scenarios, from top to bottom: first, when only the encoder is pruned; second, when only the decoder is pruned; and third, when both the encoder and decoder are pruned simultancously.
a E Jene]
Susreel
+ a [E[genc)
Starget)
encoder only
Esparsity =
a E [aaee Suargel
+ a [gaee] Surget
decoder only
0 [[gnn]]
E [gdec]]
2 29urget
of (E(genel + E[gaae]
29 2 jointly
As noted by Wang et al. (2020), Starget is gradually increased during training. Therefore, let the loss for the downstream task be Lowsm, the overall loss function that aim to minimize is:
L Lowsm + Csparsity
3.4 CONTEXT-AWARE GATE PREDICTOR
As shown in Fig. [ the gate probability is calculated using Gate Predictors. In Peng et al. (203b)) two types of Gate Predictors are proposed: GlobalGP and LocalGP. GlobalGP calculates the gate
4

---

Published as a conference paper at ICLR 2025
probability based the encoder's input, which is also fed into the first layer of the encoder. In contrast, LocalGP provides a Gate Predictor for each layer, computing the probability based on the imput to that specific layer. While both methods have shown promising results, opted for GlobalGP due to its implementation simplicity. The detailed process of calculating gate probability in Appendix G
To handle multiple languages and tasks simultaneously, we created vectors representing the language and task, combined them with the speech features, and used them as input the Gate Predictors. These vectors are combinations of one-hot vectors representing the language and task. For example, there are two languages, French and German, and tasks including speech recognition and transla- tion between them, the language conditions are [0, 1] for French and [1, 0J for German, and the task conditions are [0, 0, 1] for speech recognition, [o, 1, 0] for French to German translation, and [1, O, for German to French translation. Combining these, tasks such as French speech recognition can be expressed as [0, 1, 0, 0, 1], and similarly [0, 1, 1, means translating French to German.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset This study employs the Europarl-ST (Iranzo-Sánchez et al. 2020) dataset to evaluate model performance across multiple languages. The corpus was compiled from debates held in the European Parliament between 2008 and 2012. We utilized version 1.1 of the dataset, which com- prises speech data nine languages. For our experiments, we selected German, French, and Italian, which consist of approximately 20 hours of speech data. As the europarl-ST dataset is not part of the OWSM training data, we deemed suitable for evaluating the model under multi-lingual and multi-task settings.
Task We fine-tuned the OWSM model with pruning objective across ASR and ST tasks. The experiments were designed to compare two pruning strategies: one in which the model was trained on ASR and ST tasks independently, and another where both tasks were integrated during training- Additionally, we investigated the effects of sparsifying the model in three configurations: sparsifying only the Encoder, only the Decoder, and both simultaneously.
Evaluation In this experiment, evaluated the ASR task using Word Error Rate (WER) and the ST task using BLEU scores. For each language, we prepared models with sparsity ratio of 10%, 30%, 50%, 70%, and 90%, and assessed their performance. Additionally, we used baseline model that was fine-tuned with all modules retained for comparison. Note that the sparsity level refers to the ratio of activated modules to the total number of modules, not the number of parameters in the model. In all experiments, we performed auto-regressive decoding with a beam size of 5.
The model sparsity observed in this experiment is visualized using heat maps, where each cell rep- resents the gate probability for all modules across all layers. During model validation, the activation frequency each gate accumlated, and the average computed to derive the expected activation probability per module.
4.2 RESULTS
4.2.1
MULTI-LINGGAL ASR
Figure shows the WER for German. In Figures we present the visualization of E. Dene and E[Gaec for each module when the encoder and decoder were pruned separately. Figure 5 illustrates the [[[ene]] and E[gaee] when encoder and decoder were jointly pruned. We analyzed the across varied Starget and languages, and found no significant differences. Therefore, we focus on the German ASR results here. For complete heatmaps and WER table, refer to Appendix[H H We considered the possibility that the differences in the amount of data used for pre- pre-training the OWSM model across languages might affect our results and conducted additional training accordingly. The results are provided in Appendix B
5

---

Published as a conference paper at ICLR 2025
0
4.40
100
8
8
80
WER (%)
12
60
16
40
A
C
F1
F2
C
F1
F2
20
0
44 8
0.1
0.3
0.5
0.7
0.9
16 2012 8
Suaget
1611
enc
dec
enc-dec
bbaseline
F2
Figure 2: Comparison of sparsity ratio and WER. The label ene indicates pruning of the encoder only, dec indicates pruning of the de- coder only, and enc-dec indicates simultane- ous pruning of both. The WER is evaluated for German as the Starget varies from 0.1 to 0.9. The baseline WER is 13.5. Decoder pruning retains WER even at high sparsity, while encoder pruning significantly degrades it.
Figure 3: Visualization of E[gene when pruned separately with German and French ASR. The columns represents the Starget with Starget being 0.5 and 0.7 from left to right. The first row represents the result for German ASR, and the second row repre- sents the French ASR. The y-axis within the heatmap represents the depth, where top is the first layer. The label A indi- cates glob-ATT, C indicates cgMLP, F/ in- dicates FFNI, and F2 indicates FFN2.
4
1612 8
1612 8
8
1612 8
1612
SELF
SRC
FFN
SELF
SRC FFN
A
C
F1
F2
SELF
SRC
FFN
Figure 4: Visualization of E[gas] in German ASR, when decoder was pruned separately. The columns represent the Surget. with Stargtt being 0.7 and 0.9 from left to right. The la- bel SELF indicates self-ATT, SRC indicates src-ATT, and FFN indicates FFN. The other settings are consistent with those in Figure 3
Figure Visualization of [[gene] and E [aee when they were pruned jointly. The Stargat for this figure 0.7. The left image corre- sponds the encoder, and the right image corresponds to the decoder. The other set- tings consistent with those in Figure 3 and Figure
Inference Performance We found that pruning the decoder side did not harm WER, even with high sparsity ratios, where pruning encoder modules greatly deteriorate the WER in high sparsity ratio. By analyzing results alongside the module heatmap, we observed decline in encoder accuracy at Staggt 0.7, specifically when cgMLP started to be pruned, underscoring the critical role of cgMLP in ASR tasks. In contrast, observing the decoder side with a Starett = 0.9, where a substantial number of FFNs have been pruned, we find that WER does not deteriorate severely as in the ST case discussed section 4.2.2 This finding supports our initial question that ASR may not require large-scale decoders the same extent ST.
Sparse Encoder Analysis Referring to Figure. the encoder's pruning strategy remained consis- tent across languages. Additionally, the highly polarized colors in Figure. 3 indicate that the gate probabilities concentrated at extreme values, suggesting minimal variation in module selection based on speech characteristics. Interestingly, pruning approximately 50% of the encoder modules led more biased module selection, favoring cgMLP activations. This highlights the erucial role of local context captured by cgMLP, challenging the current architectural convention that equally balances both.

---

Published as a conference paper at ICLR 2025
10
0
0.1
0.3
0.5
0.7
0.9
1612 8
16 2012 8
Starget
enc
dec
enc-dec
baseline
SELF
FFN
SRC
Figure 6: Comparison of sparsity ratio and BLEU scores for French-to-German transla- tion. BLEU is evaluated for encoder-only, decoder-only, and encoder-decoder pruning, with baseline score of 12.2. Similar ASR, the decoder retains performance better than the encoder high sparsity.
Visualization of E[genc] and Idee when they were pruned jointly. The columns represents the Starget with Starget being 0.5 and 0.7 from left to right. The first row represents the encoder, and the second row represents the decoder. The other settings are consistent with those in Figure
Figure 7:
Sparse Decoder Analysis In conditions where 90% of the modules were pruned, src-ATT was prioritized over self-ATT and FFN in Figure [ indicating its essential role in inference. Given that src-ATT is the module responsible for incorporating audio information, this behavior is understand- able. At a 70% sparsity ratio, we observed a typical flow in the TransformerDeccnder where the process moves from self-ATT src-ATT, and then FFN. The heatmap indicates that groups of modules can be computed in chunks, with several self-ATT modules processed together, followed by block of src-ATT modules, and then group of FFN modules. These findings suggest that incorporating chunk-wise computation could improve the efficiency of conventional decoder archi- tectures.
Combined Encoder-Devoder Sparsity Analysis We found that several portions of the encoder exhibit a similar architecture that of the Conformer. That is, the processing sequence progresses from the FFN to the glob-ATT, then to the cgMLP, and back to the FFN. This finding suggests that the Conformer architecture is effective in speech foundation models, particularly when the model size is constrained.
Compared to Figure 4 and Figure we observed that when pruning applied only the decoder, sre aetenttnnion layers in the early part of the decoder are often skipped. However, when both the en- coder and decoder are pruned together, the earlier sre-attention layers in the decoder become more active. This difference appears to stem from whether the encoder's 's full capacity is available. When encoder fully utilized, the decoder computes more self-attention and FFN layers before src- attention to incorporate additional contextual information from the output tokens. In contrast, when encoder capacity is limited, the decoder compensates by performing self-attention and FFN compu- tations directly the audio features to capture details that may have been missed by the encoder. Additionally, by analyzing the number of active modules in the decoder, we found differences in the number of FFN layers processed before src-attention, which further supports this interpretation. These findings are also supported by visualizations provided in the Appendix H
4.2.2 MULTI-LINGUAL ST
Figure 6 presents the BLEU scores for the French-to-German translation tasks. Figure visual- ize the [[[ene] and [[gtee] for each module when the encoder and decoder were pruned separately. We also analyzed the g across varied Surget and tasks, and found no significant differences. There- fore, focus the French-to-German German ST results here. Refer to the Appendix H for complete heatmaps and table of BLEU score. The analysis on combined encoder-decoder settings are also in the Appendix H

---

Published as a conference paper at ICLR 2025
Inference Performance Focusing on the encoder side in Figure 6 the BLEU scores remained
relatively stable even with Surget = 0.5. On the other hand, from Figure and Figure BLEU scores began degrade when usage of cgMLP modules dropped when surget becomes 0.7. This models in ST, consistent with ASR tasks. A notable
highlights the importance of convolution-base difference from ASR is that pruning the decoder also deteriorates model performance. We found that in the ST decoder, the FFN tends to be retained in computation. As the model removes the FFN, the BLEU score also degrades. This observation supports our initial hypothesis that the decoder plays more critical role in ST compared to ASR.
Sparse Encoder Analysis In Figure 7 we observed increased importance of egMLP, similar to the findings in ASR. However, unlike ASR, the utilization of FFN2 decreased in the earlier layers of the ST task. Nevertheless, there were no significant differences in module selection within the encoder structure. This indicates that it may not be necessary to modify the encoder when designing models intended handle both ASR and ST tasks simultaneously.
Sparse Decoder Analysis From Starget = 0.5 to Starget = 0.7, the number of activated src-ATT decreased significantly in Figure 7 with this reduction being larger than ASR. Additionally, be- came apparent that self-ATT were computed over broader range of depths compared to ASR. This suggests that while ASR places greater importance on audio information. ST requires self-ATT and FFN more than audio information for translation. This increased reliance can be attributed to the non-monotonic relationship between input audio and output text ST, necessitating greater use of self-ATT and FFN to capture complex dependencies.
In Figure 7 we observed that self-ATT layers are more frequently activated before sre-ATT in ST compared to ASR. We hypothesize that this difference arises from the distinct priorities of each task. In ASR, the primary focus on integrating audio features directly, as each computation sre-AT ATT increases the prominence of audio information a higher importance on alignment text information through self-ATT before src-ATT to effectively map different languages. As a result, self-ATT layers activated earlier better contextualize before Src-ATT, reflecting the task-specific demands of aligning cross-modal information. These findings underscore how the allocation of self-ATT and sr--ATT computations is influenced by the differing requirements of ASR and ST.
weighted sum. In contrast, ST seems to place
4.2.3 PRUNING BY JOINT ASR AND ST
Figure 8 represents the WER for German ASR, and Figure rrepresnts the BLEU score for French- to-German translation task. Figure 9 represents the BLEU score for French-to-German translation task. Same as the single-task settings, we also analyzed the & across varied Surget and tasks, but could not find differences in module selection across languages and tasks. Therefore, focus the French-to-German ST results here. Refer to the Appendix H for a complete heatmaps and tables for other tasks and languages. The analysis of the case where the encoder and decoder were pruned separately is included in Paragraph 4.2.3
Inference Performance When ASR and ST tasks were trained simultaneously, slight degrada- tion WER and BLEU was observed in Figures X and particularly when pruning was applied to both the encoder and decoder. However, found that when only the decoder pruned, perfor- mance was better maintained compared to other settings. These findings suggest that, for large-scale speech foundation models trained multiple tasks, focusing on decoder pruning is more effective strategy for preserving accuracy across various tasks.
Combined Srroodr--eccoder Sparsity Analysis In Figure 10 the processing order observed was FFN, followed by Src-ATT, and then self-att, particularly when Surget is 0.7. This order contradicts the typical processing sequence of Transformer decoder and the observations made in 4.2.1 These results suggest that the conventional processing order of Transformer decoders may not be optimal for speech foundation models trained multi-task data.
8

---

Published as a conference paper at ICLR 2025
150
15
100
WER
50
0
0.1
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Sorer
enc/single
dec/single
enc-dec/single
enc/single
dec/single dec/joint
encdec/single
enc/joint
dec/jaint
enc-dec/joint
enc/joint
encdec/joint
baseline
baseline
Figure 8: Comparison of Suaget and WER on German ASR. This figure compares results when using only ASR data versus using both results
Figure 9: Comparison of Stageet and BLEU on French-to-German ST. This figure compares results when using only ST data versus us- ing both ASR and ST data. The labels are the same as Figure g Combined with Fig- ure X this suggests that decoder pruning multitask settings maintains competitive per- formance compared to task-specific pruning.
ASR and ST data. single refers obtained using only ASR data, and joint in- cludes results from pruning that also incor- porates ST data. The other settings are sistent with those Figure
12
14
ATT
CGMLP FINI
FFNZ
ATT
CGMLP
FNN1
FFNZ
ATT
OGMLP FINI
FFNZ
10
12
14
SELF-ATT SRC-ATT
FFN
SELF-ATT SRC-ATT FFN
SELF-ATT SRCATT
FFN
Figure 10: Visualization of [g[ane] and E[gase] when they were pruned jointly. The columns represent the Surgets with Surget being 0.3, 0.5, and 0.7 from left to right. The first represents the encoder, and the second row represents the decoder. The other settings are consistent with those Figure
4.3 INFERENCE EFFICIENCY
We measured the actual inference time of the pruned model to analyze the effect of pruning on infer- ence speed. We used one A40 GPU and 16 CPUs for each inference run. We employed vectorized Since previous experiments
beam search (Seki et al. 2019) for decoding, where the beam size showed no variation in module selection across languages or tasks, we focused on one language and task, specifically German ASR for measuring inference time in each case. Table presents the trends in metrics, inference time, and FL.OPs as function of module sparsity. It is important to note that we ignored the first run of inference, as contains initialization processes that make it slower.
The results show that reducing the decoder modules by 50% improves latency while maintaining accuracy for both ASR and ST tasks. Specifically, we achieved a 34.3% reduction in inference time with degradation in BLEU for ST, and 28.6% reduction with only 2.8% WER increase for ASR. Since OWSM uses auto-regressive inference with vectorized beam search, the decoder handles the majority of the computational load. This is reflected in the significant reduction in FLOPs observed during pruning, as the decoder processes each output token individually, treating the beam size the batch size. In this analysis, set the beam size 5, meaning the encoder's

---

Published as a conference paper at ICLR 2025
Table 1: Metries and Elapsed time on inference for ASR (German) and ST Crench-to-Germann across varying Surget- The models were trained on task-specife training datasets. The table compares elapsed time, GFLOPs, and metries (WER for ASR and BLEU for ST) at different Sunget for the enc, dec, and enc-dec. Baseline results are: ASR 9.28 seconds and 13.5 WER, and 3781 GFLOPs; ST 10.54 seconds, 12.2 BLEU, and 3409 GFLOPs. Each row represents different sparsity target, showing the impact on inference time and output quality as the sparsity increases. ET refers to the elapsed time. We use fvcore library to estimate the GFLOPs.
ASR (German)
Encoder
Decoder
EncDec
sparsity
ET
GFLOPs
WER
ET
GFLOPs
WER
ET
GFLOPs
WER
10%
9.07
3697
14.4
9.33
3268
15.3
10.39
2843
14.3
30%
9.21
3690
14.8
7.24
2293
15.9
7.09
2249
15.4
50%
8.92
3669
16.3
6.62
1713
20.4
5.99
1698
17.8
70%
9.15
3633
4.79
1272
17.2
5.49
1139
28.8
90%
8.52
3613
80.8
4.80
625
24.5
5.22
682
80.8
ST (Fremch-to-Germann
Encoder
Decoder
EncDec
sparsity
ET
GFLOPs
BLEU
ET
GFLOPs
BLEU
ET
GFLOPs
BLEU
10%
10.42
3369
12.3
10.72
2718
12.9
10.58
2861
12.2
30%
10.08
3357
11.9
8.27
2015
13.0
9.27
2093
11.6
50%
10.27
3335
12.3
6.92
1521
12.2
7.09
2023
9.4
70%
10.38
3311
5.38
1063
10.3
6.58
5.2
90%
10.44
3298
2.5
4.27
504
5.3
4.81
549
0.4
batch size 1, while the decoder's is As a result, pruning the decoder not only reduces FLOPs but also has more pronounced impact on inference speed compared to pruning the encoder.
CONCLUSION
In this work, proposed novel context-aware dynamic pruning method for speech foundation models that adapts pruning dynamically during inference. With the pruned model, we successfully accelerated the inference of speech foundation models, particularly without any degradation in the ST task. Through detailed analysis of the model structures that emerge after pruning, we identi- fied the efficiency of the Transformer decoder and Conformer, while also uncovering an interesting computational flow when the model was pruned in multi-task settings, Although this study focused on the speech domain, our approach can be readily extended to foundation models in other fields, such as NLP and computer vision.
ACKNOWLEDGEMENTS
This study was supported by the BRIDGE program of the Cabinet Office, Govermment of Japan. Also, we used the Bridges2 system at PSC and Delta system at NCSA through allocation CIS210014 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Sup- port (ACCESS) program, which supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY
You can download the OWSM-v3.1 we employed this experiment from the huggingface hub All of our experiments are conducted with ESPnet (Watanabe et al. 2018). Based on the training configuration of OWSM-v3.1, we added or modified the following configuration:
encoder: e_branchhfomme_ttoke__ooodddddd
decoder: transfoomer__eccder_.oooeooooone
tau_ini: 1
tau tau_end: 0.1 tau_coooldow__sepp. 15000
sparsity_init: 0.0
sparsity_end: 0.3
optim: adamw optim_conff
1r: 0.00001
weight_decay: 0. 000001
acheduler: warmupir scheduler_conf: warmup_steps: 6000
Several configurations were added to the original ESPnet. Each configuration is as follows:
tau_ini / tau_end: The initial and final temperatures for Gumbel-Softmax.
tau wwoldown stops:
steps: The iteration number for the temperature of Gumbel-Softmax, Target sparsity gradually increases to sparsity _nnd.
sparsity_ing iinit/ sparsity_ent The initial target sparsity and after the warmup.
sparsity warmup steps: Warmup steps for the sparsity. The target sparsity will be gradually increased to reach sparsity_ends
The class set for the encoder and decoder is the extended class of E- B-Branchformer and Trans- formerDecoder to incorporate pruning in this study. Other configurations are same as the OWSM- v3.1, and you can refer to all settings in huggingface hub
REFERENCES
Jose M. Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 222-2270, 2016.
Amazon. The amazon nova family of models: Technical report and model card, Amazon Technical Reports, 2024.
Anthropic. Claude 3 haiku: our fastest model yet, 2024.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint, 1511 1.06997 2016.
Alexandre Bittar, Paul Dixon, Mohammad Samragh, Kumari Nishu, and Devang Naik. Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer en- coder. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 10386-10390, 2024.
htteess//nngiinffeeee.////eeeess v3. leff
httper//anggiggooo..o//eeett// v3. ebt
train
s2t 2tt___eff_cc//__siii11222222 pieeeeeeeeeeieieeeeeeeeeeeee_e__ 1r2e- 41172e-_wwrmmp6666 warmup6kk. flashattn_r
bpe50000/config..yam

---

Published as a conference paper at ICLR 2025
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th In- ternational Conference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 527-536. PMLR, 2017.
Feng-Ju Chang, Jing Liu, Martin Radfar, Athanasios Mouchtaris, Maurizio Omologo, Ariya Ras- trow, and Siegfried Kunzmann. Context-aware transformer transducer for speech recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 503-510, 2021.
Heng-Jui Chang, Shu-Wen Yang, and Hung-yi Lee. Distilhubert: Speech representation learning by layer-wise distillation hidden-unit bert. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 7087-7091. IEEE, 2022.
Heng-Jui Chang, Ning Dong. Ruslan Mavlyutov, Sravya Popuri, and Yu-An Chung. Colld: Con- trastive layer-to-layer distillation for compressing multilingual pre-trained speech encoders. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10801-10805, 2024.
Tang Changli, Yu Wenyi, Sun Guangzhi, Chen Xianzhao, Tan Tian, Li Wei, Lu Lu, MA Zejun, and Zhang Chao. SALMONN: Towards generic hearing abilities for large language models. In The International Conference Learning Representations (ICLR), 2024.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, (666):1505-1188 2022. doi: 10. 10.110//STSP.2222..33388
Shi Chen and Qi Zhao. Shallowing deep networks: Layer-wise pruning based on feature represen- tations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12):3048-300 2019.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07990 2023.
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal repre- sentations of speech. arXiv preprint arXiv:2205. 12446, 2022.
DeepSeek-AI. Deepseek-v3 technical report, 2025.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. Technical report. Kyutai, September 2024.
Shaojin Ding, David Qiu, David Rim, Yanzhang He, Oleg Rybakov, Bo Li, Rohit Prabhavalkar, Weiran Wang, Tara N. Sainath, Zhonglin Han, Jian Li, Amir Yazdanbakhsh, and Shivani Agrawal. Usm-lite: Quantization and sparsity aware fine-tuning for speech recognition with universal speech models. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10756-10760, 2024.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In The International Conference on Learning Representations (ICLR), 2020.
Yonggan Fu, Yang Zhang, Kaizhi Qian, Zhifan Ye, Zhongzhi Yu, Cheng-I Jeff Lai, and Celine Lin. Losses can be blessings: Routing self-supervised speech representations towards efficient multilingual and multitask speech processing. In Advances in Neural Information Processing Systems, volume 35, pp. 20902-20920, 2022.
Sanchit Gandhi, Patrick von Platen, and Alexander M Rush. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. arXiv preprint arXiv:2311.00433 2023.
12

---

Published as a conference paper at ICLR 2025
Gemini Team Google. et al. gemini: family of highly capable multimodal models, 2023.
Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint, 1603.08983, 2016.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2016.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. Eie: Efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium Computer Architecture (ISCA), pp. 243-254, 201
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. In The International Conference on Learning Representations (ICLR), 2016b.
Song Han, Huizi Mao, and William J. Dally. Pruning convolutional neural networks for resource efficient inference. In The International Conference on Learning Representations (ICLR), 2017.
Babak Hassibi, David Stork, and Gregory Wolff. Optimal brain surgeon: Extensions and perfor- mance comparisons. In J. Cowan, G. Tesauro, and Alspector (eds.), Advances in Neural Infor- mation Processing Systems, volume 6. Morgan-Kauut Kauffann, 1993.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter prun- ing criteria for deep convolutional neural networks acceleration. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2006-2015, 2020.
Shujie HU, Long Zhou, Shujie LIU, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xun- ying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2204. 000566 2024.
J. Tranzo-Sánchedd J. A. Silvestre-Cerdee J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8229-8233, 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax. In
The International Conference on Learning Representations (ICLR), 2017.
Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in re- current neural networks. In The International Conference on Learning Representations (ICLR), 2017.
Cheng-I Jeff Lai, Yang Zhang, Alexander H. Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David D. Cox, and James R. Glass. PARP: prune. adjust and Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 21256-21278 2021.
re-prune for self-supervised speech recognition.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Borgan-Kaufman 1989.
Jaesong Lee, Jingu Kang, and Shinji Watanabe, Layer pruning on demand with intermediate etc. In Interspeech 202/, pp. 3745-3749, 2021.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In The International Conference Learning Representations (ICLR), 2017.
13

---

Published as a conference paper at ICLR 2025
Qi Li, Hengyi Li, and Lin Meng. Feature map analysis-based dynamic enn pruning and the acceler- ation on fpgas. Electronics, 11(18), 2022.
Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Hui Li, and Wen-Huang Cheng. Lightweight deep learning for resource-constrained environments: A survey. ACM Comput. Surv., 56(10), jun 2024. ISSN 0360-0300.
Jing Liu, Rupak Vignesh Swaminathan, Sree Hari Krishnan Parthasarathi, Chunchuan Lyu, Athana- sios Mouchtaris, and Siegfried Kunzmann. Exploiting large-scale teacher-student training for on-device acoustic models, 2021.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through lo regularization. In The International Conference on Learning Representations (ICLR), 2018.
Yen-Ju Lu, Jing Liu, Thomas Thehaud, Laureano Moro- Voro--elazzuez, Ariya Rastrow, Najim Dehak, and Jesus Villalba. Ca-sstr: Condition-aware self-supervised learning representation for general- ized speech processing, 2024.
Jon Macoskey, Grant P. Strimel, and Ariya Rastrow. Bifocal neural asr: Exploiting keyword spotting for inference optimization. ICASSP 2021 2021 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 5999-6003, 2021a.
Jonathan Macoskey, Grant P. Strimel, Jinru Su, and Ariya Rastrow. Amortized neural networks for low-latency speech recognition. In Interspeech 2021, pp. 4558-4562 2021b.
Meta. The Ilama herd of models, 2024.
Hieu Duy Nguyen, Anastasios Alexandridis, and Athanasios Mouchtaris. Quantization aware train- ing with absolute-cosine regularization for automatic speech recognition. In Interspeech 2020, pp. 3366-3370, 2020.
OpenAl. Introducing chatgpt, 2022.
OpenAl. Gpt-4 technical report, 2023.
OpenAl. Gpt-4o system car, 2024.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In Proc. ICASSP, 2015.
Yifan Peng, Kwangyoun Kim, Felix Wu, Prashant Sridhar, and Shinji Watanabe. Structured pruning of self-supervised pre-trained models for speech recognition and understanding. In IEEE Inter- national Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pp. 1-5. IEEE, 2023a.
Yifan Peng, Jaesong Lee, and Shinji Watanabe. 13d: Transformer architectures with input-dependent dynamic depth for speech recognition. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023b.
Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. Dphubert: Joint distillation and pruning of self- self-supervised speech models. In Naomi Harte, Julie Bersdnn,errdsen, and Gareth Jones (eds.), 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, pp. 62-66. ISCA, 2023c.
Yifan Peng. Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, Wangyou Zhang, Yui Sudo, Muhammad Shakeel, Jee- Weon Jung, Soumi Maiti, and Shinji Watanabe. Reproducing whisper-style training using an open-source toolkit and publicly available data. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8, 2023d.
Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. OWSM-CTC: An open encoder- only speech foundation model for speech recognition, translation, and language identification. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), August 2024a.
14

---

Published as a conference paper at ICLR 2025
Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Sha- keel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee weon Jung, and Shinji Watanabe. Owsm v3.1: Better and faster open whisper-style speech models based e-branchformer. In Interspeech 2024, pp. 352-356, 2024b.
Krishna C. Puvvada, Piotr Zelasko, He Huang, Oleksii Hrinchuk, Nithin Rao Koluguri, Kunal Dhawan, Somshubra Majumdar, Elena Rastorgueva, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balam, and Boris Ginsburg. Less is more: Accurate speech recognition translation without web-scale data. In Interspeech 2024, pp. 3964-3968, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever (eds.). Robust Speech Recognition via Large-Scale Weak Supervision, volume 2023 of Proceed- ings of Machine Learning Research, 2023.
Saumya Y. Sahai, Jing Liu, Thejaswi Muniyappa, Kanthashree M. Sathyendra, Anastasios Alexan- dridis, Grant P. Strimel, Ross McGowan, Ariya Rastrow, Feng-Ju Chang, Athanasios Mouchtaris, and Siegfried Kunzmann. Dual-attention neural transducers for efficient wake word spotting in speech recognition, 2023.
Jin Sakuma, Tatsuya Komatsu, and Robin Scheibler. MLP-based architecture with variable length input for automatic speech recognition, 2022.
Kanthashree Mysore Sathyendra, Thejaswi Muniyappa, Feng-Ju Chang, Jing Liu, Jinru Su, Grant P. Strimel, Athanasios Mouchtaris, and Siegfried Kunzmann. Contextual adapters for personalized speech recognition in neural transducers. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8537-8544 2022.
Hiroshi Seki, Takaaki Hori, Shinji Watanahe, Niko Moritz, and Jonathan Le Roux. Vectorized heam search for TTC-aattention-basee speech recognition. In Interspeech 2019, pp. 3825-3829, 2019.
Grant Strimel, Yi Xie, Brian John King, Martin Radfar, Ariya Rastrow, and Athanasios Mouchtaris. Lookahead when it matters: Adaptive non-causal transformers for streaming neural transducers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 32654-32676, 23-29 Jul 2023.
Voxforge.org. Free and open source speech recognition engines (linux, windows and mac) vox- forge.org.
Haoyu Wang, Siyuan Wang, Wei-Qiang Zhang, Hongbin Suo, and Yulong Wan. Task-agnostic structured pruning of speech representation models. In Naomi Harte, Julie Beaod-Berr,,sn, and Gareth Jones (cds.), 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, pp. 231-235 ISCA, 2023.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Bonnie Webber, Trevor Cohn, Yulan Hc, and Yang Liu (cds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6151-6162. Association for Computational Linguistics, November 2020.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. Espnet: End-to-end speech processing toolkit. In Proc. Interspeech, pp.
2207-22 2211, 2018
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Yi Xie, Jonathan J. Macoskey, Martin Radfar, Feng-Ju Chang, Brian King, Ariya Rastrow, Athana- sios Mouchtaris, and Grant Strimel. Compute cost amortized transformer for streaming asr. In Interspeech 2022, pp. 3043-3047, 2022.
15

---

Published as a conference paper at ICLR 2025
Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, and Boris Ginsburg. Effi- cient sequence transduction by jointly predicting tokens and durations. In Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning 23-29 Jul 2023.
Research, pp.
Gong Yuan, Luo Hongyin, H. Liu Alexander, Karlinsky Leonid, and R. Glass James. Listen, think, and understand. The International Conference on Learning Representations (ICLR), 2024.
A PROBLEM FORMULATION OF DYNAMIC PRUNING WITH STRAIGHHT GUMBEL-SOFTMAN ESTIMATOR
Our task formulation follows (Louizos al. 2018). but differs in the formulation of the used to compute model complexity, where we employ the SGSE function. Specifically, given an input speech signal v; model parameters 0, model function f()). and pruning mask 2, the output token y computed using the pruned parameters as follows:
ÿ = f(((0). = 0 2E (0,
where |0] denotes the number of model parameters. Given a dataset, we define the loss function and the to norm as |10|| = E[z]. The training loss is then formulated as:
loss N ( ELLrEtD) u) + Allā
where N is the batch size and À is a hyperparameter that encourages model sparsity. Here, |00lo serves as a sparsity-inducing regularization term. However, learning as a binary variable makes difficult to compute gradients for the skipped modules, which prevents parameter updates. To address this, (Louizos et al.) 2018) applies reparameterizzation trick to represent in a differentiable form ž, allowing for simultaneous optimization of both 0 and 2 by defining |l0l|o B[a].
However, this method results being a continuous value, which leads to discrepancies between training and inference computations. To mitigate this issue, employ SGSE, which ensures that remains binary even during training. Specifically, given a trainable global gate predictor G(-) and context information C, ā can be expressed using the SGSE function as:
z = SGSE(G((z, C))
B
DATA SIZE
The language-specific data volumes used for training OWSM-v3.1 are shown in Table including additional languages from the Appendix experiments. Although the Italian dataset is smaller than German and French, pruning trends remain similar. For performance details, see Appendix H We created new dataset by integrating Voxforge (Voxforge.org) with Europarl-ST, covering the same
Table 3: Additional dataset with Voxforge. We show WER for German (deu), French (fra), and Italian (ita). Decoder pruned from OWSM- v3.1 model.
Table 2: Data size used in OWSM-v3.1 pre- training for each language.
Language amount (h)
sparsity
deu
fra
ita
French
2489
0%
14.2
11.1
12.3
German 3704
10%
14.8
12.8
13.3
Italian
707
30%
15.1
12.9
13.6
Hungarian
97
50%
19.3
16.0
18.2
Bulgarian
18
70%
17.5
15.1
15.8
90%
25.5
23.8
21.6
16

---

Published as a conference paper at ICLR 2025
Table Additional results for Hungarian (hun) and Bulgarian (bul). We show WER for these two languages when encoders and decoders were pruned separately. The WER for full fine- tuned model 33.2 for Hungarian and 23.5 for Bulgarian.
Table 5: Reproduced I3D results using the Lib- riSpeech dataset. We show the Word Error Rate (WER) the test-clean set for varying numbers of encoder layers.
Encoder
Decoder
# Layers
WER
sparsity
hun
bul
hun
bul
36
8.7
10%
38.7
38.2
35.1
29.5
34
8.7
30%
42.5
37.7
34.1
24.2
28
8.8
50%
42.1
38.3
40.1
29.1
70%
54.4
44.2
36.3
26.5
90%
89.2
78.8
42.1
36.2
2-
4
6
8
10
10
12
12
14
SELF
SRC
(a) Encoder
(b) Decoder
Figure 11: The sparsity plot for Hungarian ASR shows the results when the encoder and decoder were pruned separately, with the target sparsity ratio set to 70%. The pruning patterns exhibit notable differences compared to those observed in high-resource languages.
languages. Training and evaluation follow Section 4 and ASR results appear in Table 3 Perfor- mance degradation across sparsity levels comparable between Europarl-ST and the joint dataset.
To assess the impact on underrepresented languages, we selected Hungarian and Bulgarian from Fleurs (Conneau et al.) 2022). Table shows WER results. Decoder pruning lowered accuracy, as with high-resource languages like French and German, but encoder pruning had a greater impact. Even at lower sparsity, encoder pruning caused notable performance drops, highlighting its crucial role in preserving accuracy for low-resource languages. With less training data, effective input feature capture essential, while decoder is less sensitive to sparsity due to its reliance on encoder representations.
We developed a baseline on Peng et al. (203b)) using a Transformer-encooder model trained on Lib- riSpeech (Panayotov et al. 2015) and fine-tuned it on the LibriSpeech- 100h subset. The reproduced WER results on the test-clean set are shown Table 5 Our parameter settings yielded sparsity levels of approximately 95% and 78%, indicating that performance remains unaffected up around 20-25% sparsity, consistent with our experimental results.
C LAYER-LEVEL PRUNING
To compare performance with other pruning methods, we conducted layer-level pruning experi- ments. An example pruning pattern for layer skipping is shown in Figure 12 Table[ WER results for German ASR with decoder-only pruning. Up to 70% sparsity, skipping at the layer and module levels shows significant difference. At lower sparsity, the layer-level approach performs better, while at higher sparsity, the module-level approach excels. This suggests that at higher spar- sity, optimizing the roles and order of individual modules is more effective than skipping entire
17

---

Published as a conference paper at ICLR 2025
Table 6: WER on module-level pruning and layer-level pruning. Results from German ASR, with only decoder pruned (left) and only encoder pruned (right).
Encoder Pruned
Decoder Pruned
sparsity
Module
Layer
Module
Layer
10%
14.5
31.3
15.8
15.4
30%
32.0
16.4
15.2
50%
16.8
34.2
16.2
16.5
70%
25.1
40.2
18.3
18.7
90%
47.3
108.6
26.4
99.9
Table 7: Comparison of GFLOPs between module skip and layer skip at different target sparsity levels when the encoder is pruned. We used the fvcore library to calculate the FLOPs. FLOPs were computed over multiple utterances, and the average value was taken. For reference, the FLOPs for the model without pruning were 3781.
sparsity
Module Skip (GFLOPs)
Layer Skip (GFLOPs)
10%
3697
3697
30%
3690
3692
50%
3669
3666
70%
3633
3640
90%
3613
3620
4
6
8
10
12
14
16
A
C
F1
F2
Figure 12: The sparsity plot when layers were skipped from encoder with the target sparsity ratio set to 70%. This model was trained on German-to-Italian ST.
layers. For the encoder, accuracy degradation significant, suggesting that layer-level pruning is unsuitable for speech foundation models. Unlike module-level pruning, layer-level pruning forcibly removes even MLP, which plays critical role in encoding.
We also computed FLOPs when the encoder is pruned, shown in Table Since different modules are used in module-level vs. layer-level pruning, we analyzed the impact. For encoder pruning, module-level pruning results in slightly lower FLOPs, reinforcing its advantage in efficiency.
D UNSTRUCTURED PRUNING
We investigated the impact of pruning strategies: unstructured vs. structured pruning. Table compares magnitude- unstructured pruning and PARP. Fine-tuning was performed on a dataset including ASR and ST, and WER was evaluated on German ASR. Even with unstructured pruning, decoder pruning preserved performance. However, compared to module-level pruning at the same sparsity, module-level
on the mncoder-maintainee accuracy more effectively.
18

---

Published as a conference paper at ICLR 2025
Table WER results for German ASR with different unstructured pruning techniques.
Model
WER (%)
Original model (without fine-tuning)
24.6
Unstructured pruning (sparsity 0.1)
24.5
Unstructured pruning (sparsity 0.3)
24.5
PARP (sparsity 0.1, applied to encoder)
31.2
PARP (sparsity 0.3, applied to encoder)
31.8
PARP (sparsity 0. applied to decoder)
19.8
PARP (sparsity = 0.3. applied to decoder)
16.4
E SPARSITY LOSS
In Peng et al. (202b)), pruning is applied to the Transformer encoder by targeting the self-attention and feed-forward network modules. The sparsity loss function, Esparsity is defined as:
1 2N
① BTT + OFEN
(1)
Ssparitty = A
where A OFFN the gate probabilities of each module in the I-th layer, and the N is the num- ber of layers. À is a constant loss weight for the sparsity and the value determined heuristically. Peng et al. (023b) aims to achieve model sparsity by controlling the magnitude of the loss through X. In Peng et al. (023b), different values of X will lead to different inference costs, as the final sparsity ratio of the model is controlled by sparsity loss.
(!)
(1)
We initially tried the fixed value for A. However, the fixed did not allow the model to achieve the desired sparsity, particularly when attempting prune over 70% of the modules. To address this, Wang et al. (2020) introduces Lagrange multiplier λ₁ and A2, and the sparsity loss is defined as:
Epenaity \ı Starget) + >2(g Starget)
where A and X2 are updated based on the model's sparsity.
Here, Lppnnlty can take a negative value when - S Surget S 0 (Details are penatty becomes negative, it complicates solving the minimization problem when combined with the ASR and ST losses. To address this issue, we employed function calculate the absolute value of Starget. so that the Cpenaity remains non-negative. For the sake of simplicity, we set the λ = A2 1 and employed constant on top of it. Starting from 1, we gradually increase the unless it reaches the desired sparsity. The visualization of each cost function is in the Appendix. Thus, the sparsity loss we used in this experiment becomes Eq. When calculating the Liparsity a was gradually increased following Wang et al. (2020).
F VISUALIZATION ON SPARSITY LOSS
The purpose of Lagrange multipliers in Wang al. 2020) is to make the Cpenatty more aggressive penalty term. So we hypothesized that simply introducing regularization term, which has similar role, we make the model prune the desired number of modules. Figure illlsstraes the graphs of various penalty terms. The x-axis represents 9t, Surget and the y-axis represents Lypursit-
The Epenilyy Appendix can rewritten as a quadratic function:
Epenalty = (r + )
This is represented by the green curve in Figure 13 They move the vertex of this green curve when updating the two Lagrange multipliers. In our implementation, we set À = X = and introduced an overall coefficient increase the value of this coefficient when the difference between the actual sparsity and the target sparsity exceeds certain threshold (0.05). Increasing this coefficient lowers the vertex of the green line in Figure| [3 For example, setting a 5 results in the red curve, which has a steeper slope for 0.
Thus, Epenalty becomes: Lpenatt a(x + 1)2²
In our approach, we
19

---

Published as a conference paper at ICLR 2025
Eppriity
Figure 13:
Visualization on different Espariity Wang et al. (2020) moves the green line to maximize the loss. Our method simply shifts the vertex downward. For example, by lowering the vertex of the green line, we transform it into a penalty function resembling the red line.
Algorithm Gate Predictor
Spooled Average
Average over time dimension, Ppooled RD Concatenate conditional info, R D+ D+ Daeei
< Concat(C, poooled)
logit t Reshape(G(rr)
Reshape G(x) to logit I _
f SGSE(logit, axis =
b Compute SGSE function Select second column from Gumbel-Softna output
+ g|:11
G CONTEXT-AWARS GATE PROBABILITY
The gate probability is computed using separate predictor for each module. Let G() denote the gate predictor, with speech feature input VGP E RTxD condition C RDC and total layers J.. Here, 7 is the number of frames, D the speech feature dimension, and DC the condition dimension. The gate probability g for a module is computed as in Algorithm with gt, [E[g]
To prevent early learning instability, we avoid random weight initialization for the Gate Predictor. Random initialization may remove 50% of modules early, hindering gradual reduction. Instead, we adjust the final layer bias so that g initially outputs values near 1, ensuring full activation. This setup allows gradual reduction in active modules during training. Following Peng et al. (2023b). we use two-layer MIP with an intermediate size of 32.
H HEATMAPS AND TABLES
Table 9: Comparison of WER (%) for French, German, and Italian using encoder-sparsifie, model, decoder-spinn -parssiid model, and jointly sparsified model.
Sparse Encurer
Spure Decoder
Jointl Sparsified Frecnder-Deaddee
Raseline
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
164
26.5
Gemmae
13.5
14.4
148
16.3
21.0
80.8
15.3
15.9
20.4
17.2
24.5
14.3
15.4
17.8
28.8
80.8
Italian
12.8
14.5
14.4
16.5
21.5
86.0
13.8
13,9
20.1
15.4
26.5
13.5
14.7
17.5
32.5
860
20

---

Published as a conference paper at ICLR 2025
16 8 44
1612 8 4
1612 8 4
1612 8 4
1612 8
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
A
C
F1
F2
0
0
8 4
1612 4
4
4
1612 8
8
8
1612
12
1612
16
C
FI
F1
F2
A
F1
F2
A
1612 0
0
1612 4
4
1612 8
1612
16
C F1
F2
A
F2
FLF2
Figure 14: Visualization of [[gene] when encoder pruned separately. The columns represents the Starget. with Starget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the French, the second row represents German, and the third row represents Italian. The other settings are consistent with those in Figure
0
1612 8 4
1612 8
16.2 8
1612 8
16
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
1612 0
0
1612 8
1612
8
1612
1612
0
0
1612 8 4
1612 8
1612 8
1612 8
1612
SELF SRC FFN
SELF SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 15: Visualization of E[gaee] when decoder pruned separately. The columns represents
the Staget, with Surget being 0, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
French, the second row represents German, and the third represents Italian. The other settings are consistent with those in Figure
Table 10: BLEU score each speech translation direction for Sparse Encoder, Sparse Decoder, and Jointly Sparsified Encoder-Decoder
Spuzse Encoder
Spane Decoder
Jointy Spanitied Encoder Decoder
tr
Teseline
0.5
0.9
dc
12.3
12.3
11.9
13.3
9.2
2.5
12.9
13.0
12.2
10.3
5.3
12.2
11.6
9.4
5.2
0.4
tra
13.5
1238
12.7
12.3
10.2
1.3
12.5
11.8
12.0
9.3
3.1
12.8
12.0
8.5
4.2
0.0
deu
7.5
9.4
9.1
8.7
7.0
8.5
6.6
2.2
9.3
9.2
9.0
3.1
8.9
8.5
6.3
3.2
3.1
21

---

Published as a conference paper at ICLR 2025
16 8 4
8 4
1612 4
1612 8
1612
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
0
4 0
1612 4
1612 8 4
1612 4
8
8
1612
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
0
0
1612 4
4
1612 8
1612
1612 8
1612
SELF
Figure 16: Visualization of E[gene] when encoder and decoder was pruned jointly. The columns
represents the Surget, with Staget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the French, the second row represents German, and the third row represents Italian. The other settings are consistent with those Figure
1612 8 4
1612 8
1612 8
1612 8
1612 8
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
1612 8 4
4
1612 8
1612 8
8
1612
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
4 0
0
4
1612 4
1612 4
1612 8
1612 8
1612 8
SELF
SRC
FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 17: Visualization of Eigae)] when encoder and decoder was pruned jointly. The columns
represents the Surget, with Starget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the French, the second row represents German, and the third row represents Italian. The other settings consistent with those Figure
Table 11: BLEU score on each speech translation direction and WER for ASR tasks for Sparse Encoder, Sparse Decoder, and Jointly Sparsified Encoder- -ecoder. The leftmost column represents source speech's language and the corresponding target text's language.
Sparse Encoder
Sparse Decoder
Jointly Spaninted Enceder Decodar
metric
baseline
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
fi-fr
WER
11.0
11.6
11.9
13.1
22.3
46.8
12.6
13.4
13.2
15.0
25.9
11.3
12.4
14.5
24.2
107.0
Teddo
BLEU
11.2
3.4
8.6
7.9
3.9
2.0
11.3
11.6
10.2
9.0
5.8
10.7
9.9
8.3
5.5
0.6
Ie-it
BLEU
13.0
2.5
12.3
4.7
10.2
7.8
13.6
26.4
de
BLEU
8.4
6.1
5.0
1.5
8.2
7.7
4.0
8.0
6.0
de-it
BLEU
6.4
6.1
6.0
5.7
3.4
1.2
6.9
7.0
6.2
4,9
2.6
4.9
3.7
1.7
0.
Bet
WER
13.2
14.4
15.3
17.3
37.6
66.0
14.
153
15.0
16.7
28.9
13.9
15.1
17.8
36.7
162.6
dede
BLCU
9.8
6.5
6.2
1.2
1066
10.4
9.8
5.8
7.1
8.7
7.4
22

---

Published as a conference paper at ICLR 2025
0
4 0
16 8
1612 8
1612
1612
1612
C
4
1612 8
1612 8
1612 8
1612 8
1612 8
A
C
F1
F2
F1 F2
A
F1
F2
F1
F2
A
C
F2
Figure 18: Visualization of E[gene] when encoder pruned separately. The columns represents
the Surget. with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the French-to-German, the second represents French-to-Italian translation. The other settings consistent with those in Figure]
1612 8 4.4
4
4
1612 8
8
8
1612
1612
1612
SELF SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
0
44
1612 4
1612
1612
1612
1612
SELF SRC FFN
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 19: Visualization of E[gas]] when decoder was pruned separately. The columns represents
the Starget with Staget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
French-to-Germal the second row represents Manch-to-Italian translation. The other settings are consistent with those Figure
0
0
A
1612 8
8
1612 8
8
1612
1612
1612
C
F1
F2
F1
F2
A
FI
F2
A
4
4
4
16 8 4
16 4
16 8
8
8
1612 12
1612
A
C
F1
F2
C
FI
F2
A
C
F1
F2
C
F2
A
C
Fl
F2
Figure 20: Visualization of Elgenc] when encoder was jointly pruned with decoder. The columns represents the Surget. with Starget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the French-to-German the second row represents French-to- -tallan translation. The other settings are consistent with those in Figure 3
23

---

Published as a conference paper at ICLR 2025
0
4
1612 8
1612 8
1612
1612
1612
SRC
SRC
SRC
1612 8 4
1612 8
1612 8
1612
1612
SELF SRC
FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 21: Visualization of E[gdel when decoder was jointly pruned with encoder. The columns
represents the Surgets with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row
represents the Wrench-to-German, the second row represents French-to-Italian translation. The other
settings are consistent with those in Figure 3
O
1
4
4
4
8
8
1612 B
B
1612 8
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
C
F1
F2
0
0
0
1612 4
1612 4
4
4
1612 8
1612 8
1612
C FI
FL F2
F1
F2
Figure 22: Visualization of E[gene] when encoder was pruned separately. The columns represents
the Stargtt with Starget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left right. The first row represents the German-to Seattttt--rrcccheh the second row represents German- to-aantt--talian translation. The other settings are consistent with those Figure
0
1612 8
1612 8
8
1612
1612
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
1612 8 4
4
4
16 4
1612 8
1612 8
1612
SELF SRC FFN
SELF SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SAC
FFN
Figure 23: Visualization of B[gaee] when decoder was pruned separately. The columns represents
the Surget. with Staret being 0, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
German-to- -ormantt--French. the second row represents Serman-to-Italian translation. The other settings are consistent with those in Figure
24

---

Published as a conference paper at ICLR 2025
0
0
4
16 8
1612 8
1612 8
1612
1612
C
F2
1612 8
1612 8
16 8
1612 8
1612 8
A
C
F1
F2
FI F2
A
F1
P2
F1
F2
A
F1
P2
Figure 24: Visualization of E[gead when encoder was jointly pruned with decoder. The columns represents the Surgets with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the German-to-Frent torroehh the second row represents Geraannto-Italian translation. The other settings are consistent with those in Figure 3
4
4
8
1612 8
1612 8
8
1612 8
1612
1612
SELF SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
0
4
1612 4
1612 8
1612 8
1612
1612
SELF SRC FFN
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 25: Visualization of E[gae] when decoder was jointly pruned with encoder. The columns
represents the Surget, with Staget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the German-to- -o-ennhhhh the second row represents German-tor to-i--too-ttaan translation. The other settings are consistent with those Figure
0
8
8
8
8
1612
16
1612
1612
1612
C
F1
F2
F1
F2
A
F1
F2
A
F2
16 4
4
1612 4
4
16 4
8
1612 8
12
A
C
F1
F2
C
F1 F2
A
C
F1
F2
C
F1
F2
A
C
F1
F2
Figure 26: Visualization of E[gene] when encoder was pruned separately. The columns represents the Surget. with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Italian-to-Freme the second row represents Italian-to-German translation. The other settings are consistent with those in Figure
25

---

Published as a conference paper at ICLR 2025
0
0
1612 8 44
1612 8
1612
1612
1612
SRC
SRC
SRC
4
1612 8
1612 8
8
1612
1612
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 27: Visualization of E[gdee] when decoder pruned separately. The columns represents
the Surget. with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
Selian-to-Frenchh the second row represents Italian-to-German translation. The other settings consistent with those in Figure]
8
4
4
B 4
1612
8
1612
8
1612
1612
1612
8
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
A
C
F1
F2
0
0
0
4
4
1612 8
4
8
1612
1612
1612
1612
C
F1
FL F2
F1 F2
Figure 28: Visualization of E[gene] when encoder was jointly pruned with decoder. The columns
represents the Surget, with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Italian-to- Frenchh the second row represents Italian-to- -o-ammnn translation. The other settings are consistent with those in Figure 3
1612 8
1612 8
1612 8
1612 8
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
4
4
4
4
16 4
1612 8
1612 8
1612 8
1612 8
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 29: Visualization of E[gae] when decoder was jointly pruned with encoder. The columns
represents the Surget, with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Soiian-to-Frenchh the second row represents Halian-to-German translation. The other settings are consistent with those in Figure 3
26

---

Published as a conference paper at ICLR 2025
0
1612 8 44
1612
1612
1612
1612
C
F2
4
1612 8
1612 8
1612 8
1612 8
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 30: Visualization of E[gene] and [[gaee] when encoder and decoder pruned separately for
French ASR. The columns represents the Surget, with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the encoder, the second row represents the decoder. The other settings are consistent with those in Figure 3
4
4
4
8
8
8
8
1612
1612
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
F1
F2
0
0
4
1612 4
1612 8
8
1612
1612
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC FFN
Figure 31:
Visualization of E[genc] and E Elgate] when encoder and decoder was pruned jointly for
French ASR. The columns represents the Surgets with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the encoder, the second row represents the decoder. The other settings are consistent with those in Figure 3
8
8
8
8
1612
12
1212
1612
1612
C
F1
F2
F1
F2
A
F1
F2
A
4
4
4
4
16 4
16 8
8
1612 8
8
12
16
A
C
F1
F2
C
F1
F2
A
C
F1
F2
F1
F2
A
C
F1
F2
Figure 32: Visualization of E[gene] when encoder was pruned separately. The columns represents the Surget. with Sunget being 0. 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Beanchtt--Germnn the second row represents French-to-Italin translation. The other settings are consistent with those in Figure
27

---

Published as a conference paper at ICLR 2025
0
4
1612 8
1612 8
1612
1612
1612
SELF
SRC
SRC
SRC
4
1612 8
1612 8
1612 8
1612
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 33: Visualization of E[gdee] when decoder pruned separately. The columns represents
the Surget. with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
Trench-to-German, the second represents French-to-Italian translation. The other settings consistent with those in Figure]
8 4
4
4
8
1612 8
8
1612
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
A
F1
F2
0
0
4
4
4
4
8
8
1612
1612
1612
1612
1612
C F1
FL F2
FI F2
Figure 34: Visualization of E[gese] when encoder was jointly pruned with decoder. The columns represents the Surget, with Staget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
the second row represents French-to-Itale translation. The other
settings are consistent with those in Figure 3
0
1612 8
8
8
1612
1612-
1612
1612
C
F1
F2
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
A
F2
4
4
4
1612 4
1612 8
16 8
1612 8
1612 8
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 35: Visualization of E[ga<] when decoder was jointly pruned with encoder. The columns
represents the Surget. with Staget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row
represents the French-to-German the second row represents French-to- Italian translation. The other settings are consistent with those in Figure
28

---

Published as a conference paper at ICLR 2025
0
1612 8 44
1612
1612
1612
1612
C
F2
4
1612 8
1612 8
1612 8
1612 8
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 36: Visualization of E[gene] and E[gaee] when encoder and decoder pruned separately for
German ASR. The columns represents the Suagett with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the encoder, the second row represents the decoder. The other settings are consistent with those in Figure 3
4
4
4
8
8
8
8
1612
1612
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
F1
F2
0
0
4
1612 4
1612 8
1612 8
8
1612
1612
SELF SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC FFN
Figure 37:
Visualization of E[gene] and E E[gdee] when encoder and decoder was pruned jointly for
Genman ASR. The columns represents the Starget. with Surget being 0.1. 0.3, 0.5. 0.7, and 0.9 from left to right. The first row represents the encoder, the second row represents the decoder. The other settings are consistent with those in Figure 3
8
8
8
1612
12
1212
1612
1612
C
F1
F2
F1
F2
A
F1
F2
A
4
4
4
4
16 4
16 8
8
1612 8
8
12
16
A
C
F1
F2
C
F1
F2
A
C
F1
F2
F1
F2
A
C
F1
F2
Figure 38: Visualization of E[gene] when encoder was pruned separately. The columns represents the Surget. with Suaget being 0. 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the German-to- -t--Frenhh the second row represents German-to- -t-alin translation. The other settings are consistent with those in Figure
29

---

Published as a conference paper at ICLR 2025
0
4
1612 8
1612 8
1612
1612
1612
SELF
SRC
SRC
SRC
4
1612 8
1612 8
1612 8
1612
1612
SELF SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 39: Visualization of E[gdee] when decoder pruned separately. The columns represents
the Surget. with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
German-to-Frente French, the second row represents German-to-Italian translation. The other settings consistent with those in Figure]
4
4
8
8
1612 8
8
1612
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
A
F1
F2
0
0
4
1612 4
4
4
1612 8
8
1612
1612
1612
C FI
FL F2
FI
F2
Figure 40: Visualization of E[gene] when encoder was jointly pruned with decoder. The columns represents the Surget, with Staget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the German-to- to-eencchh the second row represents German-to to-alia translation. The other settings are consistent with those Figure
1612 8
1612 8
8
1612
1612
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
4
4
4
4
1612 4
1612 8
16 8
1612 8
1612 8
SELF SRC FFN
SELF
SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 41: Visualization of E(ga<] when decoder was jointly pruned with encoder. The columns
represents the Surget, with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row
represents the Corman-to-Frenche the second row represents German-to- to--lalaan translation. The other settings consistent with those in Figure 3
30

---

Published as a conference paper at ICLR 2025
1612 8 44 0
1612
1612
1612
1612
C
F2
1612 8 4
1612 8
1612 8
1612 8
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 42: Visualization of E[genc] and E[gaec] when encoder and decoder pruned separately
for Italian ASR. The columns represents the Surget, with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left right. The first row represents the encoder, the second row represents the decoder. The other settings are consistent with those in Figure 3
4
4
4
8
8
8
8
1612
1612
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
F1
F2
0
4
1612 4
1612 8
8
1612
1612
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC FFN
Figure 43: Visualization of E[genc] and E E[gdee] when encoder and decoder was pruned jointly for
Italian ASR. The columns represents the Staget, with Starget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the encoder, the second row represents the decoder. The other settings are consistent with those in Figure 3
8
8
8
8
1612
12
1212
1612
1612
C
F1
F2
F1
F2
A
F1
F2
A
4
4
4
4
16 4
1612 8
8
1612 8
8
12
16
A
C
F1
F2
C
F1
F2
A
C
F1
F2
F1
F2
A
C
F1
F2
Figure 44: Visualization of E[gene] when encoder was pruned separately. The columns represents the Surget. with Suaget being 0. 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Italianto-Frrem the second row represents Italian-to-German translation. The other settings are consistent with those in Figure
31

---

Published as a conference paper at ICLR 2025
0
1612 8 4
1612 8
1612
1612
1612
SELF
SRC
SRC
SRC
1612 8 4
1612 8
1612 8
1612
1612
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 45: Visualization of E[gdee] when decoder pruned separately. The columns represents
the Surget. with Surget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the
Sellan-to-Frenchh the second row represents Italian-to-German translation. The other settings consistent with those in Figure]
4
4
8
8
1612 8
8
1612
1612
1612
1612
A
C
F1
F2
F1
F2
A
C
F1
F2
F1
F2
A
F1
F2
0
0
4
1612 4
4
4
1612 8
8
1612
1612
1612
C
FI
FL F2
A
F1
F2
Figure 46: Visualization of E[gen]] when encoder was jointly pruned with decoder. The columns represents the Surget, with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Italian-to- Frenchh the second row represents Italian-to- -o-ammnn translation. The other settings are consistent with those in Figure 3
1612 8
1612 8
8
1612
1612
1612
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
4
4
4
4
1612 4
1612 8
16 8
1612 8
1612 8
SELF SRC FFN
SELF
SRC FFN
SELF
SRC
FFN
SELF
SRC
FFN
SELF
SRC
FFN
Figure 47: Visualization of E[gae] when decoder was jointly pruned with encoder. The columns
represents the Surget, with Suaget being 0.1, 0.3, 0.5, 0.7, and 0.9 from left to right. The first row represents the Siiiin-to-Frenchh the second row represents Halian-to-German translation. The other settings are consistent with those in Figure 3
32


Reviews:
[{"review_id": "4EMw0SSXNn", "reviewer": "Reviewer_pWFs", "summary": "This research paper introduces a context-aware dynamic pruning technique for speech foundation models, enabling the models to adjust their structure based on contextual factors like language and task during inference. The authors showcase the effectiveness of their approach in a multilingual, multi-task setting, achieving approximately 30% reductions in inference time while maintaining BLEU score, ST task, and little degradation on the WER, ASR task. Additionally, they analyze the pruned model structure, offering insights into the varying importance of specific modules for different tasks and highlighting potential optimizations for speech processing architectures.", "questions": "Typo ? [line 130] if out validation data >> if validation data\n\nI think [at line 285] the sentence is incomplete. \n\nAt line 285, the authors make a statement \"This supports our initial question regarding the need for a large-scale decoder in ASR systems\". This is different form the results in the paper, pruning the decoder a lot does not effect ASR performance much compared to encoder as shown in Figure 2. \n\nIn Figure 1, it is not clear how mean pooling can be applied to the input of the transformer decoder given its autoregressive nature. \n\nNot able to understand after reading the paper, if the time reduction is from using the joint finetuned model or individual task finetuned model. I think it is for individual task finetuned models. \n\n[Please correct me if I am wrong]. The paper simply extends the work of Peng et al. (2023b) by utilizing the model structure of a speech foundation model to address multilingual and multi-task scenarios and therefore lacks major novelty. Please provide more detailed comparison of their approach to previous work and their novel contributions beyond Peng et al. (2023b).\n\nNot suited for joint pruning? As per the results, it is better to prune decoder in multitask setting. But authors report for the ASR and ST tasks SRC-ATT and SELF-ATT are individually important in the decoder. Therefore, the motivation to prune the modules in the joint finetuning is diluted.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper's main strength lies in its analysis of the importance of different modules for various speech processing tasks. The findings indicate that, within the encoder-decoder framework, pruning the decoder is more effective for speech translation (ST) and automatic speech recognition (ASR) tasks. The paper is well-written and easy to understand with minor typos and incomplete sentences.", "weaknesses": "The size of dataset (20 hours) used is too small to make general claims. \n\nThe authors do not compare their results with any other pruning methods. Like the effect of simply removing the layers, instead of different modules within, on the performance on the given the size of the dataset used in the study. \n\nWhile the paper focuses on reducing inference time, it doesn't address the computational overhead introduced by the dynamic pruning process itself and searching the best sparsity number during finetuning. \n\nLack major novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0vCZ52hDg9", "reviewer": "Reviewer_AdTV", "summary": "The work proposes a dynamic pruning method to make large speech models more efficient during inference. Instead of using a fixed pruning structure, the authors adaptively prune parts of the model based on the specific context—such as the language, task, and speaker characteristics—while keeping the model’s core structure intact. This approach aims to reduce inference computational costs without losing accuracy, especially in multilingual and multi-task scenarios. Experimental results show that the proposed method achieves up to 35% faster inference with minimal impact on accuracy for tasks like automatic speech recognition (ASR) and speech translation (ST). Key contributions include the development of a context-aware pruning mechanism and a detailed analysis of how different tasks and languages benefit from tailored pruning strategies.", "questions": "- How significant are the results? Can you provide information on the statistical significance of your results, such as confidence intervals? Reporting confidence intervals or statistical tests would strengthen the findings, especially since many works in adaptation and fine-tuning report these metrics to assess consistency and robustness.\n- Have you observed any patterns in performance across different subtasks or languages, particularly with varying sparsity targets?\n- Does your method require specifying the source language at inference, or can the model identify the language automatically, as Whisper does?\n- Would it be possible to provide code or configuration files to facilitate reproducibility?\n\nAs a side note, I recommend adding baseline (without pruning) to the tables as this would help visualization and understanding.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The approach is novel, introducing a dynamic pruning method with learned masking that adapts based on language and task contexts. The use of language tags to inform mask generation is particularly innovative, with strong potential for real-world impact.\n\n2) The method is technically well-executed, with careful design of the module-level pruning strategy and gate predictor. Experiments cover multiple tasks (ASR and ST), languages, and configurations (encoder-only, decoder-only, and joint pruning), demonstrating the method’s generalizability across varied settings. The authors include detailed subtask-specific performance analyses, which reassures method does not introduce potential bias. Also authors show particular attention to avoiding data contamination—a notable strength given the field’s reproducibility challenges.\n\n3) This work is promising for deploying large speech models in resource-limited environments, reducing inference time without sacrificing much accuracy. The context-aware pruning method offers an efficient, adaptable solution that could inspire further research into resource-saving models in NLP, computer vision, and other domains.", "weaknesses": "1) **Limited Baseline Comparisons:** The paper lacks comparisons with other dynamic pruning methods, particularly layer-wise pruning. The claim that finer-granularity pruning, like kernel or layer pruning, would disrupt model adaptability (lines 149-152) could be misleading, as studies such as [1] show that layer-wise pruning can maintain or even enhance model performance. Providing evidence for this claim, for instance by adding layer-wise pruning as a baseline, would provide valuable context for assessing the benefits of context-aware pruning, especially since there is a risk the current model is overly complex, potentially reducing its efficiency.\n2) **Reproducibility:** The paper is clear enough to support basic reproduction. However, providing the code would greatly benefit the community and improve reproducibility. Additionally, a brief explanation and citation for Gumbel-Softmax [4] would enhance clarity, as this technique, though widely known, is central to the gate predictor setup.\n3) **Need for more In-Depth Ablation Discussion:** While the paper analyzes different pruning configurations (encoder-only, decoder-only, joint pruning) with heatmaps for various subtasks being provided in the Appendix, it lacks discussion on how different languages or tasks are affected by these setups. Some languages may benefit more or less from the pruning process, potentially relating to their resourcefulness levels in OWSM pre-training data. Such insights could clarify the impact of context-aware pruning across subtasks.\n4) **Interpretability of Pruning Choices:** The paper discusses the interpretability of context-based pruning but could offer deeper insights. For example, quantitative metrics or visualizations on why certain modules are pruned would help clarify the gate predictor’s influence on pruning decisions and whether these choices are consistent across contexts. For example, you could study the correlation between pruning decisions and specific linguistic features of the input. Additionally, analyzing the impact of pruning only specific parts of the layers (e.g., self-attention vs. FFN) could improve understanding, as prior studies [2, 3] suggest these components may have varying importance across tasks and have different weight on the computational costs. \n5) **Contribution Clarification:** Network pruning using learned masks is not particularly novel, and have been studied in various domains, including self-supervised models for language and speech. Expanding on these methods in the related work section and clarifying the novelty of this specific approach would strengthen the paper’s positioning.\n\n### References\n[1] Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., & Santoro, A. (2024). Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. \n\n\n[2] Zhang, B., Bapna, A., Sennrich, R., & Firat, O. (2021). Share or not? Learning to schedule language-specific capacity for multilingual translation. ICLR 2021.\n\n\n[3] Ferraz, T. P., Boito, M. Z., Brun, C., & Nikoulina, V. (2024). Multilingual Distilwhisper: Efficient Distillation of Multi-Task Speech Models Via Language-Specific Experts. ICASSP 2024. \n\n\n[4] Jang, E., Gu, S., & Poole, B. (2017). Categorical Reparameterization with gumbel-softmax. ICLR 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTJBWaNicO", "reviewer": "Reviewer_Eam6", "summary": "The paper proposes a method to prune (specifically structured pruning) Whisper-like speech foundation models. The authors first show the dependency of optimal pruned networks on several auxiliary characteristics of the task, languages, etc. and finally show that their method can achieve similar performance on benchmark datasets with similar performance.", "questions": "- Is elapsed time the correct metric for comparing model efficiency? In my experience, any torch implementation does not yet improve efficiency in terms of flops as torch is not yet designed for it. Can the authors show metrics in terms of FLOPs maybe?\n- Please add wha conclusion we should draw from a figure to the caption of each figure. This makes the figures difficult to follow.\n\n don't have much questions. The findings presented in the paper are sound. However, I do not find anything much novel about the paper. Though novelty is not always that is required, but specifically and extension of an existing method to multilingual and multi-task scenarios is not very interesting to me. I would appreciate if the authors can point out whats novel in their \"novel context-aware pruning technique\" (except the gating technique) which is difficult for me to understand.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well written and is easy to read.\n- The paper targets an important task of efficiency in speech models. As speech models continue to grow, efficiency is important and is an important research area.\n- The proposed modification to the Context-Aware Gate Predictor is novel. To handle multiple languages and tasks simultaneously, the authors created vectors representing the language and task, combined them with the speech features, and used them as input to the Gate Predictors.\n- The analysis provided on sparse encoders and decoders is novel. The extension to multilingiuality and multi-task is new and promising.", "weaknesses": "- A major weakness of the paper is the lack of baselines and comparison with prior-art. While I understand that the paper proposes a structured pruning mechanism, the ASR community has built several unstructured pruning mechanisms [1,2,3]. Unstructured pruning methods have a motivation similar to this paper and [1] also show that they achieve similar performance with high amounts of pruning. These methods have not been discussed or compared neither used as baselines. \n- The paper feels like an analysis paper. Most methods applied are borrowed from Peng et al. (2023b), who also propose module-wise structured pruning. This paper just applies it to multilingual and multi-task scenarios and explores how a large-scale speech foundation\nmodel adapts its structure based on context.\n- The scope of the experimental section is rather narrow. The authors just show results on the Europarl-ST dataset and OWSM. A broader scope with more datasets and models would have been appreciated. Can Whisper be used? Can the authors expand the datasets employed for evaluation?\n- (Minor) I am not amazed by the results. It  is well known that these networks are sparse. This paper proposes a method to prune these networks. In my experience, pruning is not done extensively as it might lead the model to loose performance in other areas beyond benchmark datasets. For example, given every layer stores different kinds of information, pruning a layer based on ASR results might lead useful information for another language/task to deteriorate. I understand this is an academic paper and nevertheless the performance shown in this paper are somewhat promising. This point will not affect my scores.\n\n### Citations\n[1] Lai, Cheng-I. Jeff, et al. \"Parp: Prune, adjust and re-prune for self-supervised speech recognition.\" Advances in Neural Information Processing Systems 34 (2021): 21256-21272.    \n[2] Lodagala, Vasista Sai, Sreyan Ghosh, and Srinivasan Umesh. \"Pada: Pruning assisted domain adaptation for self-supervised speech representations.\" 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023.    \n[3] Fu, Yonggan, et al. \"S $^ 6$-DAMON: Unlocking Structured Sparsity in Self-Supervised Speech Models via Data-Model Co-Compression.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qbRLn7uX3W", "reviewer": "Reviewer_hGS7", "summary": "This paper addresses the problem of structured pruning for speech foundation models in the presence of context. The structured pruning is dynamically adjusted based on context during inference.\n\nFor each structured component in the model, a module selector estimates a mask, framed as a two-class classification problem, using Gumbel softmax. Experimental results on Speech Transcription and Speech Translation demonstrate the proposed approach’s effectiveness in improving inference efficiency without compromising performance.\n\nThe structured pruning also reveals key characteristics of several modules in the OWSM 3.1 architecture. Notably, pruning a decoder yields a significant speed-up compared to pruning an encoder.", "questions": "1. In Table 1, the inference time for the encoder-decoder model with s_target = 0.1 exceeds the baseline (10.39 vs. 9.28). Does this indicate that model selection (pruning) is computationally expensive?\n\n2. How critical is Gumbel-softmax for dynamic pruning? Could a simpler sigmoid-based selection function perform comparably in this setting? How are the temperatures for Gumbel-softmax adjusted?\n\n3. Based on this study, can we reasonably infer that the E-Branchformer architecture contains additional components that could be pruned for efficiency, potentially aligning it more closely with the Conformer architecture?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper underscores the effectiveness of the proposed structured pruning method with robust experimental results. \n\n- The ablation studies are thoughtfully designed, and each section provides detailed insights into the impact of various components in encoder, decoder, and joint pruning. \n\n- This work may inspire further research into new model architectures.", "weaknesses": "The original OWSM 3.1 model is a multilingual ASR model that covers a wide range of languages, each represented within the model's parameters. For high-resource languages, such as the one chosen in this study, pruning a significant portion of the network is often feasible without substantial performance degradation. However, for low-resource languages, the effects of similar pruning may lead to severe performance deterioration. Analyzing these effects could help clarify the limitations of this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)