Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
Authors: Yonggan Fu
Yang Zhang
Kaizhi Qian
Zhifan Ye
Zhongzhi Yu
Cheng-I Lai
Yingyan Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: automated speech recognition
self-supervised learning

Abstract: Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S$^3$-Router framework, which for the first time discovers that simply discarding no more than 10% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S$^3$-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. We believe S$^3$-Router has provided a new perspective for practical deployment of speech SSL models. Our codes are available at: https://github.com/GATECH-EIC/S3-Router.

Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
Yonggan Fu', Yang Zhang?, Kaizhi Qian', Zhifan Ye', Zhongzhi Yu! Cheng-I Lai', Yingyan (Celine) Lin'
GGerrgia Institute of Technology, 'MIT-IBM Watson AI Lab, "Rice University, "MIT CSAIL {yfu314, celine. 1im] @gatech. edu [yang. zhang2, chang2,kqian) (iqian))@iim com (ryio)Orice. edu (0lai24)mmit..o
Abstract
Self-supervised learning (SSL) for rich speech representations has achieved empiri- cal success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of large amount of transcribed speech and thus has driven growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be severe multilingua//mulittskk scenarios requiring simultaneously recognizing multiple languages executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims enhance the practical usage of speech SSL models towards win-win in both enhanced efficiency and alleviated overfitting via our proposed S -oouer framework. which for the first time discovers that simply discarding no more than 10% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, $ S can serve as an all-in-one technique to enable (1) new finetuning scheme, (2) an efficient mulillngual/mmutitass solution, (3) a state-of-the-eee ASR pruning technique, and (4) new tool quantitatively analyze the learned speech representation. We believe RR-Router has provided a new perspective for practical deployment speech SSL models. Our codes are available at: https: // thub. Cee//GATEC---EA EIC/S3 S3-Router.
Introduction
Deep neural network (DNN) breakthroughs have tremendously advanced the field of Automatic Speech Recognition (ASR). However, one major driving force for powerful DNNs, i.c., the availability of large amount of training data, is not always possible for ASR. This is because collecting large- scale transcriptions costly, especially for low-resource spoken languages around the world, limiting the wide application of deep ASR models. Fortunately, recent advances in self-supervised learning (SSL) for rich speech representations [1, 2, 3, 4, 5, 6, 7, 8, 9] have achieved empirical success in low-resource ASR, where SSL models pretrained finetuned on low-resource transcribed speech to match the accuracy of their supervised counterparts.
audio data without transcriptions can be
However, there exists dilemma between the trends of speech SSL models and the growing demand for speech processing applications on the edge. While advanced speech SSL models become increas- ingly larger to learn more generalizable features, it is highly desired to process the captured speech
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

signals in real time on edge devices, which have limited resources and conflict with the prohibitive complexity of existing speech SSL models. Such an efficiency concern would be more severe in multilingua/mmulitaak scenarios where simultaneously recognizing multiple languages or executing multiple speech processing tasks required: if one separate model is finetuned for each target language/task, the storage and computational cost will be significantly increased, thus prohibiting the practical deployment of existing speech SSL models. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on a low-resource speech corpus [10, 11, 12, 13, 14], limiting the achievable accuracy improvement brought about by more parameters and thus the achievable performance-efficiene
trade-off.
In this work, we aim to facilitate the practical usage of speech SSL models towards win-win in enhanced efficiency and alleviated overfitting for boosting task accuracy under low-resource settings. Excitingly. we develop a framework, dubbed self-Supervised Speech Representation Router (S Router), that can serve as an all-in-one technique to tackle the aforementioned challenges and largely enhance the practical usage of speech SSL models, contributing (1) new finetuning scheme for downstream speech processing, i.e., finetuning the connections of the model structure via learning a binary mask on top of pretrained model weights, which notably alleviates model overfitting and thus improves the achievable accuracy over standard weight finetuning methods under low-resource setting; (2) a multiinggul//mulittskk technique via learning manguage-taask-speciff binary masks on top of shared model weights inherited from SSL pretraining; (3) competitive ASR pruning technique to trim down the complexity of speech SSL models while maintaining task accuracy; and (4) new tool to quantitatively analyze what is encoded in speech SSL models thanks to the learned masks' binary nature on top of shared model weights. We summarize our contributions below:
We propose a framework dubbed Ro-Rou, which offers alternative to the mainstream finetuning of model weights that finetunes the structure of speech SSL models, by learning a binary mask on top of the pretrained model weights and integrating it with novel mask initialization strategy customized for the pretrain-fine fitrtiinf paradigm;
We are the first to discover that discarding no more than 10% of weights without finetuning pretrained model weights can achieve better task performance as compared to the mainstream method of weight finetuning on downstream speech processing tasks. Notably, our method can scale well to even larger models;
We extend our RooRer framework for enhanced deployment efficiency, contributing (1) novel mulillingul/mmulitaak solution, and (2) a competitive pruning technique achieving bet- ter performancc-efficieee efffciency trade-offs than state-of-the-an (SOTA) ASR pruning techniques;
We demonstrate the capability of $ -Router as tool to quantitatively understand what is encoded in speech SSL models thanks to the binary nature of the learned masks.
S 3--Ruteer has opened up new perspective for empowering efficient multilingual/mmltitsskk speech processing and enhancing our understanding about what is encoded in speech SSL models.
2 Related Work
Automatic speech recognition. Early ASR systems [15, 16, 17, 18, 20] were mainly based on the combinations of hidden Markov models (HMM) with Gaussian mixture models or DNNs, and often contain multiple modules (e.g., an acoustic model, language model, and a lexicon model) trained separately. Recent works process raw audio sequences end-to-end, including CTC [21] models [22, 23, 24, 25, 26), recurrent neural network(RNNN)
[27, 28, 29, 30], sequence- to-sequence models [31, 32, 33, 34, 35, 36), and transformer-based models [37, 38, 39j. Specifically, transformer-basee models have been widely adopted as speech SSL models [6, 9, 7].
Self-supervised learning for speech representation. Considering the high cost of collecting large-scale transcriptions, learning rich speech representations via SSL has become crucial and promising for empowering low-resource ASR. Early works [40, 41, 42, 43, 44, 45, 46, 47] build generative models for speech with latent variables. Recently, prediction-ban based SSL methods have become increasingly popular, where the models are trained to reconstruct the contents of unseen frames [48, 49, 50, 51, 52, 53, 1, 2] or contrast the features of masked frames with those of randomly sampled ones [3, 4, 5, 6, 7, 8, 9]. We refer the readers to a recent survey [54] for more details. Among prior arts, [55] is a pioneering work relevant to our method, and adopts masking as an alternative to weight finetuning on pretrained language models for natural language processing (NLP).

---

Input
Our R-Router Framework
Output
MM
Header1
Hello World
Header2
Header3
English
Speech Signal
Hola Mundo
Spanish
Speech Signal B
Shared
Weights
Speech Signal
Language-specific Masks
Mandarin
An overview of our Rd-RRuter framework, which receives multilingual speech signals denoted as A, B, and C here and then outputs the corresponding text transcript of predication, based on one shared weight model together with prgguage-/task-speeeee binary masks.
Figure 1:
Nevertheless, our S 2e-Rotter is non-trivially different from [55] in that (1) we target SSL models in the speech domain and the learned speech representation is required to be generalizable across different spoken languages under cross-lingual transfer setting, where the speech SSL models suffer from a higher risk overfitting on downstream low-resource tasks as compared to NLP, making our findings non-trivial contributions; (2) S Rooter features a win-win in both efficiency and accuracy thanks to our proposed mask initialization strategy, which plays a crucial role in maintaining task accuracy under high sparsity and enables the extension of S towards competitive pruning technique with SOTA accuracy; and (3) we further develop $ Router for multilingua//multien speech processing and simple yet effective tool for analyzing language-wise similarities.
Multitask learning for speech processing. There exists a growing demand for DNNs to simultane- ously process multiple tasks [56, 57). In the speech domain, previous works attempt to train single model to solve multiple tasks [58, 59, 60, 61, 62, 63) or use an auxiliary task to enhance the accuracy of primary task [64, 65, 66, 67, 68, 69]. Motivated by the success of SSL in low-resource downstream tasks, multitask/multiiinguaal learning has been adopted both SSL pretraining [70, 71, 72, 73, 7] to enrich learned speech representations, and finetuning scenarios [74]. For example, a recent relevant work [73] adopts hanguage-adaptiv pretraining on top of [7], where different sparse sub-networks are activated for different languages during multilingual pretraining and the gradients accumulated on the shared super-network to learn better multilingual representations. In contrast, Ro-Router is fundamentally different, as it targets the finetuning stage of given pretrained speech SSL model and only tunes the connections of the model structure without tuning the SSL pretrained weights.
ASR pruning. As ASR models become increasingly overparameterized for abstracting generalizable representations, pruning large-scale ASR models has drawn a growing attention. Early works trim down either the decoding search spaces [75, 76, 77, 78, 79, 80] or HMM state space [81]. Modern ASR paradigm has gradually shifted its focus to pruning end-to-end ASR models [82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93]. Recently, [94, 95, 96] prune speech SSL models towards more efficient low-resource ASR, e.g., PARP [94] proposes a finetuning pipeline to identify the existence of lottery tickets within speech SSL models. In addition to the boosted pruning efficiency over PARP [94], RRoRou emphasizes the role of sparsity in encoding Lunguage-task-speeiin information which can empower multitask/multiinnuual speech processing without tuning the pretrained weights.
3
The Proposed RRouue Framework
3.1 Drawn Inspirations from Previous Work
Recent works [97, 98, 99] find that sub-networks featuring a decent inborn accuracy and adversarial robustness are hidden within randomly initialized networks without any weight training. Specifi- cally, 197, 98] show that sub-networks with decent accuracy, even matching that of their dense networks, can be identified from randomly initialized networks, and [99] shows even stronger
3

---

evidence: merely updating the sparsity patterns of model connections without modifying the randomly initiated model weights can produce both accurate and adversarially robust models. These pioneering works imply that tuning the connections of the model structure, which can be characterized by the learned connection sparsity patterns, can be as effective as training the model weights. We hypothe- size that model sparsity not only can favor model efficiency, but also can serve as similar role, i.e, another optimization knob, as model weights to encode aanguage-taask-speiitt information.
3.2 Formulation and Optimization of S*-Rte
Inspired by the aforementioned intriguing hypothesis, we propose the Sh-Router framework to empower efficient multilingual and multitask speech processing on top of SSL speech representations.
Overview. As shown in Fig. I, given the raw audios of different spoken languages (or different tasks), R*-Router finetunes the connection patterns of the model structure for each target spoken language/task via optimizing tanguage-/tas-sspeeee given speech SSL model, instead of finetuning the model weights as adopted in the common pretrain- finetune paradigm. Specifically, the learned binary masks of each language/task are multiplied with the shared model weights mask out some connections within the given speech SSL model, where the remaining connections (or the induced connection sparsity) encode unguagg-/task-speccee information for different spoken languages and downstream tasks. Note that for each language or task, only one set of binary masks and one lightweight header, c.g., the classification head for ASR which is naturally non non-shareable across languages due to diverse dictionary sizes, need to be independently trained, incurring negligible overhead 5.33% storage of the whole backbone model).
binary masks on top of the shared weights of a
Formulation. Formally, our S -Router framework can be formulated as:
arg min 174
Σ (ELDUEDD
(e(f(m + ASSL, ve). s.t. |/mello & Re
(1)
where (xr, ve) are the input audio and corresponding transcriptions/labels of a spoken language/task / in downstream dataset D,, and 0SSL is the SSL pretrained weights of the given speech SSL model f. Specificallyy, the mask set m, applied on top of the model weights 0SSL, and optimized to minimize the loss function 4,, c.g., a CTC loss [21] for ASR, subject to an Lo sparsity constraint, where the number of non-zero elements in m. limited to k2, which serves as a hyperparameter information encoded in the
for controlling and balancing (1) the amount of Hunguage-/task-specien connection sparsity and (2) model efficiency. Intuitively, the sparsity is 0% or 100%, no new information is introduced during finetuning on the corresponding new/downstream language/task.
Optimization. To differentiably optimize my in Eq. (I), we binarize m, and activate only its top Ae elements during forward, while all the elements in m, are updated via straight-through estimation [100] during backward. More specifically, during forward, we binarize m, to my via enforcing its top he elements to I and other elements to O, thus the forward function becomes f O OssL, 2t), which is the same for the inference process; during backward, we directly propagate the gradients from the binary mask in m, i.e., Din. Bmgy" gradient-based manner.
Al R BI thus m, be learned in
3.3 How to Initialize the Masks in S RRouter?
Importance of mask initialization. A low learning rate is commonly adopted during finetuning to ensure effective inheritance of the SSL speech representations, resulting in merely smaller changes in each set of the masks my in Eq. (1) as compared to training from scratch. Therefore, the mask initialization strategy plays an important role for the quality of the finally optimized masks in S Router. Next, we discuss the pros and cons of two intuitive mask initialization strategies:
① Random initialization (RI). We empirically find that adopting commonly used random initializa- tion [101] in S -Roueer can achieve a decent accuracy under a low sparsity. However, since no prior knowledge of the speech SSL model is utilized, the accuracy can largely drop with a high sparsity under random mask initialization, limiting extending S -ooueer to be a practical pruning technique.
@ Weight magnitude based initialization (WMI). As weights magnitude can quantify the impor- tance weights as commonly used in pruning [102, 103, 104, 105, 106), taking the magnitudes of the given SSL pretrained weights |00ssz|| in Eq. I as the initial values of masks might help utilize

---

the knowledge learned during SSL pretraining. However, we empirically find that doing so causes worse trainability, i... the ranking of mask values is then more stable during finetuning than that under random initialization, and learned binary masks seldom change and mostly stick to their initial values. This may inhibit the optimization process and lead to sub-optimal learned masks.
@ Proposed Order-Prese Peeder-Preeerving Random Initialization (ORI). To marry the best of both above initialization strategies, we propose new one, which can be viewed as weight rank based initializa- tion. In ORI, we first acquire mask values of the same dimension as the shared weights via random initialization like [101], and then perform magnitude- -ased sorting to assign a larger mask value to weight elements with a larger magnitude. Thus, the ranking order between the mask values of the weight elements is the same as that of their magnitudes. In this way, the mask trainability is maintained while the learned speech SSL model knowledge can be exploited (see Sec 4).
3.4 -Router is Useful in Various Application Scenarios
new finetuning scheme. S offers new and equally effective finetuning scheme as finetunes model connections given a speech SSL model. As large-scale speech SSL models tend to overfit when finetuning their weights under low-resource setting, we hypothesize that the binary optimization of the mask patterns in Eq. (1) can serve as regularization and thus alleviate overfitting.
An efficient multilingual and multitask method. S can naturally enable multilingual and multitask speech processing by merely switching among its learned language-/task-peecil binary masks. One advantage that since the gradients of different spoken languages tasks can now be independently accumulated on their corresponding masks without interfering each other, the commonly observed gradient conflict issue [107] can be alleviated.
A new pruning technique. Since S So-Router encodes language/task--peeceee information via binary masks top of shared model weights, it naturally introduces sparsity into the given model, and thus can naturally serve as pruning technique. To further improve the achievable accuracy-efficiens trade-off and more fairly benchmark with SOTA ASR pruning methods, we propose a variant of S dubbed S Routerrr-- which first finetunes the and then prunes the model connections based on the learned binary masks in Eq. (I).
weights on the downstream audios
An effective and simple tool to analyze what is encoded across ggguuag//taskspecifff speech SSL models. Another exciting advantage of S Ro-Roueer is that it can provide a quantitative metric about how the pretained SSL speech presentation is utilized by different spoken languages/tasks via masking out languagesstusks-spccifi weights.
4 Ro-Rourer: Discarding <10% Weights is All You Need
4.1 Experiment Setup
Here we evaluate -ooRur as new finetuning scheme for downstream speech processing.
Models. We adopt wav2vec 2.0 base/large (war2vee2-base/lage)) [6] and data2vec [108] pretrained on LibriSpeech 960 hours [109] and xlsr [7] pretrained on 128 languages sampled from Common- Voice [110] as our speech SSL models.
Datasets, We consider 10 speech processing tasks, including low-resource English ASR on Lib- riSpeech [7] with only 10min/1h/100 labeled data, following the dataset split in [6, 94], and low- resource phoneme recognition on Common Voice [1] with 1h labeled data per language, following the dataset split in [111, 7], as well speech processing tasks from SUPERB [112].
Finetuning settings. Our code is built on top of fairseq [113] and we follow the standard finetuning settings for each task, i.c., the default configurations in fairseq [113] for ASR/phoneme recognition and those in SUPERB [112] for other tasks. In particular, all our experiments on ASR/phoneme recognition adopt an Adam optimizer with initial learning rate of 5e-5 plus tri-stage schedule [6] and we finetune wav 2wweee-aeeeeeaarrreeeeeaaa
for 12k/1 1k/10kk20kk steps on the 10m/1h/100 11hhhhh11hhh splits, respectively, and xlsr is finetuned for 12k steps for each spoken language. It takes about 10/24/24 GPÚ hours to finetune waw2/eec2bbss//arrg//t base/large/xlsr for 12k steps with our 3 -Rouer. We do not freeze all the layers except the final linear layer for the first 10k steps [6], following [94].
Ro-Router settings. If not specifically stated, -RoRouter only tunes the connections of (i.e., applies learnable masks on) the feed-forward networks (FFNs) of transformer structures [114] and fixes all
5

---

Rooutr on test-clean
S*Router on test-other
Weight ft on test-clean
Weight on test-ather
50.0
35
WER (96
42.5
40.0
0.1
0.2
0.4
0.5
0.5
0.0
0.1
0.2
03
0.5
0.6
0.0
0.1
02
0.3
0.4
0.5
0.6
wav2vec2-base
Libri-10m
(b) wav2vec2-base Libri-1h
wav2vec2-base
Libri-10h
WER (96
0.2 0.05
0.04
0.25
0.05
0.07
0.58
0.09
Sparsity
Sparsity
Sparsity
(e) wav2vec2-large @ Libri-10m
(f) wav2vec2-large @ Libri-1h
(g) wav2vec2-large Libri-10h
Figure 2: Benchmark our S Router and standard weight finetuning on the test-cleanttest-othr sets
of LibriSpeech on top of wav2vec2- haw2vec2-base/large under different low-resource settings.
Dutch
Mandarin
Spanish
Tatar
Russian
20100
PER (96)
1,0
Router Wtight.ft
0.22
0.34
600
1.00
0.32
0.1
0.3
0.02
5.04
306
0.58
0032
GON
0.00
3.00
0.22
0.34
6:06
1.00
Italian
Kyrgyz
Turkish
Swedish
France
PER (96)
30
160
Sparsity
Sparsity
Sparsity
Sparsity
Sparsity
Figure 3: Benchmark our Roouur and weight finetuning on xlsr across 10 spoken languages.
other weights, which is empirically found be optimal based on the ablation studies in Sec. In addition, if not stated, we adopt our ORI mask initialization by default and do not apply language models for fair benchmark of ASR performance, following [94].
4.2 Benchmark on Low-resource English ASR
Finetuning wav2vec2-hase. We apply our -oouter wav2vec2-base under different low- resources settings as shown in Fig. (a)~(c). We can observe that our $3 33-Rouer can consistently outperform the standard weight finetuning in terms the achievable WER, i.e., the lowest WER the corresponding optimal sparsity. In particular, our S9-Router achieves a 1. 172%/2..44 reduction in word eгrоr (WER) under sparsity ratio of 8% the test-clean/test-otherr set of LibriSpeech, respectively. when being finetuned on 10min labeled data. This indicates that finetuning connections via our 3 RRouter can be a competitive alternative with reduced WER for finetuning weights under low-resource settings, which provides new paradigm for finetuning speech SSL models.
A Allaioon studies of mask initialization. We equip $ 33-Router with different mask initialization schemes in Sec. 3.3 and show their achievable WER on low-resource English ASR Tab. 1. We can observe that (1) the proposed ORI mask initialization consistently outperforms the other schemes in terms of the achievable WER, and (2) even random mask initialization can match or surpass the performance of standard weight finetuning. We also provide their complete sparsity- WER trade-offs in the appendix and find that weight magnitude based initialization favors larger sparsity ratios, while is harder to overturn the ranking between masks via gradients, resulting in inferior achievable WER.

---

Scalability to larger speech SSL models. We apply R to larger model wav2vec- large. As shown in Fig. 2 (g), we can see that the achievable WER of S still con- sistently outperforms the standard weight fine- tuning across all downstream datasets, e.g., a 1. 1.99% WER reduction under 4% spar- sity on the test-clean/test-ofe set, respectively. when being finetuned on the Ih labeled data. Consistent results xlsr provided in the appendix.
Table I: Benchmark different mask initialization
schemes of Ro-Router with weight finetuning on LibriSpeech test-clean/test-other sets.
Method
Libri-10m
Libri-1h
Libri-10h
Weight ft
38 48/44/9998
18.963/26.
9.7/18.19
RI
36.98/43.42
18. 18/19/26.1
9.78/18.17
WMI
40.09/46.63
19.99/27.16
11.17/18.97
ORI (Ours)
36.68/42.64
17.95/25.47
9.66/17.27
Insights. This set of experiments indicates that 1 our -9-Router features a good scalability to larger speech SSL models as well as a good generality for different pretraining schemes, and @ our method effectively reduces the overfitting on more overparameterized speech SSL models according to the larger performance gains, thanks to the regularization effect of the binary optimization process in Eq. (1), making our method an appealing solution for more advanced speech SSL models.
Add language models (LMs). We further ap- ply a 4-gram LM [115] as the decoder for both our R*-Router the optimal sparsity ratio in Fig. and the weight finetuning baselines. In particular, we adopt the official 4-gram language model (LM) [115] with beam size of 50, an LM weight of 2, and a word insertion penalty -1. As shown in Tab. 2, our S Ro-Router still consistently outperforms weight finetuning with more notable reductions on wav2vec2-large e.g., a
Table Benchmark our S Router with standard weight finetuning when being equipped with 4- gram LM [115].
Model
Method
Libri-10m
Libri-1h
Libri-10h
wav2vec2 -bse
Weight n
26.55/32.95
11,54/18.45
6.65/13..7
R*-Router
25.28/31.07
11,28/18.33
6.42/13.14
way2vec2 -large
Weight It
24.83/29.05
11.18/15 80
5.52/10.31
S*-Rotter
23.17/25.83
9,74/13.77
4.98/9.27
1. 1.66% WER reduction on the test-clean/test- clean/test-other set of LibriSpeech when being finetuned on 10min labeled data.
Scalability to speech models pretrained by other SSL. paradigms. To validate the generalization capability of our Ro-Router across speech models pretrained by other SSL paradigms, we further apply our method on the
Table 3: Benchmark our Rooutr under different
sparsity ratios with weight finetuning on top of data2vec on LibriSpeech test-clean/test-othee sets.
Method Libri-10m
Libri-Ih
Libri-10h
Standard n
30,75/34.612
14.15/19.61
7,28/13.11
SOTA speech SSL model data2vec [108] un- der different sparsity ratios, which features new SSL pretraining paradigm based on self- distillation. Here we follow the same finetuning setting as wav2vec2-base, which is also the de-
Sa. Router@0.07
30.78/35.17
14.09/19.72
7.56/13.39
Router
30.70/34.45
13.96/1941
7.43/13.34
S Ronter@0.19
29.86/34.09
13.92/19.43
7,23/13.25
S. Router@0.10
31. 31.30/3..07
14.27/20.10
7,05/12.98
fault finetuning setting of data2vec in fairseq [113]. As shown in Tab. 3, we can observe that our So-Router still achieves lower WER across all resource settings, e-g., a O. 0.89% WER reduction on LibriSpeech test-clean when being trained with 10m labeled speech. This indicates that our S can generally serve as a competitive finetuning paradigm independent of the SSL scheme.
4.3 Benchmark on Low-resource Cross-lingual Transfer
We further evaluate our $ -oouter under two cross-lingual transfer settings, including (1) finetuning the multilingual pretrained xisr on different low-resourc, languages in Common Voice [110], and (2) a high-to-low resouree transfer setting where the monolingual (English) pretrained wav2vec2-baas is finetuned on multiple low-resource languages in Common
Cross-lingual transfer xlsr. As shown in Fig. 3, our Rooter consistently outperforms standard weight finetuning in terms of the achievable phoneme error rate (PER) across all the 10 spoken languages, e.g., a 1. 1../2.019 1% PER reduction on Dutch/Mandarin, respectively.
High-to-low resource transfer on wav2vec2- base As shown in Tab. 4, on top of the English pretrained wav2vec2-base, finetuning the con- nections with S Router still wins the lowest achievable PER over the baseline. Consistent results on wav2ve22-large are in the appendix.
Table 4: Benchmark our &ooRur and weight finetuning on wav2vec2-base and Common Voice.
Language
Dutch
Mandarin
Spanish
Tatar
Russian
Weight
19.82
26.67
13.86
11.14
17.05
8 Router
18.51
26.10
13.37
10.94
16.33
Language
Italian
Kyrgyz
Turkish
Swedish
France
Insights. This set of experiments implies that for each downstream spoken language, there exist
Weight ft R
19.27
13.41
15.70
20.81
19.35
18.29
12.30
14.82
19.64
17.94
7

---

Table Benchmark S*-Router with standard weight finetuning 8 tasks from SUPERB [112].
Catagory
Content
Speaker
Paralinguistics
Semantics
Keyword Spotting
Speaker Identification
Speaker Verification
Speaker Diarization
Emotion Recognition
Intent Classification
Slot Filling
Speech Translation
Task
Metric
Ace
Acc 1
EER
DER
Acc t
Acc f
F1 1
BLEU
Weight
66.8
7.24
7.41
18.85
8 -Ronter
95.7
71.07
6.79
7.18
62.13
92.6
88,76
19.01
Ope Spar. 0.1
0.06
0.1
0.08
0.1
0.1
0.1
0.08
decent subnetworks for processing its Einguage-speciffe information even in monolingual pretrained speech SSL models. In another words, properly learned sparsity can encode sunguage-specinn information with competitive ASR performance.
4.4 Benchmark on More Downstream Speech Processing Tasks
We further evaluate our SooRouter via finetuning wav2vec2-base on 8 speech processing tasks from SUPERB [112], covering different aspects of speech show the achievable task performance as well as the corresponding optimal sparsity ratios Tab. standard weight finetuning across all
We
and find that method surpasses the task performance tasks via discarding < 100% weights, indicating that properly learned sparsity can also effectively encode task sask-specific information. We provide the sparsity-performanee performance rrade-offs in the appendix.
4.5 Empowering Multilingual and Multitask Speech Processing
Since all the aforementioned results achieved on the same speech SSL model via $ 3-oRur share pretrained weights, it can simultaneously enable multilingual and multitask speech processing, thanks to the independent accumulation of task-specific gradients that avoids gradient conflicts [107], of superior scalability with the number of tasks. For example, compared to independent weight finetuning for each language/task. S So-Router can simultaneously support 11 languages in Sec. 4.2/ 4.3 and 8 tasks in Sec. 4.4 using wav2vec2- while achieving win-win in both accuracy (as validated in Sec. 4.2/ 4.3/ 4.4) and efficiency, i.c., more than 88.5% reductions in model parameters.
4.6 Benchmark S RRoRur with Adaptor Tuning
We further benchmark our S Roouur with adap- tor tuning [55], which has emerged as an al-
Table 6: Benchmark S RouRtr with adaptor tuning [1 on top of wav2vec2-base across Lib-
ternative finetuning scheme. In particular, we reproduce the speech adaptor design [1]6], following their open-sourced implementation, on wav2vec2-base and benchmark with our re-
riSpeech test-clean/other and Common Voice.
Method
Libri-10m
Libri-Ih
Libri-10h
Standard
38.40/44.99
18.96/26.80 30
9.70/18.1
Adaptor [116]
41.82/49.01
21.19/28,0 92
12.26/19.67
ported results of Router under the best spar- sity settings for different languages. As shown in Tab. 6, still consistently achieves the lowest WER/PER across all languages and resource settings, e.g., a 5.14% WER reduction on LibriSpeech test-clean when being trained 100ri/peech--0mm/1, respectively.
R*-Rue
36.68/4.664
17.95/25.47
9.77/17.27
Method
Dutch
Spanish
Mandarin
observe that our 3R-Router
Standard ft
19.82
13.86
26.67
Adaptor [116]
22.63
15.89
29.03
RR-Router
18.51
13.37
26.10
Insight. Under a low-resource setting, it is hard for adaptor tuning to maintain a comparable accuracy over standard weight finetuning, while our S often outperforms standard weight finetuning as shown in Sec, 4.2, thanks to binary optimization process on the masks, which can potentially regularize the learning process and lead to better downstream performances as compared to finetuning the overparameterized model weights. This indicates that R is better finetuning scheme over adaptor tuning, especially under low-resource settings.
5 S Routerr-: Pruning ASR Models for Enhancing Efficiency
Setup. For evaluating S²-Router-P (see Sec. 3.4), we conduct two modifications: (1) both FFN and self-attention (SA) modules are pruned for a fair comparison, and (2) we adopt weight magnitude

---

LibriSpeech-10ml
LibriSpeech-1h
LibriSpeech-10h
100
100
80
OMP
IMP
60
PARP
WER
73
S Rooterr
40
20
20
Sparsity
Sparsity
Sparsity
Figure Benchmark our S S9-Router-P aguuter-Paaainst OMP, IMP, and PARP [94] for pruning wav2vec2-base on LibriSpeech. The WER on the test-clean set is reported.
based mask initialization for reasons: firstly, it features the best scalability to high sparsity among the three initialization schemes; secondly, since the weights have been finetuned, their magnitudes can serve indicate their importance on the downstream speech. Ablation studies for pruning under different mask initialization schemes are in the appendix.
For baselines, benchmark with three ASR pruning meth- ods, i.e., one-shot/ieeratiee magnitude pruning (OMP/MP), and the SOTA method PARP [94] under the best setting (dubbed PARP-P), and directly adopt their reported results in [94].
OMP
60
PARP
SP-Router-P
50
PER
Pruning wav2vec2-base on LibriSpeech. As shown in Fig. 4, we can observe that (1) our S -Router-- consistently achieves the most competitive -yR-sptastty trade-offss across the two models and three datasets, c.g., a 6.46% lower WER over PARP under sparsity ratio of 0.7 with 10min labeled data, and (2) our method features a better scalability to more stringent low- resource scenarios according to the large performance gains LibriSpeech-100
0.2
0.4
0.6
0.8
Sparsity
Figure 5:
Benchmark our S3-
Router-P against OMP and PARP for pruning on Mandarin.
Pruning wav2vec2-bass on Mandarin @ Common Voice. As shown in Fig. 5, consistent observations can be drawn that our method still wins the WER-sparsity trade-offs. More pruning results are provided in the appendix.
Router:: What is Encoded in Speech SSL Models?
6.1 What is The Roles of Different Modules in Speech SSL Models?
tune the connections subset the modules the speech SSL model via our So-Router to study their contributions with other modules fixed in terms of both weights and connections.
FEN vs. SA. We tune the connections of FFN, SA, or both on top of wav2vec2-bass and LibriSpeech- 1h as shown in Tab. 7. We can ob- serve that tuning the connections of FFN only wins the lowest achievable WER, even outper- forming tuning both FFN and SA. This indicates that the SSL pretrained SA modules are general- izable enough to capture temporal relationship between tokens for downstream speech thus only FFN needs to be tuned for encoding new information about downstream tasks.
Table 7: The achievable WER on LibriSpeech test-
clean/test-other when finetuning different modules.
Sparsity
SA only
FFN only
Both
0.1
24.39/30.79
18.62/26.12
19.26/27.36
0.2
23.06/30.22
19.54/27.07
18.99/27.38 338
0.3
22.78/30.21
19.59/27.58
19.65/28.44 40
0.4
23.22/31.28
20.02/28.97
19.78/27.98
To study the roles of the blocks at different depths, we uniformly divide the 12 blocks in wav2vec2-basd into 4 groups and only tune the connec- tions of one group. As shown in Tab. 8, we indicate the tunable group 1 and other fixed ones as We can observe that generally tuning later groups achieves lower WER and tuning the st/2nd groups only will result in unacceptably high WER, aligning with our intuition that early blocks speech SSL. models extract generalizable phonetic features while later blocks capture task task-specific features thus are required to be tuned. In
Roles of different blocks.
Table Apply So-Router on different groups of blocks. WERs LibriSpeech test-clean/test-oree are reported.
Group
Libri-1h
Libri-10h
[1,0,0.0]
85.68/92.12
77.65/86.1 12
[0.
54.96/67.17
42.26/56.24 24
[0,0,
24.57/32.68
14.95/22.96
[0,0,0.1]
35.42/4
20.88/27.29

---

addition, tuning the 3rd group achieves the lowest WER, even close to that of tuning all groups, which we assume is because the features starting from the 3rd group are task specific and not generalizable across downstream tasks thus demandingly require to be finetuned.
6.2 How Much New Information is Learned by Finetuning?
We visualize the optimal sparsity ratio for achieving the lowest WER on LibriSpeech- 10m/1h/10h, which serves as an indicator about the amount of new information learned by finetuning, across wav2vec2-based weee-/////ll/r//ls:ss in Tab. 9. We can observe that (1) larger speech SSL models reach their optimal performance under lower sparsity, i.c., relatively less amount of tuning could lead to their decent downstream performance thanks to their overparameterization,, and (2) finetuning more resources leads to higher optimal sparsity, indicating that speech SSL models have more confidence to update the SSL pretrained representations given more resources.
Table The optimal sparsity ratios for the lowest WER across different models and resources.
Model
Libri-10m
Libri-Ih
Libri-10h
wav2vec2-buse
0.08
0.10
0.20
wav2vec2-large
0.04
0.04
0.06
xlsr
0.04
0.05
0.07
6.3 How is The Learned Masks Correlated to Phonetics?
Given the decent performance achieved by the learned masks, we are interested their correlations with human expertise in phonetics. In partieular, we wonder whether the similarity between the learned masks of two languages aligns with that between their phoneme inventories.
Setup. Given the 11 languages adopted in Sec. 4.2 and 4.3, we calculate layer-wise cosine similarities of their learned masks, under sparsity ratio 0. with near-optimal performance, between each pair of them. We pick the mask similarity of the first layer as our metric without losing generality. We further measure the cosine similarity between their corresponding phoneme inventories acquired from Phoible [117], cross- lonsu-iin phonological inventory database which covers over 2000 languages. Following [118], we combine the inventories for all languages to a shared phoneme inventory and use binary vector to indicate the phone inventory of each language, thus the language- wise phonetic similarities can be calculated as the cosine similarities between their corresponding binary vectors.
Results and analysis. We visualize the two sim- ilarity metries each language pair in Fig. and find that the Pearson Correlation Coeffi- cient [119] and the Spearman correlation coef- ficient [120] between the two similarity metrics are 0.527 and 0.548, respectively. This indicates that the similarities of our learned masks non- trivially correlated with that of phonetics while the former also provides new insights in the eyes of speech SSL models, which could shed light on future advances in zero-shot cross cross-lingual transfer [1 [118]. We provide more insightful cor- relation analysis as well as the visualization of layer-wise mask similarities between different languages in the appendix.
0.921
Mask Cosine Similarity
0.920
0.919
0.918
0.917
0.3
0.4 0.5 0.5
Phoneme Inventory Cosine Similarity
Figure 6: Visualizing the correlation between the similarity of the learned masks and that of the cor- responding phoneme inventories of language pairs.
7 Conclusion
Motivating by the empirical success of SSL speech representations in low-resource speech processing, we propose Sa-Router facilitate the practical usage of speech SSL models via finetuning their con- nections instead of weights and encoding language-/task-specff information via sparsity. Extensive experiments validate that Roouur not only serves as a stronger alternative with alleviated overfitting and enhanced accuracy for standard weight finetuning, but also empowers efficient multilingual and multitask speech processing. Our insights could enhance our understandings about what is encoded in speech SSL. models and thus shed light on future advances in SSL speech representations.
Acknowledgements
The work supported by the National Science Foundation (NSF) through the CCRI program (Award number: 2016727) and an IBM faculty award received by Dr. Yingyan Lin.
10

---

References
[1] Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun- Chann Hsieh, Shang-War Wen Li, and Hung- yi Lee. Audio albert: A lite bert for self-supervised learning of audio representation. arXiv preprint arXiv:2005.0857 2020.
[2] Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprini arXiv:2012.06659, 2020.
[3] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
[4] Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve. Lior Wolf. Fierre-Emmanuc Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech representations the time domain. arXiv preprint arXiv:2007.0099 00099,,,,,, 2020.
[5] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsuper- vised pre-training for speech recognition. arXiv preprint 1rXiv::904.05860 2019.
[6] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0:
A framework for self supe-supe learning of speech representations. Advances in Neural Information Processing Systems, 13:12449-12466 2020.
[7] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. 1......3999 13979, 2020.
[8] Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. Effectiveness of self-supervised pre-training for speech recognition. arXiv preprint arXiv:1911.03912. 2019.
[9] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. /EEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3461 2021.
[10] Wenxin Hou, Han Zhu, Yidong Wang, Jindong Wang, Tao Qin, Renjun Xu, and Takahiro Shinozaki. Exploiting adapters for cross-lingual low-resource speech recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:317-329, 2021.
[11] Meng Cai, Yongzhe Shi, Jian Kang, Jia Liu, and Tengrong Su. Convolutional maxout neural networks for low-resource speech recognition. In The 9th International Symposium on Chinese Spoken Language Processing, pages 133-137. IEEE, 2014.
[12] Meng Cai, Yongzhe Shi, and Jia Liu. Stochastic pooling maxout networks for low-resource speech recognition. In 2014 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3266-3270. IEEE, 2014.
[13] William Chan and Ian Lane. Deep convolutional neural networks for acoustic modeling in low resource languages. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 2056-2060. IEEE, 2015.
[14] Yajie Miao and Florian Metze. Improving low low-resource ed-dnn-hmm using dropout and multilingual dnn training. In Interspeech, volume 13, pages 2237-2241, 2013.
[15] Fei Sha and Lawrence Saul. Large margin hidden markov models for automatic speech recognition. Advances in neural information processing systems, 19, 2006.
[16] Shuiyang Mao, Dehua Tao, Guangyan Zhang, PC Ching, and Tan Lee. Revisiting hidden markov models for speech emotion recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6715-6719, IEEE, 2019.
[17] Stephen Adams and Peter A Beling. A survey of feature selection methods for gaussian mixture models and hidden markov models. Artificial Intelligence Review, 523):1139-1179, 2019.

---

[18] Xian Tang. Hybrid hidden markov model and artificial neural network for automatic speech recognition. 2009 Pacific-Asia Conference on Circuits, Communications and Systems, pages 682-685. IEEE, 2009.
[19] John S Bridle. Alpha-nets: A recurrent 'neural' network architecture with a hidden markov model interpretation. Speech Communication, 91):83-92, 1990.
[20] Navdeep Jaitly, Patrick Nguyen, Andrew Senior, and Vincent Vanhoucke. Application of pretrained deep neural networks to large vocabulary speech recognition. 2012.
[21] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376,
2006.
[22] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint 4rXiv:1412.5567, 2014.
[23] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng. Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning, pages 173-182. PMLR, 2016.
[24] Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, pages 1764-1772. PMLR, 2014.
[25] Yajie Miao, Mohammad Gowayyed, and Florian Metze. Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 167-174. IEEE, 2015.
[26] Florian Eyben, Martin Wöllmer, Bjöm Schuller, and Alex Graves. From speech to letters-using a novel neural network architecture for grapheme based asr. In 2009 JEEE Workshop on Automatic Speech Recognition & Understanding, pages 376-380. IEEE, 2009.
[27] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arX1v:/211.3711, 2012.
[28] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645-6649, Ieee, 2013.
Speech recognition with deep
[29] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-ent to-end speech recognition with mn-transducer. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 193-199. IEEE, 2017.
[30] Linhao Dong, Shiyu Zhou, Wei Chen, and Bo Xu. Extending recurrent neural aligner for streaming end-to-end speech recognition mandarin. arXiv preprint a8Xiv: 000.:06326.. 2018.
[31] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention- based models for speech recognition. Advances in neural information processing systems, 28, 2015.
[32] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention btaeet large vocabulary speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4945-4949. IEEE, 2016.
[33] William Chan, Navdeep Jaitly, Quoe Le, and Oriol Vinyals. Listen, attend and spell: neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4960 4964. IEEE, 2016.
12

---

[34] Yu Zhang, William Chan, and Navdeep Jaitly. Very deep convolutional networks for end-to-en speech recognition. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4845-4849 IEFE, 2017.
[35] Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, et al. State-of-the-art speech recognition with sequence-to-sequene models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4774-4778. IEEE, 2018.
[36] Rohit Prabhavalkar, Tara N Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung- Cheng Chiu, and Anjuli Kannan. Minimum word error rate training for attention- based models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4839-4843. IEEE, 2018.
[37] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: 2020.
transformer for speech recognition. arXiv preprini 1rXii:2005.08/000
[38] Linhao Dong, Shuang Xu, and Bo Xu. Speech-transforme
no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5884-5888. IEEE, 2018.
[39] Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer-ban acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874-6878. IEEE, 2020.
[40] Wei-Ning Hsu, Yu Zhang, and James Glass. Learning latent representations for speech generation and transformation. In INTERSPEECH, 2017.
[41] Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron den Oord. Unsupervised speech representation learning using wavenet autoencoders. IEEF/ACM transactions on audio, speech, and language processing, 27(12):2041-2055 2019.
[42] Aaron van den Oord, Oriol Vinyals, al. Neural discrete representation learning. In NeurIPS, 2017.
[43] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In NeurlPS, 2017.
[44] Janek Ebbers, Jahn Heymann, Lukas Drude, Thomas Glarner, Reinhold Haeb- BuebbU and Bhiksha Raj. Hidden markov model variational autoencoder for acoustic unit discovery. In INTERSPEECH, 2017.
[45] Thomas Glarner, Patrick Hanebrink, Janck Ebbers, and Reinhold Hacb-Umbach Full bayesian hidden markov model variational autoencoder for acoustic unit discovery. In INTERSPEECH, 2018.
[46] Sameer Khurana, Shafiq Rayhan Joty, Ahmed Ali, and James Glass. A factorial deep markov model for unsupervised disentangled representation learning from speech. In ICASSP, 2019.
[47] Sameer Khurana, Antoine Laurent, Wei-Ning Hsu, Jan Chorowski, Adrian Lancucki, Rieard Marxer, and James Glass. A convolutional deep markov model for unsupervised speech representation learning. arXiv preprint arXiv:2006.02547, 2020.
[48] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. arXiv preprint /7Xiv:1904.032200, 2019.
[49] Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In ICASSP, 2020.
[50] Yu-An Chung and James Glass. Improved speech representations with multi-target autoregres- sive predictive coding. arXiv preprint arXiv:2004.05274 2020.
13

---

[51] Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. Deep contextualized acoustic representations for semi-supervisse speech recognition. In ICASSP, 2020.
[52] Weiran Wang, Qingming Tang, and Karen Liveseu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. In ICASSP, 2020.
[53] Andy Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders, In ICASSP, 2020.
[54] Shuo Liu, Adria Mallol-Ragolta Emilia Kathan, Bin Hu, and Bjoern W Schuller. Audio self-supervised learning: A survey. arXiv preprint arXiv:2203.01 0003000205, 2022.
Kun Qian, Xin Jing, Alexander
[55] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking as an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004.1 12406, 2020.
[56] Rich Caruana. Multitask learning. Machine Learning, 1997.
[57] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv:/706.05098 2017.
[58] Yi-Chen Chen, Po-Han Chi, Shu-wen Yang. Kai-Wei Chang, Jheng-hao Lin, Sung-Feng Huang, Da-Rong Liu, Chi-Liang Liu, Cheng- Kuang Lee, and Hung-yi Lee. Speechnet: A universal modularized model for speech processing tasks. arXiv preprint arXiv:2105.03: 2021.
[59] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 301-308. IEEE, 2017.
[60] Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Almost unsupervised text to speech and automatic speech recognition. In International Conference Machine Learning, pages 5410-5419, PMLR, 2019.
[61] Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and John R Hershey, Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks. In Sixteenth Annual Conference of the International Speech Commanication Association, 2015.
[62] Tomi Kinnunen, Lauri Juvela, Paavo Alku, and Junichi Yamagishi. Non-parallel voice conversion using i-vector plda: Towards unifying speaker verification and transformation. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5535-5539. IEBE, 2017.
[63] Mingyang Zhang, Xin Wang, Fuming Fang, Haizhou Li, and Junichi Yamagishi. Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet. arXiv preprint 1XXi:1990311338 2019.
[64] Abhinav Jain, Minali Upreti, and Preethi Jyothi. Improved accented speech recognition using accent embeddings and multi-task learning. Interspeech, pages 2454-2458, 2018
[65] Yi-Chen Chen, Zhaojun Yang, Ching-Feng Yeh, Mahaveer Jain, and Michael L. Seltzer. Aipnet: Generative adversarial pre-training of accent-invariant networks for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6979-6983. IEBE, 2020.
[66] Wei-Ning Hsu, Yu Zhang, Ron Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu, and James Glass. Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5901-5905. IEEE, 2019.
14

---

[67] Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al. Transfer learning from speaker verification to multispeaker text-to-spece speech synthesis. Advances in neural information processing systems, 31, 2018.
[68] Jing- Xuan Zhang, Zhen- Ling, and Li-Rong Dai. Non- Non-parallel conversion with disentangled linguistic and speaker representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:540-5555 552, 2019.
voice
[69] Jing-Xuan Zhang, Zhen-Hua Ling, Yuan Jiang, Li-Juan Liu, Chen Liang, and Li-Rong Dai. Im- proving sequence-to-sequeee voice conversion by adding text-supervision. In ICASSP 2019- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6785-6789, IEEE, 2019.
[70] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski. Joao Monteiro. Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recogni- tion. In ICASSP 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1989-6993, IEEE, 2020.
[71] Salah Zaiem, Titouan Parcollet, and Slim Essid. Pretext tasks selection for multitask self- supervised speech representation learning, arXiv preprint arXiv:2107.00594 2021.
[72] Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Learning problem-agnostic speech representations from multiple self-supervised tasks. In INTERSPEECH, 2019.
[73] Yizhou Lu, Mingkun Huang, Xinghua Qu, Pengfei Wei, and Zejun Ma. Language adaptive cross-lingual speech representation learning with sparse sharing sub-networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6882-6886. IEEE, 2022.
[74] Yi-Chen Chen, Shu-wen Yang, Cheng- Kuang Lee, Simon See, and Hung-yi Lee. Speech representation learning through self-supervised pretraining and multi-task finetuning. arXiv preprint arXiv:2 10.09930, 2021.
[75] Sherif Abdou and Michael S Scordilis. Beam search pruning in speech recognition using posterior probability-base confidence measure. Speech Communication, 42(3-4):409-428, 2004.
[76] Janne Pylkkönen. New pruning criteria for efficient decoding. In Ninth European Conference Speech Communication and Technology, 2005.
[77] Vesa Siivola, Teemu Hirsimaki, and Sami Virpioja. On growing and pruning kneser- ney smoothed n-gram models. IEEE Transactions Audio, Speech, and Language Processing, 1515:11617-1624 2007.
[78] Tianxing He, Yuchen Fan, Yanmin Qian, Tian Tan, and Kai Yu. Reshaping deep neural network for fast decoding by Speech and Signal Processing (ICASSP), pages 245-249 IEEE, 2014.
In 2014 IEEE International Conference on Acoustics,
[79] Hainan Xu, Tongfei Chen, Dongji Gao, Yiming Wang, Ke Li, Nagendra Goel, Yishay Carmiel, Daniel Povey, and Sanjeev Khudanpur. A pruned mnlm lattice-rescoring algorithm for automatic speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5929-5933. IEEE, 2018
[80] Yuekai Zhang, Sining Sun, and Long Ma. Tiny transducer: A highly-efficient speech recog- nition model on edge devices. In ICASSP 2021-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6024-6028. IEEE, 2021.
[81] Hugo Van Hamme and Filip Van Aelten. An adaptive-beam pruning technique for continuous speech recognition. In Proceeding of Fourth International Conference on Spoken Language Processing, ICSLP*96, volume 4, pages 2083-2086. IEEE, 1996.
15

---

[82] Dong Yu, Frank Seide. Gang Li, and Li Deng. Exploiting sparseness in deep neural networks for large vocabulary speech recognition. In 2012 IEEE International conference on acoustics, speech and signal processing (ICASSP), pages
IEEE, 2012.
[83] Sankaran Panchapagesan, Daniel S Park, Chung- Chiu, Yuan Shangguan, Qiao Liang, and Alexander Gruenstein. Efficient knowledge distillation for run-transducer models. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 5639-5643, IEEE, 2021.
[84] Yuan Shangguan, Jian Li, Qiao Liang, Raziel Alvarez, and Ian McGraw. Optimizing speech recognition for the edge. arXiv preprint arXiv: (ttiv::999./2400000 12408, 2019.
[85] Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang. Dynamic sparsity neural networks for automatic speech recognition. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6014- 6018. IEEE, 2021.
[86] Ganesh Venkatesh, Alagappan Valliappan, Jay Mahadeokar, Yuan Shangguan, Christian Fuegen, Michael L Seltzer, and Vikas Chandra. Memory- speech recognition on smart devices. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8368 8372. IEEE, 2021.
[87] Yangyang Shi, Yongqiang Wang. Chunyang Wu, Ching-Fere Ych, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer. Enformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6783-6787. IEEE, 2021.
[88] Shengqiang Li, Menglong Xu, and Xiao-Lei Zhang. Efficient conformer-based speech recog- nition with linear attention. arXiv preprint arXiv:2104.06865, 2021.
[89] Stefan Braun and Shih-Chii Liu. Parameter uncertainty for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 5636-5640, IEEE, 2019.
[90] Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint arXiv:1704.05111 2017.
[91] Dawei Gao, Xiaoxi He, Zimu Zhou, Yongxin Tong, Ke Xu, and Lothar Thiele. Rethinking pruning for accelerating deep inference at the edge. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 155-164, 2020.
[92] Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech, pages 2365-2369, 2013.
[93] Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur. Semi-orthogonal low-rank matrix factorization for deep neural networks. In Interspeech, pages 3743-3747, 2018.
[94] Cheng-I Jeff Lai, Yang Zhang, Alexander H Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, and Jim Glass. Parp: Prune, adjust and re-prune for self-supervised speech recognition. Advances in Neural Information Processing Systems, 34, 2021.
[95] Lodagala VSV Prasad, Sreyan Ghosh, and S Umesh. Pada: Pruning assisted domain adaptation for self-supervised speech representations. arXiv preprint arXiv:2203.16900 2022.
[96] Yingzhu Zhao, Chongjia Ni, Cheung- Chhun Leung, Shafiq Joty, Eng Siong Chng, and Bin Ma. A unified speaker adaptation approach for asr. arXiv preprint arXiv:21 10.08545, 2021.
[97] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in randomly weighted neural network? In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 11893-11902, 2020.
16

---

[98] Eran Malach, Gilad Yehudai, Shai Shalev-Schwarte and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning all you need. In International Conference on Machine Learning. pages 6682-6688 6691. PMLR, 2020.
[99] Yonggan Fu, Qixuan Yu, Yang Zhang, Shang Wu, Xu Ouyang, David Cox, and Yingyan Lin. Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks. Advances in Neural Information Processing Systems, 34, 2021.
[100] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1308.3333..43. 2013.
[101] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[102] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. arXiv preprint 1700001510000149, 2015.
[103] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[104] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint urXiv:1608. 08710,7010000 2016.
[105] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1810.05270, 2018.
[106] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1eXiv:/803.03663 2018
[107] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[108] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli, Data2vec: A general framework for sulf-supervised learning speech, vision and language. arXiv preprint arXiv:2202.0333 2022.
[109] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. 2015 IEEE international conference on acousticx, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[110] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A speech corpus. arXiv preprint arXiv:
2019.
[111] Morgane Riviere, Armand Joulin, Emmann-E Mazaré, and Emmanuel Dupoux. Unsu- pervised pretraining transfers well across languages. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7414-7418. IEEE, 2020.
[112] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.0000
2021.
[113] Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:/904 /00.0//904.0/388, 2019.
[114] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
17

---

[115] Kenneth Heafield, Ivan Pouzyrevsky. Jonathan H Clark, and Philipp Koehn. Scalable modified kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 190-696,770 2013.
[116] Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Be- sacier. Lightweight adapter tuning for multilingual speech translation. arXiv preprint arXiv:2106. 01463, 2021.
[117] Steven Moran and Daniel McCloy, editors. PHOIBLE 2.0. Max Planck Institute for the Science of Human History, Jena, 2019.
[118] Heting Gao, Junrui Ni, Yang Zhang, Kaizhi Qian, Shiyu Chang, and Mark Hasegawa-Johnson Zero-shot cross- lingual phonetic recognition with external language embedding. In Proc. Interspeech, pages 1304-1308, 2021.
[119] Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer, 2009.
[120] Philip Sedgwick. Spearman's 's rank correlation coefficient. Bmj, 349, 2014.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Sec. 6.3, we discuss the potential of supporting zero-shot cross-lingual transfer as our future work, which is a new problem created by work.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work proposes a framework to empower efficient multilingual and multitask speech processing, which does not notably correlate with any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] We will release
code, data, and instructions upon acceptance so that we ensure ease of use. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1 and the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We report the average results for clarity of the figures. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] because the experiments are not computationally intensive.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The used dataset and the license
information can be found via the cited reference.
(c) Did you include any new assets either in the supplemental material or as a URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No]
18

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
19


Reviews:
[{"review_id": "yHT-NhdepOo", "reviewer": "Reviewer_2h6A", "summary": "This paper proposes modifying the fine-tuning procedure of speech SSL models like wav2vec 2.0 by applying a binary masking based finetuning where each weight is either kept or masked to 0 instead of modifying the model weights directly. This follows a similar recently proposed technique in the NLP domain. The paper's main contributions is application to speech domain, demonstrating that this approach can work well for ASR as well as other speech tasks such as diarization. All of this is achieved by simply learning to mask about 10% of the weights, which leads to greater efficiency of storing task-specific models. The authors propose a new method to choose masks in which they randomly choose masks for each layer but initialize the weights for the masks based on the magnitude of the weights, and show that this outperforms random masking and magnitude based masking.", "questions": "Have you tried combining weight fine-tuning with your binary masking scheme? For example, you can fine-tune weights followed by learning binary masks, or learning binary masks followed by weight fine-tuning, or do it jointly.", "limitations": "None that I can see.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Interesting and impactful approach showing that a light weight fine-tuning can outperform full model fine-tuning for speech related tasks and lead to more efficient finetuned model storage (or alternatively, somewhat sparse models).\n- Thorough evaluation with different model sizes, in both mono-lingual and multi-lingual scenarios and across multiple tasks.\n\nWeaknesses:\n- This is not a super novel work given that it mostly transfers findings from NLP, but nonetheless the demonstrated effectiveness of the approach for speech tasks and a better performing mask initialization technique should be of interest to the community.\n- The presentation is overly dramatic - perhaps the authors should reduce the use of terms like \"win-win\" \"excitingly\" etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VOuOtHRrxrJ", "reviewer": "Reviewer_13fn", "summary": "This paper proposed a novel method that fine-tunes speech SSL models by discarding the connections rather than changing the weight values. Experiments were conducted on simulated low-resource multilingual ASR tasks as well as several other speech tasks. An interesting finding was that the learnt masks were linearly correlated to the phone inventories of each individual language.  ", "questions": "1. The authors claimed that\n\"where the speech SSL models suffer from a higher risk of overfitting on downstream low-resource tasks as compared to NLP\"\nIs there an evidence or proof for this?\n\n2. Similarly,\n\"As large-scale speech SSL models tend to overfit when finetuning their weights under a low-resource setting, we hypothesize that the binary optimization of the mask patterns in Eq. (1) can serve as regularization and thus alleviate overfitting\"\nDo the authors think regularisation methods (such as L1/L2 regularisors, dropout, early stopping, and smaller learning rates, or simply only fine-tune a small parts of the model) will make a difference here?\n\n", "limitations": "1. Compare with other adaptation/adaptor methods.\n2. Study other types of speech-based SSL models, if the findings are claims to be generic.\n3. Based on the current hardware, routing within sparse models (e.g. MoE models or Pathway models) would be more efficient than routing each individual connection within a densely-connected model, unless the findings indicate way more connections can be discarded (e.g. more than 70%) -- this would allow to use sparse rather than dense matrix structures and ops.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is very carefully written and easy to follow.\n2. The idea of discarding connections is possibly still novel for speech SSL models.\n3. Many experiments were done and mostly the results were positive.\n\nWeaknesses:\n\n1. In an alternative view, the proposed S3-Router method is a binary-mask-based task adaptor approach. As the authors claimed, mainly it required to build such masks for all fully-connected layers, which results in a large number of trainable adaptor parameters, in addition to the task-specific output layers. This is a very inefficient way to build such adaptors compared to many other existing alternatives. So the more proper baselines to compare with, are other alternative adaptation methods instead of simply fine-tuning all model encoder parameters.\n\n2. Although it is claimed that \n\n\"However, there exists a dilemma between the trends of speech SSL models and the growing demand for speech processing applications on the edge. While advanced speech SSL models become increasingly larger to learn more generalizable features, it is highly desired to process the captured speech signals in real-time on edge devices, which have limited resources and conflict with the prohibitive complexity of existing speech SSL models.\"\n\nThe S3-Router actually increases computational cost rather than reduces it, since masks are required to be applied every time, and the masked weights still need to be used via normal GEMM functions due to the design of the current hardware. \nThis also results in high storage costs. Users either need to store the resulted many binary mask parameters and apply them every time with extra computation cost (although the binary masks can be stored in bits), or an extra set of parameters need to be stored for every task, which contradicts with the claim\n\n\"For example, compared to independent weight fine-tuning for each language/task, S3-Router can simultaneously support 11 languages in Sec. 4.2/ 4.3 and 8 tasks in Sec. 4.4 using one wav2vec2-base while achieving a win-win in both accuracy (as validated in Sec. 4.2/ 4.3/ 4.4) and efficiency, i.e., more than 88.5% reductions in model parameters.\"\n\nI believe other task adaptation methods can be way more efficient in terms of both computation and storage costs. Most of the advantages the authors claimed are general benefits of using adaptors/adaptations (e.g. \"One exciting advantage is that since the gradients of different spoken languages or tasks can...\"), which have been studied extensively in the speech community in the past three decades (& these works are completely ignored by this paper).\n\n3. The results are also disappointing. Although so many extra parameters are introduced, it was found that only 10% of the connections can be discarded, which is not \"non-trivial contributions (claimed by the authors)\" and possibly not useful in practice at all.  In general, I feel this paper overclaims. \n\n4. Although the findings were claimed to be generic (e.g. discarding less than 10% connections), actually only one type of speech SSL model (Wav2Vec 2 related models) was studied, which is not convincible. \n\n5. The results were not compared with other published numbers on the same tasks (e.g. those from SUPERB).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "KqJuUP1VrcI", "reviewer": "Reviewer_nV4F", "summary": "This paper presents a novel fine-tuning method for downstream speech tasks, notably automatic speech recognition (ASR). The key idea of its method is to learn binary masks on the connectivities of pretrained weights instead of directly updating the weight values. The binary masks are obtained by discretizing learnable masks (which have the same shape as weights) during the forward pass. The masks can be optimized via stochastic gradient descent, where straight-through is used to approximate the  gradient estimation. The training objective contains a sparsity constraint, which also makes the proposed fine-tuning method to become a pruning technique. ", "questions": "1. I am curious what kinds of regularization methods can be applied apart from the sparsity constraint. For example, has/have the author(s) try combining S3-router with dropout?\n\n1-2. Can you add noise to the mask (dropping out the mask)?\n\n2. The authors mentioned that S3-router can be seen as an all-in-one technique for on-device applications. Achieving such good results on multiple language tasks by only using less than 10% of the pretrained model for each task is impressive. However, if there are multiple language tasks, and if each (language | task)-specific mask vary a lot by each different task, then I wonder there will be a diminishing return in the pruning aspect. Because, in order to work well for a variety of tasks, the task specific mask should specialize (and should different from masks for other tasks), then this means each mask would be sparse, but the union set of sparse masks will not. Therefore, the whole pretrained model should be uploaded to the device. And as the authors pointed out the resource situation is quite limited. If a set of downstream tasks involve many different languages, I assume that the learned masks shouldn't be too close to each other, as the key idea of this new fine-tuning scheme and pruning technique is to learn (language | task)-specific mask. What is your opinion about this matter?  \n\n3. Can you share more details about training? There are no hyper-parameters, optimization details and training time in the paper.\n", "limitations": "The experimental results are strong, but at the same time, the proposed method is only demonstrated with empirical results. I am a bit wary of generalization capability of this method.\n", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall idea of the paper is simple but quite interesting. It's surprising to see how effective this method is, without changing pretrained weights directly, but only learning connectivity patterns. The experimental results look strong, so empirically the method seems to be working quite well. It would be nice if the authors can add a bit more refined explanation and analysis on why this method works well. \n\nThere are too many contents and citations, which hurt the readability. It is quite often distracting, because I have to open other papers to find out the details. It would be nice if the authors can make the paper to become more self-contained and make it simpler to read. For example, having to be forced to open a cited paper to check the size of the pretrained model (wav2vec) is a bit frustrating. Also, one of the most important details of a new method: \"so how can I train this?\" part is missing. The loss function is trivially given as an example, e.g., CTC loss. Having many experimental results on a paper is not a bad thing, but the training procedure and how the results were obtained are also important details. They shouldn't be omitted to have more space to add a few more tables.\n\nAs a reviewer I am pointing out that I am concerned about the reproducibility of this work as training details are not shared.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept