PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
Authors: Cheng-I Lai
Yang Zhang
Alexander H. Liu
Shiyu Chang
Yi-Lun Liao
Yung-Sung Chuang
Kaizhi Qian
Sameer Khurana
David Daniel Cox
James R. Glass
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Speech Recognition
Self-Supervised Learning
Lottery Ticket Hypothesis
Pruning

Abstract: Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results. However, directly applying widely adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is suboptimal in the computational cost needed. Moreover, we show that the discovered subnetworks yield minimal performance gain compared to the original dense network.
We present Prune-Adjust-Re-Prune (PARP), which discovers and finetunes subnetworks for much better performance, while only requiring a single downstream ASR finetuning run. PARP is inspired by our surprising observation that subnetworks pruned for pre-training tasks need merely a slight adjustment to achieve a sizeable performance boost in downstream ASR tasks. Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational advantage and performance gain of PARP over baseline pruning methods.
In particular, on the 10min Librispeech split without LM decoding, PARP discovers subnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease compared to the full model. We further demonstrate the effectiveness of PARP via: cross-lingual pruning without any phone recognition degradation, the discovery of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and its applicability to pre-trained BERT/XLNet for natural language tasks1.

PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
Cheng-I Jeff Lai', Yang 7 Alexander H. Liu'; Shiyu Chang2.4+
Yi-Lun Liao', Yung-Sung Chuang! Kaizhi Qian?, Sameer Khurana'
David Cox2. James Glass!
MIT CSAIL, 2MIT-IBM Watson Al Lab, NNational Taiwan University, 4UC Santa Barhara
clai240mit.edd
Abstract
Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve low-resource ASR results. However, directly applying widely adopted pruning methods such the Lottery Ticket Hypothesis (LTH) is suboptimal in the compu- tational cost needed. Moreover, we show that the discovered subnetworks yield minimal performance gain compared the original dense network.
better
We present Prune-Adjust-Ie Re- Prune (PARP), which discovers and finetunes subnet- works for much better performance, while only requiring a single downstream ASR finetuning run. PARP inspired by our surprising observation that subnetworks pruned for pre-training tasks need merely slight adjustment to achieve a sizeable performance boost in downstream ASR tasks. Extensive experiments on low- resource ASR verify (1) sparse subnetworks exist in monoo-ingual/multi--in pre-trained speech SSL, and (2) the computational advantage and performance gain of PARP over baseline pruming methods.
In particular, the 10min Librispeech split without LM decoding, PARP discovers subnetworks from wav2vec 2.0 with an absolute 10.9% WER decrease compared the full model. We further demonstrate the effectiveness of PARP via: cross cross-lingual pruning without any phone recognition degradation, the discovery of multi Minttilinngual subnetwork for 10 spoken languages in I finetuning run, and its applicability to pre-trained BERT/XLNet for natural language tasks
Introduction
For many low-resource spoken languages in the world, collecting large-scale transcribed corpora very costly and sometimes infeasible. Inspired by efforts such as the IARPA BABEL program, Automatic Speech Recognition (ASR) trained without sufficient transcribed speech data has been a critical yet challenging research agenda in speech processing B1 42 22 21] Recently. Self- Supervised Speech Representation Learning (speech SSL) has emerged
promising pathway toward solving low-resource ASR 84 25 110 6 127 [55] 27]. Speech SSL involves pre-training a speech representation module large-scale unlabelled data with self-supervised learning objective, followed by finetuning on a small amount of supervised transcriptions. Many recent studies have demonstrated the empirical successes of speech SSL on low-resource English and multi-lingual ASR, matching systems trained on fully-supervised settings 6 29 127 [26] Prior research attempts, however, focus pre-training objectives
129 scaling
up speech representation modules 5
3
pre- training data selections
108
78 or
"quual contribution.
Project webpage: https://people. www.s.aaaaaa.............. http:///..........a///aaa///////
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

applications of pre- trainer speech representations 26 62 94
23
63
29
76
120
122
4 86 59 66
In this work, we aim to develop an orthogonal approach that is complementary to these existing speech SSL studies, that achieves 1) lower architectural complexity and 2) higher performance (lower WER) under the same low-resource ASR settings.
556 T02
30
as well as the more recently proposed Lottery Ticket Hypothesis (LTH) 3 provide potential solution that accomplishes both objectives. According to LTH, there exists sparse subnetworks that can achieve the same or even better accuracy than the original dense network. Such phenomena have been successfully observed various domains:
Neural network pruning 65
49
39
Natural Language Processing (NLP) [23 19, NS 80 Computer Vision (CV) [ 45 and many others. All finding sparse subnetworks with comparable or better performance than the dense network. Given the lack of similar studies on pruning self self-supervised ASR, we intend to fill this gap by finding sparse subnetworks within a pre-trained speech SSL, that can achieve superior performance to the full pre-trained model on downstream ASR tasks.
However, directly applying widely-adopted pruning methods, such as One-Shot Magnitude Pruning (OMP) and Iterative Magnitude Prun- ing (IMP) 49 39/ to pre-trained speech SSL suffers from two challenges. First, adopting these methods in the conventional prun- ing framework is extremely time-consuming for SOTA speech SSL models. OMP and IMP involve more than one round of finetuning on downstream tasks (c.f. Figure D and finetuning for ASR is time- consuming and computationally demanding I The second challenge that we do not observe any performance improvement of the sub- networks over the original dense network with OMP or IMP. Figure 5] shows the WER under low-resource scenarios of the subnetworks identified by OMP (purple line) and IMP (blue dashed line) at different sparsity levels. None of the sparsity levels achieves a visible drop in WER compared to the zero sparsity case, corresponding to the original dense network. These two challenges have prompted us to ask lak-du do there exist sparse subnetworks within pre-trained speech SSL with improved performance on low-resource ASR? How can we discover them efficiently in single downstream finetuning run?
Nm of Divettunnn ASR meeurr
Computational Cost of Pruning Methods
GMP
IMS PARP
20
180 Sparsty 1%4
50
60
70
Figure 1: Number of ASR finetuning iterations needed (y-axis) versus target sparsi- ties (x-axis) for each down- stream task/language. Cross- referencing Figure B indi- cates that IMP requires lin- early more compute to match the performance (either sparsi- ty/WER) of PARP.
We propose magnitude- based unstructured pruning method 11, termed Prune-Ajjust-Re-Pune (PARP), for discovering sparse subnetworks within pre-trained speech SSL. PARP consists of the following two steps:
1. Directly prune the SSL pre-trained model at target sparsity, and obtain an initial subnetwork and an initial pruning mask.
2. Finetune the initial subnetwork on target downstream task/language. During finetuning, zero out the pruned weights specified by the pruning mask, but allow the weights be updated by gradient descent during backpropogation. After few number of model updates, re-prune the updated subnetwork at target sparsity again.
Step provides an initial subnetwork that is agnostic to the downstream task, and Step 2 makes learnable adjustments by reviving pruned out weights. A formal and generalized description and its extension are introduced in Section 3 Different from pruning methods 49 39] J PARP allows pruned-out weights to be revived during finetuning. Although such high-level idea introduced in [48], we provide an alternative insight: despite its flcxibility, Step 2 only makes minimal adjust- ment to the initial subnetwork, and obtaining good initial subnetwork in Step is the key. We empirically show in Section 3 that any aank-agi subnetwork surprisingly provides a good basis for Step 2, suggesting that the initial subnetwork can be cheaply obtained either from a readily available task/language or directly pruning the pre- rre-trained SSL model itself. In addition, this observation allows us perform gross-lingue pruning (mask transfer) experiments, where the initial subnetwork is obtained via a different language other than the target language.
Our Contributions. We conduct extensive PARP and baseline (OMP and IMP) pruning experiments on low-resource ASR with -ingull (pre-trained wav2vee 2.0 (6) and cross- cross-lingual (pre-trained
XLSR-53 (2) transfer. PARP finds significantly superior speech SSL subnetworks for low-resource
2 Standard wav2vec 2.0 finetuning setup d on any Librispeech/L LitpLLibbiiiii -ibri-light splits requires at least 50~ V100 hours, which is more than 50 times the computation cost for finetuning a pre- pre-trained BERT on GLUE [06].
2

---

ASR, while only requiring a single pass of downstream ASR finetuning, Due to its simplicity, PARP adds minimal computation overhead existing SSL downstream finetuning.
We show that sparse subnetworks exist in pre-trained speech SSL when finetuned for low-resource ASR. In addition, PARP achieves superior results to OMP and IMP across all sparsities, amount of finetuning supervision, pre-trainet model scale, and downstream spoken languages. Specifically, on Librispeech 10min without LM decoding, PARP discovers subnetworks from wav2vee 2.0 with an absolute 10.9% WER decrease compared to the full model, without modifying the finetuning hyper- prper-pprameters or objective (Section 4.
Ablation studies on demonstrating the importance of PARP's initial subnetwork (Section
PARP minimizes phone recognition error increases in cross- woos-lingual mask transfer, where subnetwork pruned for ASR in one spoken language is adapted for ASR in another language (Section 3 PARP can also be applied efficient multi-lingual subnetwork discovery for 10 spoken languages (Section
Last but not least, we demonstrate PARP's effectiveness on pre-trained BERT/XI.Net, miti- gating the cross-task performance degradation reported in BERT-Ticket 19 (Section 4.5
Significance. Findings of this work not only complement and advance current and future speech SSL for low-resource ASR, but also provide new insights for the rich body of pruning work.
2 Preliminaries
2.1 Problem Formulation
Consider the low-resource ASR problem, where there is only a small transcribed training set (z. y) Dr. Here z represents input audio, and y represents output transcription. Subscript / E 21, 2.. represents the downstream spoken language identity. Because of the small dataset size, empirical risk minimization generally does not yield good results. Speech SSL. instead assumes there is much larger unannotated dataset x E Do. SSL pre-trains a neural network f (x; 0), where 0 E Rd represents the network parameters and d represents the number of parameters, on some self-supervised objective, and obtains the pre-trained weights 0o- f(x; 0n) is then finetuned on downstream ASR tasks specified by downstream loss (0), such as CTC, and evaluated on target dataset Dj.
Our goal is discover subnetwork that minimizes downstream ASR WER on Dj. Formally, denote m E {0, 1)dd as binary pruning mask for the pre-trained weights fg. and 6 as the finetuned weights on D;- The ideal pruning method should learn (m, 8¹), such that the subnetwork f(;; m @ 6') (where is element-wise product) achieves minimal finetuning (i) loss on D;.
2.2 Pruning Targets and Settings
We adopted pre-trained speech SSL wav2vec2 and xlsr for the pre-trained initialization 0g.
wav2vec 2.0 We took wav2vec 2.0 base trav2vec2-basee and large (vav2vec2-1asd large) pre-trained on Librispeech 960 hours [6. During finetuning, a task specific linear layer is added on top of wav2vec2 and jointly finetuned with CTC loss. More details can be found in Appendix
XLSR-53 (xlar) shares the same architecture, pre-training and finetuning objectives as wav2vec2-large. xlsr is pre-trained on 53 languages sampled from Common BABEL, and Multilingual LibriSpeech, totaling for 56k hours of multi-lingual speech data.
We consider three settings where wav2vec2 and xlsr are used as the basis for low-resource ASR:
LSR: Low-Resource English ASR. Mono-lingual pre-training and finetuning English pre- trained speech SSL such as wav2vec2 is finetuned for low-resource English ASR.
H2L: High-to-Low Resource Transfer for Multi-lingual ASR. Mono-lingual pre-training and multi-lingual finetuning - a speech SSL pre-trained on a high-resource language such as English is finetuned for low-resource multi-lingual ASR.
CSR: Cross-lingual Transfer for Multi-lingual ASR. Multi-lingual pre-training and finetuning cross-ling lingull pretrained speech SSL such as xlsr is finetuned for low-resource multi-lingual ASR.
2.3 Subnetwork Discovery in Pre-trained SSL
One obvious solution to the aforementioned problem in Section 2.1 is directly apply pruning with rewinding to oo. which has been successfully applied to pre-trained BERT 19 and SimCLR [[]]
3

---

All pruning methods, including our proposed PARP, are based on Unstructured Magnitude Pruning (UMP) [39 I.l. where weights of the lowest magnitudes are pruned out regardless of the network structure to meet the target sparsity level. We introduce four pruning baselines below, and we also where weights 0g are randomly eliminated.
provide results with Random Pruning (RP)
Task-Award Subnetwork Discovery is pruning with target dataset Dj seen in advance, including One-Shot Magnitude Pruning (OMP) and Iterative Magnitude Pruning (IMP). OMP summarized as:
1. Finetune pretrained weights lg on target dataset Dr to get the finetuned weights 0'
2. Apply UMP on 6' and retrieve pruning mask m.
IMP breaks down the above subnetwork discovery phase into multiple iterations in our case multiple downstream ASR finetunings. Each iteration itself is an OMP with fraction of the target sparsity pruned. We follow the IMP implementation described in BiRT-Ticket [19, where each iteration prunes out 10% of the remaining weights. The main bottleneck for OMP and IMP is the computational cost, since multiple rounds of finetunings are required for subnetwork discovery.
Task-Agnostic Subnetwork Discovery refers to pruning without having seen Dj nor in advance. One instance is applying UMP directly Og without any downstream finetuning to retrieve m, referred to as Magnitude Pruming at Pre-trained Initailizations (MPI). Another case is pruning weights finetuned for a different language i.e. applying UMP on l for the larget language l; in our study, we refer to this as cross- rross-lingual mask transfer. While these approaches do not require target task finetuning, the discovered subnetworks generally have worse performance than those from OMP IMP.
The above methods are only for subnetwork discovery via applying pruning mask m on fp. The discovered subnetwork (2; @ Bu) needs another downstream finetuning to recover the pruning loss iee. finetune f(;;; m @ 0n) on Dr.
3 Method
In this section, we highlight our proposed pruning method, PARP (Section 3.1 its underlying intuition (Section 3.2). and an extension termed PARP-P (Section 33
3.1 Algorithm
We formally describe PARP with the notations from Section
A visual overview of PARP is Figure
Algorithm Prrue-Adjust--ie CedjusstRe-Pruntt Re-Pre (PARP) to target sparsity 8
1:
Assume there are N model updates in target task/language l's downstream finetuning. Take a pre-trained SSL (2; 0o) model. Apply task- agn-stoo subnetwork discovery, such as MP at target sparsity s to obtain initial subnetwork f(; mo G 0g). Set m = mg and variable mi 0.
3: repeat
4: Zero-out masked-out weights in 0, given by m. Lift up such that whole 0m1 is updatable.
5:
Train [x; 0w1) for model updates and obtain (x:; 0w2)-
6:
Apply UMP on [(x; 0,2)) and adjust mn accordingly. The adjusted subnetwork is f(x; m C 0n2). Set
variable 74 Tt2.
7: until total model updates reach N.
8: Return finetuned subnetwork fx;; m C 0N).
Empirically, we found the choice of n has little impact. In contrast OMP/IMP/MPI, PARP allows the pruned-out weights to take gradient descent updates. A side benefit of PARP is it jointly discovers and finetunes subnetwork in a single pass, instead of two or more in OMP and IMP.
3.2 Obtaining and Adjusting the Initial Subnetwork
PARP achieves superior or comparable pruning results as task-aware subnetwork discovery, while inducing similar computational cost as aank-agi subnetwork discovery. How does it get the best of both worlds? The key is the discovered subnetworks from task-aware and task-agnostic prunings have high, non-trivial overlaps in LSR, H2L, and CSR. We first define Intersection over Union (IOU) for quantifying subnetworks' (represented by their pruning masks m and m²) similarity:
IO m²) A [m 1) (m = =1]
(1)
[m = 1) U (m = 1)]
"hhis step is referred to as subnetwork finetuiin/re-trraiing in the pruning literature 75 g3
By default, MPI is used for obtaining the initial subnetwork for PARP and PARP-P unless specified otherwise.

---

Take H2L and CSR for instance, Figure 2 visualizes language pairs' OMP pruning mask IOUs on wav2vec2 and xlsr. Observe the high overlaps across all pairs, but also the high IOUs with the MPI masks (second to last row). We generalize these observations to the following:
For any sparsity, any amownt of finetuning superviston, any pre-training model scale, and any downstream spoken languages, the non-zero ASR pruning masks obtained from task-agnostic subnetwork discovery has high IOUs with those obtained from task-aware subnetwork discovery.
Observation 1
Observation suggests that any task-agnostic subnetwork could sufficiently be good initial subnet- work in PARP due to the high similarities. In the same instance for H2L and CSR, we could either take MPI on wav2vec2 and xlsr, or take OMP on a different spoken language as the initial subnetworks. Similarly in LSR, we take MPI on wav2vec2 as the initial subnetwork. The underlying message is the initial subnetwork can be obtained cheaply, without target task finetuning.
Now, because of the high similarity, the initial subnetwork (represented by its pruning mask my) needed merely slight adjustment for the target downstream task. While there are techniques such as dynamic mask adjustment [48), important weights pruning (7).. and deep rewiring [10), we provide an even simpler alternative suited for our setting. Instead of permanently removing the masked-out weights from the computation graph, PARP merely zeroes them out. Weights that are important for the downstream task (the "important weights") should emerge with gradient updates; those that are relatively irrelevant should decrease in magnitude, and thus be zero-outed at the end. Doing so circumvents the need of straight-through estimation or additional sparsity loss, Table 1 of 997
3.3 PARP-Progressive (PARP-P)
An extension to PARP is PARP-P, where the second stands for Progressive. In PARP-P, the initial subnetwork starts at a lower sparsity, and progressively prune up to the target sparsity s in Step 2. The intuition is that despite Observation [ not any subnetwork can be good initial subnetwork, such as those obtained from RP, or those obtamed at very high sparsities in MPI/OMP/MMP We show later that PARP-P especially effective in higher sparsity regions, e.g. 90% for LSR. Note that PARP-P has the same computational cost as PARP, and the only difference is the initial starting sparsity in Step 1.
nanguage Nack IOU Sparsity in wevZvec 2.0
Language Mask at 50% Sparsty
53
2007
2005
0396
3,964
6.00
0.54
0501
5005
6507
0.353
0.35
"Zrpet Language masis
LUBG
5.6.
EM2
2W12
5.5
0.71
0372
0574
3589
3089
8067
8965
0311
6.00
300
3527
8505
5922
053
0.966
3:05
3286
5587
5.55
0.91
0.58
0.55
0.543
02%
0730
9742
0095
60022
0.54
0.40
3981
3509
035
DRI
05.00
5991
0300
0332
2230
9.30
8200
9.00
6.30
8333
0.303
0330
0300 0330
0230
3230
9330
9,00
0,303
8.3
(333
0300
Sourc wngge age OWP masks
SourcH Languagge CM01 masbs
Figure 2: IOUS over all spoken language pairs' OMP pruning masks on finetuned wav2vec2 and xlsr. Second to last row is the IOUS between OMP masks and the MPI masks from pre-trained wav2vec2 and xlsr. Here, we show the IOUs at 50% sparsity, and the rest can be found in Appendix [T] Surprisingly any sparsities, there is a high, non-trivial (c.f. RP in the last row), similarity (>90% between all spoken language OMP masks, as well as with the MPI masks. Language IDs are in Appendix 9
4
Experiments and Analysis
4.1 Comparing PARP, OMP, and IMP on LSR, H21, and CSR
Our experimental setup can be found in Appendix g We first investigate the existence of sparse subnetworks in speech SSL. Figure 3 shows the pruning results on LSR. Observe that subnetworks discovered by PARP and PARP-P can achieve 60~80% sparsities with minimal degradation to the full models. The gap between PARP and other pruning methods also widens as sparsities increase. For instance, T compares PARP and PARP-P with OMP and IMP at 90% sparsity, and PARP-P has a 40% absolute WER reduction. In addition, observe the WER reduction with PARP in the low sparsity
5

---

regions on the 10min split in Figure 3 The same effect is not seen with OMP, IMP, nor MPI. Table compares the subnetworks discovered by PARP with the full wav2vec2 and prior work on LSR under the same setting Surprisingly, the discovered subnetwork attains absolute 10.9%/12.66 WER reduction over the full wav2vec2-large. We hypothesize that the performance gains are attributed to pruning out generic, unnecessary weights while preserving important weights, which facilitates training convergence. In other words, PARP provides additional regularization effects to downstream finetuning, We also examined the effectiveness of IMP with different rewinding starting points as studied [40 23, and found rewinding initializations bear minimal effect on downstream ASR, Full
rewinding details are in Appendix
10
wav2vec2 10min
wav2vec2
wav2ver2
10h
100
100
MPI
139 WER mio LM
90
80
90
80
OMP
80
IMP
70
PARP
60
40
PARP-P
50
20
40
20
40
100
40
40
60
wav2vec2 large + 10min
wav2vec2 large + -hh
wav2vec2 large 10h
90
80
raw WER vio UM
60
60
40
50
40
20
20
40
60
100
20
40
100
40 60
100
Spersity (precentage of weights prured out)
Spersily 196)
Spersily 196)
Figure 3: Comparison of different pruning techniques on LSR (wav2vec2 with 10min/lh/10h Librispeech finetuning splits). PARP (black line) and PARP-P (black dashed line) are especially effective under ultra-low data regime (e.g. 10min) and high-sparsity (70-100%) regions.
Table 1: WER comparison of pruning LSR: wav2vec2-basd
Table 2: WER comparison of PARP for LSR with previous speech SSL results Lib- rispeech 10min. PARP discovers sparse sub- networks within wav2vec2 with lower WER while adding minimal computational cost to the
90% sparsity with 10h fine- tuning on Librispeech without LM decoding. At 90% sparsity, DMP/IMP/PI perform nearly bad as RP. sub-finetuning stands for subnetwork finetuning.
original ASR finetuning.
Method
# ASR finetunings
test clean
test other
Method
test clean
lest other
RP + sub-finetuning
1
94.5
96.4
Continuous BERT 3 +
LM
49.5
66.3
MPI + sub-finctuning
1
93.6
96.1
Discrete BERT 3 LM
16.3
25.2
OMP + sub-finetuning
2
92.0
95.3
wav2vec2- base reported
6
46.9
50.9
IMP sub-finetuning
10
89.6
93.9
wav2ver2- large reported
43.5
45.3
PARP (90% - 90%)
1
83.6
90.7
wav2vec2-base replicated
49.3
53.2
vav2vec2-large replicated
46.3
PARP-P
70% - 90%
1
51.9
69.1
wav2vec2- base w/ 10% PARP
38.0
44.3
60% + 80% + 90%
2
33.6
53.3
wav2vec2-large w/ 10% PARP
33.7
37.2
Next, we examine if the pruning results of LSR transfers to H2L and CSR. Figure 4 is pruning H2L and CSR with 1h of Dutch (nl) finetuning, and the same conclusion can be extended to other spoken languages. Comparing Figures B
and 4 we notice that shapes of their pruning curves are different, which can attributed to the effect of character versus phone predictions. Comparing left and center of Figure 4 we show that PARP and DMP reach 50% sparsity on H2I. and 70% sparsity on CSR with minimal degradations. Furthermore, while PARP is more effective than OMP on H2L. for all sparsities, such advantage is only visible in the higher sparsity regions on CSR. Lastly, Table 3 compares the subnetworks from H2L and CSR with prior work. Even with as high as 90% sparsities either settings, subnetworks from PARP and OMP out-performs prior art.
We underscore again that LM decoding/self-trrinng are not included isolate the effect of pruning.

---

wav2vec2 + Dutch (nl)
xisr53 + Dutch (nl)
Joint-Finetune +
Dutch (nl)
RP
50
OMP PARP-P
w2v-MPI
Faw PER wio
80
OMP
MPI
40
50
60
W2v-OMP
w2v-PARP
PARP
xisr-PARP
40
20
30
20
10
60
100
Sparsity (%6)
Sparsity (%)
Sparsily (36
Figure 4: Comparison of pruning techniques on H2L & CSR with 1h of Dutch (nl) ASR finetuning. (Left)) Pruning H2L (wav2vec2-bas + nl). (Center) Pruning CSR (xlsr + n!). (Right) Pruning contly-finetune wav2vec2- base and xlsr on nl. Trend is consistent for other spoken languages.
Table 3:
Comparing subnetworks discovered by OMP and PARP from wav2vec2-bas and xlsr with prior work on H2L and CSR. PER is averaged over 10 languages.
wav2vec 2.0 10min IOU
1.00
0.96
0.96
0.94
Method
Pre-training
Sparsity
PER
0.92
MPI-OMP
Bottleneck
Babel-1070h
44.9
CPC 14
LS-100h
0%
50.9
MPI-IMP
Modified CPC 31
LS-360h
0%
44.5
MPI-PARP
0.84
wav2vec2 base
LS-960h
0%
18.7
wav2vec2 +
LS-960h
70%
41.3
Spersity (%)
wav2vec2
LS-960h
90%
40.1
Figure 5: PARP's final subnetwork and its initial MPI subnetwork exceeds 99.99% IOU after 20% sparsity (black
xler reported 29
56,000h
0%
7.6
xlsr replicated
56,000h
0%
9.9
xler DMP
56,000h
90%
33.9
line).
xler + PARP-P 56,000h
90%
22.9
4.2 How Important is the Initial Subnetwork (Step 1) PARP?
Obtaining a good initial subnetwork (Step 1) is critical for PARP, as Adjust & Re-Prune (Step 2) is operated top of it. In this section, isolate the effect of Step from Step 2 and examine the role of the initial subnetwork in PARP. Figure 0 shows shows PARP with random subnetwork from RP, instead of subnetwork from MPI, as the initial subnetwork. PARP with random initial subnetwork performs nearly as bad as RP (grey line), signifying the importance of the initial subnetwork.
Secondly, despite Observation 1 MPI in high sparsity regions (e.g. 90% in LSR) is not good initial subnetwork, since the majority of the weights are already pruned out (thus is hard to be recovered from). From Figure 3] PARP performs only par contrast, PARP-P starts with a relatively lower sparsity (e.g. 60% or 70% MPI), and progressively prunes up the target sparsity. Doing so yields considerable performance gain (up over 50% absolute WER reduction). Third, as shown in Figure 5 there -99.99% IOU between the final "adjusted" subnetwork from PARP and its initial MPI subnetwork after 20% sparsity, confirming Step 2 indeed only made minimal "adjustment" to the initial subnetwork.
worse than IMP in high sparsity regions. In
wav2vec2 base + 10min
wav2vec2 base
wav2vec2 base + 10h
100
Random
Tew WER WER wie LM
90
80
90
80
PARP-Random
80
70
PARP
70
60
50
40
40
30
20
40
20
80
20
Sparsity (%)
Sparsity 1%6]
Spensity 196)
Figure 6: PARP with random (red line) v.s. with MPI (black line) initial subnetworks in LSR.

---

Transferreee lity Lanquige Maris
50% Sparsity WaNchee
Trnnffrrap lity of Lang
Mosks at 59% Sparsity in mix2vec th PABP
PYF
+.204
180
3:5
DEST
214
LAY
E15
TMC
TCM
170
1.71
600
1.44
1.79
1.5M
2011
4.44
SIKL
J88
2/2
2051
148E
1.77
2013
2014
240
37%
17M
277
177
42/4
4.14
AMV
11M
2311
CMT
2714
2011
Inttuue Sutnettooks begellencugge
Figure 7: (Left) Cross-lingual OMP mask transfer with regular subnetwork finetuning. (Right) Cross-lingual OMP mask transfer with PARP. Last rows are RP. Values are relative PER gains over same-language pair transfer (hence the darker the bettter). Both are on H21, with pretrained wav2vec2. The same observation is observed on CSR with pretrained xlsr in Appendix 12
4.3 Are Pruning Masks Transferrable across Spoken Languages?
Is it possible to discover subnetworks with the wrong guidance, and how transferrable are such subnetworks? More concretely, we investigate the transferability of OMP pruning mask discovered from source language by finetuning its subnetwork on another target language. Such study should shed some insights on the underlying influence of spoken language structure on network pruning that similar language pairs should be transferrable. From a practical perspective, consider pruning for an unseen new language in H2L, we could deploy the readily available discovered subnetworks and thus save the additional finetuning and memory costs.
In this case, the initial subnetwork of PARP is given by applying OMP on another spoken language. According to Observation L PARP's Step 2 is effectively under-going cross-lingual subnetwork adaptation for the target language. Figure 7 shows the transferability results on H2L with pre-trained wav2vec2-base On the left is a subnetwork at 50% sparsity transfer with regular finetuning that contains subtle language clusters- for example, when finetuning on ru, source masks from es, fs,ft it. ky, n/ induces much higher PER compare that from sv-SE, ir, 11, zh-7W. On the right of Figure we show that there is no cross-lingual PER degradation with PARP, supporting our claim above.
4.4 Discovering a Single Subnetwork for 10 Spoken Languages
major downside of pruning pre-trained SSL models for many downstream tasks is the exponential computational and memory costs. In H2L and CSR, the same pruning method needs to be repeatedly re-run for each downstream spoken language at each given sparsity. Therefore, we investigate the possibility of obtaining single shared subnetwork for all downstream languages. Instead of finetuning separately for each language, we construct a joint phoneme dictionary and finetune wav2vec2 and xlsr on all 10 languages jointly in H2I. and CSR. Note that PARP with noint-finetuning can retrieve a shared subnetwork in a single run. The shared subnetwork can then be decoded for each language separately. The right side of Figure 4 illlustrates the results.
Comparing joint-finetuning and individual-finetuning in H2L, we found that the shared subnetwork obtained via OMP has lower PERs between 60~80% but slightly higher PERs in other sparsity regions; in CSR, the shared subnetwork from OMP has slightly worse PERs at all sparsities. Comparing PARP found that while PARP is effective in the setividuualfinnttuni (left of Figure 4) its shared subnetworks are only slightly better than OMP in both H2L and CSR (right of Figure The smaller performance gain of PARP over OMP in pruning jointly-finctuned expected, since the important weights for each language are disjoint and joint-finee finetfinetunig may send mixed signal to the adjustment step in PARP (see Figure R for better illustration).
to DMP in joint- finttfiietunin,,
setting
finetuned models is
4.5
Does PARP work on Pre-trained BERT/XLNet?
We also analyzed whether Observation holds for pre-trained BERT/XLNet on 9 GLUE tasks. Surprisingly, we found that there are also high (>98%) overlaps between the 9 tasks' IMP pruning masks. Given this observation, we replicated the cross-task subnetwork transfer experiment (take subnetwork found by IMP at task and finetune for task B) BERT-Ticket [19] on tre-trained BERT/XLNet with PARP. Table 4 compares PARP (averaged for each target task) to regular finetuning,
8

---

hinting the applicability of PARP to more pre- pre-trained NLP models and downstream natural language tasks. Detailed scores and figures are in Appendix
Table 4: Comparison of cross-task transfer on GLUE (subnetwork from source task A is finetuned for target task B). Numbers are averaged acc. across source tasks for each target task.
Averaged transferred subnetworks performance linetuned for
Method
CoLA
MRPC
QNLI
QQP
RTE SST-2 STS-B WNLI MNLI
70% sparse subnetworks from pre-trained BERT
Same-task Transfer (top line)
38.89
75.57
88.89
89.95
58.37
89.99
87.34
53.87
82.56
Cross-task Transfer with PARP
28.48
75.98
87.12
90.40
59.69
89,59
86.25
54.62
81.61
Regular Cross-task Transfer 19
10.12
71.94
86.54
88.50
57.59
88.80
80.27
54.03
80.48
70% sparse subnetworks from pro-trained XLNNtt
Same-task Transfer (top line)
29.92
76.47
89.62
90.74
59.21
92.2
80.78
42.25
85.16
Cross-task Transfer with PARP
30.09
77.56
87.10
90.66
58.88
91.73
83,80
52.11 42.25
83.87
Regular Cross-task Transfer
11.47
74.16
85.21
89.11
$5.80
90.19
75.61
82.65
(A) Task-Aware Subnetwork Discovery
task neurons
task B neurons
task neurons
pruned out
SSL Pretrained
multiple downstteam finetunings
not pruned out, thus good subnetworks
task A data
(B) Prune-Adjjst-Re-PPunn (PARP)
single downsteam fineturing
PARP steps
Init subnetwork
Pruning w/o task info
Adjust & Re-Prune w/ task info
SSL Pretrained
Figure 8: Conceptual sketch of pruning the few task-specific important weights in pretrained SSL. (A) Task-aware subnetwork (iscovery(DMP/IIM)) is more effective than task-agnostic pruning (MPI) since it foresees the important weights in advance, via multiple downstream finetunings, (B) PARP starts with an initial subnetwork given by MPI. Observation [ suggests that the subnetwork only off by the few important weights, and thus Step 2 revives them by adjusting the initial subnetwork.
4.6 Implications
Observation iss consistent with the findings of probing large pre-trained NLP models, that pre- pre-trained SSL models are pver-paramerrized and there exist task-oriented weights/neurons, Figure 2 implies that these important weights only account for small part of the pre-trained speech SSL.. In fact, large body of NL. work dedicated to studying task-oriented weights pre-trained models. To name
pruned out these important weights/neurons via probing and quantifying contextualized representations. Based on Observation I we can project that these NLP results should in general transfer to speech,
few, 37 35
| measured, 7
B4 61
leveraged, [8] 46 visualized, and [05
36 [
see pioneering studies g 24 23 However, different from them, PARP leverages important weights
for UMP the whole network structure instead of just the contextualized representations.
We could further hypothesize that good pruning algorithm avoids pruning out task-specific neurons in pre-trained SSL. (67 48 779,, see Figure g This hypothesis not only offers an explanation why PARP is effective in high sparsity regions and cross-lingual mask transfer, also suggests that an iterative method such as IMP is superior to OMP because IMP gradually avoids pruning out important weights in several iterations, at the cost of more compute Finally, we make connections to prior work that showed RP prevail [1I 19 neurons are less likely to get "accidentally" pruned and thus accuracy is preserved even with RP.
? under certain threshold and setting, task-specific
FFrom Section 6 3 "iterative pruning is computationally intensive, requiring training a network 15 or more times consecutively for multiple trials." From Section of (48): "several iterations alternate pruning and retraining are necessary to get a fair compression rate on AlexNet, while each retraining process consists of millions of iterations, which can be very time consuming."

---

5 Related Work
Modern Speech Paradigm and ASR Pruning. As model scale 100
6 50 58, 96 95
90
39
121 68 and model pre-training 127 29 60 57
63 55
T18 14
83
86
109 have
become the two essential ingredients for obtaining SOTA performance in ASR and other speech tasks, applying and developing various forms of memory-effciiet algorithms, such as network pruning, to these large-scale pre-trained models will predictably soon become an indispensable research endeavor. Early work on ASR pruning can be dated back to pruning decoding search
spaces [1I 11 100 52 116
[28] and HMM state space [003.. Since the seminal work of Yu et
al. 122 ASR pruning has focused primarily on end-to-end network architecture: [9 M4] applied pruning and quantization to LSTM-based RNN-Transducers, [85] applied knowledge distillation
to Conformer-basee TNN-dransdssers, [104 99 70]
designed efficient architecture/mechaniie
for LSTM, Transformer, Conformer-basse ASR models, 32 applied pruning to Deep Speech, [12] introduced SNR-basee probabilistic pruning on LSTM-based CTC model, 4% proposed entropy-
regularizer for LSTM- -badedd ASR model, 111 87 applied SVD on ASR models' weight matrices. We emphasize that our work is the first on pruning large self-supervised pre-trained models for low-resource and multi- multi-lingual ASR. In addition, to our knowledge, none of the prior speech pruning work demonstrated the pruned models attain superior performance than its original counterpart.
Conclusions
We introduce PARP, a simple and intuitive pruning method for self-supervisee speech recognition. We conduct extensive experiments on pruning pre-trained wav2vec 2.0 and XLSR-53 under three low-resource settings, demonstrating (1) PARP discovers better subnetworks than baseline pruning methods while requiring a fraction of their computational cost, (2) the discovered subnetworks yields over 10% WER reduction over the full model, (3) PARP induces minimal cross-lingual subnetwork adaptation errors, (4) PARP can discover a shared subnetwork for multiple spoken languages in one pass, and (5) PARP significantly reduces cross-tast adaptation errors of pre-trained BERT/XLNet. Beyond the scope of our study, we aspire PARP as the beginning of many future endeavours on developing more efficient speech SSL models.
Broader Impact. The broader impact of this research work making speech technologies more accessible in two orthogonal dimensions: (i) extending modern-day speech technology to many under-explored low-resource spoken languages, and (ii) introducing a new and flexible pruning technique to current and future speech SSL frameworks that reduces the computational costs required for adapting (finetuning) them to custom settings. We do not see its potential societal harm.
Limitations and Future Work
We make clear of the major limitations of our work, and the full list is in Appendix 19 The basis of all the pruning methods in the study is unstructured magnitude weight pruning. Although sparsity explicitly enforced in the models, we do not suggest that the sparse models are more memory or energy efficient than the original dense models. We do believe that our methodology and results should provide meaningful insights and be easily extended upon to more advanced unstructured or structured pruming methods. We are also curious of the possibility of finetuning or storing modern speech SSL models on local hardware devices.
Results on cross cross-lingual mask transfer on pre-trained wav2vec 2.0 in Section is limited to ASR. We do not claim pruning masks be transferrable across speech tasks (e.g. prune wav2vec2 for speaker ID and transfer for ASR). We provide a pilot cross-task mask transfer study on 3 speech tasks (phone recognition, speaker recognition, slot-filling) in SUPERB 120 and results is in Appendix [6
We claim PARP could improve the downstream ASR performance over the full wav2vec 2.0, yet we do not claim it as a plug-and-play method into any SOTA ASR pipeline, such as 126 to get performance boost. We provide a preliminary experiment on combining PARP and transformer- decoding in Appendix [5 Nonetheless, due resource limitations and isolate the effect of pruning, it remains upon investigations on the complete effects of speech pruning in different setups.
A Akknowledmments
We thank IBM for the donation to MIT of the Satori GPU cluster, and John Cohn for maintaining the cluster. We also thank Lucy Chai, Wei-Ning Hsu, Desh Raj. Shu-wen Leo Yang, Abdelrahman Mohamedm, Erica Cooper, and anonymous reviewers for helpful suggestions and paper editing. This work is part of the low resource language learning project funded by the MIT-IBM Waston Al Lab.
10

---

References
[1] Sherif Abdou and Michael S Scordilis. Beam search pruning in speech recognition using posterior probability-hase confidence measure. Speech Communication, 42(3-4):409-428, 2004.
[2] Junyi Ao, Rui Wang. Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, et al. Speecht5: Unified-modal encoder-doeoder pre-training for spoken language processing, arXiv preprint (tfi::2/100072000 2021.
[3] Alexci Baevski, Michael Auli, and AbdeIrahman Mohamed. Effectiveness of self-superrised wre-training for speech recognition. arXiv preprint urXiv:191.039112 2019.
[4] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recognition. arXiv preprint (rtxii:2005./10000 2021.
[5] Alexei Baevski, Steffen Schneider, and Michael Auli, vg-wav2vee: Self-supervised learning of discrete speech representations. XrXiv preprint arXiv: 191.0555, 2019.
[6] Alexci Bacvski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006. /477,, 2020.
[7] Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. arXiv preprint arXiv:/81 2018.
[8] Yonatan Belinkov, Ahmed Ali, and James Glass. Analyzing phonetic and graphemic representations in end-to to-etoddddddd automatic speech recognition. arXiv preprint 1907.04224 2019.
[9] Yonatan Belinkov and James Glass. Analyzing hidden representations in end-to-end automatic speech recognition systems. arXiv preprint arXiv:1709.04444 2017.
[10] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint urXiv:1711.05..6 2017.
[11] Davis Blalock, Jose Javier Gouzalez Ortiz. Jonathan Frankle, and John Guttag. What the state neural network pruning? arXiv preprint arXiv:2003.030.:5 2020.
[12] Stefan Braun and Shih-Chii Liu, Parameter uncertainty for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5636-5640. IEEE, 2019.
[13] Steven Cao, Victor Sanh, and Alexander M Rush. Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104003514, 2021.
[14] William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. Speechstew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint arXiv:2104.02133 2021.
[15] Xuankai Chang. Takashi Mackaku, Pengcheng Guo, Jing Shi, Yen-Ju Lu, Aswin Shanmugam Subra- manian, Tianzi Wang, Shu-wen Yang, Yu Tsao, Hung-yi Lee, et al. An exploration of self-supervised pretrained representations for end-to-end speech recognition. arXiv preprint arXiv:2110. 10.04590, 2021.
[16] Sanyuan Chen, Yu Wu, Zhuo Chen, Jian Wu, Jinyu Li, Takuya Yoshioka, Chengyi Wang, Shujic Liu, and Ming Zhou. Continuous speech separation with conformer. In ICASSP 202 1022- IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5749-5753. IEEE, 2021.
[17] Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, et al. Unispeech-sat: Universal speech representation learning with speaker aware pre-training. arXiv preprint arXiv:21 10.05552, 2021.
[18) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv preprint arXiv:2012.0665 2.69088,. 2020.
[19] Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu, Yang Zhang. Zhangyang Wang. and Michael Carbin. The lottery ticket hypothesis for pre-trained hert networks. arXiv preprint arXiv:2007. 12223, 2020.
[20] Yi-Chen Chen, Shu-wen Yang, Cheng Kuang Lee, Simon See, and Hung-yi Lee. Speech representation learning through self-supervised pretraining and multi-tas, finetuning, arXiv preprint arXi::21000099:0 2021.
[21] Jaejin Cho. Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, and Takaaki Hori. Multilingual soquence-to-sqqeneee speech recognition: architecture, transfer learning, and language modeling. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 221-527. IEEE, 2018
[22] Jan Chorowski, Grzegorz Ciesielski. Jaroslaw Dzikowski, Adrian Lancucki, Ricard Marxer, Mateusz Opala, Piotr Pusz, Pawel Rychlikowski, and Michal Stypulkowski. Aligned contrastive predictive coding. arXiv preprint arXiv:2104.11946 2021.

---

[23]
Shammur Absar Chowdhury. Nadir Durrani, and Ahmed Ali. What do end-to-end speech models learn about speaker. language and channel information? layer-wise and neuron-level analysis. arXiv preprint TrXiv:2107.00433 2021.
[24] Yu-An Chung, Yonatan Belinkov, and James Glass, Similarity analysis of self- self-supervised speech representations. In ICASSP 2021 1222 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3040 3044, IEEE, 2021.
[25] Yu-An Chung. Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv: 1904.03240,, 2019.
[26] Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass. Unsupervised cross-modal aligmment speech and text embedding spaces. arXiv preprint 1rXiv:1805.07467,, 2018.
[27] Yu-An Chung. Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre- training. arXiv preprint arXiv:2108.066 06209, 2021.
[28] Yu- An-An Chung, Chenguang Zhu, and Michael Zeng. Splat: syeech-language joint pre-training for spoken language understanding. arXiv preprint ar arXiv:2010.02299 2020.
[29] Alexis Conneau, Alexei Bacvski, Ronan Collohert, Ahdelrahman Mohamed, and Michael Auli. Unsu- pervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. 13399, 2020.
[30] Erica Cooper, Wen-Chin Huang. Tomoki Toda, and Junichi Yamagishi. Generalization ability mos prediction networks. arXiv preprint urXiv:2110.02633 2021.
[31] Jia Cui, Xiaodong Cui, Bhuvana Ramabhadran, Janice Kim, Brian Kingsbury, Jonathan Mamou, Lidia Mangu, Michael Picheny. Tara N Sainath, and Abhinav Sethy. Developing speech recognition systems for corpus indexing under the iarpa babel program. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6753 6757. IEEE, 2013.
[32] Jia Cui, Brian Kingsbury. Bhuvana Ramabhadran, Abhinav Sethy. Kartik Audhkhasi. Xiaodong Cui, Ellen Kislal, Lidia Mangu, Markus Nusshaum- Michael Picheny, et al. Multilingual representations for low resource speech recognition and keyword search. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 259-266. IRFF, 2015.
[33] Xiaodong Cui, Brian Kingsbury, Jia Cui, Bhuvana Ramabhadran, Andrew Rosenberg, Mohammad Sadegh Rasooli, Owen Rambow, Nizar Habash, and Vaibhava Goel. Improving deep neural network acoustic modeling for audio corpus indexing under the iarpa babel program. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
[34] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104, 080666,6,.6....8 2021.
[35] Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass, What is one grain of sand in the desert? analyzing individual neurons in deep nip models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 6309-6317, 2019.
[36] Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4908-4926, 2020.
[37] Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov, Analyzing individual neurons in pre- vo--traine language models. arXiv preprint arXiv:2010. 02690.06555 2020.
[38] Radek Fer, Pavel Matějka, František Grézl, Oldřich Plchot, Karel Veselý, and Jan Honza Černocký. Multilingually trained bottleneck features in spoken language recognition. Computer Speech & Language. 46:252 267, 2017.
[39] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural 2018.
networks. arXiv preprint
[40] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear made connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[41] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint 1rXiv:1902.09554, 2019.
[42] Mark JF Gales, Kate M Knill, Anton Ragni, and Shakti Rath. Speech recognition and keyword spotting for low fow-resource languages: Babel project research at cued. In Fourth International Workshop on Spoken Language Technologies for Under-Resourcee Languages (SLTU-2014), pages 16 66-23, International Speech Communication Association (ISCA), 2014.
12

---

[43] Dawei Gao, Xiaoxi He. Zimu Zhou, Yongxin Tong, Ke Xu, and Lothar Thiele. Rethinking pruning for accelerating deep inference at the edge. In Proceedings the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 155-164, 2020.
[44] Heting Gao, Junrui Ni, Yang Zhang, Kaizhi Qian, Shiyu Chang, and Mark Hasegawa-Johnson, Zero-shot cross-lingual phonetic recognition with external language embedding. Proc. Interspeeck 1021, pages 1304-1308, 2021.
[45] Sharath Girish, Shishira R Maiya, Kamal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivastava. The lottery ticket hypothesis for object recognition. arXiv preprint Seti::0022.06443, 2020.
[46] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah, Multimodal neurons in artificial neural networks. Distill, 6(3)=300, 2021.
[47] Anmol Gulati, James Qin, Chung-Chune Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer:
cnnvolutionauugmennn transformer for speech
recognition. arXiv preprint arXiv:2005. 00000000000,,..... 2020.
[48] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, arXiv preprint 160::6608.44933 2016.
[49] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv: 0aee::156.020000 02626, 2015.
[50] Wei Han, Zhengdong Zhang. Yu Zhang. Jiahui Yu, Chung- Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv preprint Xiv: 200v::000.0000 0319,, 2020.
[51] Babak Hassibi and David G Stork. Second order derivatives for network pruming: Optimal bruin surgeon. Morgan Kaufmann, 1993.
[52] Tianxing He, Yuchen Fan, Yanmin Qian, Tian Tan, and Kai Yu. Reshaping deep neural network for fast decoding by node- monde-puning. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 245-249. JEEE, 2014.
[53] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2/066.7744, 2021.
[54] Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana L.khhomanekko, Qiantong Xu, Vincel Pratap, Jacob Kahn, Ann Lee, Ronan Collohert, Gabriel Synnaeve, et al. Rohust wav2vee 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprint 104.01024 2021.
[55] Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can a bad teacher benefit asr pre-training? In /CASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6533-6537. IEEE, 2021.
[56] Wen-Chin Huang, Shu-Wen Yang, Tomoki Hayashi, Hung- Yi Lee, Shinji Watanahe, and Tomoki Toda. S3prI-ve: Open-source voice conversion framework with sulf-super speech representations. arXiv preprint 2 06.00000 2021.
[57] Dongwei Jiang, Wubo Li, Miao Cao, Ruixiong Zhang. Wei Zou, Kun Han, and Xiangang Li. Speech simelr: Combining contrastive and reconstruction objective for self self-supervised speech representation learning. arXiv preprint arXiv:2010.1 13991, 2020.
[58] Naoyuki Kanda, Guoli Ye, Yu Wu, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka. Large-scale pre-training of end-to-end multi-talker ast for meeting transcription with single distant microphone. arXiv preprint arXiv:2103. /6776, 2021.
[59] Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.08 032644999999...0..000 2021.
[60] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880-2894 2020.
[61] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of hert. arXiv preprint arXiv: 1908,08593, 2019.
[62] Cheng-I Lai. Contrastive predictive coding based feature for automatic speaker verification. arXiv preprint arXivv1904.0155 1904.0157, 2019.
[63] Cheng-1 Lai, Yung-Sung Chuang, Hung- Yi Lee, Shang- Wen Li, and James Glass, supervuper spoken language understanding via supe-sid speech and language model pretraining. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7468-740 7472. IEEE, 2021.
13

---

[64] Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte. Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, et al. Generative spoken language modeling from raw audio. arXiv preprint urXiv:2102.01192, 2021.
[65] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598 0088 1990.
[66] Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct speech-to to-speech translation with discrete units. arXiv preprint 1xXiv:2107.05604, 2021.
[67] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Tomr. Snip: Single-shot network pruning based on connection sensitivity, arXiv preprint 1rXivv/8100023400 2018.
[68] Bo Li, Ruoming Pang, Tara N Sainath, Anmol Gulati, Yu Zhang, James Qin, Parisa Haghani, W Ronny Huang, and Min Ma. Scaling end-to-end models for large-scale multilingual ast. arXiv preprint 2021.
arXiv:2104.1483
[69] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:160s.087/0, 2016.
[70]
Shenggiang Li, Menglong Xu, and Xiao-Lei Zhang. Efficient conformer-based speech recognition with linear attention. arXiv preprint arXiv:2104.06663 2021.
[71] Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06633 2020.
[72] Alexander Liu, Yu-An Chung, and James Glass. Non-auloregressive predictive coding for learning speech representations from local dependencies. arXiv preprint Sidiv:2011.00000 2020.
[73] Andy Liu, Shang- Wen Li, and Hung-yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2351- 2366, 2021.
[74] Andy Liu, Shu- wen Yang. fo-Han Chi, Po-chun Hsu, and Hung yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419-6423, TERR, 2020.
[75] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 1810.05000 2018.
[76] Takashi Maekaku, Xuankai Chang, Yuya Fujita, Li-Wei Chen, Shinji Watanabe, and Alexander Rudnicky. Speech representation learning combining conformer epe with deep cluster for the zerospeech challenge 2021. arXiv preprint arXiv:2107.05899 2021.
[77] Eran Malach, Gilad Ychudai, Shai Schwar-Sc and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need, In International Conference on Machine Learning, pages (682-6691. PMLR, 2020.
[78] Yen Meng. Yi-Hui Chou, Andy T Liu, and Hung-yi Lee. Don't speak too fast: The impact of data bias on sulf-supervised speech models. arXiv preprint arXiv: 10.07957, 2021.
[79] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1126--12222 2019.
[80] Rajiv Movva and Jason Y Zhao. Dissecting lottery ticket transformers: Structural and behavioral study of sparse neural machine translation. arXiv preprint arXiv:2009.15 13270, 2020.
[81] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. arXiv preprint xxXiv22066110332 2020.
[82] Sharan Narang, Erich Flsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint WiXiv:170.0051999 2017.
[83] Edwin Ng. Chung-Cheng Chiu, Yu Zhang, and William Chan. Pushing the limits of non-autoreeressive speech recognition. arXiv preprint arXiv:2104.05:: 2021.
[84] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 10X.vv:80770374: 2018
[85] Sankaran Panchapagesan, Daniel Park, Chung -Cheng Chiu, Yuan Shangguan, Qiao Liang, and Alexan- der Gruenstein. Efficient knowledge distillation for rnn- transducer models. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages IEEE, 2021.
14

---

[86] Adam Polyak, Yossi Adi, Jade Copet. Eugene Kharitonov. Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint urXiv:2104,00 00355, 2021.
[87] Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur. cemi-orthogonal low-rank matrix factorization for deep neural networks. In Interspeech, 2018
pages
[88] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561. 2020.
[89] Vincel Pratan, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Massively multilingual asr: 50 languages, model, billion parameters. arXiv preprinl arXiv:2007.000 000001, 2020.
[90] Vineel Pratap. Qiantong Xu, Jacob Kahn, Gilad Avidov, Tatiana Likhomanenko, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Scaling up online speech recognition using convnets. 2020.
arXiv preprint arXiv:2001.092
[91] Janne Pylkkönen. New pruning criteria for efficient decoding. In Ninth European Conference on Speech Communication and Technology. 2005.
[92] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's 's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1199--19922, 2020.
[93] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and finc-tuning in neural network pruning. arXiv preprint arXiv:2003 02388, 2020.
[94] Morgane Rivière, Armand Joulin, Eeerre- Mazaré, and Emmanuel Dupoux. Unsupervised pretraining transfers well across languages. In ICASSP 2020-2020 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7414-7418. IEEE, 2020.
[95] Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of ceneral-purposs audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 3875 3879. IEEE, 2021.
[96] Ramon Sanabria, Austin Waters, and Jason Baldridge. Talk, don't write: A study of direct speech-based image retrieval. arXiv preprint arXiv:2104. 01894, 2021.
[97] Victor Sanh. Thomas Wolf, and Alexander M Rush. Movement pruning: Adaptive sparsity by fine-tuning. arxiv preprint arXiv:2005. 07663, 2020.
[98] Yuan Shangguan, Jian Li, Qiao Liang, Raziel Alvarez, and Ian McGraw. Optimizing speech recognition for the edge. arXiv preprint Xiv: 12408, 2019.
[99] Yangyang Shi, Yongqiang Wang. Chunyang Wu, Ching-Fenn Feng Yeh, Julian Chan, Frank Zhang, Due Le, and Mike Seltzer. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021-2021 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6783-6787. IEFF, 2021.
[100] Vesa Siivola, Teemu Hirsimaki, and Sami Virpioja. On growing and pruning kneser-ney smoothed n--rrmm models. IEEE Transactions on Audio, Speech, and Language Processing, 15(5):1117-124, 2007.
[101] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-ena asr: from supervised to semi- supervised learning with modern architectures. arXiv preprint 1wxiv:1911.08440 2019.
[102] Liang- Hsuan Tseng, Yu-Kuan Fu, Heng-Jui Chang, and Hung-yi Lee. Mandarin Mandarin-english code swie-swichingg speech recognition with self superviidd speech representation models. arXiv preprint arXiv:2110003500 2021.
[103] Hugo Van Hamme and Filip Van Aclten, An adaptive- heam pruning technique for continuous speech recognition. In Proceeding of Fourth International Conference on Spoken Language Processing. IC- SLP'96, volume 4. pages 2083-2086. IEEE, 1996.
[104] Ganesh Venkatesh, Alagappan Valliappan, Jay Mahadeokar, Yuan Shangguan, Christian Fuegen, Michael L. Seltzer, and Vikas Chandra. Memory-efficien speech recognition on smart devices. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8368 8372. IEEE, 2021.
[105] Elena Voita. David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov, Analyzing multi- self- attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: 0905.09418. 2019.
15

---

[106] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0744: 084401761. 2018
[107] Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Yao Qian, Kenichi Kumatani, and Furu Wei. Unispeech at scale: An empirical study of pre-training method on large-scale speech recognition dataset. arXiv preprint
arXiv:2107.05233 05233,, 2021.
[108] Chengyi Wang. Yu Wu, Yao Qian, Kenichi Kumatani, Shujie I Liu, Furu Wei, Michael Zeng, and Xuedong Huang. Unispeech: Unified speech representation learning with labeled and unlabeled data, arXiv preprint arXiv:2101. 00072100..7797, 2021.
[109] Jun Wang, Max W Y Lam, Dan Su, and Dong Yu. Contrastive separative coding for self-supervised representation learning. In ICASSP 202/-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3865-3869. IEEE, 2021.
[110] Weiran Wang, Qingming Tang, and Karen Livescu. Unsupervised pre-training of bidirectional speech enenders via masked reconstruction. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 6889-6893. IEEE, 2020.
[111] Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, and Yu Wu. Wav2vec-switch: Contrastive Icarning from original-noisy speech pairs for robust speech recognition. arXiv preprint urXiv:2110.04934, 2021.
[112] Matthew Wiesner, Desh Raj. and Sanjeev Khudanpur. Injecting text and cross- lingual supervision in few-shot learning from self-supervised models. arXiv preprint arXiv:21 2021.
[113] John M Wu, Yonatan Belinkov, Hassan Sajja, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity analysis of contextual word representation models. arXiv preprint arXiv:2005.00 01 172, 2020.
[114] Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang. Dynamic sparsity neural networks for automatic speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6014 4-10. IEEE, 2021.
[115] Ji Xin, Jimmy Lin, and Yaoliang Yu. What part of the neural network does this? understanding Istms by measuring and dissecting neurons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNIP-IJCNIPP) P). pages 5827-5834, 2019.
[116] Hainan Xu, Tongfei Chen, Dongji Gao, Yiming Wang, Ke Li, Nagendra Goel, Yishay Carmiel. Daniel Povey, and Sanjeev Khudanpur. A pruned mnamm lattice-rescoring algorithm for automatic speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5929-5929999 5933, IEEE, 2018.
[117] Qiantong Xu, Alexei Baevski, and Michael Auli. Simple and effective zero-shot cross- coossllingual phoneme recognition. arXiv preprint arXiv:2109 16600, 2021.
[118] Qiantong Xu, Alexei Baevski. Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Col- lobert, Gabriel Synnaeve, and Michael Auli. Self-training and pre-training complementary for speech recognition. In ICASSP 2021-202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3030-3034, IEEE, 2021.
[119] Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with singular value decomposition, In Interspeech, pages 2365 206652236,220 2013.
[120] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-1 Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang. Guan- Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.00 00055, 2021.
[121] Zhao You, Shulin Feng, Dan Su, and Dong Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. arXiv preprint arXiv:2105.03036, 2021.
[122] Dong Yu, Frank Seide, Gang Li, and Li Deng. Exploiting sparseness in deep neural networks for large vocabulary speech recognition. In 2012 IEEF International conference on acoustics, speech and signal processing (ICASSP), pages 4409-4412. IEEE, 2012.
[123] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nip. arXiv preprint arXiv: I 1906.02768, 2019.
[124] Jiahui Yu, Wei Han, Anmol Gulati, Chung-Chor Cheng Chiu, Bo Li, Tara N Sainath, Yonghui Wu, and Ruoming Pang. Universal asr: Unify and improve streaming ast with full- modeling. arXiv preprint arXiv:2010.0603: 06030, 2020.
[125] Jiahui Yu, Wei Han, Anmol Gulati, Chung -Chnng Chiu, Bo Li. Tara N Sainath, Yonghui Wu, and Ruoming Pang. Dual-mode asr: Unify and improve streaming asr with full- context modeling. Proceedings of ICLR. 2021.
16

---

[126] Yu Zhang, Daniel S Park, Wei Han, James Qin. Anmol Gulati. Joel Shor, Aren Jansen. Yuanzhong Xu. Yanping Huang, Shibo Wang. et al. Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2 13226, 2021.
[127] Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung- Chiu, Ruoming Pang. Quoe V Le, and Yonghui Wu. Pushing the limits of semi-supervisee learning for automatic speech recognition. arXiv preprint arXiv:2010.
2020.
[128] Yuckai Zhang, Sining Sun, and Long Ma. Tiny transducer: A highly-efficient speech recognition model edge devices. In ICASSP 2021-202 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6024-6028. IEEE, 2021.
[129] Han Zhu, Li Wang, Ying Hou, Jindong Wang. Gaofeng Cheng, Pengyuan Zhang, and Yonghong Yan. Wav2vec-s: Semi-supervised pre-training for speech recognition. arXiv preprint arXiv:2110004488 2021.
17


Reviews:
[{"review_id": "z1ZsETl6vnK", "reviewer": "Reviewer_xQYW", "summary": "This paper presents a Prune-Adjust-Re-Prune (PARP) approach for finding a sparse subnetwork within a self-supervised speech model with competitive performance for the Automatic Speech Recognition (ASR) downstream task in mono-lingual and multi-lingual conditions. Despite the approach's simplicity, its downstream ASR performance (measured using WER without LM decoding) is better than other subnetwork discovery approaches (for almost all fine-tuning conditions) and the original unpruned network when 10mins of paired data is used. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed PARP approach alternates between two steps starting from the pre-trained network: a) zeroing out the smallest X % of the weights based on their magnitude. b) training the network for some update steps while allowing the zeroed-out weights in step (a) to grow again. The proposed approach is similar to previous work in NN pruning, but it is the first to be applied to self-supervised speech models. Experiments on multi-lingual ASR showed that initial subnetwork masks could be transferred between languages. Also, a quick experiment on NLP tasks showed similar behavior. The paper is well written with good experiments on the Wav2vec 2.0 pre-trained English and other lower-resource languages. \n\nThere are few comments on the presented work: \n1) The paper offers an imbalanced presentation between Speech and NLP. Given the generality of the proposed approach and continuous reference to previously proposed work on other modalities, it makes sense to offer more experiments on the same tasks. Adding Table 4 raises the confidence in the efficacy of the approach a bit. More experiments will convince the reader further. \n\n2) Using WER with no LM is accepted for seq2seq models which learn an implicit LM from the data; however, this work uses only a CTC loss leading to considerable variance in the WER when no LM is used, in particular when few amounts of paired data are used, e.g., 10mins. The authors are encouraged to share results with n-gram decoding, at least for their English models.\n\n3) The results on initial subnetwork generality to multiple tasks for the speech case need to be taken cautiously since all the other tasks are still ASR (but in other languages). The bottom part of many models trained from waveform input captures spectral representations shared between languages for the same task. Audio Event Detection tasks or Speaker Clustering tasks may be good ones to examine this point for speech.\n\n4) There is not much discussion on where the pruned weights are located in the pre-trained network. The Wav2vec2.0 approach is known to have valuable representations towards the middle of the network, with higher-level weights only beneficial for the pre-training contrastive loss with less impact on downstream applications. One reason for the similarity between the subnetwork for English and multi-lingual models could be that both use the same pre-training approach, leading to the same inductive biases, i.e., a peculiarity of the pre-training approach rather than a property of the found sub-network used with PARP. \n\n5) It will inform the readers if the authors share the number of model updates (independent of the paired dataset size) used for PARP. One hypothesis for good performance could be the long training time the model weights are exposed to the paired data (without rewinding to their original pre-trained values). It is known that longer training times with slow learning rates lead to superior performance on many tasks. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "wCbra-sV43I", "reviewer": "Reviewer_i9ds", "summary": "The goal of this contribution is to reduce the complexity of self-supervised speech representation learning systems. Following a description of the landscape of unstructured magnitude pruning methods (i.e. methods that sort weights by their magnitude and remove the lowest ones with no other consideration), authors propose PARP. PARP is based on the observation that \"task-agnostic\" masks (obtained on a pre-trained network) are very similar to ones that would be obtained after fine-tuning on a target task. Taking benefit from this observation, and unlike OMP that cascades finetune -> prune -> finetune, PARP simply perform prune -> finetune. Experiments on low resource ASR and cross-lingual training show a clear advantage of PARP over IMP, OMP and MPI within a language (less obvious for high-to-low resource transfer and cross-lingual training), but also a surprising gain in performance w.r.t. unpruned architectures. Moreover, authors show that PARP also works on another modality, by pruning BERT models for GLUE downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is remarkably well written and structured. The context and baselines are well explained, and as a reviewer unfamiliar with the pruning literature, I found the paper easy to follow and the models clearly explained. The experiments are convincing both in their scope (within and across languages, speech and NLP) and results. In particular, seeing improved performance over unpruned model with up to 50% sparsity in some cases is impressive. As a non-expert I cannot guarantee the appropriateness of the baselines (maybe some more sophisticated methods are missing), however given the baselines in the paper, PARP clearly has an advantage both in terms of training cost and performance. Overall, even though the problem addressed by this work could be considered as relatively narrow, the effort put in the redaction and presentation of the method and results, the significant performance of the system against baseline, as well as the generality of the approach (showcased by the NLP experiments) make this paper both worth reading and promising for future applications. For these reasons, I recommend this paper for acceptance.\n\nRemarks:\n* Figure 1 mentions an exponential increase in compute for IMP as the sparsity level increases. This is understandable following section 2.3 that explains how IMP prunes 10% of the remaining weights at each iteration (rather than 10% from all weights). However Figure 1 shows a linear increase in number of fine-tuning runs.\n* Section 2.2 is in past tense, while the other sections are in present tense.\n* line 151: \"Initailizations\"\n* Algorithm 1: since m is updated at every iteration, it would be easier to read by adding an index to it.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "HWLwnPwg2jr", "reviewer": "Reviewer_hRhS", "summary": "In this work, the authors build upon techniques to reduce the number of parameters in the model by finding sparse sub-networks along the lines of the “lottery-ticket hypothesis” work. In this case, however, the authors apply the proposed techniques in the context of self-supervised speech recognition by showing that the same initial masked network is suitable for fine-tuning for a number of tasks including when applied for cross-lingual transfer when used with multiple languages.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is an extremely well-written paper. In particular, I found that the motivation of the work was extremely clear, and well-supported by analyses to show that the sub-networks obtained in a task agnostic fashion had a large overlap (as measured in terms of IOU) with the task-specific sub-networks. I also found the experiments to be well-designed and comprehensive exploring different settings: pre-training and finetuning for the same language; pre-training on one language and fine-tuning on a large number of languages; pre-training and fine-tuning on multiple languages. The ablation experiments in Section 4, were interesting, and strengthen the paper greatly. In my view, the authors’ work -- particularly, the work on rapidly training models on multiple languages -- is a very impactful contribution. \n\nOne clarification question: In Figure 4, if I understand correctly, the PER w/o and LM is still ~50% even at 100% sparsity. This was somewhat puzzling to me. Shouldn’t the PER be 100%+ if all weights are zeroed-out? (I assume that 100% sparsity means that all weights are zeroed out even after finetuning).\n\nI would note that one limitation of this work is that although the authors show that they can achieve large unstructured sparsity in the model, it is not clear if the sparsity structure will be amenable to getting large speed-ups when run on a device/server. I think the paper could be strengthened by measuring actual RTF/benchmark numbers by running these models on device.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Apart from the limitations mentioned above, I do not see any potential negative societal impacts of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is interested in reducing the complexity of self-supervised speech representation learning systems, by discovering sparse subnetworks in the self-supervised architecture which perform well on downstream tasks. While reviewers acknowledge the approach is simple (basically performing a sparsifying step followed by a finetuning one), it is effective. The paper also applies the idea on BERT models.

Authors should include in the final version of the paper the additional results and findings mentioned during the discussion period.
Decision: Accept (Spotlight)