S$^6$-DAMON: Bridging Self-Supervised Speech Models and Real-time Speech Recognition
Authors: Yonggan Fu
Zhifan Ye
Shunyao Zhang
Jiayi Yuan
Zhongzhi Yu
Yingyan Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: automated speech recognition
self-supervised learning
model compression

Abstract: There has been an growing demand for deep neural network (DNN) powered automatic speech recognition (ASR) on mobile platforms for real-time speech recognition. However, ubiquitous on-device ASR systems are still hindered by two bottlenecks: (1) the lack of large-scale transcribed speech data especially for low-resource spoken languages and (2) the large gap between DNNs' prohibitive complexity and mobiles' limited resources. In parallel, speech models pretrained via self-supervised learning (SSL) have emerged to reduce the reliance on the availability of transcribed speech data, which however further enlarges the efficiency gap because they often adopt large transformers to ensure expressive speech representations. Thus, it is highly desired to trim down the complexity of speech SSL models to enable real-time on-device ASR. This is particularly challenging since only structured sparsity can favor hardware efficiency in commercial devices, under which the speech representation learned by SSL could easily be demolished. To this end, we develop a framework dubbed S$^6$-DAMON to pursue structured sparsity in speech SSL models via data-model co-compression. On the data side, leveraging both the duration of each phoneme and the pauses between the words/phonemes of human utterances, we propose a salient audio token detector, dubbed SALAD, to remove input audio tokens that are redundant; On the model side, we identify that the failure of the SOTA ASR pruning method under structured sparsity is caused by the sparsity discrepancy between finetuning/deployment and their limited learnability of sparsity distributions, and then tackle it via a new ASR pruning pipeline dubbed SAFARI, which adopts a three-step pipeline - sparsify, finetune, and adjust sparsity. Extensive experiments validate that S$^6$-DAMON can enable real-time ASR with limited transcribed speech data requirements while maintaining decent recognition performance. All source codes will be released upon acceptance.

Under review as a conference paper at ICLR 2023
S BRIDGING SELF-SUPERVISED SPEECH
MODELS AND REAL-TIME SPEECH RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
There has been an growing demand for deep neural network (DNN) powered automatic speech recognition (ASR) on mobile platforms for real-time speech recognition. However, ubiquitous on-device ASR systems are still hindered by two bottlenecks: (1) the lack of large-scale transcribed speech data especially for low-resource spoken languages and (2) the large gap between DNNs' prohibitive complexity and mobiles' limited resources. In parallel, speech models pretrained via self--upervised learning (SSL) have emerged to reduce the reliance on the avail- ability of transcribed speech data, which however further enlarges the efficiency gap because they often adopt large transformers to ensure expressive speech repre- sentations. Thus, it is highly desired trim down the complexity of speech SSL models to enable real-time on-device ASR. This is particularly challenging since only structured sparsity can favor hardware efficiency in commercial devices, under which the speech representation learned by SSL could easily be demolished. To this end, we develop a framework dubbed S SD-DAMON to pursue structured sparsity speech SSL models via madel co-compression. On the data side, leveraging both the duration of each phoneme and the pauses between the words/phonemes human utterances, we propose salient audio token detector, dubbed SALAD, to remove input audio tokens that are redundant; On the model side, we identify that the failure of the SOTA ASR pruning method under structured sparsity is caused by the sparsity discrepancy between finetniiggdepllymment and their limited learn- ability of sparsity distributions, and then tackle it via new ASR pruning pipeline dubbed SAFARI, which adopts a three-step pipeline- sparsify, finetume, and adjust sparsity. Extensive experiments validate that DDMMNN can enable real-time ASR with limited transcribed speech data requirements while maintaining decent recognition performance. All source codes will be released upon acceptance.
INTRODUCTION
Recent breakthroughs in deep neural networks (DNNs) have tremendously advanced the field of Automatic Speech Recognition (ASR), enabling sccord-breaking end-to-end ASR systems (Hannum al., 2014; Chan et al., 2016; Zhang al., 2020; Gulati et al., 2020). Considering that speech is one of the basic input modalities from users of intelligent mobile devices, there has been an increasing interest in the development and deployment of on-device ASR systems. For example, intelligent assistants (Meta-Al, 2022; Vox, 2022) are highly desired in next-generation augmented reality and virtual reality (AR/VR) devices for enabling immersive AR/VR experiences. This has called for advanced speech technologies in order to deliver accurate and real-time ASR systems.
There still remain two critical efficiency bottlenecks for ubiquitous on-device ASR systems, including (1) data efficiency: big data is often not practical for ASR since collecting transcription on large scale is costly or not even possible, especially for low-resource spoken languages and (2) model efficiency: the often limited on-device resources stand at odds with the complexity of deep ASR models, making it particularly challenging to satisfy real-time ASR requirements. To promote the aforementioned data efficiency, recent advances in self-supervised learning (SSL) for speech representation (Chi et al., 2020; Baevski et al., 2020; 2022) have demonstrated empirical success and become the de-facto paradigm for low-resource ASR, where SSL models pretrained on unlabeled audio data can be generalized to handle low-resource transcribed speech after being finetuned. However, this could further aggravate the model efficiency bottleneck giant transformers (Vaswani

---

Under review as a conference paper at ICLR 2023
Token-wise Predictions
Repeat
Blank
S*.DAMON (Ours)
22.5
IT @ A E R F
PARP
OOLL & & CA O & R D O 6 B R R C'K K & & OF N E D & O B A & A P A L E E A D é O W D R R Y ØLICE EN & & W W T C H o H A & A W H &HHAAPY HIRRREE U L A
OMP
17.5
Original
ISSPER
GIT SE L
12.5
10.0
Recognition Results
7.5
is a very fine old place of red brick softened by pale powdery lichen which has dispersed itself with happy irregularity
5.0
20
30
40
50
60
70
FLOPs Savings (%)
(a)
(b)
Figure 1: (a) An example from LibriSpeech for illustrating two types of non-salient audio tokens; (b) The trade-offs between WER on LibriSpeech test-clean and FLOPs savings achieved by different ASR compression schemes on top wav2vec2-base finetuned on LibriSpeech- 100h.
al., 2017) (e.g.. > 90M parameters) are often adopted in state-of the-art (SOTA) speech SSL models to ensure effective representation learning during SSL, making it increasingly more challenging for on-device deployment. Therefore, it is imperative to compress speech SSL models while maintaining their generalizable speech representation for delivering efficient ASR systems.
Despite the demanding need, is non-trivial to narrow the gap between large speech SSL models and constrained resources in mobile devices. First, under the SOTA pretrain-then-fieeti paradigm, most useful features are learned during the SSL stage and then pretrained speech SSL models only slightly adapt their weights to encode task-specific information during finetuning, whereas it is difficult to learn a sparsity distribution during finetuning while maintaining the fidelity of the speech representation given the low-resource downstream speech. Note that this is particularly challenging for ASR due to the more stringent low-resource settings, e.g., LibriSpeech- 10m (Panayotov et al., 2015) for ASR only contains 48 sentences for training and development, whereas the CoLA dataset (Wang et al., 2018) for natural language processing (NLP) contains 9594 sentences. Second, only structured sparsity can favor hardware efficiency in commercial mobile devices, which however will pose a more severe destruction during finetuning on the SSL speech representation learned during pretraining than unstructured sparsity, e.g., enforcing structured sparsity in the SOTA unstructured ASR pruning framework called PARP (Lai et al., 2021) will cause a >8% increase in word-error- rate (WER) under only a 20% sparsity on w baseee.ibriSeeecc-- 1h. Third, considering that ASR corresponds to sequence- seateencett-sequenee to-seueence task where the alignment between inputs and outputs is monotonic, the compression process is thus required to be more meticulous in preserving the information of useful audio frames than compressing classification-tass models.
Our Contributions. We develop a framework dubbed S*-DAMON which is the first to pursue structured sparsity in speech SSL models under low- low-resource settings via data-mode co-compression for enabling real-time on-device speech recognition.
On the data side, S -5-DAMON leverages the intrinsic redundancy in human speech. As the duration of each phoneme and the pauses between the words/phonemess of human utterances, the sampled audio frames and the corresponding extracted audio tokens, i.e., inputs for the transformers, may (1) repeat the previous tokens, or (2) stand as blank, contributing little to the final recognition (see an example in Fig. (a)). We call both non-salient audio tokens (NATs) and the first-appearing tokens that indispensable for ensuring monotonic recognition as salient audio tokens (SATs). Properly removing NATs can lead non non-trivial savings in model efficiency while better maintaining the accuracy than removing SATs, e.g., NATs account for 50.6% of the total tokens on LibriSpeech test-clean based on token-wiss wise annotations from finetuned wav2vec2-basee As only sentence-level transcriptions are annotated in ASR datasets and tosen-wise labels are not available to classify SATs/NATTs, we design a salient audio token detector called SALAD and train it in semi-supervised manner based on the pseudo token-wise labels annotated by finetuned speech SSL models on untranscribed speech. A high recall is enforced to ensure the coverage SATs, thus properly removing NATs detected by SALAD in inference can better maintain the speech representation fidelity. On the model side, we discover that the failures of SOTA ASR pruning method PARP (Lai et al., 2021) under structured sparsity caused by (1) the sparsity discrepancy between finetuning/deploymee,, i.e., PARP finds the
2

---

Under review as a conference paper at ICLR 2023
necessity of maintaining the flexibility of sparsity distributions and thus keeps all weights updatable, including pruned ones, during finetuning. resulting in a discrepancy against the hard-pruned weights during deployment. The mismatched finetuiig//deployment process could demolish the learned speech representations especially under structured sparsity; (2) the limited learnability of sparsity distributions in the PARP pipeline due the intrinsically low learning rates during finetuning. leaving the space of sparsity masks significantly under-explored. We thus propose a new ASP pruning pipeline dubbed SAFARI (i.e., sparsify, finetune, and adjust sparsity) strictly zero-out the pruned neurons during finetuning for minimizing the finetuiig/ddeployment discrepancy, followed by sparsity adjustment step to adaptively evolve the sparsity masks and thus ensure the learnability of sparsity distributions. We summarize our contributions as follows:
We propose a data-model co-compression framework, dubbed S S0-DAMON, which for the first time chases structured sparsity both input audio tokens and model structures of speech SSL models to empower real-time on-device ASR under a low-resource setting;
We develop a semi-supervised method for training a lightweight module dubbed SALAD to distinguish SATs/NAT's for the purpose of structurally removing redundant audio tokens;
We identify the underlying causes for the failures of the SOTA ASR pruning method and then develop an ASR pruning pipeline dubbed SAFARI to enable high structured sparsity in speech SSL models while maximally maintaining the ASR accuracy;
Experiments show that as compared to the SOTA ASR pruning method PARP, our S DAMON can (1) achieve a 1.96x speed-up on Pixel mobile phone with an absolute 2.49% WER reduction, and (2) win absolute 10.96% lower WER for reducing >64% FLOPs of wav2vec2-bast as shown in Fig. I (b), indicating that our method has taken an important step towards bridging speech SSL models and real-time speech recognition.
2 RELATED WORK
Automatic speech recognition. Early ASR systems (Bridle, 1990; Sha & Saul, 2006; Tang, 2009; Jaitly et al., 2012; Mao et al., 2019; Adams & Beling, 2019) mainly build on top of the combinations of hidden Markov models with Gaussian mixture models or DNNs, and often integrate multiple modules, e.g., an acoustic model, language model, and a lexicon model, that are separately trained. Driven by recent advances in DNN structures, diverse end-to-end ASR systems have been proposed, which process raw audio inputs in an end-to-ent manner, including CTC (Graves et al., 2006)-
based models (Hannun al., 2014; Amodei et al., 2016: Graves& Jaitly, 2014; Miao al., 2015; Eyben et al., 2009), recurrent neural (RN)oork Rao et al., 2017: Dong et al., 201 (b). and sequence-to--sequence models (Chorowski et al., 2015; Bahdanau et al., 2016; Chan et al., 2016; Zhang et al., 2017: Chiu et al., 2018; Prabhavalkar et al., 2018). Specifically, from the model structure perspective, transformer- based models (Zhang et al., 2020; Gulati et al., 2020; Dong et al., 201 8a; Wang et al., 2020) have been widely adopted thanks to their superior expressiveness and capabilities for modeling long-range dependencies.
(Graves, 2012; Graves et al., 2013;
Self-supervised learning for speech representation. Due to the high cost and/or impracticality of collecting large-scale speech data, the ceetrain-then-fieette
paradigm has become increasingly popular for empowering low-resource ASR. To learn rich speech representation via SSL, early works design generative models for inferring the latent variables of speech units (Hsu et al., 2017 7a; van den Oord et al., 2017: Hsu et al., 2017b; Ebbers et al., 2017: Khurana et al., 2020). Recently, prediction- based SSL methods have gained more attention, where the models are trained to reconstruct the contents of unseen frames (Chung et al., 2019; Liu al., 2020: Chi et al., 2020; Ling & Liu, 2020; Baevski et al., 2022) or contrast the features of masked frames with those of randomly sampled ones (Oord et al.. 2018; Baevski et al., 2020: Conneau et al., 2020: Baevski et al., 2019a; Hsu et al., 2021). In parallel, some works combine both predictive and contrastive objectives (Baevski et al., 2019b;a) or integrate contrastive learning and masked language modeling (Chung et al., 2021; Bapna et al., 2022). We refer the readers to the survey (Liu et al., 2022) for more details. However, SOTA speech SSL models (Baevski et al., 2020: Hsu et al., 2021; Baevski et al., 2022; Conneau et al., 2020; Chiu al., 2022) often adopt large transformers for ensuring effective representation learning during SSL and require prohibitive model sizes, making it difficult to achieve real-time speech recognition on mobile devices.
ASR pruning. To compress large-scale ASR models while maintaining their generalizable rep- resentation, ASR pruning has gained growing attention. Early works prune either the decoding
3

---

Under review as a conference paper at ICLR 2023
Salient Audio Token Detector
Transformer
Sparsify, Finetune, Adjust Sparsity
w W.....
Semi- Sumii-Spervvieed Training of SALAD
SAFARI (ours)
0 Oparsity
@ Finetune
Sparsity
Adjust
Takena
FrN
MLP JHunywin Projet
C
Pscudo Labela
Altettin
Teacher ASR Model
Criterla + Mayshan
JEAFARO preceed baw
1+1 ...
a Prune
S Addjust
Ra-grune
Dedection Results
[
SALAD
PARP
Criterle ? Migtbuck
spanity discoppecy
Trainable weights
Pruninn candidates
Audio Tokens
Spprsity mask
Previous mask
2 Grow set
(a) Chase structured sparsity in input data
Speech SSL models (c) Chase structured sparsity in model structures
Figure 2: An overview of our proposed S S6-DAMON, integrating two enabling components called (a) SALAD and (b) SAFARI for chasing structured sparsity in input data and model structures, respectively (zoom-in for better view).
search space (Abdou & Scordilis, 2004; Pylkkönen, 2005; Xu et al., 2018; Zhang et al., 2021) the
HMM state space (Van Hamme & Van Aelten, 1996). Recent works have shifted its focus to pruning end-to-end ASR models (Venkatesh et al., 2021; Shi et al., 2021; Li et al., 2021b; Braun & Liu, 2019; Gao et al., 2020; Shangguan et al., 2019; Wu et al., 2021; Ding et al., 2021). Recently, (Lai et al., 2021; Prasad al., 2022: Zhao et al., 2021) prune speech SSL models towards more efficient low- resource ASR, however, they all adopt unstructured pruning which barely favors hardware efficiency in commercial devices. (Lee et al., 2022; Chang et al., 2022) distill the knowledge of pretrained speech SSL models to lightweight student models, but require human expertise to manually design the student model, which can be impractical without utilizing the properties of speech signals, causing inferior ASR accuracy. In contrast, S -SMDAMO learns to automatically and structurally prunes the redundancy of speech SSL models, achieving triple-win in data, model, and labor efficiency.
THE PROPOSED S D'DDAMON FRAMEWORK
3.1 S6-DAMON: FRAMEWORK OVERVIEW
Rationale. Current commercial mobile devices can only benefit from structured sparsity, whereas most useful features of speech SSL models are learned during the SSL stage, which could be easily demolished when enforcing structured sparsity, especially when being finetuned under low-resource setting. To this end, our -DMMON tackles this challenge based on twofold rational: (1) instead of compressing only one dimension, $ -DDDAMON exploits the redundancy in both the input audio tokens and model structure; and (2) to preserve the monotonic aligmment between the input audio and output transcriptions,.* -DAMON identifies and then tackles the key bottlenecks for reducing the complexity of both the input and model while maintaining the fidelity of speech representation.
Overview. As shown in Fig. 2, S--DAMON performs data-model co-compression via (1) SALAD module which detects and skips redundant tokens, i.e., NATs as introduced in Sec. 1, and (2) SAFARI pipeline which pursues structured model sparsity. For compressing given speech SSL model, S -DAMON features a three-stage pipeline: finetune the speech SSL model that serves as a teacher model on the low-resource transcribed speech; @ train SALAD in a semi-supervised manner based on the annotations of the teacher model (see Sec. 3.2); and @ perform a joint optimization of input data sparsity and model sparsity on top of speech SSL models based on the SAFARI pipeline with a portion of detected NAT' removed (see Sec. 3.3 and Sec. 3.4).
3.2 DAMON: CHASE STRUCTURED SPARSITY IN INPUT DATA
Here we introduce the deign and training of our proposed SALAD module, which is to detect and skip redundant audio tokens, i.e., NAT's Note that although (Kim et al., 2021; Wang et al., 2021) have attempted to prune transformers' tokens in a layer-wise manner in the NLP domain, they do not consider the intrinsic properties of human speech and such a layer-wise dynamic skipping workload can barely favor hardware efficiency in commercial devices. In contrast, SALAD explicitly exploits the redundancy in repeated/blank regions of human utterances, favoring efficiency in commercial devices, while maintaining the monotonic alignment between input speech and output transcriptions, unique property in the speech domain, by enforcing high recall on SAT' as elaborated below.
4

---

Under review as a conference paper at ICLR 2023
SALAD's input and structure. SOTA speech SSL models (Baevski et al., 2020; Hsu al., 2021; Baevski et al., 2022) sample and convert raw audio frames into audio tokens via convolutional feature extractors, which are then processed by transformer backbone to generate corresponding contextualized representation. As the convolutional feature extractors are often fixed after SSL pretraining to ensure effective audio feature extraction (Baevski al., 2020), SALAD is applied on the extracted audio tokens after the convolutional feature extractor, and classifies each audio token as SAT/NAT. This ensures that all transformer layers can benefit from reducing the same number of audio tokens, favoring efficiency in commercial devices. Specifically, SALAD consists of four lightweight convolutional layers and outputs a binary classification between SAT and NATs, Its structure is provided in the appendix, which only contains 0.35M parameters and accounts for <0.4% floating- poatingpointtoperations (FL.Ps) of the original transformer.
SALAD's semi-supervised learning pipeline. Considering the lack of token-wise ground truth, we train S A:LAD(;;ss) in semi-supervised manner, i.e., using a finetuned ASR model to provide pscudo labels for each token on untranscribed speech (see Fig. 2 (a)). Specifically, given a speech SSL model, we first finetune it on the available transcribed speech (e.g., LibriSpeech- to create a teacher model (- (r), which is then used to annotate larger amount of untranscribed speech (e.g., LibriSpeech-100 1Ohh to acquire the pseudo character/phonent labels for each audio token. Corre- spondingly, the pseudo binary labels of SATs/NAT's can be derived for each audio token based on whether it repeats the previous token or is a blank token, serving as the training signals for SALAD.
Enforce high recall on SATs. One issue is that the consequence of classifying SAT NAT is severe, under which SATs are more likely to be mistakenly skipped in inference, harming the monotonic input/output alignment. is thus highly desired maximize the coverage of SATs (i.e., high recall on SATs) during training SALAD. We exert larger penalty when a SAT is misclassified as a NAT to simply achieve this. The training process of SALAD can be formulated as:
arg max AS
LSSLLADD) Bin(M)rr
where L is a binary cross entropy loss, 24 is the i-th audio token extracted by the convolutional feature extractor, Bin denotes transformation from pseudo character/phonemee labels to binary labels of SATs/NATs, and a; is a penalty coefficient for enforcing high recalls on SATs.
Speech SSL model finetuning with SALAD. Since skipping audio tokens would change the speed and rhythm of an input speech, a finetuning process is required to fill in the domain gap. Specifically, we finetune the target speech SSL model integrated with SALAD, where certain ratio of NATs detected by SALAD is removed before being fed into the transformer. Intuitively, although NATs are less likely to impact the monotonic input/output alignment as compared SATs, removing NATs still results in domain gaps in terms of speech speed and rhythm between pretraining and finetuning.
Implementation. To balance ASR accuracy and efficiency, we set skip ratio sr for NATs in all the input audio when finetuning with SALAD, i.e., for the NATs detected by SALAD in an input audio clip, we remove the top sr sorted in terms of confidence score predicted by SALAD, and the remaining NATs and all detected SAT's are then fed into the transformer. Since different audio clips contain different percentages of NATs, which can cause different audio token lengths for samples in a batch, we pad each sample to the largest token length of each batch during finetuning.
3.3 SD-DAMON: CHASE STRUCTURED SPARSITY IN MODEL STRUCTURES
Rethink the SOTA ASR pruning method. The key spirit of PARP (Lai et al., 2021) is to adjust the sparsity mask so that the pruned weights are learnable via gradients and thus the sparsity distributions can be updated during finetuning, which is crucial for pruning speech SSL. models as the weight magnitudes inherited from SSL pretraining may not accurately indicate the importance of neurons for downstream tasks. The importance of such learnable sparsity distributions is validated in PARP with unstructured pruning against one-shov/iterative magnitude pruning (OMP/MP) (Lai et al., 2021).
Identified issues of the SOTA ASR pruning method. Directly extending PARP to structured pruning setting will cause a failure. We extend PARP's setting to structured sparsity, i.c., remove all connections from the pruned input neurons, for pruning wav2vec2-berl 20% sparsity ratio and vary the pruming intervals in terms of iterations between prune/re-prume prume. From Tab. 1, we can that (1) the original pruning interval adopted by PARP (i.e., 50 iterations) leads to
LibriSpeech- 1h under
5

---

Under review as a conference paper at ICLR 2023
an absolute 23.93% WER increase over standard way2vec2-base (18.96% WER); (2) setting a small pruming interval could lead to reduced WER over OMP; and (3) larger pruning intervals consistently cause more notable performance degradation, where the softly pruned weights diverge more from zero as they can be updated in PARP's adjustment step.
Analysis. This set of experiments indicates that although enhancing the sparsity masks' learn- ability in PARP beneficial, the resulting spar- sity discrepancy between finetuning (i.e., not exactly zero) and final deployment (i.e., hard pruming) can demolish the delicated speech rep- resentation inherited from SSL pretraining, es- recially under structured sparsity. Additionally, the flexibility of adjusting the sparsity masks in PARP still limited, e-g., only <3% elements in the sparsity masks is updated throughout the finetuning process, which aligns with PARP's observed >99% IOU between the initial/final subnetworks. This is caused by the intrinsically low learning rates during finetuning. thus updating the sparsity distributions via gradients in PARP leaves the potential of sparsity distributions largely under- exper-explored.
Table 1: Apply OMP and PARP with different pruning intervals for compressing wav2vec2-bass on LibriSpeech- Ih under 20% sparsity.
Method
Pruming Interval
10
50
Leurn -ability
No Dis crepancy
PARP
26.90
27.30
31.18 32.77
42,89
29.01
The SAFARI pipeline. The above analysis indicates that the key to pursue structured sparsity in speech SSL models is to ensure the learnability of sparsity distributions while minimizing the sparsity discrepancy between finetuning and deployment. Thus, we propose new pruning pipeline SAFARI: 1 sparsify: sparsifies given speech SSL model the target sparsity ratio sp based on the weight magnitudes; & finetune: finetune the model weights with the sparsity mask applied, i.c., the pruned weights are zero-outed without receiving gradients avoid the sparsity discrepancy; and @ adjust sparsity: the sparsity mask is adaptively adjusted for boosting the learnability of sparsity distributions. Steps and @ are iterated towards convergence. SAFARI can be viewed as an intermediate choice between OMP and PARP, marrying the former's 's stability and the latter's s learnability.
Implementation of SAFARI. There can be different ways to implement the above spirit, i.c., SAFARI. Inspired by (Evci et al., 2020), we adopt gradient magnitudes as criterion to adjust the sparsity masks in prune-and and-grow manner (see Fig. 2 (c)). Specifically, in each sparsity adjustment step, for set of neurons |N and set of pruned neurons pl ( [|P')/1 sp) in the 1-th layer, SAFARI ① selects ar neurons from N' \ pl as the pruning candidate set C¹ (C*I/IN4 = ar) based on a pruning criterion, where is predefined adjustment ratio, and @ chooses ar neurons from the joint set of pruning candidates and pruned neurons P'UC (= P + C²) to form a grow set G based on grow criterion, which allowed to be updated by the gradients of the next finetuning step. Therefore, the new sparsity mask applied in the next finetuning step is built by p + C' G², which has a constant sparsity ratio sp. More specifically, neuron, i.c., ||W lle, for the i-th neuron, as the pruning criterion (i.c., prune the smallest ones), and the corresponding gradients|
adopt the f--oomm of the weight vectors from
at W.
|le, as the grow criterion (i.e., grow the largest ones).
3.4 S JOINT DATA-MODEL CO-COMPRESSION
To perform joint optimization of input data sparsity and model sparsity, we integrate the target speech SSL model with SALAD and finetune it via the SAFARI pipeline with a portion of detected NATs removed. To push forward the achievable accuracy accuracy-efficienyy trade-off, -DDDAOO can optionally enable remi-supervised distillation mechanism to boost the ASR accuracy, especially under large compression ratio. Specifically, we distill the knowledge of the teacher model mentioned in Sec. 3.2 to the compressed model in a layer-wirg manner during finetuning on top of mixed dataset composed of both transcribed speech and untranscribed speech, where the pseudo labels on untranscribed speech are annotated by the teacher model. Note that the teacher model is only finetuned on the limited transcribed speech. The semi supervisd distillation process can be formulated as:
arg max
Σ Z ZEDTUDU 1=1
(b), + Σ CCTC(bb())
(2)
0
DT
where Df and DU are the transcribe/huntrasccibedd speech, respectively, MSE and CTC are the loss functions, 1/ = (S 0s); is the hidden representation for the remained tokens in the 1-th layer of the compressed model M, h/ (z) S o (x; 0r)) is the corresponding hidden representation of the teacher model and S is a selection operator for only calculating the MSE loss on the remained tokens determined by SALAD.

---

Under review as a conference paper at ICLR 2023
D Dy=Libri-1h Du=None
D Dy=Libri-1h Dy=Libri-10h
Dy=Libri- B Dy=
48
DAMON (Ours)
ST-DAMON
(Dund
1977
PARP
+0
GMP
Origina
50
Drigina
DMP
WER (%6
26
WER
10
22
30
60
20
30 50
70
00
FLOPs Savings (96
FLOPs Savings (%)
FLOPs Savings (%)
(a)
(b)
(c)
Figure 3: Benchmark our S and SAFARI with SOTA ASR pruning methods PARP and
OMP on way2vec2-base with transcribed LibriSpeech- Ih and different untranscribed resources. "w/ sd" denotes applying the semi-supervised distillation for finetuning the original model, which could notably reduce the WER when the untranscribed LibriSpeech- 100h is available.
4
EXPERIMENTAL RESULTS
4.1 EXPERIMENT SETUP
Models and datasets. We evaluate our SR-DAMON on four SOTA speech SSL models, including way2vec2-base/large (Baevski et al., 2020), data2vec (Baevski et al., 2022), and hubert (Hsu et al., 2021) pretrained LibriSpeech-960h (Panayotov al., 2015). We evaluate the compression effectiveness under different resource settings, including ASR on LibriSpeech- 1h/ 1a/10h/100h following the split in (Baevski et al., 2020), and phoneme recognition (PR) on different spoken languages from Common Voice (Ardila et al., 2019) with 1h transcribed speech per language, following the split in (Conneau et al., 2020). In addition to ASR, we also consider six speech processing tasks from SUPERB (Yang al., 2021). For results on LibriSpeech, we report the WER on test-clean by default.
Finetuning settings: We implement S -DAMON on top of fairseq (Ott et al., 2019) and we follow the default finetuning settings for each task as elaborated in Appendix. B.
For SALAD training, we adopt the same training schedule as finetuning the speech SSL model weights and the o; in Eq. 1 is 10 for penalizing mistakes SATs otherwise 1. For SAFARI, we adjust the sparsity every 50 iterations and fix the sparsity mask after 10k iterations for all experiments. By default, the adjustment ratio ar is set the same as the target sparsity sp if not specifically stated, which is justified by the ablation study in Appendix. A.2. We adopt NAT skip ratio of 0.4/0.6/0.8 (i.e., of NATs removed) and note that the final ratio of skipped tokens to the total tokens would depend on the amount of NATs detected in the given speech.
-SDDAMON settings:
4.2 BENCHMARK WITH SOTA ASR PRUNING METHODS
Considering feed forward networks (FFNs) more sensitive to structured pruning than self-attention (SA) as observed in Appendix. A. thus for both our method and baselines, given a target sparsity sp, by default set their sparsity satisfy (SpSa + spFFN)/2 which achieves better ASR accurary with comparable FLOPs as compared to uniformly setting a sparsity of sp. For PARP, we adopt its best-performed setting for structured pruning, i.e., update the sparsity every one iteration for maximally avoiding the sparsity discrepancy issue.
sp and Spsa spFFN 0.2,
Benchmark on English ASR under different low-resource settings. We benchmark our s DAMON with OMP and PARP under different combinations of transcribed data Dx and untranscribed data D& for finetuning with -umi-vupee distillation as described in Eq. 2. We fix the Dt as LibriSpeech- Ih and vary the resources in Dy- We adopt NAT skip ratio of 0.4~00.8 and a sparsity ratio of 0.2-0.4 for our DAMON and a sparsity ratio of 0.1 0~150..5 for other ASR pruning baselines.
Observation and analysis. As shown in Fig. 3, we can observe that (1) our -DDAMON consistently outperforms PARP and OMP by notable margin, e.g., an absolute >7% WER reduction as compared to PARP for achieving >64% FLOPs savings wav2vec2-bass with LibriSpeech- h Dr/Dv; (2) our 6 DAMON shows decent scalability under more stringent low-resource settings where PARP/OMP fail to achieve acceptable recognition effectiveness,, e.g., absolute up-to-34% lower WER over PARP when only LibriSpeech- 1h is available; (3) our method can achieve a comparable WER (+0.5%) with 40.8% reduction in FLOPPs compared to the original wav2vec2-bass with
7

---

Under review as a conference paper at ICLR 2023
only 1h transcribed speech plus 100h untranseribed speech available; (4) enabling both SALAD and SAFARI can consistently win a better WER-FLOPs trade-off over SAFARI, especially under more stringent low-resource settings, e.g., DD-DAMON achieves absolute 4.51% lower WER over SAFARI for reducing >64% FLOPs on LibriSpeech- lh. This indicates that input data sparsity is critical dimension for structurally trimming down the complexity of speech SSL models.
Benchmark under more resources. As shown in Fig. (b), given more downstream resources, e.g., the transcribed LibriSpeech-1000 our -S-DAMON (1) still outperforms PARP and OMP, e.g., an absolute 10.96% lower WER for reducing >64% FLOPs on wav2vec2-base as compared to PARP, and (2) reduces 46% FLOPs of the original wav2vec2- with comparable WER (+0.57%).
Benchmark under unstructured sparsity. We fur- ther validate the scalability of our SAFARI to unstruc- tured sparsity via benchmarking with the reported results of PARP, OMP, IMP, and MPI (i.e., magnitude pruning at pretrained initializations) in (Lai et al., 2021) on LibriSpeech-Il Ih without any distillation. As shown in Fig. 4, we can observe that our SAFARI can still outperform all baseline methods, especially under large sparsity ratios, e.g., an absolute 8.61% lower WER under 80% sparsity over PARP. This indi- cates that (1) the sparsity discrepancy issue still exists in unstructured pruning under larger sparsity ratio, and (2) our SAFARI pipeline consistently shows its superiority as an ASR pruning paradigm over PARP under both pruuctureduusstructue sparsity patterns.
100
MPI
90
OMP
80
IMP
WER (%)
70
SAFARI (Ours)
60
10
20
30
40
50
60
70
80
Sparsity (%)
Figure 4: Benchmark our SAFARI with
PARP, OMP, IMP, and MPI for unstructured pruning on www2vec2-base/Libriiit base/LibriSpeehh lh.
Benchmark with distillation- based models. We benchmark with the reported ASR results Distil- HuBERT (Chang et al., 2022) and FitHuBERT (Lee al., 2022) for compressing hubert and wav2vec2- base on LibriSpeech- 100h. As shown in Tab. 2, S DAMON achieves an absolute 7.04% lower WER with 8.7% fewer parameters
Table 2: Benchmark S -D-DOONN with two distillation- efficient ASR models.
Method
Model
Params (M)
WER (%)
Disti
hubert
23.49
13.37
FitHuBERT
wav2vec2 bubert
22.49
14.77
22.49
12.66
compared to the strongest baseline FitHuBERT, indicating that given speech SSL model, trimming down its complexity in a top-down manner may achieve better compression effectiveness than manually designing an efficient model from scratch without exploiting the intrinsic properties of human speech.
-AMMAMN
wav2vec2
20.53
7.73
(Ours)
bubert
20.53
7.94
Benchmark on more speech SSL models. We further extend our S -D-MOMOO to more models, i.e., data2vec (Baevski et al., 2022) and wav 2vec2-laaee on LibriSpeech- 1h (i.e., no Dr). As shown in Fig. 5. we can observe that (1) our method shows consistent effectiveness across models and starting from better speech SSL models like data2vec could lead to better WER- FL.OPs trade-offs, and (2) according to the comparison between wav2vec2-base/large structurally compressing a larger speech SSL model may not result in better WER-FLOP trade-offs than compressing a smaller one as aggressively compressing pretrained model could demolish the speech representation learned by the delicate SSL pretraining process. Therefore, a smaller model plus mild structured sparsity is favored.
St DAMON en wy-base on wi-large
S Sf-DAMMON on dataZver
WER (%)
35
30 GFLOPs
Figure 5: Apply S SS-DAMON on way2vec2-basselarge and data2vec.
4.3 EXTENSION TO OTHER SPEECH PROCESSING TASKS
Benchmark on SUPERB. Although efficient ASR our main focus, we also evaluate our method on more speech processing tasks from SUPERB. In particular, we transfer the compressed models with LibriSpeech- 1h/100h Dr of our S .D-MO and PARP to perform six speech processing tasks on SUPERB (Yang et al., 2021). As shown in Tab. 3, we observe that (1) our method wins four out of six tasks over the original wav2vec2-basse with >=55.07%/44.3 4.% FI.OPs/parameter reductions, and (2) our method consistently outperforms PARP across all the tasks. This indicates that our 56 -DAMON can potentially serve general compression technique for speech processing.
8

---

Under review as a conference paper at ICLR 2023
Table Benchmark our S9-DAMON with PARP on six speech processing tasks from SUPERB.
FLOPs Saving (%)
Params
ER
QbE
IC
SF
ASV
Method
Saving (%)
(Aec f)
(Ace
(MaxTWV
(Ace f)
(F1 f)
(EER
Original
0.00
0.00
0.626
0.962
0.053
0.966
0.874
0.061
PARP
53.52
57.14
0.622
0.957
0.075
0.967
0.851
0.065
DDDMMONN
57.07
44.39
0.654
0.964
0.118
0.982
0.867
0.063
(Ours)
69.98
57.14
0.641
0.957
0.122
0.983
0.855
0.063
Table 4: Measure the latency of delivered models by our method and PARP on a Google Pixel 3 mobile phone. "FE" denotes the feature extractor and "Trans." denotes the transformer backbone. All models are finetuned on transcribed LibriSpeech- 100h and "sr"/"sp" are the adopted skip ratio/sparsity.
Params (M)
WER
Lat. (ms)
Lat. (ms)
Speed-up
Speed-up Overall
Method
1 sp
(%)
Conv FE
Trans.
on Trans.
RTF
Original
-/-
94.74
5.50
2270.1
6678.5
1.00x
1.00x
0.895
-/0.3
56.77
7.82
2270.1
4809.4
1.39x
1.26x
0.708
PARP
-/0.4
45,80
11.69
2270.1
3756.6
1.77x
1.48x
0.603
0,6/0.2
69.01
6.07
2270.1
3834.5
1.74x
1.47
0.610
0.6/0.3
56.77
6.53
2270.1
3318.7
2.01x
1.60x
0.559
SD-DAMON
0.8/0.3
56.77
6.98
2270.1
3061.1
2.18%
1.68x
0.533
(Ours)
0.8/0.4
45.80
7.98
2270,1
2377.9
2
1.93
0.465
0.8/0.7
20.53
9.20
2270.1
819.1
8.15x
2.90x
0,309
4.4 REAL-DEVICE MEASUREMENT OF -9-DAMON
To validate the real-device efficiency of SS-DAMON' delivered models, we measure their latency on a Google Pixe13 mobile phone for processing a 10s audio segment with a 16k sampling rate. We also report a real-time- factor (RTF) defined as the inference time divided by utterance duration (Gondi, 2022). As shown in Tab. 4, our S -SDDAMON can achieve (1) a 1.96x speed-up over PARP with an absolute 2.49% lower WER, and (2) 1.60 speed-up over the original wav2vec2- with comparable WER (+1.03%) or 2.90x speed-up up while maintaining the absolute WER within 10%. This indicates that our method can outperform SOTA ASR pruning methods in real-device efficiency and significantly bridge the gap between speech SSL models and real-time speech recognition.
4.5 ABLATION STUDIES
Benchmark with random, uniform, and other adaptive token skipping methods. To validate the skipping strategies made by SALAD, we benchmark with three token skipping methods: (1) random skip, (2) uniform skip with similar effect as reducing the sampling rate, and (3) layer-wise adaptive skip (Wang et al., 2021) based attention scores, which is originally designed for NLP. We finetune way2vec2-base with each of the skipping methods on LibriSpeech-In Ih. We control their skip ratios to ensure a comparable FLOPs saving. As shown in Tab. 5, our SALAD consistently wins the lowest WER under comparable FLOPs and the adaptive skip method can hardly surpass the simple uniform skip strategy, indicating that without considering the intrinsic properties of human speech, the monotonic alignment between input speech and output transcriptions can be casily demolished.
Table Benchmark with differ- ent token skipping strategies.
FLOPs Saving (%)
WER
Method
(%) )
24.3
19.17
SALAD
32.0
20.21
Uniform Skip
20.0
19.89
30.0
24.63
Random Skip
20.0
43.98
30.0
79.87
Adaptive
15.0
25.658
Skip
22.5
36.193
CONCLUSION
Both the lack of large-scale transcribed speech data for low-resource spoken languages and the prohibitive model complexity hinder ubiquitous DNW-po ASR systems on mobile platforms. This work proposes 5 S0-DAMON to tackle both challenges via effectively pruning speech SSL models to enable real-time on-device ASR. Specifically, S -D-DMMNN integrates SALAD and SAFARI to pursue structured sparsity in both input data and model structures, respectively, where the former exploits and intrinsic properties of human speech and the latter reduces the sparsity discrepancy between finetuiig/ddeployment and enhances the learnability of sparsity distributions. S SD-DAMON has enabled the deployment of speech SSL models on mobile devices based on our extensive experiments and can shed light future innovations efficiency-orientedd speech SSL paradigms.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sherif Abdou and Michael S Scordilis. Beam search pruning in speech recognition using posterior probability- confidence measure. Speech Communication, 42(3-4)):09 -48499999 2004.
Stephen Adams and Peter A Beling. A survey of feature selection methods for gaussian mixture models and hidden markov models. Artificial Intelligence Review, 52(3):179-1179, 2019.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference machine learning, pp. 173-182 PMLR, 2016.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. arXiv preprint 1rXiv:1912.06677 2019.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. Effectiveness of self-supervised pre- training for speech recognition. arXiv preprint arXiv:1911.03912 2019a.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453, 2019b.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 12:112499-2240 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to- end attention-ban based large vocabulary speech recognition. 2016 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 4945-4949, IEEE, 2016.
Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv:2202.01774 2022.
Stefan Braun and Shih-Chii Liu. Parameter uncertainty for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5636-5640. IEEE, 2019.
John S Bridle. Alpha-nets: A recurrent 'neural' network architecture with a hidden markov model interpretation. Speech Communication, 911:83--9, 1990.
netrralnntwoork
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4960 4964. IEEE, 2016.
Heng-Jui Chang. Shu-wer Yang, and Hung-yi Lee. Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7087-7091. IEEE, 2022.
Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Shang- Wen Li, and Hung-yi Lee, Audio albert: A lite bert for self-supervised learning of audio representation. arXiv preprint mrXiv:2005.08555. 2020.
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J Weiss, Kanishka Rao, Ekaterina Gonina, et al. State-of-the-ar speech recognition with sequence-to-sequene models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4774 IEFF, 2018
10

---

Under review as a conference paper at ICLR 2023
Chung-Cheen Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self sele-vuper learning with uundom-projectin quantizer for speech recognition. arXiv preprint arXiv:2202.01855. 2022.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. Advances in neural information processing systems, 28, 2015.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. arXiv preprint /90.033900 2019.
Yu-An Chung, Yu Zhang, Wei Han, Shung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for supervupe speech pre-training. arXiv preprint 2rXiv:2108.06222 2021.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross- -inguul representation learning for speech recognition. arXiv preprint arXiv:2006. 13979, 2020.
Shaojin Ding, Tianlong Chen, and Zhangyang Wang. Audio lottery: Speech recognition made ultra-lightweight, noise-robust, and transferable. In International Conference on Learning Repre- sentations, 2021.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer:: a no-recurrence sequence-to-seqqeeec model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888. IEEE, 2018a
Linhao Dong, Shiyu Zhou, Wei Chen, and Bo Xu. Extending recurrent neural aligner for streaming 2018b.
end-to-end speech recognition in mandarin. arXiv preprint
Janek Ebbers, Jahn Heymann, Lukas Drude, Thomas Glarner, Reinhold Raj. Hidden markov model variational autoencoder for acoustic unit discovery. In INTERSPEECH, 2017.
and Bhiksha
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Florian Eyben, Martin Wöllmer, Björn Schuller, and Alex Graves. From speech to letters-using novel neural network architecture for grapheme based asr. In 2009 IEEE Workshop on Automatic Speech Recognition & Understanding, pp. 376-380.10 IEEE, 2009.
Dawei Gao, Xiaoxi He, Zimu Zhou, Yongxin Tong, Ke Xu, and Lothar Thiele. Rethinking pruning for accelerating deep inference at the edge. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 164-164, 2020.
Santosh Gondi. Wav2vec2. 0 on the edge: Performance evaluation. arXiv preprint arXiv:2202.05999, 2022.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint 2012.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, pp. 1772-1772. PMLR, 2014.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Alex Graves, Ahdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649) leee, 2013.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: canvolution-augmentte transformer for speech recognition. arXiv preprint arXiv:2005.08/000 2020.

---

Under review as a conference paper at ICLR 2023
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos. Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.55577 2014.
Wei-Ning Hsu, Yu Zhang, and James Glass. Learning latent representations for speech generation and transformation. In INTERSPEECH, 2017a.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In NeurIPS, 2017b.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Navdeep Jaitly, Patrick Nguyen, Andrew Senior, and Vincent Vanhoucke. Application of pretrained deep neural networks to large vocabulary speech recognition. 2012.
Sameer Khurana, Antoine Laurent, Wei-Ning Hsu, Jan Chorowski, Adrian Lancucki, Ricard Marxer, and James Glass. A convolutional deep markov model for unsupervised speech representation learning. arXiv preprint 20XXi:20006025477, 2020.
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. arXiv preprint arXiv:2107.00910 2021.
Cheng-I Jeff Lai, Yang Zhang, Alexander H Liu, Shiyu Chang. Yi-Lun Liao. Yung-Sune Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, and Jim Glass, Parp: Prune, adjust and re-prune for self-supervised speech recognition. Advances in Neural Information Processing Systems, 34, 2021.
Yeonghyeon Lee, Kangwook Jang, Jahyun Goo, Youngmoon Jung, and Hoirin Kim. Fithubert: Going thinner and deeper for knowledge distillation of speech self-suup learning. arXiv preprint arXiv:2207.0055: 2022.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, and Yingyan Lin. Hw-nas-benchs Hardware-aware neural architecture search benchmark. arXiv preprint arXiv:2103. 10584, 2021a.
Shenggiang Li, Menglong Xu, and Xiao-Lei Zhang. Efficient conformer- speech recognition with linear attention. arXiv preprint 1rXiv:2/04.0680 2021b.
Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXivv2012.06659, 2020.
Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP, 2020.
Shuo Liu, Adria Mallol-Ragolae Emilia Cabeeerr,,,,,ioo,, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bjoemn W Schuller. Audio self-supervised learning: A survey. arXiv preprint 0:Xi::2003.01205. 2022.
Shuiyang Mao, Dehua Tao, Guangyan Zhang, PC Ching, and Tan Lee. Revisiting hidden markov models for speech emotion recognition. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6715-6719. IEEE, 2019.
Meta-Al.
New and more. www.:a/www...........ooooooooooo 2022.
advances
in
speech
recognition to power ex- facebook. fttpoo// htt.o/ook.cco/boo////oooo
periences
Yajie Miao, Mohammad Gowayyed, and Florian Metze. Eesen: End-to-end speech recognition using deep rnn models and wast-based decoding. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 167-174. IEEE, 2015.
12

---

Under review as a conference paper at ICLR 2023
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Rohit Prabhavalkar, Tara Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Chees Chiu, and Anjuli Kannan. Minimum word error rate training for attention-base sequence-to-sequeee models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4839 4843. IEEE, 2018.
Lodagala VSV Prasad, Sreyan Ghosh, and S Umesh. Pada: Pruning assisted domain adaptation for self-super speech representations. arXiv preprint arXiv:2203. 2022.
Janne Pylkkönen. New pruning criteria for efficient decoding. In Ninth European Conference on Speech Communication and Technology, 2005.
Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-end speech recognition with mun-transducer. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 193-199. IEEE, 2017.
Fei Sha and Lawrence Saul. Large margin hidden markov models for automatic speech recognition. Advances in neural information processing systems, 19, 2006.
Yuan Shangguan, Jian Li, Qiao Liang, Raziel Alvarez, and Ian McGraw. Optimizing speech recognition for the edge. arXiv preprint arXiv: 1909.1248, 2019.
Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching- Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6783-6787 IEEE, 2021.
Xian Tang. Hybrid hidden markov model and artificial neural network for automatic speech recogni- tion. In 2009 Pacific-Asia Conference on Circuits, Communications and Systems, pp. 682-685. IEEE, 2009.
Aaron den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
Hugo Van Hamme and Filip Van Aelten. An adaptive-beam pruning technique for continuous speech recognition. In Proceeding of Fourth International Conference on Spoken Language Processing.
ICSLP'96, volume pp. 2083-2086 IEEE, 1996.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Ganesh Venkatesh, Alagappan Valliappan, Jay Mahadeokar, Yuan Shangguan, Christian Fuegen, Michael L. Seltzer, and Vikas Chandra. Memory-efficien speech recognition on smart devices. In ICASSP 2021-202 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8368-8377 8372. IEEE, 2021.
Vox.
Mark zuckerberg away.
wants to build a voice assistant that blows
alexa
https://www wwxp//wwwwwooooooooooooooooxotow wwt.:/cooeereeeeee 92897777/ mark-zuse wtwp::/www..oooooeeesssssaeeeeee
and
siri
2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804 07461, 2018.
13

---

Under review as a conference paper at ICLR 2023
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on Srigh-Performncc Computer Architecture (HPCA), pp. 97-110. IEEE, 2021.
Yongqiang Wang, AbdeIrahman Mohamed, Due Le. Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3574-6878. IEEE, 2020.
Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang. Dynamic sparsity neural networks for automatic speech recognition. ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6014-6018. IEEE, 2021.
Hainan Xu, Tongfei Chen, Dongji Gao, Yiming Wang, Ke Li, Nagendra Goel, Yishay Carmiel, Daniel Povey, and Sanjeev Khudanpur. A pruned rnnim lattice-rescornng algorithm for automatic speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp.
IEEE. 2018.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2/05.0105/ 2021.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and mn-t loss. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833. IEEE, 2020.
Yu Zhang. William Chan, and Navdeep Jaitly. Very deep convolutional networks for end-to-end speech recognition. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4845-4849. IEEE, 2017.
Yuekai Zhang, Sining Sun, and Long Ma. Tiny transducer: A highly-efficient speech recognition model on edge devices. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6024 6028. IEEE, 2021.
Yingzhu Zhao, Chongjia Ni, Cheung- Chi Leung, Shafiq Joty, Eng Siong Chng, and Bin Ma. A unified speaker adaptation approach for asr, arXiv preprint 1000045,, 2021.
14

---

Under review as a conference paper at ICLR 2023
A MORE ABLATION STUDIES OF -
A.1 MODULE-WIST SENSITIVITY TO STRUCTURED PRUNING
We apply SAFARI wav2vec2- base top of LibriSpeech- 1h and vary the sparsity in FFN and SA under a comparable FLOPs. As shown in Tab. 6, we consistently find that FFNs are more sensitive structured pruning especially under large sparsity, which may be because task-specific information is mostly learned by FFNs during finetuning thus their sufficient complexity is crucial. Therefore, by default their sparsity satisfy (sPSA + sppFN)/2 sp and spsA SPFEN 0.2 for given sparsity sp in Sec. 4.
Table 6: Benchmark the WERs with varied FFN/SA sparsity www.veee--base/liiriipppeee
FFN/SA sparsity
WER (%)
FFN/SA sparsity
WER
FFN/SA
WER
GFLOPs
GFLOPs
GFLOPs
(%)
sparsity
(%)
0.2/0.2
33,024
23.19
0.3/0.3
26.96
23.19
0.4/0.4
21.52
31.97
0.1/0.3
33.96
22.13
0.2/0.4
27.61
24.52
0.3/0.5
21.89
29.72
0.3/0.1
32.016
25.19
0.4/0.2
26.23
31.05
0.5/0.3
21.08
47.77
A.2 THE CHOICE OF ADJUSTMENT RATIOS
Table 7: Benchmark the WERs with varied ad- justment ratios ar under different sparsity sp.
We vary the adjustment ratio or under differ- ent sparsity sp on top of wav2vec2-base and LibriSpeech- lh. As shown in Tab. 7, we observe that the optimal ar varies for different sp and in general larger sparsity calls for higher learnabil- ity of sparsity distributions. Therefore, we set ar sp by default in Sec. 4.
ar sp
0.2
0.3
0.4
0.1
23.75
26.53
33.14
0.2
22.86
28.65
31.97
0.3
22.96
23.19
30.76
0.4
23.18
26.79
30.74
A.3 THE NECESSITY OF ENFORCING HIGH RECALL ON SATS
To validate this, train two SALADs w/ and
Table Validate the necessity of RAT.
w/o recall-aware training (RAT), which are next ap-
Setting
w/o RAT
w/ RAT
plied on wav2vec2-base with different NAT sr on LibriSpeech-Ih. As shown in Tab. 8, explicitly en- foreing high recall on SATs results in consistent lower WER especially under larger ST, validating the
Ace (%)
79.38
75.69
Recall (%)
64.38
89.08
NAT sr=0.4
19.56
18.42
necessity of maximally covering all SATs.
NAT sr=0.6
21.63
19.17
NAT sr=0.8
23.89
20.21
A.4 CROSS-LINGUAL TRANSFER OF SALAD
Considering the semi-supervised training scheme of SALAD requires large set of untranscribed speech, which may not be available for some spo- ken languages, evaluate whether the SALAD trained on English can be directly transferred to par-
Table 9: Evaluate the phoneme recognition rate (PER) when applying SALAD trained on English to other languages under different sr.
ST
Dutch
Spanish
Mandarin
detect SATs/NATs for other languages. ticular. we transfer SALAD trained on untran- scribed LibriSpeech- 100h to pursue the input spar- sity for finetuning wav2vec2-base. on Dutch, Span- ish, and Mandarin from Common Voice (Ardila et al., 2019). As shown in Tab. 9, we can see that SALAD trained on English can transfer well to other languages, e.g., achieve comparable
19.82
13.86
26.67
0.2
18.89
13.76
26.61
0.4
19.16
13.85
26.89
0.6
19.55
13.99
26.84
0.8
20.09
14.32
28.46
lower PER under 0.6 sr,, indicating that SALAD can extract general phonetic features that can be shared across spoken languages.
15

---

Under review as a conference paper at ICLR 2023
B MORE DETAILS ABOUT EXPERIMENT SETUP
Finetuning settings. We implement S on top of fairseq (Ott et al., 2019) and we follow the default finetuning settings for each task, i.e. the default configurations in fairseq for ASR/PR and those in SUPERB (Yang et al., 2021) for other speech processing tasks. In particular, all experiments on ASR/PR are trained for 22//15k/20k/80k steps on the ham//hh/OOh/100 splits using an Adam optimizer with an initial learning rate of 5e-5 plus a tri-stage schedule (Baevski et al., 2020). We do not freeze all the transformer layers for the first 10k steps (Baevski al., 2020), following LLai et al., 2021). In addition, considering the transformer backbone accounts for >90% parameters in the speech SSL. models, all the reported FL.OssPPrams savings and sparsity ratios are relative the transformer, following PARP (Lai et al., 2021).
Mcasurement settings. For the measurement on the Google Pixel 3 mobile phone, all Pytorch models are converted to ONNX and then compiled to the TFLite format, following (Li et al., 202 la). We separately compile (a) the convolutional feature extractor + SALAD, and (b) the transformer backbone, where the output of (a) is fed into (b) as its input. The latency on both (a) and (b) well as the overall speed-up are reported in Sec. 4.4.
16


Reviews:
[{"review_id": "knR1rgDcs3", "reviewer": "Reviewer_jv1R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applies two approaches for improving the speed of ASR models: learning to identify salient tokens and skip those that are identified not salient, and weight pruning. The results show some improvements on the SUPERB set and LibriSpeech-100h. The authors also benchmark the inference speed on Pixel 3 phone and demonstrate good speedup with the technique.\nTwo questions: \n(1) in section 4.5, when comparing different skipping methods, are the models also fine-tuned with those skipping methods? Or are they only applied during inference?\n(2) On the SUPERB set, why would the quality be better than the uncompressed model?\n\nA more recent related work on self-supervised learning for speech is missing:\nSelf-supervised learning with random-projection quantizer for speech recognition. ICML 2022", "summary_of_the_review": "The proposed approach shows good improvement compared with PARP, but the empirical studies are a bit limited. There should also be more comparisons with other algorithms. The technical novelty is generally limited.", "strength_and_weaknesses": "Strength: the results in the paper show significant improvement compared to the previous work.\nWeakness:\n(1) The comparisons with the previous approach are limited, making it less convincing.\n(2) The comparisons with other pruning algorithms are also lacking.\n(3) SALAD is only slightly better than a simple uniform skipping.\n(4) The proposed pruning algorithm is a minor extension of PARP, making its novelty limited.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f9E0d6GKNJ", "reviewer": "Reviewer_V9mU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of on-device ASR self-supervised learning. In this scenario, there is a large unlabeled dataset is available for unsupervised pre-training, and a much smaller supervised set for fine-tuning. Previous models such as HuBERT, data2vec use a large transformer model for the feature pre-training. Such a transformer is resource hungry for the scenario of running on an mobile phone or an embedded device.\n\nThis work proposes 3 different components. First, there is a method to prune the input tokens. The paper uses a teacher ASR model to train a small classifier which predicts which input token is salient. Examples of salient tokens are the first of the repeating letters, or the first of several silences.\n\nSecond proposed component is the method to sparcify the model. The paper builds upon the PARP and proposes an iterative procedure of sparcification and finetuning steps.\n\nFinally, the third proposed model combines both previous models to use them for semi-supervised distillation from a teacher model.\n\nThe paper reports the results on Librispeech and Superb corpora and then evaluates the latency on a real device.", "summary_of_the_review": "Given the previous assessment, now I recommend reject.\n\nBut this paper has a potential, and here are the ways to improve it:\n  - Align the experimental results to the prior literature: HuBERT, data2vec, wav2vec, PARP. If it is not possible, provide a disclaimer (for example, that the baseline was trained on a larger corpus and therefore had better performance).\n  - Tie the different components together.", "strength_and_weaknesses": "Strengths:\n  - The paper tackles a challenging problem.\n  - The proposed models are sound. The design makes sense and motivated well\n  - The paper provides evaluations on several datasets: Librispeech and Suberb\n  - The paper evaluates the latency on a real device\n\nWeaknesses:\n  - There is some lack of focus in this paper: there are three models introduced. Each one is complex enough to have a small paper. This leads to a very condensed writing and each model is not described well.\n  - The results are hard to compare to previous works. For example, the Fig. 3 models are trained in a different setup than wave2vec2 or HuBERT, therefore not comparable. The PARP paper's setup is also a bit different than presented here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BuUDPfKyuH", "reviewer": "Reviewer_JZjt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two methods for efficient ASR modeling, the token/frame skipping and structured pruning. The token skipping is achieved by applying a binary classifier to detect whether a token is salient. The structured pruning is an iterative process of sparsifying, fine-tuning and recovering, until reaching a certain sparsity level. Experimental results demonstrates the effectiveness in terms of FLOPs savings against WER increases, in comparison with other popular pruning methods.\n", "summary_of_the_review": "Overall, though the papers' core ideas are not novel, the combination and comprehensive experimental evaluation show that it is working well. One concern is that no source code nor trained model provided for reproduction.", "strength_and_weaknesses": "Pros:\n1. The paper is clearly written and easy to follow. The illustrative figures and examples are helpful to understand the ideas.\n2. Examples are conducted on different backbone models including wav2vec2 and data2vec, with different sizes.\n3. The ablation study compared with random removing frames or adaptive removing, are very useful to establish a sense of how the proposed method performs compared with baselines.\n4. The pruning performance is impressive compared with other methods. I guess it is due to the token skipping\n\nCons:\n1. The idea of both methods are incremental, though combining them and show they works is the value of this paper.\n2. In figure 3 (a), why the S6-DAMON does not have values below 40% FLOPs? Is it that the WER is worse than the others?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject