Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: null
review_content: I'd like to thank the authors for their extensive experimental results added based on the review feedback. The authors are encouraged to add these findings/results in the main paper and/or the appendix. I'll raise my rating for the paper from 5 -> 7. 
review_title: Update based on author feedback

final_score: rating: 7: Good paper, accept
review_content: summary: The goal of this contribution is to reduce the complexity of self-supervised speech representation learning systems. Following a description of the landscape of unstructured magnitude pruning methods (i.e. methods that sort weights by their magnitude and remove the lowest ones with no other consideration), authors propose PARP. PARP is based on the observation that "task-agnostic" masks (obtained on a pre-trained network) are very similar to ones that would be obtained after fine-tuning on a target task. Taking benefit from this observation, and unlike OMP that cascades finetune -> prune -> finetune, PARP simply perform prune -> finetune. Experiments on low resource ASR and cross-lingual training show a clear advantage of PARP over IMP, OMP and MPI within a language (less obvious for high-to-low resource transfer and cross-lingual training), but also a surprising gain in performance w.r.t. unpruned architectures. Moreover, authors show that PARP also works on another modality, by pruning BERT models for GLUE downstream tasks.
main_review: The paper is remarkably well written and structured. The context and baselines are well explained, and as a reviewer unfamiliar with the pruning literature, I found the paper easy to follow and the models clearly explained. The experiments are convincing both in their scope (within and across languages, speech and NLP) and results. In particular, seeing improved performance over unpruned model with up to 50% sparsity in some cases is impressive. As a non-expert I cannot guarantee the appropriateness of the baselines (maybe some more sophisticated methods are missing), however given the baselines in the paper, PARP clearly has an advantage both in terms of training cost and performance. Overall, even though the problem addressed by this work could be considered as relatively narrow, the effort put in the redaction and presentation of the method and results, the significant performance of the system against baseline, as well as the generality of the approach (showcased by the NLP experiments) make this paper both worth reading and promising for future applications. For these reasons, I recommend this paper for acceptance.

Remarks:
* Figure 1 mentions an exponential increase in compute for IMP as the sparsity level increases. This is understandable following section 2.3 that explains how IMP prunes 10% of the remaining weights at each iteration (rather than 10% from all weights). However Figure 1 shows a linear increase in number of fine-tuning runs.
* Section 2.2 is in past tense, while the other sections are in present tense.
* line 151: "Initailizations"
* Algorithm 1: since m is updated at every iteration, it would be easier to read by adding an index to it.
 
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a Prune-Adjust-Re-Prune (PARP) approach for finding a sparse subnetwork within a self-supervised speech model with competitive performance for the Automatic Speech Recognition (ASR) downstream task in mono-lingual and multi-lingual conditions. Despite the approach's simplicity, its downstream ASR performance (measured using WER without LM decoding) is better than other subnetwork discovery approaches (for almost all fine-tuning conditions) and the original unpruned network when 10mins of paired data is used. 

main_review: The proposed PARP approach alternates between two steps starting from the pre-trained network: a) zeroing out the smallest X % of the weights based on their magnitude. b) training the network for some update steps while allowing the zeroed-out weights in step (a) to grow again. The proposed approach is similar to previous work in NN pruning, but it is the first to be applied to self-supervised speech models. Experiments on multi-lingual ASR showed that initial subnetwork masks could be transferred between languages. Also, a quick experiment on NLP tasks showed similar behavior. The paper is well written with good experiments on the Wav2vec 2.0 pre-trained English and other lower-resource languages. 

There are few comments on the presented work: 
1) The paper offers an imbalanced presentation between Speech and NLP. Given the generality of the proposed approach and continuous reference to previously proposed work on other modalities, it makes sense to offer more experiments on the same tasks. Adding Table 4 raises the confidence in the efficacy of the approach a bit. More experiments will convince the reader further. 

2) Using WER with no LM is accepted for seq2seq models which learn an implicit LM from the data; however, this work uses only a CTC loss leading to considerable variance in the WER when no LM is used, in particular when few amounts of paired data are used, e.g., 10mins. The authors are encouraged to share results with n-gram decoding, at least for their English models.

3) The results on initial subnetwork generality to multiple tasks for the speech case need to be taken cautiously since all the other tasks are still ASR (but in other languages). The bottom part of many models trained from waveform input captures spectral representations shared between languages for the same task. Audio Event Detection tasks or Speaker Clustering tasks may be good ones to examine this point for speech.

4) There is not much discussion on where the pruned weights are located in the pre-trained network. The Wav2vec2.0 approach is known to have valuable representations towards the middle of the network, with higher-level weights only beneficial for the pre-training contrastive loss with less impact on downstream applications. One reason for the similarity between the subnetwork for English and multi-lingual models could be that both use the same pre-training approach, leading to the same inductive biases, i.e., a peculiarity of the pre-training approach rather than a property of the found sub-network used with PARP. 

5) It will inform the readers if the authors share the number of model updates (independent of the paired dataset size) used for PARP. One hypothesis for good performance could be the long training time the model weights are exposed to the paired data (without rewinding to their original pre-trained values). It is known that longer training times with slow learning rates lead to superior performance on many tasks. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this work, the authors build upon techniques to reduce the number of parameters in the model by finding sparse sub-networks along the lines of the “lottery-ticket hypothesis” work. In this case, however, the authors apply the proposed techniques in the context of self-supervised speech recognition by showing that the same initial masked network is suitable for fine-tuning for a number of tasks including when applied for cross-lingual transfer when used with multiple languages.
main_review: Overall, this is an extremely well-written paper. In particular, I found that the motivation of the work was extremely clear, and well-supported by analyses to show that the sub-networks obtained in a task agnostic fashion had a large overlap (as measured in terms of IOU) with the task-specific sub-networks. I also found the experiments to be well-designed and comprehensive exploring different settings: pre-training and finetuning for the same language; pre-training on one language and fine-tuning on a large number of languages; pre-training and fine-tuning on multiple languages. The ablation experiments in Section 4, were interesting, and strengthen the paper greatly. In my view, the authors’ work -- particularly, the work on rapidly training models on multiple languages -- is a very impactful contribution. 

One clarification question: In Figure 4, if I understand correctly, the PER w/o and LM is still ~50% even at 100% sparsity. This was somewhat puzzling to me. Shouldn’t the PER be 100%+ if all weights are zeroed-out? (I assume that 100% sparsity means that all weights are zeroed out even after finetuning).

I would note that one limitation of this work is that although the authors show that they can achieve large unstructured sparsity in the model, it is not clear if the sparsity structure will be amenable to getting large speed-ups when run on a device/server. I think the paper could be strengthened by measuring actual RTF/benchmark numbers by running these models on device.

limitations_and_societal_impact: Apart from the limitations mentioned above, I do not see any potential negative societal impacts of this work.
review_title: null

Meta-review: This paper is interested in reducing the complexity of self-supervised speech representation learning systems, by discovering sparse subnetworks in the self-supervised architecture which perform well on downstream tasks. While reviewers acknowledge the approach is simple (basically performing a sparsifying step followed by a finetuning one), it is effective. The paper also applies the idea on BERT models.

Authors should include in the final version of the paper the additional results and findings mentioned during the discussion period.
Decision: Accept (Spotlight)