Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel fine-tuning method for downstream speech tasks, notably automatic speech recognition (ASR). The key idea of its method is to learn binary masks on the connectivities of pretrained weights instead of directly updating the weight values. The binary masks are obtained by discretizing learnable masks (which have the same shape as weights) during the forward pass. The masks can be optimized via stochastic gradient descent, where straight-through is used to approximate the  gradient estimation. The training objective contains a sparsity constraint, which also makes the proposed fine-tuning method to become a pruning technique. 
strengths_and_weaknesses: Overall idea of the paper is simple but quite interesting. It's surprising to see how effective this method is, without changing pretrained weights directly, but only learning connectivity patterns. The experimental results look strong, so empirically the method seems to be working quite well. It would be nice if the authors can add a bit more refined explanation and analysis on why this method works well. 

There are too many contents and citations, which hurt the readability. It is quite often distracting, because I have to open other papers to find out the details. It would be nice if the authors can make the paper to become more self-contained and make it simpler to read. For example, having to be forced to open a cited paper to check the size of the pretrained model (wav2vec) is a bit frustrating. Also, one of the most important details of a new method: "so how can I train this?" part is missing. The loss function is trivially given as an example, e.g., CTC loss. Having many experimental results on a paper is not a bad thing, but the training procedure and how the results were obtained are also important details. They shouldn't be omitted to have more space to add a few more tables.

As a reviewer I am pointing out that I am concerned about the reproducibility of this work as training details are not shared.
questions: 1. I am curious what kinds of regularization methods can be applied apart from the sparsity constraint. For example, has/have the author(s) try combining S3-router with dropout?

1-2. Can you add noise to the mask (dropping out the mask)?

2. The authors mentioned that S3-router can be seen as an all-in-one technique for on-device applications. Achieving such good results on multiple language tasks by only using less than 10% of the pretrained model for each task is impressive. However, if there are multiple language tasks, and if each (language | task)-specific mask vary a lot by each different task, then I wonder there will be a diminishing return in the pruning aspect. Because, in order to work well for a variety of tasks, the task specific mask should specialize (and should different from masks for other tasks), then this means each mask would be sparse, but the union set of sparse masks will not. Therefore, the whole pretrained model should be uploaded to the device. And as the authors pointed out the resource situation is quite limited. If a set of downstream tasks involve many different languages, I assume that the learned masks shouldn't be too close to each other, as the key idea of this new fine-tuning scheme and pruning technique is to learn (language | task)-specific mask. What is your opinion about this matter?  

3. Can you share more details about training? There are no hyper-parameters, optimization details and training time in the paper.

limitations: The experimental results are strong, but at the same time, the proposed method is only demonstrated with empirical results. I am a bit wary of generalization capability of this method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes modifying the fine-tuning procedure of speech SSL models like wav2vec 2.0 by applying a binary masking based finetuning where each weight is either kept or masked to 0 instead of modifying the model weights directly. This follows a similar recently proposed technique in the NLP domain. The paper's main contributions is application to speech domain, demonstrating that this approach can work well for ASR as well as other speech tasks such as diarization. All of this is achieved by simply learning to mask about 10% of the weights, which leads to greater efficiency of storing task-specific models. The authors propose a new method to choose masks in which they randomly choose masks for each layer but initialize the weights for the masks based on the magnitude of the weights, and show that this outperforms random masking and magnitude based masking.
strengths_and_weaknesses: Strengths:
- Interesting and impactful approach showing that a light weight fine-tuning can outperform full model fine-tuning for speech related tasks and lead to more efficient finetuned model storage (or alternatively, somewhat sparse models).
- Thorough evaluation with different model sizes, in both mono-lingual and multi-lingual scenarios and across multiple tasks.

Weaknesses:
- This is not a super novel work given that it mostly transfers findings from NLP, but nonetheless the demonstrated effectiveness of the approach for speech tasks and a better performing mask initialization technique should be of interest to the community.
- The presentation is overly dramatic - perhaps the authors should reduce the use of terms like "win-win" "excitingly" etc.

questions: Have you tried combining weight fine-tuning with your binary masking scheme? For example, you can fine-tune weights followed by learning binary masks, or learning binary masks followed by weight fine-tuning, or do it jointly.
limitations: None that I can see.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a novel method that fine-tunes speech SSL models by discarding the connections rather than changing the weight values. Experiments were conducted on simulated low-resource multilingual ASR tasks as well as several other speech tasks. An interesting finding was that the learnt masks were linearly correlated to the phone inventories of each individual language.  
strengths_and_weaknesses: Strength:
1. The paper is very carefully written and easy to follow.
2. The idea of discarding connections is possibly still novel for speech SSL models.
3. Many experiments were done and mostly the results were positive.

Weaknesses:

1. In an alternative view, the proposed S3-Router method is a binary-mask-based task adaptor approach. As the authors claimed, mainly it required to build such masks for all fully-connected layers, which results in a large number of trainable adaptor parameters, in addition to the task-specific output layers. This is a very inefficient way to build such adaptors compared to many other existing alternatives. So the more proper baselines to compare with, are other alternative adaptation methods instead of simply fine-tuning all model encoder parameters.

2. Although it is claimed that 

"However, there exists a dilemma between the trends of speech SSL models and the growing demand for speech processing applications on the edge. While advanced speech SSL models become increasingly larger to learn more generalizable features, it is highly desired to process the captured speech signals in real-time on edge devices, which have limited resources and conflict with the prohibitive complexity of existing speech SSL models."

The S3-Router actually increases computational cost rather than reduces it, since masks are required to be applied every time, and the masked weights still need to be used via normal GEMM functions due to the design of the current hardware. 
This also results in high storage costs. Users either need to store the resulted many binary mask parameters and apply them every time with extra computation cost (although the binary masks can be stored in bits), or an extra set of parameters need to be stored for every task, which contradicts with the claim

"For example, compared to independent weight fine-tuning for each language/task, S3-Router can simultaneously support 11 languages in Sec. 4.2/ 4.3 and 8 tasks in Sec. 4.4 using one wav2vec2-base while achieving a win-win in both accuracy (as validated in Sec. 4.2/ 4.3/ 4.4) and efficiency, i.e., more than 88.5% reductions in model parameters."

I believe other task adaptation methods can be way more efficient in terms of both computation and storage costs. Most of the advantages the authors claimed are general benefits of using adaptors/adaptations (e.g. "One exciting advantage is that since the gradients of different spoken languages or tasks can..."), which have been studied extensively in the speech community in the past three decades (& these works are completely ignored by this paper).

3. The results are also disappointing. Although so many extra parameters are introduced, it was found that only 10% of the connections can be discarded, which is not "non-trivial contributions (claimed by the authors)" and possibly not useful in practice at all.  In general, I feel this paper overclaims. 

4. Although the findings were claimed to be generic (e.g. discarding less than 10% connections), actually only one type of speech SSL model (Wav2Vec 2 related models) was studied, which is not convincible. 

5. The results were not compared with other published numbers on the same tasks (e.g. those from SUPERB).
questions: 1. The authors claimed that
"where the speech SSL models suffer from a higher risk of overfitting on downstream low-resource tasks as compared to NLP"
Is there an evidence or proof for this?

2. Similarly,
"As large-scale speech SSL models tend to overfit when finetuning their weights under a low-resource setting, we hypothesize that the binary optimization of the mask patterns in Eq. (1) can serve as regularization and thus alleviate overfitting"
Do the authors think regularisation methods (such as L1/L2 regularisors, dropout, early stopping, and smaller learning rates, or simply only fine-tune a small parts of the model) will make a difference here?


limitations: 1. Compare with other adaptation/adaptor methods.
2. Study other types of speech-based SSL models, if the findings are claims to be generic.
3. Based on the current hardware, routing within sparse models (e.g. MoE models or Pathway models) would be more efficient than routing each individual connection within a densely-connected model, unless the findings indicate way more connections can be discarded (e.g. more than 70%) -- this would allow to use sparse rather than dense matrix structures and ops.
review_title: null

Meta-review: metareview: reviewers like that
* paper is well written and interesting
* experiments are solid

this paper should be accepted.
confidence: Certain
Decision: Accept