Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Many self-supervised speech representation learning methods use masked prediction at their core. This paper proposes Ask2Mask (ATM) approach for informed masking during learning through a supervised teacher model that provides frame-level posteriors probabilities of linguistic output units. The frame-posterior probabilities extracted from the supervised teacher model are also used as utterance weights to favor ones with high confidence.
main_review: Improving training effectiveness through better masking is an important research direction for speech representation learning. 

Strengths:
The paper presents a robust experimental setup with three SOTA models (W2V, HuBERT, W2V-BERT) evaluated in two sizes L/XL on multiple datasets (LS, AMI, CHIME-6, Tedlium, and Commonvoice.

Weaknesses:
There are three major points here:
1) The paper presents an incremental improvement over the baseline systems with minor novelty. 
2) The proposed approach relies on access to labeled data during the pre-training stage, which departs from the standard setup for self-supervised learning. The pre-training stage in self-supervised learning is task agnostic to open the door for utilizing the learned representations in a wide array of downstream tasks.
3) The observed gains are modest using the proposed approach. With 100h of labeled data, semi-supervised methods (either from random weights or initialized from self-supervised models) would bring much lower WERs.

General comments:
Equation 5 encourage sampling from places with high confidence, which may lead to oversampling silence and vowel segments. How would you encourage the selection of diverse positive samples? 
The primary motivation for filtering data in semi-supervised learning is to remove noisy inputs and ones with bad teacher labels/confidence. A similar goal can be achieved in self-supervised learning using voice activity detection (VAD) to remove long silences and other signal processing methods. Dealing with data problems in self-supervised learning has also been studied in (https://hal.archives-ouvertes.fr/hal-03070411/document)
One way to benefit from a supervised teacher is to benefit from the teacher segmentations and labels, not only confidence scores, for masking and negative sampling as done here (https://arxiv.org/pdf/2103.05149.pdf)

summary_of_the_review: The paper presents an approach for improving self-supervised learning approaches using a supervised teacher model. The problem is important, but the proposed method and the results aren't convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors proposed to use an external scorer to weight the frames to be masked for the MSM loss. Idea is interesting as not all speech frames are of equal importance. 
main_review: I personally think that authors have a good point here, how to do better mask selection in self-supervised learning. Obviously, not all speech frames are
of equal importance. Studies, such as in automatic speaker recognition have been made, where it was clearly found out that different phones carried 
different amount of speaker recognition cues than others. In ASR, you can take non-speech region vs vowel and you get a big difference in importance. 

However, the way it was done here, I believe is not correct. First of all, the idea of self-supervised learning is to learn task independent features that
can be then used for any task. So, then in principle, the learning signal should be something that does not have any one task in mind. But okay, lets say
your intention is to do only ASR and your chosen model, of course then you can proceed as described in the present paper. But then your need to compare 
with other ASR models that use _similar_ amount (and the same) labeled training data. Which in your case is the data used in the scoring model. So SoTA 
end-to-end mehtods need to be used (and ESPnet is a good way to start in looking for implementations https://github.com/espnet/espnet). 

And finally, if you look at the Table 6, the difference in WER betwene proposed, that uses extra data, vs baseline is not too great. Only in CHIME we see
some real improvement. 

Minor comments: 

- L_div is not explained in more detail. 
- Equation (3) is not needed. 
- This sentence is very suspicious: "The scorer’s training data is chosen such that it is matches the target data condition". Wouldn't it be better
that the whole learned model is as general as possible? Could you elaborate on this?
- In Table 2, could you add the metric (WER) in the caption. 
- Please replace string "%WER" with "WER (%)". Also in the same vein it would be important to make all captions self-contained, so that 
reader should be able to understand the Table and Figure without referencing to the the main text. 

I have read the authors rebuttal and discussion and due to sufficient answer to my concerns I have raised the score. 
summary_of_the_review: Authors add more labeled training data, as an external scorer, and still results are significantly better. Authors need to compare against SoTA models where all models see exactly the same data.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This study investigates the use of an external frame-synchronous CTC-based ASR system to get confidence scores for frames in masked speech modeling which is called AskToMask (ATM). These confidence scores are used in two ways: 
1) as a distribution to determine the segment to mask (higher confidence means higher probability), 
2) the utterance level average of the per-frame confidence scores is used as a scaling factor of the per-utterance loss. 

The experiments perform unsupervised pre-training using these two proposed methods and then performs supervised fine-tuning for the final ASR task. Experiments on Librispeech and AMI show that confidence score-based selection of masking improves the final WER as compared to using uniform distribution in the mask selection (the first approach versus baseline). The second approach provides marginal gains on top of the first approach. 
main_review: Strengths: 

- Experiments are extensive to evaluate different aspects of the proposal (amount of masking vs. WER, type of scorer vs WER, effect of ATM approach vs. ATM+scaling approach)
- The method is relatively easy to implement with existing open source tools. 

Weaknesses:

Notation of the equations needs some updates, for example:
- Eq. 2. $L$ is used both as an index and as a set of codebook indices
- Eq. 5. The denominator should use a summation index different than $t$

The novelty is somewhat limited. Having a weighting mechanism instead of uniform distribution for masking is relatively straightforward extension of the existing masked based modeling approaches. 

Some details are missing. Please refer to the questions below:
1) Introduction: “The scoring model is necessarily a speech recognition model trained on small amount of data” -> it could have been a large model on a known larger dataset. Would a better scorer help with ATM?
2) “ignoring utterances with low-confidence” -> Not exactly true, to the reviewer's understanding, confidence level is just a scaling factor. Or do the authors put a threshold on scores?
3) In the introduction of Section 5, which part of AMI is used as the training subset? Without mentioning this detail, it is hard to see why it can show the domain generalization aspect. Since the experiments usually mention LS fine-tune -> test on LS test, and AMI fine-tune and test on AMI test set, the generalization claim is not very clear.
4) Details of fine-tuning is missing. Do the authors use, for instance, w2v-BERT-L + ATM based embeddings and train a new ASR model? Or do the authors add a new output layer to map input speech to word-piece units? 
5) In Table 4 and its explanation, do the authors refer to match and mismatch based on the dataset used in pre-training and fine-tuning?

Minor typos: Introduction, first paragraph: pesudo -> pseudo;
Confidences cores -> confidence scores

Please add reference number to CHiME-6 dataset in Section 4.1.
summary_of_the_review: The paper uses ASR based scores to mask high-confident regions in masked speech modeling. Using this weighting idea is not very novel even though it may be the first implementation in the speech domain. The main strength of the paper is that it provides extensive comparison for different aspects of the approach. The writing may benefit from some notational fixes and addition of some implementation details as discussed in the detailed review. From the discussion above, it can be seen that there are more weaknesses as compared to the strengths.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a weighted masking scheme for learning speech representations. Weights in this scheme are surrogates of confidence scores obtained from an external ASR system. The paper also describes how utterance-level confidence scores can be incorporate to down-weight contribution from utterance that are likely to be problematic for an ASR system. 

The paper makes use of Librispeech dataset for learning representations and Librispeech and AMI datasets for training external ASR systems. The paper evaluates both schemes on Librispeech, AMI, and some other datasets. Experimental results show that the proposed schemes offer small-to-medium gains over the unweighted reference masking scheme. 


main_review: Strengths
- paper introduces confidence scores into masked modelling
- paper shows that even simplest of schemes can lead to small-to-medium gains
- comprehensive performance comparison across many speech data sets
- large-scale, non-trivial, models and data sizes

Weaknesses
- limited technical novelty
- technical quality of paper (numerous typos (eq 2, ), undefined terms ({\bm E}^{rm}), sloppy notation (1,...,k-1 does not equal K), confusing notation (eq 5)); section 3.1 is meant to be the key section but you are making a very limited attempt to be clear not just for speech but also more general audiences 
- limited discussion about the use of confidence scores in ASR
- lack of any discussion or investigation regarding suitability of softmax probabilities as surrogates of confidence scores
- your claim about gains up to 11.6% seems to apply to CV task if you compare 12.1% of SpeechStew and your ATM+S approach. If this is true your statement in the abstract is misleading at best. Please focus on comparing unweighted and weighted schemes. Please make it clear when you are comparing your best number and someone else's best numbers. 
summary_of_the_review: This paper explores the use of weighted masking for learning speech representation. The proposed scheme relies on confidence scores obtained from external ASR systems to decide which speech segments should and should not be used for masked modelling. An extensive empirical investigation shows that weighted masking is a promising direction for learning accurate speech representations. 

Although the technical novelty is limited, this paper introduces weighted masking by means of confidence scores (very popular subject) into speech representation learning area. Unfortunately, this paper currently contains a number of technical issues (see above). Furthermore it offers a limited insight about the quality of confidence scores used and sensitivity of the proposed scheme to this important in practical applications factor (e.g. limited resource languages where WERs of typical external ASR systems range wildly).
review_title: null

Meta-review: This paper presents an approach that uses ASR-based scores to guide masking high-confident blocks for speech representation learning. As most of the reviewers mentioned, it is an incremental improvement over baseline systems with limited novelty. About the use of confidence scores which is a key factor of the method, it lacks enough discussion on its quality and sensitivity.
Decision: Reject