Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper propose local prior matching to leverage a language model to use unlabeled speech data to improve an ASR system. This is a worthy goal. The details of the proposal were a bit hard for me to understand. The proposed method reminded me of "posterior regularization" (K. Ganchev et al. 2010), but I could not understand Section 2.2 well enough to draw a direct link. I encourage the authors to condense 2.2 and make it clearer what, exactly, Local Prior Matching is.

The paper presents extensive, interesting results. I do want to point that they seem to be considerably off of the LibriSpeech state of the art, e.g. see K. Irie et al. Interspeech 2019.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work proposed a distillation approach which use ASRs to generate hypotheses for unsupervised data, run a LM to get probability for the hypothesis, and perform distillation with the resulting probability. The ASRs being used for generating hypotheses can be either a model trained with the supervised data or the student model, and can switch between the two during training. In the experiments, ASR models are pre-trained with the subset of Librispeech data and use the rest of Librispeech data as unsupervised data, and the LM is trained with Librispeech LM data. The experiments shown the proposed approach improve baseline model trained with the Librispeech subset significantly.

The use of LM to provide soft target is a good idea as LMs can utilize unsupervised text data as opposed to the requirement of training a strong teacher model with paired data, and can be easily integrated with existing distillation approaches for ASRs. The switching to the student model for generating hypotheses when it outperforms the pre-trained ASR also makes a good sense. The overall novelty however is a bit limited compared to the existing work, as the major contribution is to propose to use LMs as teacher rather than ASRs, with the rest of the design to be similar to existing works.

The paper relates their method to self-supervised learning, yet I find it having stronger correlation with existing distillation approaches, and can be better understood through the distillation perspective.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Overview:
This paper is dedicated to proposing a self-supervised objective, local prior matching (LMP), for speech recognition. This approach can take advantage of vase quantities of unlabeled speech data. What' more, the objective is simple to implement and theoretically well-motivated. In the paper, based on a supervised pretrained model, it then finetunes 360 hours with unlabeled data and LPM reduces the WER consistently. They also conduct extensive ablation experiments to show the effect of their self-supervised approach.

Strength Bullets:
1. I think this self-supervised learning objective (LMP) is very novel. The motivation that the source of indirect supervision on processing unlabeled speech comes from prior knowledge about the world and the context of the speech makes sense to me. The author combines the Bayesian method to build the model which is aligned with the motivation. They also provide clear and well-organized derivations. 
2. The paper performs extensive ablation studies over all components, including beam size, mixing ratio, LPM weights, model update strategies, model initialization, length filtering and choice of language models. It provides convincing evidence of the effect of each component. 
3. It provides interesting experiments results to study the relationship between the amount of unlabelled data for self-supervision and final performance. And LPM can surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data.


Weakness Bullets:
1. The paper only evaluates their method on LibriSpeech dataset. Although this dataset is popular, one or two more datasets will be more convincing. 
2. For bayesian based methods, it is well known that it performs badly in high dimensional space. The reason is that we can not sample enough data points to obtain a good posterior estimation. Could you provide more analysis about the quality of posterior with different amount of sampled data?
3. For the experiment between the amount of unlabelled data for self-supervision and final performance, it would be better the author can provide a curve with more results.

Recommendation:
I think it is a good paper. The proposed approach is useful. This is a weak accept. 
review_title: Official Blind Review #2

Meta-review: The paper proposed local prior matching that utilizes a language model to rescore the hypotheses generate by a teacher model on unlabeled data, which are then used to training the student model for improvement. The experimental results on Librispeech is thorough. But two concerns on this paper are: 1) limited novelty: LM trained on large tex data is already used in weak distillation and the only difference is the use of multiply hypotheses. As pointed out by the reviewers, the method is better understood through distillation even though the authors try to derive it from Bayesian perspective. 2) Librispeech is a medium sized dataset, justifications on much larger dataset for ASR would make it more convincing. 
Decision: Reject