Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper investigates an unsupervised learning approach based on bi-directional contrasive predictive coding (CPC) to learning speech representations.  The speech representations learned using 1k and 8k hours unlabeled data based on CPC are shown to be helpful in semi-supervised learning ASR tasks in terms of sample efficiency, WER and cross-domain robustness. The reported work is interesting and may have value to the speech community.  Regarding the paper, I have the following concerns. 

1.  In terms of semi-supervised learning ASR, I think any proposed approach should compare with the "naive" way of doing it. That is, use a high-performance ASR model to decode the unlabeled data and use the decoded pseudo-truth as the ground truth to train an acoustic model with an appropriate capacity.  In my experience,  many of the "novel" approaches can not outperform this "naive" method.  I would like to see this as a baseline for the semi-supervised learning experiments. 

2. In sec. 3.1 on the setting of unsupervised learning, the authors state that "all audio signals have a sampling rate of 16KHz". This is obviously not true for the Switchboard data in Table 6 in Appendix A, which has a sampling rate of 8KHz as they are telephony signals.   The authors should clarify. 

3.  It is not clear to me why the authors use two different ASR models (DeepSpeech2 small and TDNN). Why not stick to one architecture but adjust the model capacity?  

4.   I wonder if the latent features learned by CPC can be complementary to the conventional features such as logmel ? How does it perform if  the two are simply concatenated as the input to the acoustic model? 

P.S.  rebuttal read. I will stay with my score.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes an unsupervised method for learning representations of speech signals using contrastive predictive coding. 
The authors provide results for the speech recognition task, in which they trained their model on up to 8000 hours of speech. The authors provide results on several English benchmark datasets in addition to four low-resource African language datasets. 
The authors compared their method to the traditional signal processing representations and show that the proposed method is superior. 

My main concern with this submission is its novelty.
The proposed method was previously explored in [1] and presented similar results. If I understand it correctly, the main novelty in this work is the usage of bi-directional models together with more data. However, it is not clear what made the improvements. Considering the fact that such an approach was suggested recently by [1], a detailed comparison with uni-directional models is needed.
For example, in Table 2, the authors provide results for WSJ dataset, however, with no LM decoding. Can the authors provide experiments of WSJ while using LM similarly to [1]?  Moreover, if the authors wanted to eliminate the effect of LM as they stated in the paper, why not calculating Character Error Rates instead or in addition to Word Error Rates? Again, as done in [1], and in many other papers in the field [2]. 

Additionally, in Table 1 and Table 5, the error rates seem pretty high, especially for the baseline model, did the authors investigated different architectures/stronger ones for these tasks? Different representations such as LogFilterBanks / MFCCs?

I'm willing to increase my score, in case the authors will address my concerns. However, at the moment, I do not see much novelty in this paper comparing to previous work. Additionally, the authors are missing an essential comparison to previous work so we could better understand the contribution of this paper. 

Minor comments: "using a simpler convolutional architecture than is common" -> should be rephrased.


[1] Schneider, Steffen, et al. "wav2vec: Unsupervised Pre-training for Speech Recognition." arXiv preprint arXiv:1904.05862 (2019).

[2] Adi, Yossi, et al. "To Reverse the Gradient or Not: an Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition." ICASSP, 2019.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
Overview:

This work uses contrastive predictive coding (CPC) to learn unsupervised speech representations on large amounts of unlabelled speech data and then uses the resulting features in downstream speech recognition systems. Unlabelled data is obtained from several sources (spanning different languages). Supervised systems are then built on top of these features and sample-efficiency and cross-domain robustness is investigated using English data sets. Finally, the approach is applied to four African languages.

Strengths:

Firstly, the paper is very clearly written and motivated. Secondly, a very relevant problem is tackled in a systematic way; compared to transcribed resources, unlabelled resources are much easier to collect and more widely available. This paper shows that these unlabelled resources can be of great benefit in downstream tasks and on languages where few resources are available. Thirdly, the experiments are carried out very systematically to support the claims of the paper: that bidirectional CPC-based feature learning improves same efficiency (they show that much less labelled data is required to achieve the same performance as when using more substantial labelled data with conventional features), and that it improves robustness to out-of-domain data. They perform these experiments on both English and truly low-resource languages.

Weaknesses:

There are two main weaknesses to the paper. Firstly, as the authors note themselves, unsupervised CPC-based speech feature learning was developed and considered in previous work, and has also been subsequently investigated by others. The main technical contribution is therefore only in changing the unidirectional architecture to bidirectional. Secondly, the paper does a very poor job of linking this work with previous work. The work in [1] is very related. In Section 5, the ZeroSpeech challenges are mentioned briefly (with a single citation), but over the last decade there has been substantial work in this community specifically looking at exactly the main problem addressed in this paper (unsupervised speech representation learning). It would be of great benefit to situate this work within that context, and I would recommend that the paper at least mention [2] to [9].

Overall assessment:

Although technical novelty is limited (first weakness), I think there is novelty in the paper's systematic experimental investigation, including ASR experiments on truly low-resource languages. The conclusions of this work also has practical implications for the ASR community. The second weakness can be addressed by amending Section 5. I therefore assign a "Weak Accept" to the paper.

Questions, suggestions, typos, grammar and style:

- In Figure 1, it might be useful to indicate the autoregressive nature of the context vectors by adding arrows in-between the $c$ blocks on the top left. (In the text it says an RNN is used.)
- p.7: "... are suitable for driving recognition different languages ...". A typo or grammatically incorrect sentence.
- p. 9: "Tts without t" -> "TTS without T"
- p. 9: "african" -> "African" (check all citations for capitalization)

Missing references:

1. https://arxiv.org/abs/1904.03240
2. A. Jansen et al. A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition. ICASSP, 2013.
3. Badino, L., Canevari, C., Fadiga, L., & Metta, G. (2014). An auto-encoder based approach to unsupervised learning of subword units. in ICASSP.
4. Versteegh, M., Anguera, X., Jansen, A. & Dupoux, E. (2016). The Zero Resource Speech Challenge 2015: Proposed Approaches and Results. In SLTU-2016 Procedia Computer Science, 81, (pp 67-72).
5. Renshaw, D et al. (2015). A Comparison of Neural Network Methods for Unsupervised Representation Learning on the Zero Resource Speech Challenge. Interspeech.
6. R. Thiolliere et al. A  hybrid  dynamic  time  warping-deep  neural  network  architecture  forunsupervised acoustic modeling. Interspeech. 2015
7. https://arxiv.org/abs/1811.08284
8. https://arxiv.org/abs/1702.01360
9. https://arxiv.org/abs/1709.07902

review_title: Official Blind Review #3

Meta-review: The paper focuses on learning speech representations with contrastive predictive coding (CPC). As noted by reviewers, (i) novelty is too low (mostly making the model bidirectional) for ICLR (ii) comparison with existing work is missing.
Decision: Reject