Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper presents a method for unsupervised representation learning of speech. The idea is to first learn discrete representation (vector quantization is done by Gumbel softmax or k-means) from audio samples with contrastive prediction coding type objective, and then perform BERT-style pre-training (borrowed from NLP). The BERT features are used as inputs to ASR systems, rather than the usual log-mel features. The idea, which combines those of previous work (wav2vec and BERT) synergetically, is intuitive and clearly presented, significant improvements over log-mel and wav2vec were achieved on ASR benchmarks WSJ and TIMIT. Based on these merits, I suggest this paper to be accepted.

On the other hand, I would suggest directions for investigation and improvements as follows.

1. While I understand that vector quantization makes the use of NLP-style BERT-training possible (as the inputs to NLP models are discrete tokens),  there are potential disadvantages as well. One observation from the submission is that the token set may need to very large (from tens of thousands to millions) for the system to work well, making the BERT training computationally expensive (I noticed that the BERT model is trained on 128 GPUs). Also, without BERT pre-training, using directly the discrete tokens seems to consistently give worse performance for ASR. I think some more motivations or explorations (what kind of information did BERT learn) are needed to understand why that is the case.

2. Besides the computational expensive-ness of the three-step approach (vector quantization, BERT, acoustic model training), the combined model complexity is large because these steps do not share neural network architecture. A more economical approach is to use BERT-trained model as initialization for acoustic model training, which is the classical way how RBMs pre-training were used in ASR.

3. One concern I have with discrete representation is how robust they are wrt different dataset. The ASR datasets used in this work are relatively clean (but there does exists domain difference between them). It remains to see how the method performs with more acoustically-challenging speech data, and how universally useful the learned features are (as is the case for BERT in NLP).

4. Another curious question is whether the features would still provide as much improvement when a stronger ASR system than AutoSeg (e.g., Lattice-free MMI) is used.

Overall, while I think the computational cost of the proposed method is high, rendering it less practical at this point, I believe the approach has potential and the result obtained so far is already significant.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Though rather dense in its exposition, this paper is an interesting contribution to the area of self-supervised learning  based on discrete representations. What would make it stronger imo is to address the issue of how much is gained from a discrete vs. continuous representation. The authors take it as a given that discrete is good because it allows us to leverage work in NLP. That makes sense -- but at what cost?

"Table 4 shows that our first results are promising, even though they are not as good as the state of the art." The state of the art on LibriSpeech is not Mohamed at al. 2019. See e.g. Irie et al. Interspeech 2019 for better result

The Conclusion is very sparse. "In future work, we are planning to apply other algorithms requiring discrete inputs to audio data": can  you elaborate?
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper proposes a way to pre-train quantized representations for speech. The approach proposed is a two-stage process: 1. train a quantized version of wav2vec [my understanding is that wav2vec is the same thing as CPC for Audio except for using a binary cross-entropy loss instead of InfoNCE softmax-cross entropy loss]. the authors propose to use gumbel softmax / VQ codebook for the vector quantization.
2. once you have a discrete representation, you could train BERT (as if it were a seq of language tokens). this makes a lot of sense especially given that CPC / wav2vec recovers phonemes and quantizing the phonemes will recover a language-like version of the raw audio. And running BERT across those tokens will allow you to capture the dependencies at the phoneme level. 

After pre-training, the authors use the learned representations for speech recognition. They compare this to using log-mel filterbanks. 

The results (WER / LER) is lower for the proposed pipeline compared to using dense wav2vec representation for n-gram and character LM.  It also makes sense that BERT helps for the k-means (vq) setting since the number of codes is large. 

The authors also cleverly adopt/adapt span-BERT which is more suited to this setting.

I think this paper presents a useful contribution as far as improving speech / phoneme recognition using self-supervised learning goes, and also has useful engineering aspects in terms of combining CPC and BERT. I would like to see this paper accepted.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: 
Overview:

This paper considers unsupervised (or self-supervised) discrete representation learning of speech using a combination of a recent vector quantized neural network discritization method and future time step prediction. Discrete representations are fine-tuned by using these as input to a BERT model; the resulting representations are then used instead of conventional speech features as the input to speech recognition models. New state-of-the-art results are achieved on two datasets.

Strengths:

The core strength of this paper is in the results that are achieved on standard speech recognition benchmarks. The results indicate that, while discritization in itself does not give improvements, coupling this with the BERT-objective results in speech features which are better in downstream speech recognition than standard features. I think the main technical novelty is in combining discritization with future time step prediction (but see the weakness below).

Weaknesses:

The main weakness of the paper is that it does not situate itself within existing literature in this area. Over the last few years, researchers in the speech community have invested significant effort in learning better speech representations, and this is not discussed. See e.g. [1]. Even more importantly, very recently there has been a number of papers investigating discrete representations of speech; see the review [2]. Some of these papers specifically use VQ-VAEs [3]. [4] actually compares VQ-VAE and the Gumbel-Softmax approach. These studies should be mentioned. This paper is different in that it incorporates future time step prediction. But context prediction has also been considered before, also for speech [5, 6, 7]. This paper can be situated as a new contribution combining these two strands of research. In the longer run it would be extremely beneficial to the community if this approach is applied to the standard benchmarks as set out in [2].

As a minor weakness, some parts of the paper is not described in enough detail and the motivation is weak or not exactly clear (see detailed comments below).

Overall assessment:

I think the results as well as the new combination of existing approaches in the paper warrants publication. But it should be amended significantly to situate itself within the existing literature. I therefore award a "weak accept".

Detailed questions and suggestions:

- Section 1: As motivation for this work, it is stated that "we aim to make well performing NLP algorithms more widely applicable". As noted above, some NLP-like ideas (such as prediction of future speech segments, stemming from text-based language modelling) have already been considered within the speech community. Rather than motivating the work in this way, it might be helpful to focus the contribution as a combination of future time step prediction and discretization (both of which have been considered in previous work, but not in combination).
- Section 4: Would it be possible to train the vq-wav2vec model jointly with BERT, i.e. as one model? I suspect it would be difficult since, for the masking objective, the discrete units are already required, but maybe there is a scheme where this could work.
- Section 2.2: Similarly to the above question, would there be a way to incorporate the BERT principles directly into an end-to-end model, e.g. by randomly masking some of the continuous input speech?
- Section 3.3: What exactly does "mode collapse" refer to in this context? Would this be using only one codebook entry, for instance?
- Section 6: It seems that in all cases to obtain improvements from discritization, BERT is required on top of the vq-wav2vec discrete symbols. Is it possible that the output acoustic model is simply better-matched to continuous rather than discrete input (direct vq-wav2vec gives discrete while BERT gives continuous)? Would it make sense to train the wav2vec acoustic model on top of the vqvae codebook entries (e) instead of directly on the symbols?

Typos, grammar and style:

- "gumbel" -> "Gumbel" (throughout; or just be consistent in capitalization)
- "which can be mitigated my workarounds" -> "which can be mitigated *by* workarounds"
- "work around" -> "workaround"

Missing references:

1. Versteegh, M., Anguera, X., Jansen, A. & Dupoux, E. (2016). The Zero Resource Speech Challenge 2015: Proposed Approaches and Results. In SLTU-2016 Procedia Computer Science, 81, (pp 67-72).
2. https://arxiv.org/abs/1904.11469
3. https://arxiv.org/abs/1905.11449
4. https://arxiv.org/abs/1904.07556
5. https://arxiv.org/abs/1904.03240
6. https://arxiv.org/abs/1807.03748 (this paper is cited)
7. https://arxiv.org/abs/1803.08976

Edit: Based on the feedback from the authors, I changed my rating from a 'weak accept' to an 'accept'.
review_title: Official Blind Review #1

Meta-review: This paper proposes a new self-supervised pre-trained speech model that improves speech recognition performance. 
 The idea combines an earlier pre-training approach (wav2vec) with discretization followed by BERT-style masked reconstruction.  The result is a fairly complex approach, with not too much novelty but with a good amount of engineering and analysis, and ultimately very good performance.  The reviewers agree that the work deserves publication at ICLR, and the authors have addressed some of the reviewer concerns in their revision.  The complexity of the approach may mean that it is not immediately widely adopted by others, but it is a good proof of concept and may well inspire other related work.  I believe the ICLR community will find this work interesting.
Decision: Accept (Poster)