Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduces a fully unsupervised automatic speech recognition framework built upon the Wav2Vec 2.0 representations and a GAN generator/discriminator training process that, amongst others, performs on a par with previous state-of-the-art ASR models. 
main_review: Thank you for a really nice and informative read. The paper is well-structured and the appendix is supporting the claims and choices such that it seems reproducible. I believe this to be a strong contribution to the research community.

Some comments:
I'm not sure that I can make sense of the "Removing Silences" paragraph in the paper in cohesion with the findings in Appendix C?

In the "Speech Audio Representations" paragraph it would be informative to run a more in depth analysis on the features. E.g., "ON SCALING CONTRASTIVE REPRESENTATIONS FOR LOW-RESOURCE SPEECH RECOGNITION" by Borgholt et al. runs an interesting analysis in their Figure 3. Figure 2 in this paper seems rather arbitrary and needs investigation.

The paper would be significantly stronger from more datasets. E.g., what about more noisy and conversational datasets? How does the model perform in those settings?

Have you tried to set up a semi-supervised or self-supervised variant of the framework? It would be very interesting to see how the model framework would improve.

limitations_and_societal_impact: I think that the major societal impact is written into the article.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper builds upon previous work on unsupervised speech recognition. This work introduces several improvements which are crucial to achieve impressive performance on a variety of datasets. These improvements are: using a new strategy for cluster segmentation for better alignment with the phonemes, matching the number of silences for the GAN generator. Additionally, this work uses a lightweight GAN. A distinguishing features of this work comparing to prior ones is that it proposes and uses an unsupervised metric for the development set for the model selection. 

The experiments include several popular datasets such as LibriSpeech, TIMIT, as well as low resource language datasets for Tatar, Kyrgyz, Swahili. The proposed method shows performance comparable to some supervised methods and superior to previous unsupervised ones.
main_review: 
# Originality

The paper builds upon the prior works, but it introduces several distinguishing features that are crucial for the results. 

# Quality

The model described thoroughly and the architecture choices motivated well. The experiments are well motivated too. The paper shows that the experimentation is conducted rigorously. Specifically, the setup for all the datasets is described fully and should be possible to replicate. It is especially valuable that the paper conducts experiments on the low resource languages. This is where the unsupervised approach can make the difference.

# Clarity

The paper is very well written. Every section has a clear motivation. Then, the writing is simple and concise, but conveys all the information.

The only thing I found a bit lacking is that self-training (specifically, wav2vec-U + ST) is just mentioned briefly. This model seems to perform best and it makes sense to dedicate some space to describe it in details.

# Significance

In my opinion, this paper is highly significant for both speech recognition and unsupervised training communities. It is known that the unsupervised training for structured prediction (sequence-to-sequence) is hard. The speech recognition is a hard and competitive task as well. This paper can make a significant impact in both areas.

EDIT: Rebuttal acknowledged
limitations_and_societal_impact: This work has a positive social impact as it helps transcription of low-resource languages.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an unsupervised training method for speech recognition, using unlabeled audio and corresponding text. The method makes use of adversarial training, and previously proposed unsupervised learned audio representations with wav2vec 2.0.

main_review: This work builds on a lot of existing work but extends and varies it.

Many experiments have been done including ablation studies.

How to get from phonemes (or what else?) to chars/words in the Librispeech model?
What exactly is the final model to perform speech recognition? How is recognition done?
Does it use a predefined phoneme set including the lexicon (phone to word mapping) in recognition?

Why does it need silence removal if anyway some sil is kept, and sil is explicitly modeled?

Why is PCA done here? Why does it need the segmentation?

The results look very nice, and demonstrate that this is an important contribution to this area.

limitations_and_societal_impact: None that I know.
review_title: null

Meta-review: This work was recommended for acceptance by all reviewers. They praised the thoroughness and replicability of the experimental setup, as well as the significance of the results. This work has the potential to strongly impact speech recognition for low-resource languages, and constitutes a step change in our ability to build practically usable speech recognition systems.
Decision: Accept (Oral)