Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new experimental benchmark for ASR based on neural architecture search (NAS). NAS becomes one of the important machine learning/deep learning areas and has been widely studied in image classification and NLP tasks. This paper follows this trend and provides a NAS benchmark for ASR by using TIMIT. The paper also has a number of experiments by changing the neural network architectures and shows a lot of interesting findings. The paper is well written overall. 

Strengths: 
1) Providing a NAS platform for ASR
2) A lot of analysis in terms of the various architectural configurations and NAS algorithms
3) A discussion of applying such methods to the other database (librispeech). 

Weaknesses:
1) The problem is too specific to ASR. It may not gain much attention from general machine learning researchers in ICLR 
2) Not so much technical or algorithmic novelty, although I appreciate the authors' efforts for this new benchmark.
3) TIMIT is not a public/downloadable corpus, and its access is limited. I recommend the authors to try other easily accessible corpora (e.g., CMU an4 or "mini" Librispeech). 
4) The analysis is mainly based on one corpus (TIMIT), and I'm not very sure about the finding and discussions are applicable to the other database.
5) No survey about the architecture search efforts in ASR. There is a lot of literature about NAS, evolution algorithm, a black-box search of ASR architectures.
6) The optimization hyper-parameters and input feature configurations should be considered as one of the search configurations. The architectures, input feature configurations, and optimization hyper-parameters are highly correlated.

Other comments
- Does the TIMIT experiment include a phone bi-gram model? This is a standard experimental setup.
- Figure 2 is an interesting finding. The paper says that it is different behavior compared with image classification benchmarks. Please discuss it by referring to the report about the image classification benchmarks.
review_title: NAS benchmark for ASR

final_score: rating: 7: Good paper, accept
review_content: The paper presents a study on neural architectural search on speech recognition. An new approach for NAS using convolutional models on speech is presented. Using the TIMIT dataset, ~8k different architectures are trained and evaluated. A study shows that the findings on TIMIT transfer well to a large-scale dataset, Librispeech.

Pros:
- The study is clearly novel, As far as I can tell this is the first NAS paper on speech.
- The finding on the transferability between TIMIT and Librispeech is significant.
- The paper is well motivated and well situated in the literature.

Cons:
- The databases selected makes the findings and the models limited to one domain, hence limiting the significance of the paper.

Detailed comments:
- My main concern with the paper is the choice of corpora: TIMIT and Librispeech are commonly used in ASR studies, but they are both composed on only clean, read speech in English. There is no way to know if the TIMIT results also transfer to another type of speech (conversational, acted, etc), to other recording conditions or to another language. Hence, the presented findings and models are only useful to ASR research focusing on this particular domain. I would encourage the authors to plan more studies with varied data as future work.

Overall, I think the paper is a very good first step towards more NAS-benchmark studies for speech, hence I vote for acceptance despite its limitation in terms of domain.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Motivation and summary:
A lot of research works have been done for NAS-benchmark in the domain of computer vision (and also NLP). This paper introduces a NAS-benchmark dataset for ASR. The authors release a NAS-Bench dataset which would benefit both ASR model architecture search and reproducible NAS research.
This is an interesting and pioneer work in the ASR domain. The NAS-Bench-ASR is built upon TIMIT, a relatively small corpus for speech phonetic recognition. Besides the careful analysis on the designed NAS dataset, the authors also evaluated a few NAS algorithms, and showed that good cell structures identified on the TIMIT dataset aligned with some existing convolutional ASR models in the literature, and can be transferred to Libirspeech. I have below questions/concerns:

Regarding the design of the macro-Architecture:
Why the design only uses unidirectional, rather than bidirectional LSTM? Is the paper focusing on on-device deployment?

Regarding the performance on TIMIT:
The best performance on TIMIT reported in the paper (PER 18.91 on validation set and 21.05 on test set) has clear gap compared to numbers reported in the literature after year of 2013. Basically, simple stacked bi-directional LSTM CTC recognizers should be able to achieve clearly lower validation/test set PER (on 39 Phones) than numbers reported in the paper.
I understand that achieving low PER on TIMIT (and low WER on LibriSpeech) is not the goal of this paper. However, the goal of NAS search is to search for competitive (or even state-of-the-art architectures) for using and future research. While the NAS-BENCH-ASR dataset is built upon TIMIT, it would be more convincing if stronger models with better validation/test PER are identified.

Regarding early stopping:
According to the last paragraph in page 4, it seems that the authors logged the validation loss/PER for each epoch, but only logged the test metrics at the end of the training. So, the test performance is not coming from the best epoch, this might have some effect to the validation/test performance correlation study.

Regarding transferability:
The authors claimed the transferability of the identified cell structure. The evidence is the correlation between Librispeech WER and TIMIT PER. To better support this claim, the authors may consider conduct study on some other corpus like SWITCHBOARD.

review_title: Interesting research on NAS in ASR domain

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies neural architecture search for automatic speech recognition. The approach is to first search over small, reusable networks, called cells, and then applies the cells to a template network. The cells are learned with phonetic recognition on TIMIT and validated on letter recognition on LibriSpeech.

The approach strikes a good balance between having a large search space and the computation cost of the search. I will discuss a few weaknesses in detail, but these weaknesses won't be known prior to performing the experiments in the paper.

The presentation of the paper is also done well. I have no trouble following the paper from start to finish.

The weakness of the paper is the absolute PERs on TIMIT and WERs LibriSpeech. The best PER achieved in this paper, 21.1% is quite high in today's standard. In (Graves et al., 2013), the numbers are around 18%. The best WERs achieved in Figure 7 are high in the teens. In (Hsu et al., 2020), the number for training on the 100 hours of LibriSpeech is around 14%.

It is unclear if the paper uses any regularizer at all when training the models. Even adding some amount of dropout would help the final numbers.

The discrepancy between the numbers in the paper and others makes me wonder the search over the cells is a wrong direction to begin with. Maybe it is the things held fixed that play the role of achieving the best numbers. For example, the macro architecture is held fixed, the optimizer is held fixed, the learning rate schedules are more or less fixed. The macro architecture might play a critical role here. Typically, the competitive architectures require many layers of LSTMs instead of one used in the paper. It is quite discouraging that the models, discovered by NAS after spending so much compute, are not competitive to baseline models reported in other papers.


Hybrid Speech Recognition with Deep Bidirectional LSTM
Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed
ASRU, 2013

Semi-Supervised Speech Recognition via Local Prior Matching
Wei-Ning Hsu, Ann Lee, Gabriel Synnaeve, and Awni Hannun
arXiv:2002.10336
review_title: A sound paper with somewhat discouraging results

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors contribute to the NAS literature by presenting a framework that works decently well on small ASR tasks, specifically TIMIT. They make judicious decisions regard the macro and micro cells that are then swept over. They also show that there is some correlation between training for TIMIT and tasks that have more data, such as librispeech. The experiments look to have been done carefully.

My chief issue with the work is how far the results are from sota on any of the tasks. Their NAS search for TIMIT only yields PER of 21.93 on test, 19.55 on val. wav2vec 2.0 [1] gets 8.3 test, 7.4 val. Authors may argue the wav2vec results are pre-trained, and I would argue that the authors should also do that. However, [2] gets 13.8% on timit test with training being from scratch. Similarly, their best librispeech wer is 19, which is _very far_ from sota from the same paper. Even their transfer correlations between TIMIT and LibriSpeech are not very high.

It is challenging to evaluate their results when they are so far from sota. Authors should resubmit their paper with updated results.

[1] Baevski, et al. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. https://arxiv.org/pdf/2006.11477.pdf

[2] Ravanelli, et al. THE PYTORCH-KALDI SPEECH RECOGNITION TOOLKIT. https://arxiv.org/pdf/1811.07453v2.pdf


========================================================================

I thank the authors for their detailed rebuttal. However, their accuracies are still very below sota. Therefore, I am inclined to stick to my original review and rating.
review_title: results not close to sota

Meta-review: Good clarity: a NAS benchmark for ASR and results transferable across datasets. Although this is more specific to speech domain, building such a benchmark for speech is important for general NAS research, especially the papers finds different behaviors compared to image classification benchmarks. 

The main factor for the decision is the clarity and importance for NAS in speech domain. 
Decision: Accept (Poster)