NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition
Authors: Abhinav Mehrotra
Alberto Gil C. P. Ramos
Sourav Bhattacharya
Łukasz Dudziak
Ravichander Vipperla
Thomas Chau
Mohamed S Abdelfattah
Samin Ishtiaq
Nicholas Donald Lane
Venue: ICLR 2021 Poster (2021)
Keywords: NAS
ASR
Benchmark

Abstract: Powered by innovations in novel architecture design, noise tolerance techniques and increasing model capacity, Automatic Speech Recognition (ASR) has made giant strides in reducing word-error-rate over the past decade. ASR models are often trained with tens of thousand hours of high quality speech data to produce state-of-the-art (SOTA) results. Industry-scale ASR model training thus remains computationally heavy and time-consuming, and consequently has attracted little attention in adopting automatic techniques. On the other hand, Neural Architecture Search (NAS) has gained a lot of interest in the recent years thanks to its successes in discovering efficient architectures, often outperforming handcrafted alternatives. However, by changing the standard training process into a bi-level optimisation problem, NAS approaches often require significantly more time and computational power compared to single-model training, and at the same time increase complexity of the overall process. As a result, NAS has been predominately applied to problems which do not require as extensive training as ASR, and even then reproducibility of NAS algorithms is often problematic. Lately, a number of benchmark datasets has been introduced to address reproducibility issues by pro- viding NAS researchers with information about performance of different models obtained through exhaustive evaluation. However, these datasets focus mainly on computer vision and NLP tasks and thus suffer from limited coverage of application domains. In order to increase diversity in the existing NAS benchmarks, and at the same time provide systematic study of the effects of architectural choices for ASR, we release NAS-Bench-ASR – the first NAS benchmark for ASR models. The dataset consists of 8, 242 unique models trained on the TIMIT audio dataset for three different target epochs, and each starting from three different initializations. The dataset also includes runtime measurements of all the models on a diverse set of hardware platforms. Lastly, we show that identified good cell structures in our search space for TIMIT transfer well to a much larger LibriSpeech dataset.

Published as a conference paper at ICLR 2021
NAS-BENCH REPRRUCCILEE NEURAL ARCHITECTURE SEARCH FOR SPEECH RECOGNITION
REPRODUCIBLE
Abhinav Mehrotra ' Alberto Gil C. P. Ramos", Sourav Bhattacharya' it Lukasz Dudziak' 12 Ravichander Vipperla', Thomas Chau', Samin Ishtiaq', Mohamed S. Abdelfattah", and Nicholas D. Lane
SSmsuun Al Center, Cambridge University of Cambridge "Equal contribution {a. la.mehrotral, a. gilgamossooooooo sourav 1. dudziak i J samsung. com
ABSTRACT
Powered by innovations in novel architecture design, noise tolerance techniques and increasing model capacity, Automatic Speech Recognition (ASR) has made giant strides in reducing word-error-rate over the past decade. ASR models are often trained with tens of thousand hours of high quality speech data to produce state-of-the -hrtt-rrt art (SOTA) results. Industry-scale ASR model training thus remains computationally heavy and time-consuming, and consequently has attracted little attention in adopting automatic techniques. On the other hand, Neural Architec- ture Search (NAS) has gained a lot interest in the recent years thanks to its successes in discovering efficient architectures, often outperforming handcrafted alternatives. However, by changing the standard training process into a bi-level optimisation problem, NAS approaches often require significantly computational power compared to single-model training, and at the same time in- crease complexity of the overall process. As result, NAS has been predominately applied to problems which do not require as extensive training as ASR, and even then reproducibility of NAS algorithms is often problematic. Lately, a number of benchmark datasets has been introduced to address reproducibility issues by pro- viding NAS researchers with information about performance of different models obtained through exhaustive evaluation. However, these datasets focus mainly on computer vision and NL. tasks and thus suffer from limited coverage of applica- tion domains. In order to increase diversity in the existing NAS benchmarks. and at the same time provide systematic study of the effects of architectural choices for ASR, we release MAS--ench-ASSR the first NAS benchmark for ASR models. The dataset consists of 8, 242 unique models trained on the TIMIT audio dataset for three different target epochs, and each starting from three different initializa- tions. The dataset also includes runtime measurements of all the models on a diverse set of hardware platforms. Lastly, we show that identified good cell struc- tures in our search space for TIMIT transfer well to much larger LibriSpeech dataset.
time and
INTRODUCTION
Innovations in Deep Neural Network (DNN) architecture design, data augmentation techniques and continuous increase in the amount of available high quality training datasets, resulted in mas- sive reduction in ASR word-error-raa over the past decade [Amodei et al.] 2016 Kim et al. 2019 Park et al. 2019 Synnaeve et al. 2020 However, training ASR models achieve ofate-of-the performance remains challenging as it requires computationally heavy training process, e.g. often thousands GPU-hours are needed for good convergence (Amodei et al. 20166] Kahn et al. 2020
Furthermore, the requirement of hyper-parameter optimizations increases the computational loads in ASR training. Despite the system-level complexities the training procedure, the importance of novel architecture design has proven extremely important in variety of application domains includ- ing ASR (Chiu & Raffel 2018 Pratap et al.) 20020), computer vision [He et al.) (2016] Krizhevsky
al. 2012 and natural-
naturalllanguage language processing (NLP) DDvvlnn al. 2019
Vaswani al. 2017

---

Published as a conference paper at ICLR 2021
However, architecture design is non-trivial task and often depends on years of experience, domain knowledge of the researchers and is driven by empirical successes.
Over the past few years, the deep learning community is witnessing a trend in adopting automatic techniques to find neural network architectures over more traditional hand-designed alternatives. highly successful in discovering state-of-the art architectures in various com- puter vision tasks [Cai et al. 2020 Howard et al. 2019 Lee et al. [020 Real et al.] 2018: Tan et all) 2019 Tan & Le 2019]. However, many of them suffer from high computational demands, requiring a large number of architecture variations to be trained (Zoph & Le 2017]. Furthermore, NAS algo- rithms are often difficult to reproduce by different researchers, mamly due non- standtadd use of training settings, e.g., hyperparameters, and subtle variations in the architecture search spaces (Li & Talwalkar 2019 Sciuto al. 2020]. Recently, a number of attempts have been made to mitigate these problems by releasing various benchmark datasets for the NAS research community (Dong & Yang, 2020; Klyuchnikov et al., 2020) Siems et al., 2020; Ying et al.) 2019]. These datasets usu- ally provide a direct mapping between an architecture variant and its post training performances, which can be used efficiently by a NAS algorithm speeding up the search process and, at the same time, providing common, fully reproducible environment for assessment and comparison of dif- ferent algorithms. Initial attempts of creating benchmark datasets predominantly focus on image classification tasks (with only one existing work targeting NLP the time of this writing), and thus suffer from poor application coverage.
NAS algorithms
We address the lack of coverage problem by introducing a new NAS-benchmak dataset in the domain of ASR, to our best knowledge the very first of its kind. To build the dataset, we have trained 8, 242 unique convolutional neural network architectures on the TIMIT dataset [Garofolo consider convolutional architectures due to their recent successes in the domain of ASR |Pratap al.)
[993]. We
2020 Hannun al. 2019]. Moreover, convolution based architectures are computationally efficient to run on mobile devices, thus favouring real-time on-device deployment. Our dataset contains multiple runs of the entire training procedure of an architecture, spanning three initializations of the network parameters and three target epochs, amounting total of 74, 178 = 8,242 x 3 x 3 training runs. In addition to the per epoch validation and final test metrics, such as Phoneme Error desktop and embedded Rate (PER), and CTC loss, we also provide run-times of the architectures GPUs for varying batch size. Furthermore, we compare a number of NAS algorithms [Zoph & Le 2017 Real et al. 2018 Dudziak et al.) 2020, Li et al. 2017 Li & Talwalkar 2019] our search space, highlighting potential challenges and differences compared to their performances on existing NAS benchmark dataset. Lastly, we show the transferability of the top architecture cells found on TIMIT to a much larger Librispeech dataset [Panayotov et al. 2015).
In summary, the contributions of this paper are:
Design of ASR NAS Search Space. ASR NAS-Bench is first-of-its-kindl search space for convolutional speech models. facilitates the reproducible study of ASR through NAS methods and thus fills important gap in the literature. The associated dataset consists of 8, 242 unique cells and contains validation and test metrics along with model parameters, FLOPs and on-device run-time
Enabling NAS for Large-scale ASR. Prohibitive training times for non-toy ASR datasets, has prevented NAS from strongly influencing the evolution of ASR architecture design. We show that is able to support the discovery of cell structures that generalize even to large- scale datasets like Librispeech key breakthrough. We believe the methodological decisions in this paper will act as a blueprint for future work, where NAS plays a prominent role in ASR design.
ASR
Validating Existing NAS Algorithm Design. Existing understanding of NAS is grossly influ- enced by image-based tasks. By systematically benchmarking popular NAS algorithms, under a rich ASR search space, our findings provide otherwise lacking scientific support for prior results.
2 RELATED WORK
NAS Benchmarks. Ying et al. [019 inroduced NAS-Bench-101 dataset in an attempt address the difficulties in reproducing NAS research. It contains over 400K unique image classification
'The NAS-Bench datasct and the code can be downloaded from httpa:://gtthbb.com thwb.
Abh naw
2

---

Published as a conference paper at ICLR 2021
models trained the CIFARIO dataset. Despite being the biggest NAS dataset yet, not all NAS algorithms can utilize the dataset due to the restrictions it imposes on the maximum number of edges to limit the search space size. To mitigate the limitations, NAS-Benche
was introduced, which contains 15K image classification models and includes more diagnostic data [Dong & Yang. 20200. Concurrently
TAS-Bench-Ishott (Zela et al.. 2020) and NAS-Bench- 301 (Siems et al.) 2020 were introduced. NAS-Bench-Isholt focuses on benchmarking NAS with weight sharing, whereas, NAS-Bench-301 points out the need for surrogate functions for scaling and uses the DARTS search space [Liu al. 2019) with approximately 10" models. Lastly, NAS- Bench-NLPP [Klyuchnikov et al.] [020] contains models with custom recurrent cells, which are used to replace traditional layers like LSTMs. The dataset contains around 14K different architectures
NAS-Bench-200
trained on a language modeling task.
NAS in Audio Modeling. Recently, there has been an increasing interest in applying NAS in speech- related tasks, such as keyword spotting [Mo et al. 2020 Mazzawi et al. 2019]. speaker verifica-
tion (Ding et al., 2020 Quu et al. 2020]. and acoustic scene classification Li 2020]. NAS has
also been applied in ASR He al. 2020 Chen et al.) 2020 Kim al. 2020 Baruwa al. 2019).
For example,(Chan Chen et al. 2020 used a search methodology based on the vanilla DARTS to optimize CNN-based feature extractor, followed by fixed Bi-LSTM module and multiple output heads. The authors evaluated the NAS-discovered models under mono- and multi-lingual settings, using the Full Language Pack from IARPA BABEL Gales al., 2014), showing improvements over VGG-based extractor. In contrast, Kim et al. [2000] considered evolution- based search to optimize a micro-cell used within a transformer architecture. The evaluation was done using English and Ko- rean dataset of approximately 100 hours of speech each, under monolingual setting. Similarly, Hel) al. 2020] used differentiable search, using P-DARTS [Chen et al. 2019] as the base, to optimize convolutional model without the recurrent tail. Unlike the previous approaches, the evaluation is done on Mandarin, while using various training datasets spanning between 170 and 10K hours of speech. Closest to work is the work by Baruwa et al. (2019), where the authors consider both TIMIT and LibriSpeech datasets, but studied them independently rather than using TIMIT as proxy for LibriSpeech. Further differences include variations in search space design, e.g., the au- thors only considered operations arranged in a fixed feed-forward manner, and allowed interleaving convolutions with recurrent blocks. Lastly, current work on NAS for ASR focus mainly on using the search to improve prediction accuracy, however, they often lack solid foundations needed for analysing and reasoning about the overall search process and identifying its limitations. This work presents large-scale study on the effects of architectural changes on ASR models.
3
ASR NAS-BENCH
The main purpose of the ASR NAS-Bench dataset is to provide a direct mapping from an archi- tecture instance in the search-space ($3.T) to training time and final performance metries. The mapping is designed for any NAS algorithm to quickly navigate the architecture space without re- training procedure. For architecture search
quiring the time-consuming and computationally-heen we use the TIMIT dataset |aarofolo et al., [993] and conduct pilot experiment to select suitable hyperparameters used to train 8, 242 unique models.
3.1 ASR ARCHITECTURE SEARCH SPACE
In-line with existing work Liu et al. 2019; Pham al. 2018 and NAS Benchmarks YYng al.
2019 Dong & Yang 2020], we restrict our search to a small feed-forward neural network topology- space, commonly known as cells. We repeat and arrange a chosen cell to construct a predefined macro-architecture, which is then trained on the TIMIT dataset.
Micro-Architeeture. A micro-architecture or a cell, as shown in Figure [a). is represented by a directed acyclic graph (DAG). We consider DAGs with four nodes ,,. with corresponding incident tensors .1....,ta ta and allow two types of edges: main and skip connection edges. A main edge connects two successive nodes T,-1 and T; in the graph shown by the solid line-arrows in Figure a).. A skip connection edge on the other hand, can connect any two nodes T, and T., with the constraint j < and are depicted as dotted line-an inn-arrows in Figure [T]) [a). Each edge ej-; represents an operation on the tensor t,. The tensor t; is computed by summing the results of operations done by all incoming edges on T; (i.e., Ej-r where j < i).
3

---

Published as a conference paper at ICLR 2021
We consider choice of six operations for the main edges: linear operation. four convolution operations distinguished by choices of (kernel size, dilation)
Oulput (T' N)
Cell
Linear
LSTM
{(5, (5, 2), (7, (7, (7, ation, which outputs a tensor of zeros with the same shape and type as its input. Skip connection operations on the other hand, can be either the identity operation or the zero operation. We used L2 kernel regularizer with convolution operations and dropouts with linear operations. Within the convolution operations, the size of the kernels (e.g., or 7) is chosen as a trade-off between the audio context duration and the model size. Similarly two dilation factors are considered (e.g., or 2) to investigate the trade-off between audio context duration and time resolution. As the final step of cell, the value of l4 further passed through a layer normalization to produce the output of the cell. These design choices are made after we have considered the search space that can be used by the vast majority of the NAS algorithms.
and
oper-
Cell
T3
LeyerNorm
HeLU Conv10 [F4, K4, 84)
Main Operations
5x Cell
Linear
ayerNorm HeLU
1x5 Conv
1x5 Conv, dilation
Conv10 (F3, K3, S3)
1x7 Conv
1x7 Conv, dliation 2
4x Cell LeverNorm ReLU Conv1D (F2. K2 S2)
Zeroize
Skip-connection Operations
Identity
3x Cell
Zeroize
LeyerNorm ReLU Cony1D
(T,80)
(b)
Figure I: (a) Example of micro-architectures (cells) trained TIMIT. The architecture space has 8, 242 unique cells. (b) An
Macro-Architecture: One of our focuses in this benchmark is on-device deployment, so the macro- architecture is made up of convolution and unidirec- tional LSTM blocks as they are computationally ef- ficient to computation composed of four blocks followed by a unidirectional LSTM and a linear layer. Indi- vidual blocks composed of convolution layer, an activation layer (ReL.U), a layer normalization and a composition of C; (i = 2, 3. 4) search cells, with the same micro architecture across all the blocks. Each block is parametrized by three parameters F,, K; and S. These are used to define the number of filters P,, the kernel size K; and the stride S; of the convolution layer. Note that F; also the number of filters of the convolution layers inside the search cell. We use the following set of parameters to define the macro macro-architecture architecture while performing all nicro-architecture training on the TIMIT dataset: C1:4 = [3, 4, 5, 6]. FI:4 [600, 800, 1000, 1200]. K1:4 = [8. 8, 8] and S1a = [1, 1,
overview of the macro-architecture
mobile CPUs. The macro architecture, illustrated in Figure lb). is sequential
3.2 TIMIT DATASET
TIMIT [Garofolo et al. 1993 is one of the earliest datasets designed for evaluating and benchmark- ing phoneme ASR systems. It is idcally suited for neural architecture search experiments due to its small size and high quality transcriptions. It comprises of 6, 300 utterances from 630 speakers, amounting to 5.4 hours of speech. We use the standard training partition of 3, 696 utterances from 462 speakers. Following Lee & Hon [1989]. we split the test dataset into test partition, con- sisting of 24 speakers, and a validation partition. Inline with kaldi Timit-s5 recipe, the original 61 phonemes mapped into set of 48 phonemes, which forms the output layer targets for all the models. These 48 phonemes were further folded to set of 39 during evaluations [Lee & Hon 1989). We use 80-dimensional log-mel spectrograms computed over 25 ms sliding window with stride of 10 ms as input features. During training, we employ a curriculum learning strategy [Amodei et al., 2016), where we begin training by iterating twice over audio utterances shorter than 1s, then twice over audio utterances shorter than 2s as a warmup phase followed by training on the entire dataset. For efficiency, we also use a batch bucketing strategy, where a batch size of 64 is used for audio utterances smaller than 2s, and a batch size of 32 is used otherwise. We used CTC beam-search decoder with beam-size of 12. Note, we did not use any language model in our experiments in order to avoid further HPO (e.g., weighting factors) optimization. This also helped to keep the architecture search for Acoustic Model (AM) tractable. All training experiments on TIMIT report the PER validation and test partitions.
4

---

Published as a conference paper at ICLR 2021
Top 1000
Middle 1000
Bottom 1000
22.5
Best Best Validation PER
80
Best validation
20.00
22.0
velidation
70
19.75
21.5
60
19.50
21.0
200
19.25
10.00
18.75
20.0
21.0
21.4
21.6
21.8
22.2
23.20
23:25
23.25
50
70
Final
test
test
Figure 2: Correlation of best validation and final test PER for top, middle and bottom 1, 000 models.
3.3 PILOT EXPERIMENT
As different training procedures can potentially lead to substantially different results (Yig et al, 2019), in this work we employ single general training procedure for all models. Moreover, use the same parameter set, e.g.. F, K, S. learning rate, and decay across all 8, 242 models. In order to select good values of the parameters, we conducted range of training experiments, where we performed a grid-search to find good macro structure parameters and optimizer settings.
Macro Structure Parameter Selection. The macro structure considered in this work has four main blocks (see Figure T(a) and we considered the following variations in the pilot study: fil- ters F {[600, 800, 1000, 1200]. [900, 1100, 1300, 1500], [1200, 1300, 1500, 1700]]. kernel sizes K E [[6, 8, 10, 12], 8, 8, 8]. (10, 10, 10, 10], [12, 12, 12, 12]}. and time reduction via strides S E [[2, 2, 2, 1], 2, [2,
Optimizer Setting for Training. We further explored good ranges of the learning rate, decay factor, and start epoch for an exponential decay learning rate scheduler used in conjunction with the Adam
optimizer. Specifically, we considered the learning rate (LR) E [10 10 5 * 10 10 10=), the
decay factor E [0.9, 0.95, 0.99], and the LR decay start epoch E [5, 10, 15].
Finally, we randomly chose five micro architectures among all possible 8, 242 cells and for each cell we conducted the grid search created a model from a particular macro architecture, which we then trained by selecting a particular optimizer setting on TIMIT. We identified the parametric setting that resulted the lowest PER across all five cells in the experiments and used the same parameters in all latter model training experiments. The best maero structure parameters are presented above (see $3.1). whereas the best IR was 10 and the decay factor and start epoch were: (i) 0.9 and 5 for target epoch 40, (ii) 0.631 and 2 for target epoch 10, and (iii) 0.398 and for target epoch 5 respectively. The decay factor and start epoch are chosen such that the learning rate of the optimizer reaches the same value at the end of the target epoch of training.
the aforementioned parameters. Specifically, for each cell
3.4 TRAINING SETUP
Individual models are trained using TensorFlow-based training pipeline running on a single GPU. We leveraged NVIDIA V100 and P40 GPUs, and decreased training time by increasing throughput via the bucketing strategy based on the audio length. Metrics such as CTC loss and Phoneme Error Rate (PER) for TIMIT were used to measure model performances. Specifically, train and validation metrics were logged at each epoch, and the test metrics were logged at the end of training for the best model (refered to as final test PER). Our dataset contains logs of each of the 8, 242 models trained with three different seeds and for three target epochs (5, 10 and 40), thus generating a total of 74, 178 model training traces. Additionally, we computed the number of parameters and floating point operations (FLOPs) for each of the architectures and measured their latency on two commonly used hardware platforms: Tesla 1080Ti and Jetson Nano.
4 ANALYSIS OF NAS- HAS-BENCH-ASE DATASET
As described in $3.1 our cell search space is constructed by selecting 3 main operations from a set of 6 candidates, and then selecting either identity zeroize operation for the 6 skip connection edges (see Figure 1 a)). This amounts to 13, 824 (6' x 2d) possible instances for the search-space. How-
5

---

Published as a conference paper at ICLR 2021
Table 1: Summary of test ws. validation analysis on NAS-Bench-ASR, including three subgroups. Spearman-correlati between the test and validation performances are also presented.
Test PER (%)
Val. PER (%)
Test- Val
Group
Positions
Corr.
Min.
Max.
Avg.
Min.
Max.
Avg.
Overall
All
20.83
90.28
30.37+36.47
18.71
90.41
28.21
0,8519
Top
1-1000
20.83
22.29
21.87=0.52
18.71
20.54
19.52+0.51
0.2037
Middle
2000-3000
23.19
23.48
23.3640.17
19.91
22.59
21.11
0,1408
Bottom
7000-8000
32.16
84.53
65.03442.68
29.57
84.6
64.52144.21
0.9977
Table Comparison of correlations between test and validation accuracies reported in the existing image classification benchmarks MMS-Bench- NB- 28-Bench-201, and our
NAS-Bench-201
Group
NAS-Bench-101
NAS-Bench-ASR
CIFAR-10
CIFAR-100
ImageNet-16
Overall
0,989
0.993
0.987
0.997
0.8519
Top 1000
0,356
0.855
0.613
0.827
0,2037
Top 5%
0,573
0.826
0.596
0.796
0.141
Top
Bottom
Operation
400
Carvibi
Convi5.d-22
Canvl7)
200
Convi7,
Main Edge
Edge
Edge
Top
Middle
Bottom
1000
Present
500
250
Skip connection
Skip connection
Skip connection
Figure 3: Distribution selected edges for each node (first row) and distribution of skip-connections for possible edges (second row). Top, middle and bottom are grouped by final test PER.
ever, the use of zero operations introduces computational graph-isomorphismn To identify unique architectures, following (Ying et al. 2019; Dudziak et al. 2020). we first identify nodes with zero operations and disconnect them from their neighbors. Then we perform a reachability test and re- move all nodes that can not be reached from the input or output nodes of the cell. Finally, we end up with minimized graph representation, which we hash using an iterative graph invariant ap- proach (Ying) 2019). After accounting for isomorphism and discarding the empty graph from the list of valid models, found 8, 242 unique architectures, out of which 000 are without any operations.
Distribution of Model Performances. Analyzing the model training results we found that the ma- jority of the models have converged and achieved a final test PER within small range. Specifically, 83.3% 869) of the models fall below PER 26%, with the best model reaching 20.83%. The remaining 6.77% of the models form a rather evenly weell-distrrbuued tail, achieving PER values between 26% 100%. Because of the high density of points below 26% PER and overall high range of PER values, we conducted our analysis by considering three subgroups of models: TOP models are the top 1, 000 architectures according to test PER, MIDDLE are models at positions between 2, 000 and 3, 000 in the test PER ranking, and BOTTOM between 7, 000 and 8000.
PER of
Correlation Between Validation and Test PERs. Figure 2 shows the correlation between the test PER and the best validation PER. The results indicate that there is moderate correlation for top models, weak for middle ones, and strong for worst ones. An overall summary also presented in Table I In general, correlation between validation and test accuracy among the top perform- ing models is significantly lower than for image classification benchmarks, which is highlighted in "able 2 can potentially pose additional challenge to NAS algorithms.

---

Published as a conference paper at ICLR 2021
Distribution of Operations. Fig- ure (first row) presents the dis- tribution of selected operations
6680
Positoo in ranking
Position in rerking
6903
among main edges across top, mid- dle and bottom most models. We found that the top models have
2200
2903
the highest number of linear op- erations on the main edges, while the middle and bottom groups have low linear operations but higher convolution operations. Figure 3 (second row) also highlights that the worst models (bottom group) also have the highest selection of skip-conmection.. Figure g further confirms this observations show- ing how significantly worse the models in the search space become as they include less linear layers, or
Linear layers
connections
Figure 4: Ranking of models according their test PER, when they are grouped according to the number of Linear layers (left) and skip-connections (right) used in their designs.
Input
Inpori
Tepui
Comer)
Conv(7.d=2)
Conved)
Inpur
Lawar
Linear
Linear
Comet
more than 2 skip connections.
I invar
Linerr
Interestingly, we found that the mi- cro cells our search space yield- ing the best validation PER is sim- ilar the structure found the state-of-the-art convolution mod- els such as time-depti-separable (TDS) models [Pratap al. 2020]. Figure [ shows the TDS-cell and the micro-cells with best validation and test PERs (from left right). This indicates the potential of conducting NAS on smaller datasets to discover new architectures yielding state-of-the-arr results on larger datasets.
Oupui
Outpur
Cniper
Curput
Figure 5: Cell architectures in search space (from left to right): TDS-like with best validation and test PERs, and one of the efficient pareto points. Note that the TDS-like cell could also have different convolution selected as the first operation.
Latency, Params, FLOPs and PER. Figure@iinvestigates the relationship among test PER, number of parameters and latency of models in the dataset. FLOPS counts are closely correlated to the number of parameters so are not shown for simplicity. Although this figure shows only latency measured on a desktop GPU (NVIDIA GeForce GTX 1080 Ti) with batch size of 32, the NAS- Bench-ASS ASR includes latency measured on different devices with varying batch sizes. Also shown in the figure are 13 Pareto-optimal models, indicated by purple (x) markers. Also, the best models on test and validation dataset are highlighted in red markers. TDS-like models ((Praaa et al.) 2020) achieve test PER between 22.7% and 23.45% and the best validation PER model has test PER of 22.13% There are Pareto-optimal models that achieve much lower latency without significant increase in PER, for instance, the model indicated by violet best test PER model and still has test PER of 21.98%. The distribution of coloured clusters along the y axis suggests that the number of parameters is not a strong factor to determine the accuracy of models. Analogically, models with similar number of parameters can lead to very different latency.
has one-fifth of the latency of the
TRANSFERABILITY
Training an ASR architecture to achieve statc-of-the-art WER is a slow process, as it often requires iterating over tens of thousands hours of speech data, and potentially over various hyper-parameter choices. Even when distributed across multiple GPUs, this process can take days or weeks [Amodei) cet al., 2016, Kahn et al.] 2020]. Since the largest strides in lowering WERs have resulted from new architectures being developed manually by speech experts slowly over decades, it would be beneficial to scale such exploration through NAS. However, performing NAS directly on large seale ASR datasets needed for peak performance is not practical due to significantly more computational costs. An issue which is further exacerbated when one is also interested in architectures yielding lower latency and computational cost.
7

---

Published as a conference paper at ICLR 2021
80
Number of pararreters (M)
100
Best test PER model
70
Librispeech WER
model
models
60
Test PER
80
Parcta-aptimal models
Ffficient Paretn optimaal model
56
30-
30
40
PER (Val)
25
Dest WLl (1anl)
30
20
Best WER (Va
20
10
20
30
40
50
60
70
18
20
22
24
30
Latency [ms]
TIMIT PER
Figure 6: Test PER versus latency, color- coded by the number of parameters. Best models, Pareto-optimal models, and selected TDS-like models are highlighted.
Transfer between TIMIT and Lib- rispeech on validation and test datasets with high validation and test correlation of 0.85 and 0.86, respectively.
Figure 7:
For this reason, we explore how well new architectures found by NAS on TIMIT (5 hours) trans- fer to Librispeech (100 hours). Specifically, we investigate the correlation between the PER of best/random/wort /oostt performing architectures found on TIMIT, and the Label Error Rate (LER) (or WER) obtained when training the same architectures Librispeech. If a high correlation is present, this would mean that one could lower the computational requirements of NAS by running it on a smaller datasets. In our case this would mean 20x speedup in NAS, and in general this would open a new avenue for research, namely on how much one could save by running NAS on different smaller datasets, running NAS on subset of larger dataset extracted by current next generation data summarization techniques.
Librispeech Dataset. I. Librispeech (Panayotov al. 2015 is widely used ASR training and bench- marking database of about 1, 000 hours of speech derived from audiobooks. The training dataset is partitioned into clean-100 100, clean-360 and other-500 sets with the clean partitions exhibiting much higher SNR. The test and dev sets are also partitioned similarly. In our work, we have used clean- 100 partition for training and dev-clean and test-clean for validation and testing. The choice of 100 hours partition is to strike a balance between building decent large scale model, while still keeping the computational cost involved in architecture search experiments plausible. Previous literature indicates that the models trained on 100 hours partition are about 1-1.5% absolute [Panayotov et al., 2015) behind the models trained on 1, 000 hours when evaluated on test-clean set.
Training. Librispeech models were trained with 4 GPUs with Horovod distributed framework. We use 80-dimensional log-mel features computed over 25ms Hamming windows strided by 10ms. Guided by latency considerations, used vocabulary of 780 sub-word tokens output. Model performances are measured by tracking CTC loss, LER, and WER on the validation dataset during training and on test dataset post-training.
Pilot Experiments. Similarly to $3.3) we also conducted a pilot experiment to determine appro- priate macro structure parameters and hyper-parameters promote transferability hetween datasets at a reasonable compute cost, with the micro cell of the best model found on TIMIT dataset (out of 8, 242),, we conducted another grid search but instead on Librispeech dataset. Accordingly, we chose the final number of filters, kernel size, dilation, learning rate, decay factor and start epoch based lowest LER/WER Librispeech.
train the models for Librispeech. To
Correlation Analysis. For a meaningful analysis we selected 100 architectures from our search space based on their TIMIT PER, namely we took the best 20, 70 random, and the worst 10. We then plugged the micro cell architectures into the macro architecture for Librispeech. Finally, we trained these models on Librispeech with the hyper-parameters reported earlier and computed their LERs/WERS. We observe that models that didn't converge for TIMIT also didn't converge for Librispeech. For this reason, when looking at the strength of transferability between TIMIT and Librispeech we considered models that had TIMIT PER lower than 50%. As shown in Figure 7 there is a high correlation between TIMIT PER and Librispeech WER. In particular, we observe 0.85 correlation between validation metries, and 0.86 for test metrics. Thus, it is possible to do indirect NAS for cell micro architecture on Librispeech by first applying NAS on TIMIT, and then reusing the identified cell structures in macro architecture suitable for Librispeech.
8

---

Published as a conference paper at ICLR 2021
PER
45
Aug best text PUR
40
21.2
21.1
REINF
23.0
RE
20.9
BP
20
20.8
100
200
300
400
500
600
700
200
400
800
1000
1200
Trained models
Trained models
Figure 8: Performance of NAS algorithms our ASR search space, where shaded regions mark interquartile ranges. The dashed line represents the best model in the dataset across all seeds.
NAS EXPERIMENTS
Following existing NAS benchmarks, include results from rumning different NAS algorithms
our search space, using the NAS-Bench-ASR to query for accuracy of models. We consider the following standard NAS algorithms: Regularized Evolution (RE) (Real et al., 2018). Hyperband
(HYPB) Li et al. 2017 REINFORCE with an LSTM-based controller (REINF) [Zoph & Le
2017). Binary Predictor (BP) [Duzziak al. 2020], and Random Search (RAND) [i & Talwalkar
2019 We also include an algorithm based on Deep Q-Learning (QLRN), which behaves similarly RE but tries chain mutations (actions) and learns what mutations can eventually lead to better results. We report results as functions of "trained models" to unify notion of cost. Because in the full training setting we trained models for 40 epochs, an algorithm trains a model for fewer epochs than that (e.g. HYPB) we consider the cost of such training to be nı/ 40 of a "trained model", where n is the number of epochs used (0 < n s 40). We run each algorithm 100 times and present results in Figure [8 as average test PER of the best model found (all algorithms were using validation accuracy as their optimization target), and average validation PER of the most recently trained model (after smoothing the curve using exponential moving average with weight 0.9). We found that none of the algorithms were able to consistently find the best model - this is somewhat surprising given the moderate size our search space and the fact that algorithms like RE are able consistently find the best model on TAS-Bench-200 given similar budget. Another interesting observation is related to the weak correlation between validation and test accuracies mentioned the previous sections. By comparing the figures on the right and left sides (Figure 8), we can see that algorithms which tend to better optimize their rewards (validation accuracy) as search progresses are not necessarily better in the test PER objective - this is especially visible in the case of REINFORCE, which significantly outperforms random search if validation accuracy considered, but when evaluated using test PER it presents comparable results. Another surprise is weak performance of Hyperband the only algorithm that has the ability to adjust number of epochs to train models. From our observations it seems that many of models converge relatively good PER values fast (after 5-10 epochs) so we expected Hyperband to be able to leverage this property. However, it seems that in the case of our search space this is not helpful potentially because models might still change relative ordering as they are trained for more epochs despite the fact that they have already achieved decent performance.
CONCLUSIONS
We introduced pensive training/test time evaluation of model performances, while conducting NAS research in the domain of ASR. We trained 8, 242 unique models on the TIMIT dataset for three initializations and three target epochs. In addition to the common training metrics in ASR, e.g., PER, CTC-loss, we also provide information on the number of parameters, FLOPS and latency of running all the mod- els two hardware platforms for varying batch sizes. Model run-times are especially beneficial for NAS researchers interested in optimizing architectures for efficient on-device deployment. We presented comprehensive analysis of our dataset and evaluated the performances of number of NAS algorithms on it. Lastly, we show that good cell structures as identified on the TIMIT dataset transfer well to Librispeech, which paves the way for affordable NAS ASR domain.
first comprehensive dataset for allowing computationally inex-

---

Published as a conference paper at ICLR 2021
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, and et al. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin. In ICML, 2016.
Ahmed Baruwa, Mojeed Abisiga, Ibrahim Gbadegesin, and Afeez Fakunle. Leveraging End-to- End Speech Recognition with Neural Architecture Search. International Journal of Scientific Engineering Research, 10(11), 2019.
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize for efficient deployment. In ICLR, 2020.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019.
Yi-Chen Chen, Jui- Yang Hsu, Theng-KKangg Lee, and Hung yi Lee. DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recognition and Adaptation. In Interspeech, 2020.
Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. In JCLR, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ACL, 2019.
Shaojin Ding, Tianlong Chen, Xinyu Gong, Weiwei Zha, and Zhangyang Wang. AutoSpeech: Neu- ral Architecture Search for Speaker Recognition. In Interspeech, 2020.
Xuanyi Dong and Yi Yang. MAS-Bench-201.. Extending the Scope of Reproducible Neural Archi- tecture Search. In ICLR, 2020.
Lukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. BRP-NAS: Prediction-based NAS using GCNs. In NeurIPS, 2020.
M. Gales, K. Knill, A. Ragni, and Shakti P. Rath. Speech recognition and keyword spotting for low-resource languages: Babel project research at CUED. In SLTU, 2014.
John S. Garofolo, Lori Lamel, William M. Fisher, Jonathan G. Fiscus, David S. Pallett, Nancy L. Dahlgren, and Victor Zue. TIMIT Acoustic Phonetic Continuous Speech Corpus. Linguistic Data Consortium, 1993.
Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert. Sequence-to Soaicce-tt--eeeee Sequence Speech Recog- nition with Time-Depth Separable Convolutions. In Interspeech, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Liqiang He, Dan Su, and Dong Yu. Learned transferable architectures can surpass hand-designed architectures for large scale speech recognition. arXiv preprint arXiv:2008. 11589, 2020.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc Le, Mark Sandler, Bo Chen, Weijun Wang. Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for MobileNetV3. In ICCV, 2019.
Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, E. Mazaré, Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-Light: A Benchmark for ASR with Limited or No Supervision. In ICASSP, 2020.
Jihwan Kim, Jisung Wang, Sangki Kim, and Yeha Lee. Evolved Speech- FreechhTransfommer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition. In Interspeech, 2020.
Kwangyoun Kim, Kyungmin Lee, Dhananjaya Gowda, Junmo Park, Sungsoo Kim, Eunhyang S. Kim, Young- Yoon Lee, Jinsu Yeo, Dachyun Kim, Scokyeong Jung, Jungin Lee, Myoungji Han, and Chanwoo Kim. Attention based on-device streaming speech recognition with large speech corpus. In ASRU, 2019.
10

---

Published as a conference paper at ICLR 2021
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. WAS--ench-NNLP: Neural Architecture Search Benchmark for Natural Lan- guage Processing. arXiv preprint 2.Xt::0000007166 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NearIPS, 2012.
K-F Lee and H-W Hon. Speaker-independent phone recognition using hidden markov models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(11), 1989.
Royson Lee, Lukasz Dudziak, Mohamed Abdelfattah, Stylianos Venieris, Hyeji Kim, Hongkai Wen, and Nicholas Lane. Journey towards tiny perceptual super-resolution. In ECCV, 2020.
Jixiang Li, Chuming Liang, Bo Zhang, Zhao Wang, Fei Xiang, and Xiangxiang Chu. Neural Archi- tecture Search on Acoustic Scene Classification. Interspeech, 2020.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. JMLR, 18(1). 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Hanna Mazzawi, Xavi Gonzalvo, Alcks Kracun, Prashant Sridhar, Niranjan Subrahmanya, Igna- cio Lopez Moreno, Hyun Jin Park, and Patrick Violette. Improving Keyword Spotting and Lan- guage Identification via Neural Architecture Search at Scale. In Interspeech, 2019.
Tong Mo, Yakun Yu, Mohammad Salameh, Di Niu, and Shangling Jui. Neural Architecture Search for Keyword Spotting. In Interspeech, 2020.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP, 2015.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. SpecAugment: simple data augmentation method for Automatic Speech Recogni- tion. In Interspeech, 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient Neural Architecture Search via Parameters Sharing. In ICML, 2018.
Vineel Pratap, Qiantong Xu, Jacob Kahn, Gilad Avidov, Tatiana Likhomanenko, Awmi Hannun, Vi- taliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Scaling up online speech recognition using convnets. In Interspeech, 2020.
Xiaoyang Qu, Jianzong Wang, and Jing Xiao. Evolutionary Algorithm Enhanced Neural Architec- ture Search for Text-Independent Speaker Verification. In Interspeech, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V, Le. Regularized evolution for image classifier architecture search. In AAAI, 2018.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In ICLR, 2020.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. NAS- Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search. arXiv preprint arXiv:2008.0977 2020.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-End ASR: From supervised to semi-super learning with modern architectures. In ICML: Workshop on Self- supervision in Audio and Speech, 2020.
11

---

Published as a conference paper at ICLR 2021
Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-Aware Neural Architecture Search for Mobile. CVPR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Chris Ying. Enumerating unique computational graphs via an iterative graph invariant. arXiv preprint wrXiv:1902.06192 2019.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench- 101: Towards Reproducible Neural Architecture Search. In ICML, 2019.
Arber Zela, Julien Siems, and Frank Hutter. NAS- NAS-Bench- Bench-IShotl: Benchmarking and Dissecting One-shot Neural Architecture Search. In ICLR, 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. ICLR, 2017.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
Our dataset comprises of 8,242 unique models that were trained for three target epochs and each starting with three different initializations. In this appendix we present the results for the models with different initializations and target epochs.
A.1 CORRELATION BETWERN VALIDATION AND TEST PERS FOR THE RUNS WITH DIFFERENT TARGET EPOCHS
In this section present the correlation of validation and test PERs between different target epochs. Specifically, we correlate each run of target epoch 40 (i.e., runs starting with unique initialization and trained for 40 epochs) with the runs for target epoch 5 and 10. This analysis informs if reduced training (i.e., target epoch Table there is weak correlation of target epoch 40 and 5. However, there moderate correlation between target epoch 40 and 10, suggesting that the reduced training might be useful for performing NAS.
or 10) can be used as a good proxy for NAS. As shown in
Table 3: Correlation of validation and test PERs between different target epochs.
E-5
E-10
E-40
Validation
Test
Validation
Test
Run 0.3349
0.3097
0.5493
0.5029
Run 2
0.2986
0.3022
0.5138
0.4793
Run 3
0.3234
0.3362
0.526
0.5112
A.2 CORRELATION BETWEEN VALIDATION AND TEST PERS
In this section we present the correlation between the final test PER and the best validation PER for the three runs for target epoch 40 starting with different initializations, which is extension of results presented in Figure 2 As shown in Figure the results for all seeds are similar. Specifically, there is moderate correlation for top models, weak for middle ones, and strong for worst ones.
A.3 CELL ARCHITECTURES OUR SEARCH SPACE
In this section we present the micro-cell architectures in our search space that achieved best vali- dation and test PERs for the three runs starting with different initializations. Figure 10 shows the micro-cell architectures that achieved best validation PERs (in top row) and best test PERs (in bot- tom row). We observe that the micro-cell architectures top models often start with two convolution layers followed by a linear layer (which is consistent across all architectures).
A.4 IMPACT OF DIFFERENT OPERATIONS-COUNT ON MODEL PERFORMANCE
In this section we present the impact of the number of selected operation on the model performance. As shown in Figure 11 models with
convolution layers, or with no linear layer ranked as worst performing models. At the same time, models with a zero ops in the main edge have poor performance. This indicates that the top performing models comprise of linear and convolution layers. We also observe that an increase the number of skip connections deteriorates performance significantly, with surprisingly two skip connections yielding similar results to the traditional one skip connection, which aims to learn perturbation of the identity rather than generic transformation between input and output nodes.
designed such that all nodes
13

---

Published as a conference paper at ICLR 2021
Top 1000
Middle 1000
Bottom 1000
20.25
Best validation PFR.
Bost validation PES
20.00
Best validaton
20.75
19.50
20.35
21.0
21-4 23.4
21.0
21.8 22.0 22.2
25.00
25.09
23.00
23.05 05
25.45
50
60
Final test PER
Final test PER
Final best PER
Top 1000
Middle 1000
Bettom 1000
20.2
23.75
BEst yallicttion PER
23.50
Best veliceton
Bos valicction
28.8
60
29.4
20.75
10
212
21A
222
25,4023
4503 1O25.75
Final test PER
Final test DESt.
PER
Final test PER
Top 1000
Middle 1000
Bottomm 1000
20.75
Best validation PER
20.50
22.00
Best vollitation PER
20.25
Best validation
23.50
22.00
21.25
00
29.50
20.05
19.25
20.23
21.
22.0 22.2
27.4020
5023 223.0023 7023775 Final best PER
Finel lest PER
Finel test PER
Figure 9: Correlation of best validation and final test PER. Figures in the columns are for top, middle and bottom 1, 000 models. Each row represents the results for runs with unique initialization.
Impurt
Ingat
Input
Liber
Conv(7.d=2)
Canv(7)
Conv(5)
Jinear
Linear
Outpul
Output
Best Val (1)
Best Val (2)
Val (3)
Inpor
Imput
Inpar
Conv(5)
Conve5.d-22
Litear
Conv(5)
Lincar
Osipul
Output
Output
Best Test (1)
Best Test (2)
Best Test (3)
Figure 10: Cell architectures our search space with best validation and test PERs for the three runs.
14

---

Published as a conference paper at ICLR 2021
8000
300
500
600
Notton tarkinng
8000
300
8000
5000
6030
600
2006
5000
No. canvolutional layss
No. Linear layers
No. of Zara layers
No. Skp-cornectione
Posibon tonking
Positoon
Convelution wyees
No. Unear bayers
No. of Zero lnyers
No. Sep-cornections
Poobbnn 21122n
Fectton
layers
byers
loyers
Skip
Figure 11: Positions of models in the ranking according to their test PER, when they grouped according to the number of Convolution layer (first column), Linear layer (second column), Zero op (third column) and Skip-connections (last column). Each correspond to the results for with unique inititialization.
15


Reviews:
[{"review_id": "pSAjgBCpuQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivation and summary:\nA lot of research works have been done for NAS-benchmark in the domain of computer vision (and also NLP). This paper introduces a NAS-benchmark dataset for ASR. The authors release a NAS-Bench dataset which would benefit both ASR model architecture search and reproducible NAS research.\nThis is an interesting and pioneer work in the ASR domain. The NAS-Bench-ASR is built upon TIMIT, a relatively small corpus for speech phonetic recognition. Besides the careful analysis on the designed NAS dataset, the authors also evaluated a few NAS algorithms, and showed that good cell structures identified on the TIMIT dataset aligned with some existing convolutional ASR models in the literature, and can be transferred to Libirspeech. I have below questions/concerns:\n\nRegarding the design of the macro-Architecture:\nWhy the design only uses unidirectional, rather than bidirectional LSTM? Is the paper focusing on on-device deployment?\n\nRegarding the performance on TIMIT:\nThe best performance on TIMIT reported in the paper (PER 18.91 on validation set and 21.05 on test set) has clear gap compared to numbers reported in the literature after year of 2013. Basically, simple stacked bi-directional LSTM CTC recognizers should be able to achieve clearly lower validation/test set PER (on 39 Phones) than numbers reported in the paper.\nI understand that achieving low PER on TIMIT (and low WER on LibriSpeech) is not the goal of this paper. However, the goal of NAS search is to search for competitive (or even state-of-the-art architectures) for using and future research. While the NAS-BENCH-ASR dataset is built upon TIMIT, it would be more convincing if stronger models with better validation/test PER are identified.\n\nRegarding early stopping:\nAccording to the last paragraph in page 4, it seems that the authors logged the validation loss/PER for each epoch, but only logged the test metrics at the end of the training. So, the test performance is not coming from the best epoch, this might have some effect to the validation/test performance correlation study.\n\nRegarding transferability:\nThe authors claimed the transferability of the identified cell structure. The evidence is the correlation between Librispeech WER and TIMIT PER. To better support this claim, the authors may consider conduct study on some other corpus like SWITCHBOARD.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ggqKgdRtRuU", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new experimental benchmark for ASR based on neural architecture search (NAS). NAS becomes one of the important machine learning/deep learning areas and has been widely studied in image classification and NLP tasks. This paper follows this trend and provides a NAS benchmark for ASR by using TIMIT. The paper also has a number of experiments by changing the neural network architectures and shows a lot of interesting findings. The paper is well written overall. \n\nStrengths: \n1) Providing a NAS platform for ASR\n2) A lot of analysis in terms of the various architectural configurations and NAS algorithms\n3) A discussion of applying such methods to the other database (librispeech). \n\nWeaknesses:\n1) The problem is too specific to ASR. It may not gain much attention from general machine learning researchers in ICLR \n2) Not so much technical or algorithmic novelty, although I appreciate the authors' efforts for this new benchmark.\n3) TIMIT is not a public/downloadable corpus, and its access is limited. I recommend the authors to try other easily accessible corpora (e.g., CMU an4 or \"mini\" Librispeech). \n4) The analysis is mainly based on one corpus (TIMIT), and I'm not very sure about the finding and discussions are applicable to the other database.\n5) No survey about the architecture search efforts in ASR. There is a lot of literature about NAS, evolution algorithm, a black-box search of ASR architectures.\n6) The optimization hyper-parameters and input feature configurations should be considered as one of the search configurations. The architectures, input feature configurations, and optimization hyper-parameters are highly correlated.\n\nOther comments\n- Does the TIMIT experiment include a phone bi-gram model? This is a standard experimental setup.\n- Figure 2 is an interesting finding. The paper says that it is different behavior compared with image classification benchmarks. Please discuss it by referring to the report about the image classification benchmarks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aTrYxVkMOgo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors contribute to the NAS literature by presenting a framework that works decently well on small ASR tasks, specifically TIMIT. They make judicious decisions regard the macro and micro cells that are then swept over. They also show that there is some correlation between training for TIMIT and tasks that have more data, such as librispeech. The experiments look to have been done carefully.\n\nMy chief issue with the work is how far the results are from sota on any of the tasks. Their NAS search for TIMIT only yields PER of 21.93 on test, 19.55 on val. wav2vec 2.0 [1] gets 8.3 test, 7.4 val. Authors may argue the wav2vec results are pre-trained, and I would argue that the authors should also do that. However, [2] gets 13.8% on timit test with training being from scratch. Similarly, their best librispeech wer is 19, which is _very far_ from sota from the same paper. Even their transfer correlations between TIMIT and LibriSpeech are not very high.\n\nIt is challenging to evaluate their results when they are so far from sota. Authors should resubmit their paper with updated results.\n\n[1] Baevski, et al. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. https://arxiv.org/pdf/2006.11477.pdf\n\n[2] Ravanelli, et al. THE PYTORCH-KALDI SPEECH RECOGNITION TOOLKIT. https://arxiv.org/pdf/1811.07453v2.pdf\n\n\n========================================================================\n\nI thank the authors for their detailed rebuttal. However, their accuracies are still very below sota. Therefore, I am inclined to stick to my original review and rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "994e4PAoyGE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a study on neural architectural search on speech recognition. An new approach for NAS using convolutional models on speech is presented. Using the TIMIT dataset, ~8k different architectures are trained and evaluated. A study shows that the findings on TIMIT transfer well to a large-scale dataset, Librispeech.\n\nPros:\n- The study is clearly novel, As far as I can tell this is the first NAS paper on speech.\n- The finding on the transferability between TIMIT and Librispeech is significant.\n- The paper is well motivated and well situated in the literature.\n\nCons:\n- The databases selected makes the findings and the models limited to one domain, hence limiting the significance of the paper.\n\nDetailed comments:\n- My main concern with the paper is the choice of corpora: TIMIT and Librispeech are commonly used in ASR studies, but they are both composed on only clean, read speech in English. There is no way to know if the TIMIT results also transfer to another type of speech (conversational, acted, etc), to other recording conditions or to another language. Hence, the presented findings and models are only useful to ASR research focusing on this particular domain. I would encourage the authors to plan more studies with varied data as future work.\n\nOverall, I think the paper is a very good first step towards more NAS-benchmark studies for speech, hence I vote for acceptance despite its limitation in terms of domain.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4HcPPbtXgnp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies neural architecture search for automatic speech recognition. The approach is to first search over small, reusable networks, called cells, and then applies the cells to a template network. The cells are learned with phonetic recognition on TIMIT and validated on letter recognition on LibriSpeech.\n\nThe approach strikes a good balance between having a large search space and the computation cost of the search. I will discuss a few weaknesses in detail, but these weaknesses won't be known prior to performing the experiments in the paper.\n\nThe presentation of the paper is also done well. I have no trouble following the paper from start to finish.\n\nThe weakness of the paper is the absolute PERs on TIMIT and WERs LibriSpeech. The best PER achieved in this paper, 21.1% is quite high in today's standard. In (Graves et al., 2013), the numbers are around 18%. The best WERs achieved in Figure 7 are high in the teens. In (Hsu et al., 2020), the number for training on the 100 hours of LibriSpeech is around 14%.\n\nIt is unclear if the paper uses any regularizer at all when training the models. Even adding some amount of dropout would help the final numbers.\n\nThe discrepancy between the numbers in the paper and others makes me wonder the search over the cells is a wrong direction to begin with. Maybe it is the things held fixed that play the role of achieving the best numbers. For example, the macro architecture is held fixed, the optimizer is held fixed, the learning rate schedules are more or less fixed. The macro architecture might play a critical role here. Typically, the competitive architectures require many layers of LSTMs instead of one used in the paper. It is quite discouraging that the models, discovered by NAS after spending so much compute, are not competitive to baseline models reported in other papers.\n\n\nHybrid Speech Recognition with Deep Bidirectional LSTM\nAlex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed\nASRU, 2013\n\nSemi-Supervised Speech Recognition via Local Prior Matching\nWei-Ning Hsu, Ann Lee, Gabriel Synnaeve, and Awni Hannun\narXiv:2002.10336", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Good clarity: a NAS benchmark for ASR and results transferable across datasets. Although this is more specific to speech domain, building such a benchmark for speech is important for general NAS research, especially the papers finds different behaviors compared to image classification benchmarks. 

The main factor for the decision is the clarity and importance for NAS in speech domain. 
Decision: Accept (Poster)