NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition

1 INTRODUCTION . Innovations in Deep Neural Network ( DNN ) architecture design , data augmentation techniques and a continuous increase in the amount of available high quality training datasets , resulted in a massive reduction in ASR word-error-rate over the past decade [ Amodei et al. , 2016 ; Kim et al. , 2019 ; Park et al. , 2019 ; Synnaeve et al. , 2020 ] . However , training ASR models to achieve state-of-the-art performance remains challenging as it requires computationally heavy training process , e.g. , often thousands of GPU-hours are needed for good convergence [ Amodei et al. , 2016 ; Kahn et al. , 2020 ] . Furthermore , the requirement of hyper-parameter optimizations increases the computational loads in ASR training . Despite the system-level complexities in the training procedure , the importance of novel architecture design has proven extremely important in a variety of application domains including ASR [ Chiu & Raffel , 2018 ; Pratap et al. , 2020 ] , computer vision [ He et al. , 2016 ; Krizhevsky et al. , 2012 ] , and natural-language processing ( NLP ) [ Devlin et al. , 2019 ; Vaswani et al. , 2017 ] . However , architecture design is a non-trivial task and often depends on years of experience , domain knowledge of the researchers and is driven by empirical successes . Over the past few years , the deep learning community is witnessing a trend in adopting automatic techniques to find neural network architectures over more traditional hand-designed alternatives . NAS algorithms are highly successful in discovering state-of-the-art architectures in various computer vision tasks [ Cai et al. , 2020 ; Howard et al. , 2019 ; Lee et al. , 2020 ; Real et al. , 2018 ; Tan et al. , 2019 ; Tan & Le , 2019 ] . However , many of them suffer from high computational demands , requiring a large number of architecture variations to be trained [ Zoph & Le , 2017 ] . Furthermore , NAS algorithms are often difficult to reproduce by different researchers , mainly due to a non-standard use of training settings , e.g. , hyperparameters , and subtle variations in the architecture search spaces [ Li & Talwalkar , 2019 ; Sciuto et al. , 2020 ] . Recently , a number of attempts have been made to mitigate these problems by releasing various benchmark datasets for the NAS research community [ Dong & Yang , 2020 ; Klyuchnikov et al. , 2020 ; Siems et al. , 2020 ; Ying et al. , 2019 ] . These datasets usually provide a direct mapping between an architecture variant and its post training performances , which can be used efficiently by a NAS algorithm speeding up the search process and , at the same time , providing common , fully reproducible environment for assessment and comparison of different algorithms . Initial attempts of creating benchmark datasets predominantly focus on image classification tasks ( with only one existing work targeting NLP at the time of this writing ) , and thus suffer from poor application coverage . We address the lack of coverage problem by introducing a new NAS-benchmak dataset in the domain of ASR , to our best knowledge the very first of its kind . To build the dataset , we have trained 8 , 242 unique convolutional neural network architectures on the TIMIT dataset [ Garofolo et al. , 1993 ] . We consider convolutional architectures due to their recent successes in the domain of ASR [ Pratap et al. , 2020 ; Hannun et al. , 2019 ] . Moreover , convolution based architectures are computationally efficient to run on mobile devices , thus favouring real-time on-device deployment . Our dataset contains multiple runs of the entire training procedure of an architecture , spanning three initializations of the network parameters and three target epochs , amounting a total of 74 , 178 = 8 , 242 × 3 × 3 training runs . In addition to the per epoch validation and final test metrics , such as Phoneme Error Rate ( PER ) , and CTC loss , we also provide run-times of the architectures on desktop and embedded GPUs for varying batch size . Furthermore , we compare a number of NAS algorithms [ Zoph & Le , 2017 ; Real et al. , 2018 ; Dudziak et al. , 2020 ; Li et al. , 2017 ; Li & Talwalkar , 2019 ] on our search space , highlighting potential challenges and differences compared to their performances on existing NAS benchmark dataset . Lastly , we show the transferability of the top architecture cells found on TIMIT to a much larger Librispeech dataset [ Panayotov et al. , 2015 ] . In summary , the contributions of this paper are : • Design of ASR NAS Search Space . ASR NAS-Bench is a first-of-its-kind search space for convolutional speech models . It facilitates the reproducible study of ASR through NAS methods and thus fills an important gap in the literature . The associated dataset consists of 8 , 242 unique cells and contains validation and test metrics along with model parameters , FLOPs and on-device run-times1 . • Enabling NAS for Large-scale ASR . Prohibitive training times for non-toy ASR datasets , has prevented NAS from strongly influencing the evolution of ASR architecture design . We show that ASR NAS-Bench is able to support the discovery of cell structures that generalize even to largescale datasets like Librispeech – a key breakthrough . We believe the methodological decisions in this paper will act as a blueprint for future work , where NAS plays a prominent role in ASR design . • Validating Existing NAS Algorithm Design . Existing understanding of NAS is grossly influenced by image-based tasks . By systematically benchmarking popular NAS algorithms , under a rich ASR search space , our findings provide otherwise lacking scientific support for prior results . 2 RELATED WORK . NAS Benchmarks . Ying et al . [ 2019 ] inroduced NAS-Bench-101 dataset in an attempt to address the difficulties in reproducing NAS research . It contains over 400K unique image classification 1The NAS-Bench-ASR dataset and the code can be downloaded from https : //github.com/ AbhinavMehrotra/nb-asr . models trained on the CIFAR10 dataset . Despite being the biggest NAS dataset yet , not all NAS algorithms can utilize the dataset due to the restrictions it imposes on the maximum number of edges to limit the search space size . To mitigate the limitations , NAS-Bench-201 was introduced , which contains 15K image classification models and includes more diagnostic data [ Dong & Yang , 2020 ] . Concurrently to NAS-Bench-201 , NAS-Bench-1shot1 [ Zela et al. , 2020 ] and NAS-Bench301 [ Siems et al. , 2020 ] were introduced . NAS-Bench-1shot1 focuses on benchmarking NAS with weight sharing , whereas , NAS-Bench-301 points out the need for surrogate functions for scaling and uses the DARTS search space [ Liu et al. , 2019 ] with approximately 1018 models . Lastly , NASBench-NLP [ Klyuchnikov et al. , 2020 ] contains models with custom recurrent cells , which are used to replace traditional layers like LSTMs . The dataset contains around 14K different architectures trained on a language modeling task . NAS in Audio Modeling . Recently , there has been an increasing interest in applying NAS in speechrelated tasks , such as keyword spotting [ Mo et al. , 2020 ; Mazzawi et al. , 2019 ] , speaker verification [ Ding et al. , 2020 ; Qu et al. , 2020 ] , and acoustic scene classification [ Li et al. , 2020 ] . NAS has also been applied in ASR [ He et al. , 2020 ; Chen et al. , 2020 ; Kim et al. , 2020 ; Baruwa et al. , 2019 ] . For example , Chen et al . [ 2020 ] used a search methodology based on the vanilla DARTS to optimize a CNN-based feature extractor , followed by a fixed Bi-LSTM module and multiple output heads . The authors evaluated the NAS-discovered models under mono- and multi-lingual settings , using the Full Language Pack from IARPA BABEL [ Gales et al. , 2014 ] , showing improvements over a VGG-based extractor . In contrast , Kim et al . [ 2020 ] considered evolution-based search to optimize a micro-cell used within a transformer architecture . The evaluation was done using English and Korean dataset of approximately 100 hours of speech each , under monolingual setting . Similarly , He et al . [ 2020 ] used differentiable search , using P-DARTS [ Chen et al. , 2019 ] as the base , to optimize a convolutional model without the recurrent tail . Unlike the previous approaches , the evaluation is done on Mandarin , while using various training datasets spanning between 170 and 10K hours of speech . Closest to our work is the work by Baruwa et al . [ 2019 ] , where the authors consider both TIMIT and LibriSpeech datasets , but studied them independently rather than using TIMIT as a proxy for LibriSpeech . Further differences include variations in search space design , e.g. , the authors only considered operations arranged in a fixed feed-forward manner , and allowed interleaving convolutions with recurrent blocks . Lastly , current work on NAS for ASR focus mainly on using the search to improve prediction accuracy , however , they often lack solid foundations needed for analysing and reasoning about the overall search process and identifying its limitations . This work presents a large-scale study on the effects of architectural changes on ASR models . 3 ASR NAS-BENCH . The main purpose of the ASR NAS-Bench dataset is to provide a direct mapping from an architecture instance in the search-space ( §3.1 ) to its training time and final performance metrics . The mapping is designed for any NAS algorithm to quickly navigate the architecture space without requiring the time-consuming and computationally-heavy training procedure . For architecture search we use the TIMIT dataset [ Garofolo et al. , 1993 ] and conduct a pilot experiment to select suitable hyperparameters used to train 8 , 242 unique models . 3.1 ASR ARCHITECTURE SEARCH SPACE . In-line with existing work [ Liu et al. , 2019 ; Pham et al. , 2018 ] and NAS Benchmarks [ Ying et al. , 2019 ; Dong & Yang , 2020 ] , we restrict our search to a small feed-forward neural network topologyspace , commonly known as cells . We repeat and arrange a chosen cell to construct a predefined macro-architecture , which is then trained on the TIMIT dataset . Micro-Architecture . A micro-architecture or a cell , as shown in Figure 1 ( a ) , is represented by a directed acyclic graph ( DAG ) . We consider DAGs with four nodes T1 , . . . , T4 with corresponding incident tensors t1 , . . . , t4 and allow two types of edges : main and skip connection edges . A main edge connects two successive nodes Ti−1 and Ti in the graph as shown by the solid line-arrows in Figure 1 ( a ) . A skip connection edge on the other hand , can connect any two nodes Tj and Ti , with the constraint j < i and are depicted as dotted line-arrows in Figure 1 ( a ) . Each edge ej→i represents an operation on the tensor tj . The tensor ti is computed by summing the results of operations done by all incoming edges on Ti ( i.e. , ej→i , where j < i ) . We consider a choice of six operations for the main edges : linear operation , four convolution operations distinguished by choices of ( kernel size , dilation ) ∈ { ( 5 , 1 ) , ( 5 , 2 ) , ( 7 , 1 ) , ( 7 , 2 ) } , and a zero operation , which outputs a tensor of zeros with the same shape and type as its input . Skip connection operations on the other hand , can be either the identity operation or the zero operation . We used L2 kernel regularizer with convolution operations and dropouts with linear operations . Within the convolution operations , the size of the kernels ( e.g. , 5 or 7 ) is chosen as a trade-off between the audio context duration and the model size . Similarly two dilation factors are considered ( e.g. , 1 or 2 ) to investigate the trade-off between audio context duration and time resolution . As the final step of a cell , the value of t4 is further passed through a layer normalization to produce the output of the cell . These design choices are made after we have considered the search space that can be used by the vast majority of the NAS algorithms . Macro-Architecture . One of our focuses in this benchmark is on-device deployment , so the macroarchitecture is made up of convolution and unidirectional LSTM blocks as they are computationally ef- ficient to run on mobile CPUs . The macro architecture , illustrated in Figure 1 ( b ) , is a sequential computation composed of four blocks followed by a unidirectional LSTM and a linear layer . Individual blocks are composed of a convolution layer , an activation layer ( ReLU ) , a layer normalization and a composition of Ci ( i = 1 , 2 , 3 , 4 ) search cells , with the same micro architecture across all the blocks . Each block is parametrized by three parameters Fi , Ki and Si . These are used to define the number of filters Fi , the kernel size Ki and the stride Si of the convolution layer . Note that Fi is also the number of filters of the convolution layers inside the search cell . We use the following set of parameters to define the macro-architecture while performing all micro-architecture training on the TIMIT dataset : C1∶4 = [ 3 , 4 , 5 , 6 ] , F1∶4 = [ 600 , 800 , 1000 , 1200 ] , K1∶4 = [ 8 , 8 , 8 , 8 ] and S1∶4 = [ 1 , 1 , 2 , 2 ] .

Summary: The authors contribute to the NAS literature by presenting a framework that works decently well on small ASR tasks, specifically TIMIT. They make judicious decisions regard the macro and micro cells that are then swept over. They also show that there is some correlation between training for TIMIT and tasks that have more data, such as librispeech. The experiments look to have been done carefully.