user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Architecture Search for Natural Language Understanding

Paper content:
Neural architecture search (NAS) has made rapid progress incomputervision,wherebynewstate-of-the-artresultshave beenachievedinaseriesoftaskswithautomaticallysearched neural network (NN) architectures.
In contrast, NAS has not made comparable advances in natural language understanding (NLU).
Corresponding to encoder-aggregator meta architecture of typical neural networks models for NLU tasks (Gong et al. 2018), we re-deﬁne the search space, by splittingitinto twoparts:encodersearchspace,andaggregator search space.
Encoder search space contains basic operations such as convolutions, RNNs, multi-head attention and its sparse variants, star-transformers.
Dynamic routing is included in the aggregator search space, along with max (avg) pooling and self-attention pooling.
Our search algorithm is then fulﬁlled via DARTS, a differentiable neural architecture search framework.
We progressively reduce the search space every few epochs, which further reduces the search time and resource costs.
Experiments on ﬁve benchmark data-sets show that, the new neural networks we generate can achieve performances comparable to the state-of-the-art models that does not involve language model pre-training.
Neural architecture search (NAS) has recently attracted intensive attention.
On one hand, promising methodological innovation for NAS have been developed, e.g. the seminal gradient-based NAS approach DARTS (Liu, Simonyan, and Yang 2018) , followed by improvements such as SNAS (Xie et al. 2018 ), P-DARTS , PC-DARTS (Xu et al. 2019) , etc.
On the other hand, NAS has helped to discover better models to for a variety of vision tasks, e.g., image classification (Zoph and Le 2017; Zoph et al. 2017; Cai, Zhu, and Han 2018) , semantic segmentation , object detection (Ghiasi, Lin, and Le 2019) , superresolution (Ahn, Kang, and Sohn 2018) , etc.
For natural language processing tasks, NAS is relatively less studied.
Except for the general methodology-wise innovations NASNet (Zoph and Le 2016) , ENAS (Pham et al. 2018) and DARTS (Liu, Simonyan, and Yang 2018) which pay slight extra effort on searching for new RNN cells on Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org).
All rights reserved.
language modeling (LM) tasks, there is little studies tailored to the NLU task.
One such an example is the evolved transformer (So, Liang, and Le 2019) , which uses the evolutionbased NAS algorithm to search for better transformer architecture for machine translation.
Although state-of-the-art performance has been achieved on 4 machine translation tasks, the computation cost is exceedingly high since they have to evaluate a large number of models.
In fact, NAS has not been fully investigated for a wide variety of fundamental natural language understanding (NLU) tasks, such as classification (e.g. or sentiment analysis), natural language inference (NLI), sequence tagging tasks such as named entity recognition (NER).
Especially, there is no existing work on the effectiveness of one-shot architecture search (Bender et al. 2018 ) methods on NLU tasks, which could also otherwise significantly reduce the search cost as done in vision tasks.
A typical neural network architecture for NLU includes an encoder which contextualizes the embedded text inputs and extracts higher-level features, and an aggregator that aggregates the encoded inputs to a fix-length vector to make a prediction (Gong et al. 2018) .
In terms of encoders, many previous NAS literature restrict the search space to nonlinear maps such as tanh and sigmoid, and the objective to be the discovery of a new recurrent cell to form a new type of recurrent neural network (RNN).
However, other than RNNs, there are many other available encoders, for example, convolutional networks (CNN) (Kim 2014) , and attentionbased model such as transformer (Vaswani et al. 2017) , etc.
In addition, recent works e.g. star-transformer (Guo et al. 2019) have proposed more sparse versions of transformer to reduce the computational complexity and improve the generalization when there is no pre-trained language model.
In addition, as far as we know, there is no existing work on searching for an aggregator.
A collection of aggregators are available (Gong et al. 2018) .
However, one have to choose manually in a trial-and-error fashion.
In this work, we design an encoder search space that contains a rich collection of encoders.
The involved operations include:
i) the zero map and identity map;
ii) the two most commonly used RNNs, LSTM (Hochreiter and Schmidhuber 1997) and GRU (Cho et al. 2014) ;
iii) highway network (Srivastava, Greff, and Schmidhuber 2015) ;
iv) a series of convolutional networks with different kernel sizes;
v) multi-head attention from (Vaswani et al. 2017) ;
vi) startransformer (Guo et al. 2019) and its variants, which will be explained later in the next section.
The combination of encoder operations is searched in a encoder search cell, which is a directed acyclic graph (DAG) of intermediate nodes collected by the encoder operations from the encoder search space.
To further reduce the human designs, we propose to search for a suitable aggregator along with the search of encoder cell via an aggregator search cell which includes max (average) pooling, self-attention pooling and dynamic routing (Gong et al. 2018) .
The aggregator search cell is a DAG with only one step in which the only node is connected to the inputs by a mixture of aggregators.
Our search strategy is mainly based on DARTS (Liu, Simonyan, and Yang 2018) .
To reduce computation cost, we employ a progressive search space reduction strategy similar to P-DARTS .
Experiments are performed on three different kinds of NLU tasks, i.e., text classification, NLI and NER, with 5 benchmark datasets.
For fair comparison, we only compare our results with former state-of-the-art (SOTA) models without large-scale LM pre-training, or any other outside resources like knowledge bases, or any human designed features.
Results have shown that with the help of NAS on our search space, we achieve results that are comparable to the SOTA on these 5 tasks, indicating the effectiveness of NAS in the field of NLU research.
Our work contributes the field by the following aspects:
• We re-define the search space for neural architecture search in NLU tasks, by extending and modifying the encoder search space from the evolved transformer, and define the aggregator search space.
• To the best of our knowledge, we are the first to conduct NAS experiments on NLU tasks such as classification, NLI, NER tasks, with one-shot NAS.
• Our approach achieves the results that are comparable to the state-of-the-art models designed by human experts, on various NLU tasks (classification, NLI, NER), by using neural architecture search over the search space defined above.
In addition, we demonstrate the effectiveness of one-shot architecture search for NLU tasks.
• We propose a modularized version of star-transformer and its variant, thus including a sparse version of transformer into the search space, which is also novel in the literature.
The resulting advantage is that the search cost can be reduced notably and the network's generalization capability can also be improved.
Related Work Recently, a new research field named neural architecture search (NAS) has been drawing more and more attention.
The goal is to find automatic mechanisms for generating new neural architectures to replace conventional handcrafted ones.
Recently, it is widely applied to computer vision tasks, such as image classification (Zoph and Le 2017; Zoph et al. 2017; Cai, Zhu, and Han 2018) , semantic segmentation , object detection (Ghiasi, Lin, and Le 2019) , super-resolution (Ahn, Kang, and Sohn 2018) , etc.
However, NAS is less well studied in the field of natural language understanding (NLU).
Recent works (Zoph and Le 2016; Pham et al. 2018; Liu, Simonyan, and Yang 2018) search new recurrent cells for the language modeling (LM) task on the PennTreebank dataset 1 .
The recurrent cell discovered by (Liu, Simonyan, and Yang 2018) achieves the test perplexity of 56.1, which is competitive with the stateof-the-art model enhanced by a mixture of softmaxes .
The evolved transformer (So, Liang, and Le 2019) applies NAS to discover better versions of the transformer architecture.
Eploying an evolution-based search algorithm, and the vanilla transformer as the initial population, it generates a better transformer architecture that consistently outperform the vanilla transformer on 4 benchmark machine translation tasks.
Our work contributes by going beyond the RNN structure and re-defining the search space to include a richer connection of operations.
Our work is implemented on DARTS (Liu, Simonyan, and Yang 2018) and P-DARTS .
DARTS relaxes the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent.
Due to its simplicity, DARTS has inspired a series follow-up work to improve the search stability and efficiency.
Based on DARTS, P-DARTS ) divides the search process into multiple stages and progressively increase the network depth at the end of each stage.
Our work contributes to the gradient-based NAS (and more generally, one-shot NAS) research by investigating its effectiveness in discovering new NN architectures for a series of NLU tasks.
Our search space design takes advantages of the recent advances in the NLU field.
One of the most import advances in sentence encoding is the application of various self-attention mechanisms, among which the transformer (Vaswani et al. 2017 ) is the most prominent one, which has become ubiquitous in NLU research.
Specifically, the QANet ) modifies the transformer architecture to obtain the first place on the SQuaD leaderboard 2 .
The transformer is powerful due to its multi-head self-attention mechanism, which can well capture the contextual information.
However, the transformer maybe be difficult to train and generalize well on a small or medium sized data-set (Guo et al. 2019 ).
Thus, many other self-attention operations are proposed, e.g., dynamic self-attention (Yoon, Lee, and Lee 2018) and DiSAN (Shen et al. 2018) .
Recently, (Guo et al. 2019) propose the star-transformer, a sparser version of the multi-head attention model, and achieves competitive results on a series of benchmark datasets like SST-1, SNLI, CoNLL2003.
On the aggregation side, an important advancement is the application of capsule networks and dynamic routing policy in text classification Gong et al. 2018) .
Capsule networks can dynamically decide what and how much information need to be transferred from each word to the final encoding of the text sequence, thus achieving better results even with simple encoders (Gong et al. 2018 ).
Our work is built upon these work and contributes by:
i) include some of the most prominent attention based encoders and aggregators into the search space, and experiment on whether NAS can generate new architectures that have competitive results;
ii) we are the first to propose the aggregator search space;
iii) we include a modularized version of the star-transformer and its variant into the search space, thus we are the first to combine the dense and sparse multi-head self-attention operations into the same search space.
Results on SST Results on SST-1 and SST-2 datasets are listed in Table 2 .
On the SST-1, DARTS generate a network architecture (DARTS-SST-1-V0) that performs better than most of the traditional NN models.
Not that the encoder cell of DARTS-SST-1-V0 contains only RNN and CNN operations, but the exact details of combination of different level of features are impossible to design manually.
The best ar- (Le and Mikolov 2014) 48.7 87.8 MT-LSTM (F2S) 49.1 87.2 Tree-LSTM (Tai, Socher, and Manning 2015) 51.0 88.0 CNN-Tensor (Lei, Barzilay, and Jaakkola 2015) 51.2 -BiLSTM + max pooling (Gong et al. 2018) 48.0 87.0 BiLSTM + average pooling (Gong et al. 2018) 46.2 85.2 BiLSTM + self-att (Gong et al. 2018) 48.2 86.4 BiLSTM + dynamic routing (Gong et al. 2018) 50.5 87.6 Emb + self-att (Shen et al. 2018) 48.9 -DiSAN (Shen et al. 2018) 51.7 -BiLSTM + self-att (Yoon, Lee, and Lee 2018) 50.4 88.2 CNN + self-att (Yoon, Lee, and Lee 2018) 50.6 88.3 Dynamic self-att (Yoon, Lee, and Lee 2018) 50.6 88.5 Transformer (Guo et al. 2019) 50 chitecture (DARTS-SST-2-V0) we obtained on the SST-2 dataset involves a star-transformer operation and an identity map.
Note that since (Guo et al. 2019 ) did not provide results on SST-2, we use the code from fastNLP 4 to run the transformer and the original star-transformer on SST-2.
The results given by us are all the average of 10 different runs.
We can see that DARTS-SST-2-V0 can obtain results comparable to the SOTA on SST-2.
We also experiment on the transferability of the learned architectures.
From Table 2 , we can see that DARTS-SST-2-V0 performs worse than DARTS-SST-1-V0 on SST-1 with a significant margin, but DARTS-SST-1-V0 also performs competitively on SST-2.
Results on NLI tasks Among the architecture candidates derived from the search on SciTail, we find that the one obtained by accepting the null operation when it gets the highest score (DARTS-SciTail-V0) performs best.
In addition, this search run gives the average pooling as the aggregator instead of dynamic-routing.
The results are presented in Table 3 : Test accuracy (%) on the SciTail dataset.
Model ACC 600D ESIM 70.6 Decomposable Attention 72.3 DGEM 72.3 AdvEntuRe 79.0 HCRN (Tay, Luu, and Hui 2018) 80.0 DeIsTe (Yin, Schütze, and Roth 2018) 82.1 CAFE (Yin, Schütze, and Roth 2018) 83.3 MIMN 84.0 ConSeqNet 85.2 HBMP (Mihaylov et al. 2018) 86.0 star-transformer (Guo et al. 2019) 79 Table 3 .
DARTS-SciTail-V0 achieves a competitive performance on the test set, outperforming the baseline models such as ESIM and decomposable attention by a large margin.
It also outperforms the results of the star-transformer and transformer even after extensively parameters tuning.
Our model is actually the best one that has no inter-sentence attentions other than the final interaction before the prediction layer, and uses no outside resources, no manually designed features and no extra training mechanism like adversarial training.
As we can see from Figure 5 that, on the MedNLI dataset, the search gives out a architecture (DARTS-MedNLI-V0) that quite resembles the original implementation of the multi-head attention inside the transformer block, except the residual connection is replaced by a sep conv with kernel size 3.
DARTS-MedNLI-V0 performs worse than the original star-transformer, but it is better than the original transformer, and the baseline ESIM and InferSent.
We also look into the transferability between the two task.
We find that although the datasets are from different domains, the architecture searched on one performs comparable on the other.
This paper addresses NAS for a series of NLU tasks.
Corresponding to the encoder-aggregator architecture of typical NN models for NLU (Gong et al. 2018) , we redefine the search space, by splitting it into encoder search space and aggregator search space.
Our search strategy is based on DARTS (Liu, Simonyan, and Yang 2018) and P-DARTS .
Experiments shows that architectures discovered by NAS achieves results that are comparable to the previous SOTA models.
In the further, we would like to investigate one-shot architecture search on more large-scale NLU tasks.

assistant: text: Neural Architecture Search for a series of Natural Language Understanding tasks. Design the search space for NLU tasks. And Apply differentiable architecture search to discover new models