SlowGuess/ACL_2020_OCR huggingface dataset:
Also, we find that searching for the appropriate connections among cells plays a more important role in improving the model performance. We observe that the intra-cell NAS (DARTS) system underperforms the inter-cell counterpart with the same number of parameters. It is because the well-designed intra-cell architectures (e.g., Mogrifier-LSTM) are actually competitive with the NAS structures. However, the fragile connections among different cells greatly restrict the representation space. The additional inter-cell connections are able to encode much richer context.
Nevertheless, our ESS method does not defeat the manual designed Transformer-XL model on the WikiText-103 dataset, even though ESS works better than other RNN-based NAS methods. This is partially due to the better ability of Transformer-XL to capture the language representation. Note that RNNs are not good at modeling the long-distance dependence even if more history states are considered. It is a good try to apply ESS to Transformer but this is out of the scope of this work.
To modulate the complexity of the intra and intercell, we study the system behaviors under different numbers of intermediate nodes (Figure 5). Fixing the number of model parameters, we compare these systems under different numbers of the intra and inter-cell nodes. Due to the limited space, we show the result on the PTB in the following sensitivity analysis. We observe that an appropriate choice of node number (8 nodes for intra-cell and 3 nodes for inter-cell) brings a consistent improvement. More interestingly, we find that too many nodes for inter-cell architecture do not improve the model representation ability. This is reasonable
Figure 6: Perplexity on the validation data (PTB) and Mean Absolute Deviation (MAD) between edge weights and uniform distribution vs. number of training steps.
Word	Count	Δloss	Word	Count	Δloss
mcmoran	11	-0.74	the	59421	-0.009
cie.	9	-0.66	<unk>	53299	-0.004
mall	13	-0.65	<eos>	49199	-0.010
missile	23	-0.55	N	37607	-0.008
siemens	12	-0.51	of	28427	-0.008
baldwin	9	-0.51	to	27430	-0.004
nfl	21	-0.49	a	24755	-0.013
prime-time	17	-0.47	in	21032	-0.015
Table 3: Difference in word loss (normalized by word counts) on validation data when searching intra and inter-cell jointly. The left column contains the words with eight best improvements (larger absolute value of  $\Delta$ loss) and right column presents the most frequent words in the validation data.
because more inter-cell nodes refer to considering more history in our system. But for language modeling, the current state is more likely to be relevant to most recent words. Too many inputs to the gate networks raise difficulties in modeling.
We observe that our ESS method leads to a model that is easier to train. The left part in Figure 6 plots the validation perplexity at different training steps. The loss curve of joint ESS significantly goes down as the training proceeds. More interestingly, our joint learning method makes the model achieve a lower perplexity than the intra-cell NAS system. This indicates better networks can be obtained in the search process. Additionally, the convergence can be observed from the right part in Figure 6. Here we apply Mean Absolute Deviation (MAD) to define the distance between edge weights and initial uniform distribution. It is obvious that both the intra and inter-cell architectures change little at the final searching steps.
In order to figure out the advantage of inter-cell connections, we detail the model contribution on each word on the validation data. Specifically, we compute the difference in word loss function (i.e.,
(a) An intra-cell architecture found by using inter-cell connections
(b) An intra-cell architecture found without using inter-cell connections
Figure 7: Comparison of intra-cell architectures found by using and not using additional inter-cell connections
Models	F1
LSTM-CRF (Lample et al., 2016)	90.94
LSTM-CRF + ELMo (Peters et al., 2018)	92.22
LSTM-CRF + Flair (Akbik et al., 2019)	93.18
GCDT + BERTLARGE (Liu et al., 2019b)	93.47
CNN Large + ELMo (Baevski et al., 2019)	93.50
DARTS + Flair (Jiang et al., 2019)	93.13
I-DARTS + Flair (Jiang et al., 2019)	93.47
ESS	91.78
ESS + Flair	93.62
log perplexity) between methods with and without inter-cell NAS. The words with eight best improvements are shown in the left column of Table 3. We observe that the rare words in the training set obtain more significant improvements. In contrast, the most frequent words lead to very modest decrease in loss (right column of Table 3). This is because the connections between multiple cells enable learning rare word representations from more histories. While for common words, they can obtain this information from rich contexts. More inputs from previous cells do not bring much useful information.
Additionally, we visualize the learned intra-cell architecture in Figure 7(a). The networks are jointly learned with the inter-cell architecture. Compared with the results of intra-cell NAS (Figure 7(b)), the learned network is more shallow. The inter-cell architectures have deeper networks. This in turn reduces the need for intra-cell capacity. Thus a very deep intra-cell architecture might not be necessary if we learn the whole model jointly.
5.2.3 Transferring to Other Tasks
After architecture search, we test the transferability of the learned architecture. In order to apply the model to other tasks, we directly use the architecture searched on WikiText-103 and train the param
Table 4: F1 scores on CoNLL-2003 NER task. BiLSTM
Models	F1
Cross-BiLSTM-CNN (Aguilar et al., 2018)	45.55
Flair (Akbik et al., 2019)	50.20
DARTS + Flair†	50.34
ESS	48.85
ESS + Flair	52.18
Table 5: F1 scores on WNUT-2017 NER task.  $\dagger$ Obtained by training the corresponding architecture using our setup.
Models	F1
NCRF++ (Yang and Zhang, 2018)	95.06
BiLSTM-CRF + IntNet (Xin et al., 2018)	95.29
Flair (Akbik et al., 2019)	96.72
GCDT + BERTLarge (Liu et al., 2019b)	97.30
DARTS + Flair†	96.59
ESS	95.51
ESS + Flair	97.22
Table 6: F1 scores on CoNLL-2000 chunking task.  $\dagger$ Obtained by training the corresponding architecture using our setup.
eters with the in-domain data. In our experiments, we adapt the model to CoNLL-2003, WNUT-2017 NER tasks and CoNLL-2000 chunking task.
For the two NER tasks, it achieves new state-of-the-art F1 scores (Table 4 and Table 5). ELMo, Flair and BERTLARGE refer to the pre-trained language models. We apply these word embeddings to the learned architecture during model training process. For the chunking task, the learned architecture also shows greater performance than other NAS methods (Table 6). Moreover, we find that our pre-learned neural networks yield bigger improvements on the WNUT-2017 task. The difference of the two NER tasks lies in that the WNUT-2017 task is a long-tail emerging entities recognition task. It focuses on identifying unusual, previously-unseen entities in the context of emerging discussions. As we discuss in the previous part of the section, the additional inter-cell NAS is good at learning the representations of rare words. Therefore, it makes
6636
sense to have a bigger improvement on WNUT-2017.
We have proposed the Extended Search Space (ESS) method of NAS. It learns intra-cell and inter-cell architectures simultaneously. Moreover, we present a general model of differentiable architecture search to handle the arbitrary search space. Meanwhile, the high-level and low-level sub-networks can be learned in a joint fashion. Experiments on two language modeling tasks show that ESS yields improvements of 4.5 and 2.4 perplexity scores over a strong RNN-based baseline. More interestingly, it is observed that transferring the pre-learned architectures to other tasks also obtains a promising performance improvement.
Gustavo Aguilar, Adrian Pastor López-Monroy, Fabio González, and Thamar Solorio. 2018. Modeling noisiness to recognize named entities using multitask neural networks on social media. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1401-1412, New Orleans, Louisiana. Association for Computational Linguistics.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In *NAACL* 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, page 724-728.
Peter J. Angeline, Gregory M. Saunders, and Jordan B. Pollack. 1994. An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Networks, 5(1):54-65.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5359-5368, Hong Kong, China. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2017. Designing neural network architectures using reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
James Bergstra, Daniel Yamins, and David D. Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pages 115-123.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. 2018. SMASH: one-shot model architecture search through hypernetworks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 933-941.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Efficient multi-objective neural architecture search via lamarckian evolution. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceed-
6637
ings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1243-1252.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261-2269.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. 2018. Automated Machine Learning: Methods, Systems, Challenges. Springer. In press, available at http://automl.org/book.
Yufan Jiang, Chi Hu, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2019. Improved differentiable architecture search for language modeling and named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3583-3588, Hong Kong, China. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 260-270.
Liam Li and Ameet Talwalkar. 2019. Random search and reproducibility for neural architecture search. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 129.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019a. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019b. GCDT: A global context enhanced deep transition architecture for sequence labeling. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2431-2441.
Gábor Melis, Tomáš Kocisky, and Phil Blunsom. 2019. Mogrifier lstm.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018a. An analysis of neural language modeling at multiple scales. CoRR, abs/1803.08240.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018b. An analysis of neural language modeling at multiple scales. CoRR, abs/1803.08240.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018c. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Stephen Merity, Bryan McCann, and Richard Socher. 2017. Revisiting activation regularization for language rnns.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045-1048.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, pages 4092-4101.
Jack W. Rae, Chris Dyer, Peter Dayan, and Timothy P. Lillicrap. 2018. Fast parametric learning with activation memorization. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, pages 4225-4234.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2019. Regularized evolution for image classifier architecture search. volume 33, page 4780-4789. Association for the Advancement of Artificial Intelligence (AAAI).
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie-Yan Liu. 2018. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1294-1303.
David R. So, Quoc V. Le, and Chen Liang. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, ICML
2019, 9-15 June 2019, Long Beach, California, USA, pages 5877-5886.
Kenneth O. Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. *Evol. Comput.*, 10(2):99-127.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, Florence, Italy. Association for Computational Linguistics.
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018. Multi-layer representation fusion for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 3015-3026.
Paul J Werbos. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550-1560.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Yingwei Xin, Ethan Hart, Vibhuti Mahajan, and Jean-David Ruvini. 2018. Learning better internal structure of words for sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2584-2593, Brussels, Belgium. Association for Computational Linguistics.
Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In Proceedings of ACL 2018, System Demonstrations, pages 74-79, Melbourne, Australia. Association for Computational Linguistics.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. 2018. Practical block-wise neural network architecture generation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2423-2432.
Barret Zoph and Quoc V. Le. 2017. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8697-8710.
6639