Automated Concatenation of Embeddings for Structured Prediction
Authors: Xinyu Wang
Yong Jiang
Nguyen Bach
Tao Wang
Zhongqiang Huang
Fei Huang
Kewei Tu
Venue:  (2021)

Abstract: Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that better word representations can be obtained by concatenating different types of embeddings. However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever-increasing number of embedding types makes it a more difficult problem. In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, based on a formulation inspired by recent progress on neural architecture search. Specifically, a controller alternately samples a concatenation of embeddings, according to its current belief of the effectiveness of individual embedding types in consideration for a task, and updates the belief based on a reward. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model, which is fed with the sampled concatenation as input and trained on a task dataset. Empirical results on 6 tasks and 21 datasets show that our approach outperforms strong baselines and achieves state-of-the-art performance with fine-tuned embeddings in the vast majority of evaluations.

Under review as a conference paper at ICLR 2021
AUTOMATED CONCATENATION OF EMBEDDINGS FOR STRUCTURED PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that better word representations can be obtained by concatenating different types of embeddings. However, the se- lection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever- increasing number of embedding types makes it more difficult problem. In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured predic- tion tasks, based on a formulation inspired by recent progress on neural archi- tecture search. Specifically, controller alternately samples concatenation of embeddings, according to its current belief of the effectiveness of individual em- bedding types in consideration for task, and updates the belief based on a reward. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of task model, which is fed with the sampled concatenation as input and trained on task dataset. Empir- ical results on tasks and 21 datasets show that our approach outperforms strong baselines and achieves state-of-the art performance with fine-tuned embeddings in the vast majority of evaluations.
INTRODUCTION
Recent developments on pretrained contextualized embeddings have significantly improved the per- formance of structured prediction tasks in natural language processing. Approaches based on con- textualized embeddings, such as EL.Mo (Peters et al. 2018), Flair (Akbik al. 2018), BERT (Devlin) et al. 2019), and XLMMR (Conneau et al. 2020),, have been consistently raising the state-of-the-an for various structured prediction tasks. Concurrently, research has also showed that word represen- tations based the concatenation of multiple pretrained contextualized embeddings and traditional non-comtextualizee embeddings (such as word2vee (Mikolov et al.) 2013) and character embeddings (Santos & Zadrozny 2)14)) can further improve performance (Peters al. 2018 Akbik et al.) [018]] Straková et al., 2019 He & Choi 2020). Given the ever-increasing number of embedding learning methods that operate different granularities (e.g.. word, subword, character level) and with dif- ferent model architectures, choosing the best emheddings to concatenate for a specific task becomes non-trivial, and exploring all possible concatenations can be prohibitively demanding computing resources.
Neural architecture search (NAS) is active area of research in deep learning automatically search for better model architectures, and has achieved state-of-the-arr performance on various tasks in computer vision, such as image classification (Real et al.) 2019). semantic segmentation (Liu) et al., 2019a), and object detection (Ghiasi et al.) 2019). In natural language processing. NAS has been successfully applied find better RNN structures (Zoph Le| Pham et al. 2018b) and recently better transformer structures (So et al.) 2019 Zhu et al. [2020]. In this paper, we propose the Automated Concatenation of Embeddings (ACE) approach to automate the process of finding better concatenations of embeddings for structured prediction tasks, formulated as an NAS problem. In this approach, an iterative search process is guided by a controller based on its belief that models the effectiveness of individual embedding candidates in consideration for specific task. At each step, the controller samples a concatenation of embeddings according to the belief model and feeds the concatenated word representations inputs to task model, which in turn is trained the

---

Under review as a conference paper at ICLR 2021
task dataset and returns the model accuracy reward signal to update the belief model. We use the policy gradient algorithm (Williams) [992) in reinforcement learning (Sutton & Barto [992)) to solve the optimization problem. In order to improve the efficiency of the search process, also design a special reward function by accumulating all the rewards based on the transformation between the current concatenation and all previously sampled concatenations.
Our approach is different from previous work on NAS in the following aspects:
Unlike most previous work, we focus on searching for better word representations rather than better model architectures.
2.
We design a unique search space for the embedding concatenation search. Instead of using RNN in previous work of & Le (2017), we design a more straightforward controller generate the embedding concatenation. We design novel reward function in the objective of optimization better evaluate the effectiveness of each concatenated embeddings.
Our approach is efficient and practical. ACE can find strong word representation on sin- gle GPU with only a few GPU-hours for structured prediction tasks, while a lot of the NAS approaches require dozens of or even thousands of GPU-hours search for good neural archi- tecture.
The task model from ACE achieves high accuracy without the need for retraining, while in pre- vious work of NAS the resulting neural network usually requires retraining from scratch.
Empirical results show that ACE outperforms strong baselines. Furthermore, we show that when ACE is applied concatenate pretrained contextualized embeddings which are already fine-tuned on specific tasks, we can achieve state-of-the-art or competitive accuracy on 6 structured prediction tasks including Named Entity Recognition (Sundheim) 1995), Of-eeOh- tagging (DeRose) 1988), chunking (Tjong Kim Sang & Buchholz 2000). aspect extraction (Hu & Liu (004). syntactic
dependency parsing [Tesière 1959) and semantic dependency parsing (Oepen et al. (2014) over 21 datasets. Besides, we also analyze the advantage of ACE and reward function design over the baselines and show the advantage of ACE over ensemble models.
2 RELATED WORK
2.1 EMBEDDINGS
Non-contextualized embeddings, such word2vec (Mikolov et al. 2013), GloVe (Pennington al., 2014). and fastText (Bojanowski et al. (017)). help lots of NLP tasks. Character embeddings (San-) Zadrozny 20144) are trained together with the task and applied many structured prediction tasks (Ma & Hovy 2016 Lample et al.
2016 Dozat & Manning, 2018). For pretrained contextu- alized embeddings, ELMo (Peters et al.
2018). pretrained contextualized word embedding gen- erated with multiple Bidirectional LSTM layers, significantly outperforms previous state-of-the-an
approaches on several NLP tasks. Following this idea, Akbik et al. (2018) proposed Flair embed- dings, which is kind of contextualized character embeddings and achieved strong performance in sequence labeling tasks. Recently, Devlin et al. (2019) proposed BERT, which encodes contextual- ized sub-word information by Transformers and significantly improves the performance NLP tasks. Much research such as RoBERTa (Liu et al. 2019c) has focused on improving BERT model's performance through stronger masking strategies. Moreover, multilingual contextualized embeddings become popular. [Pres et al. (2019) and Wu & Dredzel (2019) showed that Multilin- gual BERT (M-BERT) could learn good multilingual representation effectively with strong cross- lingual zero-shot transfer performance in various tasks. Conneau et al. (2020) proposed XLM-R, which is trained on a larger multilingual corpus and significantly outperforms M-BERT on various multilingual tasks.
lot of
2.2 NEURAL ARCHITECTURE SEARCH
Recent progress on deep learning has shown that network architecture design is crucial to the model performance. However, designing strong neural architecture for each task requires enormous ef- forts, high level of knowledge, and experiences over the task domain. Therefore, automatic de- sign of neural architecture is desired. A crucial part of NAS is search space design, which defines
2

---

Under review as a conference paper at ICLR 2021
the discoverable NAS space. Previous work (Baker al. 2017: Zoph & Le 2017: Xie & Yuille 2017) designs a global search space (Elsken et al. 2019) which incorporates structures from hand- crafted architectures. For example, Zoph & Le (2017) designed chained-strructure search space with skip connections. The global search space usually has a considerable degree of freedom. As an example, the approach of (Zoph & Le (2017) takes 22,400 GPU-hours to search CIFAR-10 dataset. Based on the observation that existing hand-crafter architectures contain repeated structures (Szegedy al. 2016; He et al. 2016; Huang et al. 2017 Zoph al. (2018) explored cell-based search space which can reduce the search time to 2,000 Goouh.
In recent NAS research, reinforcement learning and evolutionary algorithms are the most usual ap- proaches. In reinforcement learning, the agent's 's actions are the generation of neural architectures and the action space is identical to the search space. Previous work usually applies an RNN layer (Zoph & Le 2017 Zhong et al. 2018] Zoph et al.) 2018) or use Markov Decision Process (Baker) 2017) to decide the hyper- paraeeerr each structure and decide the input order of each struc- Evolutionary algorithms have been applied to architecture search for many decades (Miller
al..
ture.
ct al.
1989 Angeline et al. 1994] Stanley & Mükkulainen 2002; Floreano et al. 2008 Jozefowicz populations through recombination and mu- tation operations and selects survivors through competing among the population. Recent work with evolutionary algorithms differ in the method parent/survivor selection and population generation. For example, Real et al. (2.17) _ [ Liu et al. (2018a), Wiistbba (2018) and Real et al. (2019) applied tournament selection (Goldberg & Deb 1991) for the parent selection while Xie & Yuille (2017)
al.
2015 The algorithm repeatedly generates
keeps all parents. SSugauuma et al. (2017) and Elsken et al. (2018) chose the best model while Real al. (2019) chose several latest models as survivors.
AUTOMATED CONCATENATION EMBEDDINGS
In ACE, a task model and a controller interact with each other repeatedly. The task model pre- dicts the task output, while the controller searches for better embedding concatenation the word representation for the task model to achieve higher accuracy. Given an embedding concatenation generated from the controller, the task model is trained over the task data and returns reward the controller: The controller receives the reward to update its parameter and samples new embedding concatenation for the task model. Figure shows the general architecture of our approach.
3.1 TASK MODEL
For the tasks model, emphasis sequence-structuree and graph-structured outputs. Given structured prediction task with input sentence æ and structured output y. we can calculate the prob- ability distribution Pyy|æ) by:
exp (Score(æ, y))
P(ylæ) (x)) exp (Score(xr y²))
where Y(x) represents all possible output structures given the input sentence æ. Depending different structured prediction tasks, the output structure y can be label sequences, trees, graphs other structures. In this paper, exemplar structured prediction tasks. We use BiLSTM-CRF model (Ma & Hovy 2016 Lample al. 2016) for sequence-structuured outputs and use BiLSTM-Biafinng model (Dozat & Manning for graph-structured outputs:
grguence-structurre and graph-structured outputs two
2017
ped x) = BiLSSM-CRR(C) y);
peraph (y|x)) = BiL.SM- -iiMM-iianfeeWW (V,
where V = |0:: ;.). V E Rdx= is a matrix of the word representations for the input sentence with words, is the hidden size of the concatenation of all embeddings. The word representation v; of i-th word is a concatenation of L types of word embeddings:
= embed (z); e; = :?:
where embed' is the model of 1-th embeddings, n; E Rdd of Rdd d' is the hidden size of embed!.
3

---

Under review as a conference paper at ICLR 2021
Previous Current Choice Choice
RReward
Controller
Action)]
Choice
f
0
X
Task Model
BERT
The main paradigm of approach is shown in the middle, where example of reward function is represented in the left and an example of a concatenation action is shown in the right.
Figure 1:
3.2 SEARCH SPACE DESIGN
The neural architecture search space can be represented set of neural networks (Elsken et al. 2019). A neural network can be represented as a directed acyelic graph with a set nodes and directed edges. Each node represents operation, while each edge represents the inputs and outputs between these nodes. In ACE, we represent each embedding candidate as a node. The input to the nodes is the input sentence ar, and the outputs are the embeddings v'. Since concatenate the embeddings as the word representation of the task model, there is no connection between nodes in our search space. Without considering the connections between nodes, the search space can be significantly reduced. For each node, there are lot of options to extract word features. Taking BERT embeddings as an example, Devlin et al. (2019) concatenated the last four layers as word features while Kondratyuk & Straka (2019) applied a weighted sum of all twelve layers. However, the empirical results (Devlin al. 2019) do not show a significant difference in accuracy. We follow the typical usage for each embedding to further reduce the search space. As result, each embedding only has a fixed operation and the resulting search space contains 2 - possible combinations of nodes.
In NAS, weight sharing (Pham et al., 2018a) shares the weight of structures in training different fixed the weight of pretrained
neural architectures reduce the training cost. In comparison, embedding candidates in ACE except for the character embeddings. Instead of sharing the parame- ters of the embeddings, we share the parameters of the task models at each step of search. However, the hidden size of word representation varies over the concatenations, making the weight sharing of structured prediction models difficult. Instead of deciding whether each node exists in the graph, we keep all nodes in the search space and add an additional operation for each node to indicate whether the embedding masked out. To represent the selected concatenation, we use binary vector [a],"' at , aL) as an mask to mask out the embeddings which are not selected:
(I) where ag is a binary variable. Since the input V is applied to a linear layer the BiLSTM layer, multiplying the mask with the emheddings is equivalent directly concatenating the selected em- beddings:
v; = [o] ; of/az az]
L W U. Σ WT vlag W e ddxh xxh and W, e Rd
W W Wzl:
(2)
Therefore, the model weights can be shared after applying the embedding mask to all embedding candidates' concatenation. Another benefit of our search space design is that unused embedding candidates and the corresponding weights in W for lighter task model after the best concatenation is found by ACE.
remove the
3.3 SEARCHING IN THE SPACE
During search, the controller generates the embedding mask for the task model iteratively. We use parameters 1 02:00..................... ...; Or. for the controller instead of the RNN structure applied in previous
approaches (Zoph Le 2017 Zoph et al., 2018). The probability distribution of selecting an
concatenation a is petl (a: 0) I pett (a:: 0z). Each element a of a is sampled independently from a Bernoulli distribution, which is defined as:
per 1: 01) (6:): petd (a. 0;01) pard (a: (:)
(3)
4

---

Under review as a conference paper at ICLR 2021
where is the sigmoid function. Given the mask, the task model is trained until convergence and returns an accuracy R on the development set. As the accuracy cannot be back poookpropggtted to the controller, use the reinforcement algorithm for optimization. The accuracy R is used as the reward signal to train the controller. 'he controller's target is maximizing the expected reward (()) (a:0) [ through the policy gradient method (Williams) 1992). In our approach, since calculating the exact expectation is intractable, the gradient of J(0) is approximated by sampling only one selection following the distribution per (a: 0) at each step for training efficiency:
VoJ(O)= =0 Vo log Praht (R
(4)
where b is the baseline function to reduce the high variance of the update function. The baseline usually can be the highest accuracy during the search process. Instead of merely using the highest accuracy of development set over the search process as the baseline, we design a reward function on how each embedding candidate contributes to accuracy change by utilizing all searched concatena- tions' development scores. We use a binary vector (a' - a'l to represent the change between current embedding concatenation a' at current time step t and a' previous time step i. We then define the reward function as:
Σ RRe (ho-Ri)lats R:)la' a'l
(5)
where r' is a vector with length L representing the reward of each embedding candidate. R and R₁ are the reward at time step and i. When the Hamming distance of two concatenations H Hamm((' (amm(aa.cr a') gets larger, the changed candidates' contribution to the accuracy becomes less no-
ticeable. The controller may be misled reward candidate that is not very helpful. We apply
discount factor to reduce the reward for two concatenations with large Hamming distance to alleviate this issue. Our final reward function is:
t-1
R
Ray Hamm .a)) a')-1laa -a'l
(6)
i=1
where 7 E (0, 1). Eq. is then reformulated as:
VoJi(0)
Rc Vo log pe (af:
3.4 TRAINING
dictionary ID to store the concatenations and validation scores. At t= 1, we train the task model with all embedding candidates concatenated. From t 2, we repeat the following steps until maximum iteration T:
To train the controller,
Sample a concatenation a' based on the probability distribution in Eq.
Train the task model with a' following Eq. 1 and evaluate the model the development set to get the accuracy Rg-
Given the concatenation a', accuracy R, and D, compute the gradient of the controller following Eq. and update the parameters of controller.
Add a' and R2 into TD, set f t
When sampling a', we avoid selecting the previous concatenation a and the all zero vector (i.e., selecting no embedding). If a' is in the dictionary DD, we compare the RG with the value in the dictionary and keep the highest one.
EXPERIMENTS
4.1 DATASETS AND CONFIGURATIONS
To show ACE's effectiveness, we conduct extensive experiments on a variety of structured prediction tasks varying from syntactic tasks semantic tasks. The tasks are named entity recognition (NER),
5

---

Under review as a conference paper at ICLR 2021
Part-Of-Speet (POS) tagging, Chunking, Aspect Extraction (AE), Syntactic Dependency Parsing (DP) and Semantic Dependency Parsing (SDP). The details of the tasks are in Appendix [A.1
We train the controller for 30 steps and save the task model with the highest accuracy the devel- opment set as the final model for testing. For all experiments, we report the averaged accuracy of runs. For other settings, please refer to Appendix A..3 for more details.
4.2 EMBEDDINGS
Basic Settings: For the candidates of embeddings on English datasets, we use the language- specific model for EL.Mo, Flair, base BERT, GloVe word embeddings, fastText word embeddings, non-contextual character embeddings (Lample et al., (2016), multilingual Flair (M-Flair), M- M-BERT and XLM-R embeddings. The size of the search space in our experiments is 2 language-specifie models of other languages, please refer to Section A..4 for more details. In AE, there is no available language-specific BERT, Flair and ELMo embeddings for Russian and there no available language-speciffs Flair and ELMo embeddings for Turkish. We use the corresponding English embeddings instead so that the search spaces these datasets are almost identical to those of the other datasets. All embeddings are fixed during training except that the character embeddings are trained over the task. The empirical results are reported Section 4.3.1
2047" For
embeddings is usual approach get better accuracy. In sequence labeling. most of the work follows the fine-tuning pipeline of BERT that connects the BERT model with a linear layer for word-level classification. However, when multiple embeddings are concatenated, fine-tuning specific group of embeddings becomes difficult. It is impractical to train multiple embeddings because of complicated Hyper-parameter settings and mas- sive GPU memory consumption. To alleviate this problem, first fine-tune the transformer-basee embeddings with the task and then concatenate these embeddings together with other embeddings in the basic setting apply ACE. The empirical results reported in Section 4.3.2
Embedding Fine-tuning: Fine-tuning transformer-bast
Table 1: Comparison with concatenating all embeddings and random search baselines on 6 tasks.
NER
POS
AE 16kes
de
en
es
Ritter
ARK
TB-v2
14Lap
14Res
15Res
es
n
ALL
83.1 92.4 88.9 89.8
90.6
92.1
94.6
82.7
88.5
74.2
73.2
74.6
75.0
67.1
67.5
RANDOM
84.0 92.6
88.8 91.9
91.3
92.6
94.6
83.6
88.
73.5
74.7
75.0
73.6
68.0
70.0
ACE
84.2 93.0
88.9 92.1
91.7
92.8
94.8
83.9
88.6
74.9
75.6
75.7
75.3
70.6
71.1
CHUNK
DP
SDP
AVG
CoNLL 2000
UAS LAS DM-ID
DM-OOD PAS-ID PAS-OOD
PSD-ID PSD-OOD
ALL
96.7
96.7
95.1
94.3
90.8
94.6
92.9
82.4
81.7
85.3
RANDOM
96.7
96.8
95.2
94.4
90.8
94.6
93.0
82.3
81.8
85.7
ACE
96.8
96.9
95.3
94.5
90.9
94.5
93.1
82.5
82.1
86.2
4.3
RESULTS
We use the following abbreviations in our experiments: UAS: Unlabeled Attachment Score; LAS: Labeled Attachment Score: ID: In-domain test set; OOD: Out-of-domain test set; F & G (2019):
Fernánddez-Gonzálezz & Gómez-Rodriguez (2019); F & G (2020): Fernández-Gonzállez & Gómez-
Rodríguezi (2020): D & M (2018): Dozat & Manning 2018). all tables, we use ISO 639-1 language codes to represent each language.
4.3.1 COMPARISON WITH BASELINES
To show the effectiveness of our approach, we compare our approach with two strong baselines. For the first one, we let the task model Iearn by itself the contribution of each embedding candidate that is helpful to the task. We set train the task model (A11). The linear layer weight W in Eq. 2 reflects the contribution of each candidate. For the second one, we the random search (Random), strong baseline in NAS (Li &
all-ones (i.e., the concatenation of all the embeddings) and
Flair embeddings have two models (forward and hackward) for each language.

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison with state-of-the-art approaches in NER and POS tagging. 1: Models are trained on both train and development set. 1 Models trained with document information. %
Results are from Conneau et al. (2020).
NER
POS
de
deus
nl
Ritter
ARK
TB-v2
Akbik et al. 2018
88.3
93.1
Owoputi et al. 2013
90.4
93.2
94.6
Baevski et al.
(2019
93.5
Gui cl al 2017
90.9
92.8
Straková 2019
85.1
93.4
88.8
92.7
Gui et al.
2018
91.2
92.4
Yu ct al. (2020)
86.4
90.3
93.5
90.3
93.7
Nguyen ct al. (2020)
90.1
94.1
95.2
85.8
92.9
89.7
92.5
XLM-R+Fine-lun
93.0
93.4
95.0
ACE+Fine-tune
87.0
90.5
93.5
91.7
94.6
ACE-Fine-tune
93.4
93.8
95.6
Comparison with state-of-the-art approaches in chunking and aspect extraction. report the results reproduced by Wei al. (2020).
Table 3:
We
CHUNK
AE
CoNLL 2000
14Lap
14Res 15Res
16Res
es
ru
tr
Akbik
2018
96.7
Xu et al.
2018
84.2
84.6
72.0
75.4
Clark et al. (2018)
Xu al.
2019)
84.3
78.0
Liu et al.
(2019b)
97.3
Wang et al. 2020al
72.8
74.3 72.9 71.8 59.3
Wang ct al.
( (002b)
Wei ct al. 2020)
82.7
87.1
72.7
77.7
XLM-R+Fine-tun
96.5
LLM-R+FFine-te
81.3
88.4
77.3
78.5
77.8
72.1
75.7 66.7 76.7
ACE+Fine-tune
ACE+Fine-tune
85.0
89.8
78.5
81.2
78.8
76.7 77.7
Talwalkar 2020). For Random, we run the same maximum iteration as in ACE. Table [ shows that
ACE outperforms both baselines in 6 tasks over 23 test with only two exceptions. Comparing Random with All 1, Randon outperforms All by 0,4 on average and surpasses the accuracy of All on 14 of 23 sets, which shows that concatenating all embeddings may not be the best solution to most structured prediction tasks. In general, searching for the concatenation for the word representation is essential most cases, and our search design can usually lead to better results compared to both of the baselines.
4.3.2 COMPARISON WITH STATE-OF-TII--AEE APPROACHES
As we have shown, ACE has an advantage in searching for better embedding concatenations. We further show that ACE is competitive or even stronger than tfte-of-the-aa art approaches. In some tasks, We have several additional settings to better compare with previous work. In NER, we also conduct comparison on the revised version of German datasets in the CoNLL 2006 shared task (Buchholz & Marsi 2006). In parsing tasks, we use XL.Net (Yang et al. 2019), which is significantly stronger than BERT in DP (Zhou & Zhao 2019). In SDP, the state-of-the-an approaches used POS tags and lemmas additional word features the network. We add these two features to the embedding candidates and train the embeddings together with the task. We use the fine-tuned BERT embeddings, XLM-R embeddings, and XLNet embeddings each task instead of the pretrained version of these embeddings as the candidates. For the NER tasks, we use the XLM-R models fine-tuned by Hugging Face instead
We additionally compare with fine-tuned XLM-R model for NER, POS tagging, chunking and AE, and compare with fine-tuned XL.Net model for DP and SDP, which are strong fine-tuned models in most of the experiments. Results are shown in Table 2 3 Results show that ACE with fine-tuned embeddings achieves state-of-the-art performance 22 out of 24 test and competitive with the state-of-the-ae art approaches in the other 2 test sets. Our approach is competitive or even stronger than the approaches using additional information in NER and DP, which shows that finding good word representation helps structured prediction tasks. We also show that ACE is stronger than the fine-tuned models, which shows the effectiveness of concatenating the fine-tuned embeddings.
Pleasee refer Appendix A.4 and B.1 for more details.

---

Under review as a conference paper at ICLR 2021
Table 4: Comparison with state-of-the-art approaches in DP and SDP. 1 For reference, they addi- tionally used constituency dependencies in training. 4: For reference, confirmed with the authors oe & Choi (2020) that they used a different data pre-processing script with previous work.
DP
SDP
PTB
DM
PAS
PSD
UAS
LAS
ID
0OD
OOD
ID
00D
Zhou & Zhao
2019
97.2
95.7
He & Choi
(2020
94.6
90.8
96.1
94.4
86.8
79.5
F & G (2019)
96.0
94.4
D & M (2018)
94.0
89.7
94.1
91.3
81.4
79.6
Hc Choi (2020)
96.8
95.3
Wang et al. (2019)
93.7
88.9
93.9
90.6
81.0
79,4
Zhang et al. (2020)
96.1
94.5
& (2020)
94.4
91.0
95.1
93.4
82.6
82.0
XLN ine-tune
97.0
95.4
XLNet+Fine-tune tunn
94.9
92.0
94.8
93,4
82.6
82.2
ACE+Fineetuun
97.2
95.7
ACE+Fine-tune
95.3
92.6
95.3
93.9
83.6
83.2
96.6
Sample Accurccy
Bext Accuracy
96
96.4
95
96.2
941
93
96
0
10
20
30
0
10
20
30
Figure 2: Comparing the efficiency of random search (Random) and ACE. The x-axis is the number of time steps. The left y-axis is the averaged best validation accuracy on CoNLL English NER dataset. The right y-axis averaged validation accuracy of the current selection.
5 ANALYSIS
5.1 EFFICIENCY OF SEARCH METHODS
To show how efficient our approach is compared with the random search algorithm, we compare the algorithm in two aspects CoNLL English NER dataset. The first aspect is the best development accuracy during training. The left part of Figure Sarreesshoow shows that ACE consistently stronger than the random search algorithm in this task. The second aspect is the searched concatenation at each time step. The right part of Figure Saareesshoo shows that the accuracy of ACE gradually increases and gets stable when concatenations sampled.
5.2 ABLATION STUDY ON REWARD FUNCTION DESIGN
To show the effectiveness of the designed reward function, we compare our reward function (Eq. (6) with the reward function without discount factor (Eq. 5) and the traditional reward func- tion (reward term in Eg. 44. We sample 2000 training sentences on CoNLL English NER dataset for faster training and train the controller for 50 steps. Table shows that both the discount fac- tor and the binary vector la' a'l for the task are helpful in both development and test datasets.
Table 5: Comparison of reward functions.
DEV
TEST
ACE
93.18 90.00
No discount (Eq. 5 Simple (Eq.
92.98 89.90
92.89 89.82
5.3 COMPARISON WITH ADDITIONAL BASELINES
We compare ACE with two more approaches further show the effectiveness of ACE, One a vari- ant of A1l, which uses weighting parameter b [b,,... ... ....... bz) passing through sigmoid function to weight each embedding candidate. Such an approach can explicitly learn the weight of each embedding in training instead of a binary mask. We call this approach A1 1 We +Weiggh.... Another one is model ensemble, which trains the task model with each embedding candidate individually and uses the trained models to make joint prediction on the test set. We voting for ensemble it is simple and fast. For sequence labeling tasks, the models vote for the predicted label at each position. For DP, the models vote for the tree of each sentence. For SDP, the models vote for each potential
8

---

Under review as a conference paper at ICLR 2021
Table 6: A comparison among A11, Random, ACE, Table Results of models with document
All+Reight and Ensemble. CHK: chunking.
context on NER. +sent/+doc: models with sentence-/doccment-leeelll embeddings.
DP UAS LAS
SDP
NER POS AE CHK
ID
OOD
de dc06 en
cs
All
92.4
90.6
73.2
96.7
96.7 95.1 94.3 90.8
All+sent
86.8 90.1 93.3 90.0 94,4
Random
92.6
91.3
74.7
96.7
96.8
95.2 94.4 90.8
ACE+sent
87.1 90.5 93.6 92.4 94.6
ACE
93.0
91.7 75.6
96.8
96.9
95.3 94.5 90.9
BERT (2019)
92.8
Ali+ht
92.7
90.4
73.7
96.7
96.7 95.1 94.3
90.7
Akbik el al. 2019
88.3
90.4
Ensemble
92.2
90.6
68.1
96.5
96.1
94.3
94.1 90.3
et al. (2020)
86.4 90.3 93.5 90.3 93.7
Ensembleder
92.2
90.8
70.2
96.7
96.8
95.2
94.3
90.7
All+doc
87.6 91.0 93.5 93.3 93.7
Ensemble
92.7
91.4
73.9
96.7
96.8
95.2
94.4
90.8
ACE+doc
88.0 91.4 94.1 95.6 95.5
labeled arc. We use the confidence of model predictions break ties if there are more than one agreement with the same counts. We call this approach Ensemble. One of the benefits of voting is that it combines the predictions of the task models efficiently without any training process. We can search all possible - model ensembles in short period of time through caching the outputs of the models. Therefore, we search for the best ensemble of models on the development set and then evaluate the best ensemble on the test set (Ensemb ledev). Moreover, we additionally search for the best ensemble on the test set for reference (Ensembletess) which the upper bound of the approach. We use the same setting as in Section 4.3.1 and select one of the datasets from each task. For NER, POS tagging, AE, and SDP, we use CoNLL 2003 English, Ritter, 16Res, and DM datasets, respectively. The results shown in Table Empirical results show that ACE outperforms all the settings of these approaches and even Ensembleeeee erest, which shows the effectiveness of ACE and the limitation of ensemble models. A1 All+AAltwww. ael++Wiight and Ensemb: embeleev are competitive in most of the cases and there is no clear winner of these approaches on all the datasets. These results show the strength of embedding concatenation. Concatenating the embeddings incorporates information from all the embeddings and forms stronger word representations for the task model, while in model ensemble, it is difficult for the individual task models to affect each other through ensemble.
5.4 ACE WITH DOCUMENT-LEVEL REPRESENTATIONS
Recently, models with document-level word representations extracted from transformer-basee em- beddings significantly outperform models with sentence-level word representations in NER (Devlin) et al. 2019; Yu et al.) 2020). To show the effectiveness of ACE with document- representations, replace the sentence-level word representations from transformer-bace based embeddings (i.e., XLM- R and BERT embeddings) with the document-level word representations. We generate document- level representations following Yu al. (2020). Results are shown in Table 7 We report the test results of Al1 to show how the gap between ACE and All changes with different kinds of represen- tations. We report the test accuracy of the models with the highest development accuracy following Yu et al.] (2020) for fair comparison. Empirical results show that the document-level representa- tions can significantly improve the accuracy of ACE. Comparing with models with sentence-level representations, the averaged accuracy gap between ACE and All is enhanced from 0.7 to 1.1 with document-level representations, which shows that the advantage of ACE becomes stronger with document-level representations.
CONCLUSION
In this paper, we propose the Automated Concatenation of Embeddings, which automatically searches for better embedding concatenation for structured prediction tasks. We design a simple search space and use the reinforcement learning with novel reward function to efficiently guide the controller to search for better embedding concatenations. We take the change of embedding con- catenations into the reward function design and show that our reward function is stronger than the simpler ones. Results show that ACE outperforms strong baselines. Together with fine-tuned embeddings, ACE achieves state-of-the-ar performance in 6 tasks over 19 out of 21 datasets.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence la- beling. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1638-1649, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguis- tics. URL httpt::t//wwet ac org/ http//nnoooooooooooooooee 1139
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 724-728, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 11L16553/1//N1-1077 URL www. aclweb. org/ anthology/
N1 119-1078
Peter J Angeline, Gregory M Saunders, and Jordan B Pollack, An evolutionary algorithm that constructs recurrent neural networks. IEEE transactions Neural Networks, 5(1):54-65, 1994.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLP-IJCNLL)
pp. 5360-5369, Hong Kong, China, November
2019. Association for Computational Linguistics. doi: 10. 10.18653/11DD19-15999 URL https:::
7 www. www.wa.wwwbb............e. org/ http:////aaaaaoooooooo/--11
1539
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In ICLR, 2017.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. ISSN 2307-387X.
Sabine Buchholz and Erwin Marsi. CoNLL-x shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL- X). pp. 149-164, New York City, June 2006. Association for Computational Linguistics. URL
https: www acl www.wwwwwwww.................w.w www.:eww...beeeeeee..e...eeeeeee aat/tontoooogyy ogtp
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. Semi-supervised quence modeling with cross-view training. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 1915-1925, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18633/1/D18-1211 URL
https::/ /tt:://www.... acl www.wwwwb.... orgg hetp//ntholllo wtw/og/-///// 1217
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaune Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18655//1/0222.aad 7/www. www.wb.wwwbbo... //w.://lwec.ccrraaatttlo///////2 aww.aacclmaai... 747
747. URLhttps:
Linguistics, 14(1), 1988. antholoory/220..ccl
Steven J. DeRose. Grammatical category disambiguattin by statistical optimization. Computational http:///wwww..ccaao.o.../////tt/
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18853/v1/N119-11 1423. URL https:
www. awwweb. http://////nnoooooolll...e 1423
Timothy Dozat and Christopher D Manning. Deep biaffine attention for neural dependency parsing. In International Conference on Learning Representations, 2017.
10

---

Under review as a conference paper at ICLR 2021
Timothy Dozat and Christopher D. Manning. Simpler but more accurate semantic dependency parsing. In Proceedings of the 56th Anmual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pp. 484 484, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18662/1//118-07777 URL https::////.......... //ss://www.../ //www. ac web. org/antho oogphhh-11yp111111777
Thomas Elsken, Jan- Kun-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. In ICLR workshop, 2018.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
Journal of Machine Learning Research, 20:1-21, 2019.
Daniel Fernánez--Gonzáler and Carlos Gómez-Rodriguez. Left-to-right dependency parsing with pointer networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 710-716, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 1076. URLhttps: www acl www.ww.wwwbbbccc.. wwtp:/tw....eeeeeeee..e...ee..ee hatgg//t/hhhg/////
N1 909--0766
Daniel Lrnnandez-Gonzāle, and Carlos Gómez-Rodríguez. Transition-based semantic dependency parsing with pointer networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7035-7046. Online, July 2020. Association for Computa-
tional Linguistics. doi: 1/22000/////00.llllmmmmmmmi eanthology/2020.ecoo -aal-mainnn 62.9
URL BtRhhtps::// www. aclweb org/
Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary intelligence, 1(1):47-62, 2008
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7036-7045, 2019.
Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Fisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 42-47, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https: 7/ //www.c. www.ww.wwwb........ wwt.:oogg/aoooooooo wttgg///atthoooy/11111/00000
David E Goldberg and Kalyanmoy Deb. A comparative analysis of selection schemes used in genetic algorithms. In Foundations of genetic algorithms, volume 1, pp. 69-93. Elsevier, 1991.
Tao Gui, Qi Zhang, Haoran Huang, Minlong Peng, and Xuanjing Huang. Part-of-speech tagging for twitter with adversarial neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2411-2420. Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1..18633/1/DD1-11566 URL https: 7/ www. ac www.wwebbcc org/ httpb/oorggaatooooo/ooooo.
Tao Gui, Qi Zhang, Jingjing Gong, Minlong Peng, Di Liang Keyu Ding, and Xuanjing Huang. Transferring from formal newswire domain with hypernet for twitter POS tagging. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2540-2549 Brussels, Belgium, October-Novente 10.18653/v1/DI1
2018. Association for Computational Linguistics. doi:
URL http:: www ac www..www.bb...m................. org http./////...toooog//1--------1
Han He and Jinho Choi. Establishing strong baselines for the new decade: Sequence tagging. syntactic and semantic parsing with bert. In The Thirty-Third International Flairs Conference, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Under review as a conference paper at ICLR 2021
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, pp. 168-177, New York, NY, USA, 2004. Association for Computing Machinery. doi: 10.1145/1014052.00000 1014073. URL https://dcci htt.///.g//0.....1
ISBN 1581 138881.
101 4052. 1014073]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. International conference on machine learning, pp. 2342-2350, 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Dan Kondratyuk and Milan Straka. 75 languages, model: Parsing Universal Dependencies univer- sally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 2779-2795, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1.118633/11/D1--12777 URL AtRhhtp:::/ www. www.eaawwebbooo www..ccccee...rr/tttoooo...... D19-1279
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 260-270, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16- 1030. URI.https: h/wp::://www... wew.eb. org/anthol N16 66-000
Liam and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2020.
Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. Exploiting BERT for end-to-end aspect- based sentiment analysis. In Proceedings of the 5th Workshop on Noisy User- nee-ggenratted Text (W-NUT 2019), pp. 3441, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. (0.18653/1/DD19-5500 959-5505
wwww wwwwwaacl... www..we..bbccmmm org/ wwwprrgg//aaoloooooooooooooooooo hgo
Chenxi Liu, Liang- Chiih Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan I. Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 82 92, 2019a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. In International Conference Learning Representations, 2018 8a.
Yijia Liu, Yi Zhu, Wanxiang Che, Bing Qin, Nathan Schneider, and Noah A. Smith. Parsing tweets into universal dependencies. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers). pp. 965-975. New Orleans, Louisiana, June 2018b. Association for Computational Linguistics. doi: 10.18653/v1/N18. 1088. URLhttps: htt::///wwww.. acl www.:w/wwwwb.................... hww..//..l....rg/tttooooog/////. N18-10
Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. GCDT: A
global context enhanced deep transition architecture for sequence labeling. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2431-24411 Flo- rence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1233.
URL https:/ wwww. c. web. web.org//nthon. hgyp 1 9233
12

---

Under review as a conference paper at ICLR 2021
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X7::9977.11922 2019c.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Reprexentations, 2018.
Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRE: In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1064-1074, Berlin, Germany, August 2016. Association for Com- putational Linguistics. doi: 10.18653/v1/P1-11 1101.
URL https: //www///wwwwwo. wwwwwww.bbb.. acrg
anthology/PI6-II00
Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. Stack- pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1403-1414, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/113 1130.
URL https: www..wwwww......w..wwwwwwwwww..w org/ a/taaaatooooy11---1 wtt.://wwwwe.....aaaacooo///////
1130
Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajië. Non-projective dependency pars- ing using spanning tree algorithms. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 523-530, Vancou- ver, British Columbia, Canada, October 2005. Association for Computational Linguistics. URL
https: www awwwab.w.............. www.org///aooooo wwtp///ataoogg////./..33
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
Geoffrey Miller, Peter Todd, and Shailesh Hegde. Designing neural networks using genetic algo- rithms. In 3rd International Conference on Genetic Algorithms, pp. 379-384, 01 1989.
Dat Quoe Nguyen, Thanh Vu, and Anh Tuan Nguyen. BERTweet: pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Dan Flickinger, Jan Hajic, An- gelina Ivanova, and Yi Zhang. Semeval 2014 task 8: Broad-coverage semantic dependency pars- ing. SemEval 2014, 2014.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao. Daniel Zeman, Silvie Cinková, Dan Flickinger, Jan Hajic, and Zdenka Uresova. Semeval 2015 task 18: Broad-coverage semantic dependency parsing. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 915-926, 2015.
Olutobi Owoputi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 380-390, Atlanta, Georgia,
June 2013. Association for Computational Linguistics. URL httph: wwww aclweb. org/ Nathology/N13-1033
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2233 New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10.18653/v 1118653/1//N1-1111 /N18-1202. URL https: / www. a web. wwabboog///nnooo ogy
13

---

Under review as a conference paper at ICLR 2021
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095-4104, 2018a.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning, 2018b.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996-5001, Florence, Italy, July 2019, Association for Computational Linguisties. doi: 10. 1/P65//111199199//9 1493. wtR.ttts::/ www. ac web. web.orrg/ammchologyy/P 9-91999-19911111999999
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 27-35, Dublin, Ireland, August 2014. Association for Computational Linguistics. doi: 10.3115/11$14-2004
URL https: / wwww w www.wwwwwbbboo wwtp:/wwwbwrrre.ooo...e http://ww....oaaaooooooo////////
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. SemEval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International
Workshop on Semantic Evaluation (SemEval 2015), pp. 486-495, Denver, Colorado, June 2015. Association for Computational Linguistics. doi: 1/S555/1/S11552000 URL http:https/// //www.
Maria Pontiki. Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-SSmaii, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotel- nikov, Nuria Bel, Salud María Jiménez-Zafra, and Gülşen Eryigit. SemEval-2011 task 5: As- pect based sentiment analysis. In Proceedings of the 10th International Workshop on Seman- tic Evaluation SeemEval-2016, pp. 19-30, San Diego California, June 2016. Association for Computational Linguistics. Lingutttsss aanthology/S66-100
doi: 20186533v1/S116 1002. URL
www.eb..lwwbcc....
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, pp. 2902-2911, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. Named entity recognition tweets: An experi- mental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1524-1534, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguisties. URL htttp:: www aclweb htt.w//wwwww./tttoooooo////////a
1141
Cicero D Santos and Bianca Zadrozny. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31 international conference on machine learning (ICML-14), pp- 1818-1826, 2014.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1599-1613, Minneapolis, Minnesota, June 2019, Association for Computational Linguisties. doi: 10. (0.18653/v1/N19-111 162.
htR.htts::/ www. a cl web. org/ http:/athholgyyyy 1t1///19911111 9-1162
David R So, Chen Liang. and Quoe V Le. The evolved transformer. In International Conference on Machine Learning. 2019.
Kenneth O Stanley and Risto Mükkulaanen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
14

---

Under review as a conference paper at ICLR 2021
Jana Straková, Milan Straka, and Jan Hajic. Neural architectures for nested NER through lineariza- tion. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguis- fics, pp. 5326-5331 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P1///////PP/// 112771799 URL https:: /www. ac web. ogy/P1 91111191111 1527
Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic programming approach to designing convolutional neural network architectures. In Proceedings of the genetic and evo- lutionary computation conference, pp. 497-504, 2017.
Beth M. Sundheim. Named entity task definition, version 2.1. In Proceedings of the Sixth Message Understanding Conference, pp. 319-3322 1995.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1992.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigmiew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Lucien Tesnière. éléments de syntaxe structurale. Editions Klincksieck, 1959.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-independee named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-20022, 2002. URL https:tp///.. //t://wwwww ac wwwwwb......... org/ wwwthhoooyy/oooo 2024
Erik F. Tjong Kim Sang and Sabine Buchholz. Introduction to the CoNLL-2000 shared task chunk- ing. In Fourth Conference on Computational Natural Language Learning and the Second Learn- ing Language in Logic Workshop, 2000. URL https: /wwww , a a web. org/ / ant ho logy/
WDC- 0726
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-indeeendent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HL.T-NACI, 2003, pp. 142-147, 2003. URL https:
77 //www. ac a web. org/
0449
Xinyu Wang, Jingxian Huang, and Kewei Tu. Second-order semantic dependency parsing with end- to-end neural networks. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 4609-4618, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P199 1/544 htLhttps: www. aclweb. org/ hatghooohhho/// P19- 11144
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, and Kewei Tu. Structure-level knowledge distillation for multilingual sequence labeling. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 3317-3330, Online, July 2020a. URLhttps
Association for Computational Linguistics. doi: 10.
7/ //www.aclwes o wttpg//nnhhoooy/22222222 acl mmalmaallaaa in. 304
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang. Huang Zhongqiang, Fei Huang, and Kewei Tu. More embeddings, better sequence labelers? In Findings of EMNLP, Online, November 2020b.
Zhenkai Wei, Yu Hong, Bowei Zou, Meng Cheng, and Jianmin Yao. Don't eclipse your arts due to small discrepancies: Boundary repositioning with a pointer network for aspect extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp.
3678-3684, Online, July 2020. Association for Computational Linguistics. doi: 18.18v53/v1/ 2020.acl- ma..amammm.............. http:t/:: / wwwwwwwww. acl wrg.borg// anthol /20p://220mmm acc mainn.
339
J J Williams. Simple statistical folliig algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 1992.
Martin Wistuba. Deep learning architecture search by neuro-cell-based evolution with function- preserving mutations. In Joint European Conference on Machine Learning and Knowledge Dis- covery in Databases, pp. 243-258. Springer, 2018.
15

---

Under review as a conference paper at ICLR 2021
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross- cros--lnguall effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-JJCNLP), pp. 833-844, Hong Kong, China, November 2019. Association for Computational Linguistics. URL https: www. ac web www.oo/ggggooogogo/onnooo 9
L. Xie and A. Yuille. Genetic enn. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1388-1397, 2017.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. Double embeddings and CNN-based sequence labeling for aspect extraction. In Proceedings of the 56th Anmual Meeting of the Associa- fion for Computational Linguistics (Volume 2: Short Papers), pp. 592-598, Melbourne, Aus- tralia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/P18-2000 URL https: / www aclweb. org/ anthology P18-2094
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. BERT post-training for review reading compre- hension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 2324-2335, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/1//N19-1222. URL
https: / www ac www.waclweb.oor. org wwwpgg/anhhoogyyyygg
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763 2019.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. Named entity recognition as dependency parsing.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 64706 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/
2020.acl- main. URL https: wtww://wwwww//... www.eb.wwbbbb....... http://wwww.weeeeb..rrraaatoaao/
mmain.
577
Yu Zhang, Zhenghua Li, and Min Zhang. Efficient second-order TreeCRF for neural dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp. 3295-3305, Online, July 2020. Association for Computational Linguistics. doi:
1tt.6653/112222.acllll acl-main. 302
URL https: /www aclweb http://www....eeeeo///ttttoooooo hwtp.ee..rrrg.aaooooo//////////
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2423-2432 2432, 2018
Junru Zhou and Hai Zhao. Head-driven phrase structure grammar parsing on Penn treebank. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 2396-2400 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19-1230 URL
www ac web. http.///g/g.aaooog////sss../
Wei Zhu, Xiaoling Wang. Xipeng Qiu, Yuan Ni, and Guotong Xie. Autotrans: Automating trans- former design via reinforced architecture search. arXiv preprint arXiv:2009.02070, 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
A DETAILED CONFIGURATIONS
We use ISO 639-1 language codes to represent languages in the table
https://en http://www.wiiiieeaaaoo///////// 1_____10_69911__ 1_codes
16

---

Under review as a conference paper at ICLR 2021
A.1 DATASETS
The details of the 6 structured prediction tasks in experiments are shown in below:
NER: We use the corpora of languages from the CONLL 2002 and 2003 shared task (Toong Kim Sang. 2002 Tjong Kim Sang & De Meulder 2003 with standard split.
POS Tagging: We use three datasets, R (Ritter et al. (201T),
(Gimpel
et al. 2011; Owooputi et al., 2013) and Tweebank-v2 (Liu et al. al. 1 8b) datasets (Ritter, ARK and TB-v2 in simplification). We follow the dataset split of (Nguyen et al. 2020).
Chunking: We use CoNLL 2000 (Tjong Kim Sang & Buchholz 2000) for chunking. Since there no standard development set for CoNLL 2000 dataset, we split 10% of the training data as the development set.
Aspect Extraction: Aspect extraction is a subtask of aspect-based sentiment analysis (Pontiki et al. 2014, 2013 2016). The datasets are from the laptop and restaurant domain of SemEval 14, restaurant domain of SemEval 15 and restaurant domain of SemEval 16 shared task (14Lap, 14Res, 15Res and 16Res in short). Additionally, we use another languages in the restaurant domain of SemEval 16 to test our approach in multiple languages. We randomly split 10% of the training data as the development set following Li al. (2019).
Syntactic Dependency Parsing: We use Penn Tree Bank (PTB) 3.0 with the same dataset pre- processing as (Ma et al. 2018).
We use DM, PAS and PSD datasets for semantic dependency parsing (Oepen et al. 2014) for the SemEval 2015 shared task (Oepen et al., 2013). The three datasets have the same sentences but with different formalisms. We use the standard split for SDP. In the split, there are in-domain test sets and out-of-domain test sets for each dataset.
Semantic Dependency Parsing:
Among these tasks, NER, POS tagging, chunking and aspect extraction are sequence-structtree outputs while dependency parsing and semantic dependency parsing the graph-sectured outputs. POS Tagging, chunking and DP are syntactic structured prediction tasks while NER, AE, SDP are semantic structured prediction tasks.
A.2 EVALUATION
To evaluate our models, We use Fl score to evaluate NER, Chunking and AE, use accuracy to evaluate POS Tagging, use unlabeled attachment score (UAS) and labeled attachment score (LAS) to evaluate DP, and use labeled Fl score to evaluate SDP.
A.3 TASK MODELS AND CONTROLLER
batch For sequence-structureed tasks (i.e., NER, POS tagging, chunking, aspect extraction), size of 32 sentences and an SGD optimizer with a learning rate of 0.1. We anneal the learning rate by 0.5 when there is no accuracy improvement on the development set for 5 epochs. We set the maximum training epoch to 150. For graph-structured tasks (i.e., DP and SDP), we use Adam (Kingma & Ba 2013) to optimize the model with learning rate of 0.002. We anneal the learning rate by 0.75 for every 5000 iterations following Dozat & Manning (2017). We set the maximum training epoch to 300. For DP, we run the maximum spanning tree McDonald et al. (2005) algorithm to output valid trees in testing. We fix the hyper-parameters of the task models.
We tune the learning rate for the controller among (0.1,0.2,0.3 0.4, 0.5) and the discount factor among 10. 0.3, 0.5, 0.7, 0.9) on the same dataset in Section 5.2 We search for the hyper-paramette through grid search and find learning rate 0.1 and a discount factor of 0.5 performs the best on the development set. The controller's parameters are initialized to all 0 so that each candidate is selected evenly in the first two time steps. We use Stochastic Gradient Descent (SGD) to optimize the controller. The training time depends on the task and dataset size. Take the English NER CoNLL dataset as an example. It takes 45 GPU hours to train the controller for 30 steps on a single Tesla P100 GPU, which is an acceptable training time in practice.
17

---

Under review as a conference paper at ICLR 2021
A.4 SOURCES OF EMBEDDINGS
The sources of the embeddings that we used are listed in Table
Table 8: The embeddings we used our experiments. The URL is where we downloaded embeddings. Note that we have confirmed that the XLM-R models fine-tuned on CoNLL 2002/2003 datasets are only trained on the training data.
GloVe
Penninatonn et 2014
slp.stan Earrl wct pren leot w/o Love
fastTest
Text
ELMo
(Other longunges)
Schmmnr
Lgggel wwn.eennnnneeeo Lextue
BERT
LEVIR 2 21
-119
M. HERT
DSWI
2019
Liqual cassed
(Geemmn)
Dare quoteen demnt
RERT (Spanish)
docuchile
30 Iin mineareeee col Bomaol In/eer
hase wheniag Caked
BERT (Turkish))
ahmdd
Councan 2020
Dunch)
11 901 mn nn
robe Terge
Finetaned
dut
XLM-R (CoNLL 02 Spanish)
Hugging Face
moe 2n Folwe Larow
Linel mned
con 102 Spal 1d
XLM-R (CoNLL 03 English)
Hugging Face
Toberta"
Ginet
(CoNL. Gemaan)
Hunoiio
Yong
TLgg mg atee 1007 Xcem Jeage Dased
B ADDITIONAL ANALYSIS
B. FINE-TUNED MODELS VERSUS ACE
To fine-tune the embeddings, we use AdamW (Loshchilov & Hutter 2018) optimizer with learning rate 5e-5 and trained the contextualized embeddings with the task for 10 epochs. We use a batch
size of 32 for BERT, M-BERT and use batch size of 16 for XLM-R and Xe.Net A comparison between ACE and the fine-tuned embeddings that we used in ACE is shown Table TO Results show that ACE can further improve the accuracy of fine-tuned models.
Table 9: A comparison between ACE and the fine-tuned embeddings that are used in ACE for NER and POS tagging.
NER
POS
de
de (Revised)
en
CS
nl
Ritter
ARK
TB-v2
BERT-Fine-tune 78.1
82.2
91.0
83.1
83.3
91.2
91.7
94.4
MBERT+Fine-und
81.9
86.2
91.3
87.6
90.7
90.8
91.5
93.9
XLM-R-Fine-tune
85.8
92.9
89.7
93.4
95.0
ACF+Fine-tune
87.0
90.5
93.5
91.7
94.6
93.4
93.8
95.6
Table 10: A comparison between ACE and the fine-tuned embeddings used in ACE for chunking and AE.
Chunk
AE
CoNLI. 2000
14Lap
14Res
15Res
16Res
n1
ru
BERT+Fine-tune
96.7
81.2
87.7
71.8
73.9
76.9
73.1
64.3
75.6
MBERT+Fine-o
96.6
83.5
85.0
69.5
73.6
74.5
71.6
58.8
XLM-R+ Fea--R-Finettttt tune
96.5
81.3
88.4
77.3
78.5
77.8
72.1
75.7
66.7
ACE+Finee-une
97.0
85.0
89.8
78.5
81.2
78.8
76.7
76.7
77.7
B.2 RETRAINING
Most of the work (Zoph & Le 2017 Zoph et al. 2018: Pham et al. 2018b: So et al. 2019 Zhu al.
2020) in NAS retrains the searched neural architecture from scratch so that the prper-parammters of the searched model can be modified trained larger datasets. To show whether searched embedding concatenation is helpful to the task, we retrain the task model with the embedding con- the dataset settings catenations the same dataset from scratch. For the experiment,
18

---

90.8 Under review as a conference paper at ICLR 2021
Table 11: A comparison between ACE and the fine-tuned embeddings that are used in ACE for DP and SDP.
DP
SDP
PTB
DM
PAS
PSD
UAS
LAS
ID
OOD
OOD
ID
OOD
FERT+Finn-tuun
96.6
95.1
94.4
91.4
94.4
93.0
82.0
81.3
ABERT+Fine-ttt 96.5
94.9
93,9
90.4
93.9
92.1
81.2
80.0
XLM-R+Fine-tun 96.6
95.1
94.3
91.1
94.5
92.8
82.0
81.6
XLNNT+Fine-tumm 97.0
95.4
94.9
92.0
94.8
93.4
82.6
82.2
ACE+Fine-tune 97.2
95.7
95.3
92.6
95.3
93.9
83.6
83,2
as in Section 5.3
We train the searched embedding concatenation of each run from ACE times
(therefore, runs for each dataset).
Table 12 shows the comparison between retrained models with the searched embedding concate- nation from ACE and All. The results show that the retrained models are competitive with ACE in SDP and in chunking. However, in another three tasks, the retrained models perform inferior ACE, which shows our approach's 's advantage. The retrained models outperform All in all tasks, which shows the effectiveness of the searched embedding concatenations.
Table 12: A comparison among retrained models, All and ACE. We use the one dataset for each task.
NER
POS
Chunk
AE
DP-UAS
DP-LAS
SDP-ID
SDP-OOD
All
92.4
90.6
96.7
73.2
96.7
95.1
94.3
Retrain
92.6
90.8
96.8
73.6
96.8
95.2
90.9 94.5
94.5
ACE
93.0
91.7
96.8
75.6
96.9
95.3
90.9
B.3 EFFECT OF EMBEDDINGS IN THE SEARCHED EMBEDDING CONCATENATIONS
There is no clear conclusion on what concatenation of embeddings is helpful to most of the tasks. We analyze the best searched embedding concatenations by ACE over different structured outputs, semantic/syntacticc type, and monolingual/mmuliinnnnnn selected by the best concatenations from all experiments of ACE are shown in Table B3 The best emhedding concatenation varies over the output structure, syntactic/semantic level of understand- ing, and the language. The experimental results show that it is essential select embeddings for each kind of task separately. However, we also find that the embeddings are strong in specific set- tings. In comparison to the eequence-structured and graph-structured tasks, we find that M-BERT and ELMo are only frequently selected in ggquence-structurrg always selected in graph-structured tasks. For Flair embeddings, the forward and backward model are evenly selected. We suspect one direction of Flair embeddings is strong enough. Therefore con- catenating the embeddings from two directions together cannot further improve accuracy. For non-contextualized embeddings, pretrained word embeddings are frequently selected in sequence- structured tasks, and character embeddings are not. When we dig deeper into the semantic and syntactic type of these two structured outputs, we find that in all best concatenations, BERT embed- dings are selected in all syntactic sequence-structuee embeddings selected in syntactic graph-structured tasks. In multilingual tasks, all best concate- nations in multilingual NER tasks select M-BERT embeddings while M-BERT is rarely selected in multilingual AE tasks. The monolingual Flair embeddings always selected in NER tasks, and XLM-R is more frequently selected in multilingual tasks than monolingual sequence-structured tasks (SS).
tasks. The percentage
each embedding
tasks while XLM-R embeddings are
tasks, and Flair, M-Flair, word, and XLM-R
19

---

Under review as a conference paper at ICLR 2021
Table 13: The percentage of each embedding candidate selected in the best concatenations from ACE. F and MF are monolingual and multilingual Flair embeddings. We count these two em- beddings are selected one of the forward/backward (fw/bw) direction of Flair selected in the concatenation. We count the Word embedding is selected if one of the fastText/GloVe Ve embeddings selected. SS: sequence-struuctrre. tasks. GS: graph-structure Syn.: Syntactic-level tasks. M-NER: Multilingual NER tasks. M-AE: Multilingual AE tasks. We only use English datasets in SS and GS. English datasets are removed for M-NER and M-AE.
tasks. Sem.: Semantic-level tasks.
BERT M-BERT Char
EL.o
F-bw
F-Aw MF
MF-hw MF-fw
Word
XLM-R
SS
0.81
0.74
0.37
0.85
0.70
0.48
0.59
0.78
0.59
0.41
0.81
0.70
GS
0.75
0.17
0.50
0.25
0.83
0.75
0.42
0.83
0.58
0.58
0.50
1.00
Sem. SS
0.67
0.73
0.40
0.80
0.60
0.40
0.53
0.87
0.60
0.53
0.80
0.60
Syn. SS
1.00
0.75
0.33
0.92
0.83
0.58
0.67
0.67
0.58
0.25
0.83
0.83
Sem. GS
0.78
0.22
0.67
0.33
0.78
0.67
0.56
0.78
0.56
0.67
0.33
1.00
Syn. GS
0.67
0.00
0.00
0.00
1.00
1.00
0.00
1.00
0.67
0.33
1.00
1.00
M-NER
0.67
1.00
0.56
0.83
1.00
0.78
1.00
0.78
0.44
0.78
0.89
M-AE
1.00
0.33
0.75
0.33
0.58
0.42
0.42
0.75
0.25
0.75
0.50
0.92
20


Reviews:
[{"review_id": "nQ-wfd0Xx7a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n \nThis paper proposes to automate the concatenation of word embeddings (obtained using different strategies) to produce powerful word representations for a given downstream task. To this end, the paper develops an approach based on Neural Architecture Search, wherein the search space is comprised of embedding candidates obtained using different concatenations. Using an accuracy-based reward function, it is showed that ACE can determine more effective concatenations. ACE is evaluated using  extensive experiments with different tasks and datasets, and it outperforms the two baselines (to different degrees) -- random search and concatenating all embeddings with no subselection.\n\n##########################################################################\n\nPositives:\n- The idea of using NAS to construct concatenated embeddings is interesting and the formulation is clearly developed in the paper.\n- The proposed approach is generic and can support different types of structured outputs (sequences, graphs etc.)\n- The search process is computationally efficient and can be even run on a single GPU.\n- A simple modification (based on a discount factor) is proposed to the reward function design that leads to non-trivial performance improvements.\n- Strong experiment design: The proposed approach is evaluated on a large suite of datasets and tasks, and in many cases \n\nConcerns:\n- While the overall idea is interesting, the design choices made in the paper are not fully justified. Since ACE already pretrains the task model for each of the embeddings independently to begin with, why not adopt a \"boosting\" style approach instead of the naive \"ALL\" baseline. It is not surprising that even random search (known to be a strong baseline) consistently outperforms \"ALL\". The key challenge in concatenating disparate emebddings is that they can predict with varying degrees of confidence in different parts of the data and sequential inclusion of embeddings could be effective. In my opinion, the baselines chosen for concatenation are weak.\n- Why is \"accuracy\" the best choice for reward design? There could be two different embeddings that could produce the same accuracy with varying levels of confidence (or empirical calibration). Unlike conventional ensemble learners, each contextualized representation is not a weak learner and hence it will be critical to take into account confidence estimates.\n\nOverall, though the paper is experimentally strong, the design choices and the baselines need to be better justified.\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease respond to the questions under concerns.\n\n##########################################################################\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aycUbSu26Y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updates after discussion/revision period:\n\nI think the revisions have improved the paper, but I'm not willing to increase my score or to fight for the paper. Overall, I think the paper represents a minor contribution, with its rigorous experimentation and some of its ideas, and that others may benefit from reading it, but I don't know that it is at the level of a typical ICLR publication. \n\n\n--------\n\nThis paper describes an approach to choosing a subset of several options for (optionally contextualized) word embeddings to use in NLP tasks. Ideas are drawn from neural architecture search (NAS) and RL. The basic idea is to maximize dev set accuracy by searching over the space of embedding sets to use. There are some tweaks, including avoiding retraining-from-scratch for each set by keeping a single generalized model with all embeddings in it (where subsets can be chosen by setting some matrices to zero). Experiments are done on many NLP tasks (tagging, chunking, NER, parsing, etc.), leading to state-of-the-art results on nearly all test sets. \n\nThis paper represents an impressive number of experiments, considering several types of embeddings and many NLP tasks/datasets. It is well-written on the whole, though there are a few things I had confusion or concern about (details below). I lean positive on this paper, as it has an interesting algorithm that is more practical than prior work in NAS and has some promising results. However, I also have a few high-level concerns, described below:\n\n1. I'm not sure if I'm thoroughly convinced of the empirical superiority of ACE. The primary baselines are All (using all embeddings always) and Random (random search over subsets of embeddings). Random is a little better than All on average, and ACE is a little better than Random on average. The average difference of 0.5 in Table 1 between ACE and Random is largely due to the 8 aspect extraction (AE) datasets, for which the differences are sometimes sizable. However, across the 17 other results in the table (tagging, NER, chunking, and parsing), none differ by more than 0.4, and the average difference between Random and ACE on those other 17 numbers is 0.17 (computed by me). Possibly statistically significant, especially because there are so many different datasets, but less impressive. If one were to deploy a method like this in practice, one would likely start by trying random search because it's so simple and doesn't require the slightly specialized learning framework and reward function in ACE.  \n\nRelatedly, I am concerned that the All baseline is not strong enough. Another natural baseline would be to start with All but then add a_l parameters as gates on the input embeddings, just as they are present in ACE. (The a_l parameters could be normalized to be between 0 and 1 by passing them each through a sigmoid before being multiplied with embedding vectors.) By having a single parameter to weight each embedding type in this way, the new version of All could switch on or off entire embedding types without adding many more parameters, which would make it more similar to the other methods. This would let us see the results of this stronger version of All (which, I would argue, is more likely to be used in practice than the current version of All). \n\n2. My second concern is about the following sentence in Sec. 4.2: \"If the pretrained contextualized embeddings are not available for a particular language, we use the pretrained contextualized embeddings for English instead.\" I find this to be a rather surprising decision, as it could add a great deal of noise for non-English languages. This could be especially problematic for the All baseline which doesn't have an easy way to switch off a noise type of embedding. It would be nice to know for which tasks/datasets this English embedding replacement was done in practice. I looked at Appendix A.4, but I wasn't able to determine from that section which embedding types were missing for which datasets. \n\n3. CoNLL 2003 does not contain gold chunk labels. It contains automatic chunk labels (as can be confirmed by checking the original paper). CoNLL 2003 should not be used for chunking experiments. Unfortunately this mistake has been repeated in many papers. Please remove all CoNLL 2003 chunking experiments. \n\nSome additional (less major) questions are below:\n\nIn Appendix B.2, why does ACE work better than retraining? I wouldn't have expected this to happen. \n\nIn Sec. 3.1, it's odd that the BiLSTM-CRF and BiLSTM-Biaffine functions only take in V as the only argument, where V is a function solely of the input x, not of the output y. Why is it not a function of y as well?\n\nIn Sec. 3.2, I find the phrasing \"concatenation with the mask\" to be a bit confusing. I don't think the mask is being concatenated; I think it's being multiplied elementwise with the embeddings. \n\nRight above Eq. (7), there is the text \"Taking m = 1\" -- what is m?\n\nIn Sec. 4.2: What exactly is meant by \"character embeddings\"? There are many ways to embed words using characters. How are the character embeddings composed to form a word embedding?\n\nHow was the random search done? I see the sentence \"For Random, we use the same training settings as our approach\", which makes me assume there were 30 steps, but I think this should be made more explicit. Since the random approach and ACE are different algorithms with different hyperparameters, it's not clear to me what is meant by using \"the same training settings\". \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzpGk_O_TP4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores a way of learning how to automatically construct a concatenated set of embeddings for structured prediction tasks in NLP. The paper's model takes up to L embeddings concatenated together and feeds them into standard models (BiLSTM-CRFs or the BiLSTM-Biaffine technique of Dozat and Manning) to tackle problems like POS tagging, NER, dependency parsing, and more.  Search over embedding concaneations is expressed as a search over binary masks of length L.  The controller for this search is parameterized by an independent Bernoulli for each mask position.  The paper's approach learns the controller parameters with policy gradient, where the reward function is (a modified version of) the accuracy on the development set for the given task. This modified reward uses all samples throughout training to effectively get a more fine-grained baseline for the current timestep based on prior samples.  Notably, the paper uses embeddings that are already fine-tuned for each task, as fine-tuning the concatenated embeddings is hard due to divergent step sizes and steep computational requirements.\n\nResults show gains over randomly searching the space of binary masks. The overall model outperforms XLM-R in a range of multilingual settings.\n\nThis paper has some nice empirical results and the simplicity of its approach is attractive. But there are two shortcomings of the paper I will discuss.\n\nMOTIVATION/COMPARISONS\n\nThe authors motivate their technique by drawing parallels to neural architecture search. But I actually think what the authors are doing more closely resembles ensembling, system combination, or model stacking, e.g.:\nhttps://www.aclweb.org/anthology/N09-2064.pdf\nhttps://www.aclweb.org/anthology/N18-1201.pdf\n\nWhen you take large Transformer models (I have to imagine that the Transformers are contributing more to the performance than GloVe and other static word embeddings -- and Table 11 supports this somewhat) and staple a BiLSTM-CRF on top of them, most of the computation is happening in the (fixed) large Transformer. Most NAS methods I'm familiar with re-learn fundamental aspects of the architecture (e.g., the Evolved Transformer), while fixing most of the architecture and re-learning a last layer or two is more suggestive of system combination or model stacking.\n\nMy main question is: did the authors try comparing to an ensemble or post-hoc combination of the predictions according to different models?  Computationally this would be cheaper than what the authors did. It's also much faster to search over 2^L-1 possibilities when checking each possibility just requires decoding the dev set rather than training the BiLSTM-CRF -- actually, this can be done very efficiently if each model's logits are cached.\n\nThere are more sophisticated variants of this like in the papers I linked above where each model has its own weights or additional inputs are used. Intellectually, I think these approaches are related, and they should be discussed and compared to.\n\nRESULTS\n\nAs for the results, Table 1's gains are small -- they are consistent over random search, but I don't find them all that convincing.  There are too many embeddings here for ALL to work well -- my guess would be that a smaller set would yield better performance.\n\nTables 2-4 show improvements over existing baselines, XLM-R, and XLNet. This performance is commendable. However, again, I don't know how this compares to ensembling across a few leading approaches (like mBERT and XLM-R for the cross-lingual tasks).\n\nCONCLUSION\n\nIn the end, I'm not sure how readily this approach will be picked up by others. Because the embeddings aren't themselves fine-tuned as part of the ensemble, it really feels more like a fine-tuned ensemble of existing models rather than true NAS. And the overhead of this approach is significant: it requires running many training runs over large collections of existing pre-trained models to get a small improvement over the current state-of-the-art. This is a possibly useful datapoint to have in the literature, but it feels like the technique isn't quite right to lead to more work in this area.\n\nMINOR:\n\n\"use BiLSTM-Biaffine model (Dozat & Manning, 2017) for graph-structured outputs\"\n\nThis is a very particular structure, namely a directed minimum spanning tree (MST), though projective trees are also possible using the Eisner algorithm. The paper should specify that it's these, and not arbitrary graphs that are being produced here.\n\n-------------------\n\nUPDATE AFTER RESPONSE\n\nThanks for the response and the additional experiments. The comparison between ACE and these other techniques is nice to see, although I'll note that both SWAF and voting shouldn't make totally independent predictions in tasks like NER, but should at least respect constraints in the label space (not sure if there were applied or not).\n\nIn the end, my opinion of this paper largely comes down to the practicality of this technique and its likelihood to be adopted more generally. This results in a large, complex model, and while I am now convinced that the authors have a better ensembling/combination technique than some others, I think it still falls short of a real \"neural architecture search\" contribution or a really exciting result.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NyAPJqvXtyH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced an interesting application of reinforcement learning in the selection of concatenation of contextual/non-contextual word embeddings.  It is clever to limit the search space on the selection of embedding sources rather than search the whole network structure, as the current strategy is much easier in the training step. The author(s) conducted many experiments that compared many other models (including SOTA models, and ablation study). Those results are pretty good and impressive.\n\nThe main concern is the necessity of using the concatenation of those contextual embeddings. Calculating different contextual embeddings will cost many computing resources and affect the model's speed. Instead of using reinforcement learning to learn the concatenation of different embeddings, why not use the ensemble model to aggregate the results from different contextual embedding-based models? For example, we can use three embeddings BERT, ELMO, Glove to build three separate models and then aggerate their predicted results, the computing speed/resource may be similar to the learned embedding concatenation model, but is it possible that the ensemble model outperforms the ACE model? More experiments of the comparison with ensemble models should be conducted to prove the necessity of ACE. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for using multiple word embeddings in structured prediction tasks. The reviewers shared the concerns that the method seems rather specific to this use case and the empirical improvements do not justify the complexity of the approach. They also questioned the definition of the method as "architecture search" vs a particular ensembling method.
Finally, I think the authors should provide more discussion of why using all the embeddings (in the sense of bias-variance tradeoffs).


Decision: Reject