Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Updates after discussion/revision period:

I think the revisions have improved the paper, but I'm not willing to increase my score or to fight for the paper. Overall, I think the paper represents a minor contribution, with its rigorous experimentation and some of its ideas, and that others may benefit from reading it, but I don't know that it is at the level of a typical ICLR publication. 


--------

This paper describes an approach to choosing a subset of several options for (optionally contextualized) word embeddings to use in NLP tasks. Ideas are drawn from neural architecture search (NAS) and RL. The basic idea is to maximize dev set accuracy by searching over the space of embedding sets to use. There are some tweaks, including avoiding retraining-from-scratch for each set by keeping a single generalized model with all embeddings in it (where subsets can be chosen by setting some matrices to zero). Experiments are done on many NLP tasks (tagging, chunking, NER, parsing, etc.), leading to state-of-the-art results on nearly all test sets. 

This paper represents an impressive number of experiments, considering several types of embeddings and many NLP tasks/datasets. It is well-written on the whole, though there are a few things I had confusion or concern about (details below). I lean positive on this paper, as it has an interesting algorithm that is more practical than prior work in NAS and has some promising results. However, I also have a few high-level concerns, described below:

1. I'm not sure if I'm thoroughly convinced of the empirical superiority of ACE. The primary baselines are All (using all embeddings always) and Random (random search over subsets of embeddings). Random is a little better than All on average, and ACE is a little better than Random on average. The average difference of 0.5 in Table 1 between ACE and Random is largely due to the 8 aspect extraction (AE) datasets, for which the differences are sometimes sizable. However, across the 17 other results in the table (tagging, NER, chunking, and parsing), none differ by more than 0.4, and the average difference between Random and ACE on those other 17 numbers is 0.17 (computed by me). Possibly statistically significant, especially because there are so many different datasets, but less impressive. If one were to deploy a method like this in practice, one would likely start by trying random search because it's so simple and doesn't require the slightly specialized learning framework and reward function in ACE.  

Relatedly, I am concerned that the All baseline is not strong enough. Another natural baseline would be to start with All but then add a_l parameters as gates on the input embeddings, just as they are present in ACE. (The a_l parameters could be normalized to be between 0 and 1 by passing them each through a sigmoid before being multiplied with embedding vectors.) By having a single parameter to weight each embedding type in this way, the new version of All could switch on or off entire embedding types without adding many more parameters, which would make it more similar to the other methods. This would let us see the results of this stronger version of All (which, I would argue, is more likely to be used in practice than the current version of All). 

2. My second concern is about the following sentence in Sec. 4.2: "If the pretrained contextualized embeddings are not available for a particular language, we use the pretrained contextualized embeddings for English instead." I find this to be a rather surprising decision, as it could add a great deal of noise for non-English languages. This could be especially problematic for the All baseline which doesn't have an easy way to switch off a noise type of embedding. It would be nice to know for which tasks/datasets this English embedding replacement was done in practice. I looked at Appendix A.4, but I wasn't able to determine from that section which embedding types were missing for which datasets. 

3. CoNLL 2003 does not contain gold chunk labels. It contains automatic chunk labels (as can be confirmed by checking the original paper). CoNLL 2003 should not be used for chunking experiments. Unfortunately this mistake has been repeated in many papers. Please remove all CoNLL 2003 chunking experiments. 

Some additional (less major) questions are below:

In Appendix B.2, why does ACE work better than retraining? I wouldn't have expected this to happen. 

In Sec. 3.1, it's odd that the BiLSTM-CRF and BiLSTM-Biaffine functions only take in V as the only argument, where V is a function solely of the input x, not of the output y. Why is it not a function of y as well?

In Sec. 3.2, I find the phrasing "concatenation with the mask" to be a bit confusing. I don't think the mask is being concatenated; I think it's being multiplied elementwise with the embeddings. 

Right above Eq. (7), there is the text "Taking m = 1" -- what is m?

In Sec. 4.2: What exactly is meant by "character embeddings"? There are many ways to embed words using characters. How are the character embeddings composed to form a word embedding?

How was the random search done? I see the sentence "For Random, we use the same training settings as our approach", which makes me assume there were 30 steps, but I think this should be made more explicit. Since the random approach and ACE are different algorithms with different hyperparameters, it's not clear to me what is meant by using "the same training settings". 

review_title: reasonably interesting, though some concerns remain

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
 
This paper proposes to automate the concatenation of word embeddings (obtained using different strategies) to produce powerful word representations for a given downstream task. To this end, the paper develops an approach based on Neural Architecture Search, wherein the search space is comprised of embedding candidates obtained using different concatenations. Using an accuracy-based reward function, it is showed that ACE can determine more effective concatenations. ACE is evaluated using  extensive experiments with different tasks and datasets, and it outperforms the two baselines (to different degrees) -- random search and concatenating all embeddings with no subselection.

##########################################################################

Positives:
- The idea of using NAS to construct concatenated embeddings is interesting and the formulation is clearly developed in the paper.
- The proposed approach is generic and can support different types of structured outputs (sequences, graphs etc.)
- The search process is computationally efficient and can be even run on a single GPU.
- A simple modification (based on a discount factor) is proposed to the reward function design that leads to non-trivial performance improvements.
- Strong experiment design: The proposed approach is evaluated on a large suite of datasets and tasks, and in many cases 

Concerns:
- While the overall idea is interesting, the design choices made in the paper are not fully justified. Since ACE already pretrains the task model for each of the embeddings independently to begin with, why not adopt a "boosting" style approach instead of the naive "ALL" baseline. It is not surprising that even random search (known to be a strong baseline) consistently outperforms "ALL". The key challenge in concatenating disparate emebddings is that they can predict with varying degrees of confidence in different parts of the data and sequential inclusion of embeddings could be effective. In my opinion, the baselines chosen for concatenation are weak.
- Why is "accuracy" the best choice for reward design? There could be two different embeddings that could produce the same accuracy with varying levels of confidence (or empirical calibration). Unlike conventional ensemble learners, each contextualized representation is not a weak learner and hence it will be critical to take into account confidence estimates.

Overall, though the paper is experimentally strong, the design choices and the baselines need to be better justified.
##########################################################################

Questions during rebuttal period: 

Please respond to the questions under concerns.

##########################################################################


review_title: AUTOMATED CONCATENATION OF EMBEDDINGS FOR STRUCTURED PREDICTION

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper explores a way of learning how to automatically construct a concatenated set of embeddings for structured prediction tasks in NLP. The paper's model takes up to L embeddings concatenated together and feeds them into standard models (BiLSTM-CRFs or the BiLSTM-Biaffine technique of Dozat and Manning) to tackle problems like POS tagging, NER, dependency parsing, and more.  Search over embedding concaneations is expressed as a search over binary masks of length L.  The controller for this search is parameterized by an independent Bernoulli for each mask position.  The paper's approach learns the controller parameters with policy gradient, where the reward function is (a modified version of) the accuracy on the development set for the given task. This modified reward uses all samples throughout training to effectively get a more fine-grained baseline for the current timestep based on prior samples.  Notably, the paper uses embeddings that are already fine-tuned for each task, as fine-tuning the concatenated embeddings is hard due to divergent step sizes and steep computational requirements.

Results show gains over randomly searching the space of binary masks. The overall model outperforms XLM-R in a range of multilingual settings.

This paper has some nice empirical results and the simplicity of its approach is attractive. But there are two shortcomings of the paper I will discuss.

MOTIVATION/COMPARISONS

The authors motivate their technique by drawing parallels to neural architecture search. But I actually think what the authors are doing more closely resembles ensembling, system combination, or model stacking, e.g.:
https://www.aclweb.org/anthology/N09-2064.pdf
https://www.aclweb.org/anthology/N18-1201.pdf

When you take large Transformer models (I have to imagine that the Transformers are contributing more to the performance than GloVe and other static word embeddings -- and Table 11 supports this somewhat) and staple a BiLSTM-CRF on top of them, most of the computation is happening in the (fixed) large Transformer. Most NAS methods I'm familiar with re-learn fundamental aspects of the architecture (e.g., the Evolved Transformer), while fixing most of the architecture and re-learning a last layer or two is more suggestive of system combination or model stacking.

My main question is: did the authors try comparing to an ensemble or post-hoc combination of the predictions according to different models?  Computationally this would be cheaper than what the authors did. It's also much faster to search over 2^L-1 possibilities when checking each possibility just requires decoding the dev set rather than training the BiLSTM-CRF -- actually, this can be done very efficiently if each model's logits are cached.

There are more sophisticated variants of this like in the papers I linked above where each model has its own weights or additional inputs are used. Intellectually, I think these approaches are related, and they should be discussed and compared to.

RESULTS

As for the results, Table 1's gains are small -- they are consistent over random search, but I don't find them all that convincing.  There are too many embeddings here for ALL to work well -- my guess would be that a smaller set would yield better performance.

Tables 2-4 show improvements over existing baselines, XLM-R, and XLNet. This performance is commendable. However, again, I don't know how this compares to ensembling across a few leading approaches (like mBERT and XLM-R for the cross-lingual tasks).

CONCLUSION

In the end, I'm not sure how readily this approach will be picked up by others. Because the embeddings aren't themselves fine-tuned as part of the ensemble, it really feels more like a fine-tuned ensemble of existing models rather than true NAS. And the overhead of this approach is significant: it requires running many training runs over large collections of existing pre-trained models to get a small improvement over the current state-of-the-art. This is a possibly useful datapoint to have in the literature, but it feels like the technique isn't quite right to lead to more work in this area.

MINOR:

"use BiLSTM-Biaffine model (Dozat & Manning, 2017) for graph-structured outputs"

This is a very particular structure, namely a directed minimum spanning tree (MST), though projective trees are also possible using the Eisner algorithm. The paper should specify that it's these, and not arbitrary graphs that are being produced here.

-------------------

UPDATE AFTER RESPONSE

Thanks for the response and the additional experiments. The comparison between ACE and these other techniques is nice to see, although I'll note that both SWAF and voting shouldn't make totally independent predictions in tasks like NER, but should at least respect constraints in the label space (not sure if there were applied or not).

In the end, my opinion of this paper largely comes down to the practicality of this technique and its likelihood to be adopted more generally. This results in a large, complex model, and while I am now convinced that the authors have a better ensembling/combination technique than some others, I think it still falls short of a real "neural architecture search" contribution or a really exciting result.
review_title: Some nice results, but I'm not fully convinced by the technique

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduced an interesting application of reinforcement learning in the selection of concatenation of contextual/non-contextual word embeddings.  It is clever to limit the search space on the selection of embedding sources rather than search the whole network structure, as the current strategy is much easier in the training step. The author(s) conducted many experiments that compared many other models (including SOTA models, and ablation study). Those results are pretty good and impressive.

The main concern is the necessity of using the concatenation of those contextual embeddings. Calculating different contextual embeddings will cost many computing resources and affect the model's speed. Instead of using reinforcement learning to learn the concatenation of different embeddings, why not use the ensemble model to aggregate the results from different contextual embedding-based models? For example, we can use three embeddings BERT, ELMO, Glove to build three separate models and then aggerate their predicted results, the computing speed/resource may be similar to the learned embedding concatenation model, but is it possible that the ensemble model outperforms the ACE model? More experiments of the comparison with ensemble models should be conducted to prove the necessity of ACE. 
review_title: Interesting, good results, but too heavy and complex

Meta-review: The paper proposes a method for using multiple word embeddings in structured prediction tasks. The reviewers shared the concerns that the method seems rather specific to this use case and the empirical improvements do not justify the complexity of the approach. They also questioned the definition of the method as "architecture search" vs a particular ensembling method.
Finally, I think the authors should provide more discussion of why using all the embeddings (in the sense of bias-variance tradeoffs).


Decision: Reject