Title: Deep contextualized word representations
Authors: Matthew E Peters
Mark Neumann
Mohit Iyyer
Matt Gardner
Christopher Clark
Kenton Lee
Luke Zettlemoyer
Venue: ICLR 2018
Keywords: representation learning
contextualized word embeddings

Abstract: We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).  Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pretrained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis.  We also present an analysis showing that exposing the deep internals of the pretrained network is crucial, allowing downstream models to mix different types of semi-supervision signals.


TL;DR: We introduce a new type of deep contextualized word representation that significantly improves the state of the art for a range of challenging NLP tasks.

Review discussion:
The authors learn token embeddings that use surrounding context by concatenating representations obtained by training a bidirectional language model, very similar to Peters et al. 2017. They learn a distribution of weights for each layer of embeddings of the pre-trained bi-lm language model. These embeddings improve accuracies over a large range of tasks.
The technical contribution of the paper seems minimal on top of Peters et al.: Learning weights for every layer of embeddings, adding dropout, and adding a regularization term to the training. They have evaluated the efficacy of these embeddings on more tasks than the original paper that introduced them. As it stands, there is not enough novelty in this paper. Answering the questions from their future work would be interesting and would add more technical depth to the paper.
We disagree about the lack of novelty from Peters et al. (2017).  The BiLM architecture is unified and improved, as compared to Peters et al., who used off-the-shelf independently trained left-to-right and right-to-left models.
Furthermore, exposing the representations from all layers in the biLM to the task models via our scalar weighting function is a completely new idea. We show it is crucial to obtain the best performance, as demonstrated in Table 2.
Overall, our method works extremely well in practice, despite its technical simplicity. We think this fact should be considered a major strength of the work.
This paper proposed a model called ‘deep contextualized’ to extract word embeddings for downstream applications. This model is simply a bi-directional language model (biLM) and the word embedding is a weighted combination of the output of the hidden layers (forward and backward).
Furthermore like previous work, the authors proposed to pre-train the biLM with a large amount of data and then use the embeddings in combination with the context-independent word embedding in neural network models (in this paper, RNN) for final applications. Their results showed consistent improvements over the baseline and the previous best systems on several tasks.
Some detailed comments:
-	I would like to see an overview figure which illustrates the biLM and its integration in downstream applications.
Furthermore, it is also interesting to see the performance difference between with and without fine-tuning of the biLM.
-	Intrinsic evaluations are missing in this paper.
Although we know that word similarity tasks are not the best way to evaluate word embeddings, it is always informative to report results on these standard tasks.
-	The baseline results in table 1 and table 2 are not consistent. Is there something wrong?
-	Did the baseline systems use pretrained embeddings or randomly initialized ones? In my opinion, the right baseline systems should use pretrained word2vec or glove embeddings.
Our overall goal in this work is to improve performance on a wide range of NLP tasks, hence the primary focus on extrinsic evaluation.
However, we did provide intrinsic evaluations of the biLM’s contextual representations compared to those from a NMT system in Section 5.3.  As contextual representations are fundamentally different than traditional word embeddings, they require a different set of intrinsic evaluations.
Moreover, it’s likely that the WSD and POS evaluations in Section 5.3 are better correlated with extrinsic performance than the traditional intrinsic word embedding evaluations. This is an interesting area for future work.
Minor points:
1.  “The baseline results in table 1 and table 2 are not consistent. Is there something wrong?”  As mentioned in the captions, Table 1 evaluates on the test set, while Table 2 shows ablations on the development set.
2.  “Did the baseline systems use pretrained embeddings or randomly initialized ones? In my opinion, the right baseline systems should use pretrained word2vec or glove embeddings.”  The baseline systems are state-of-the-art systems.  They all use pre-trained embeddings as described in Section 3.3 and the model details in the Appendix.
This paper proposes a method to learn contextualized word representations (ELMO) by pretraining a multilayer bidirectional LSTM language model and using representations from all levels of the LSTM in the input or output layer of a supervised task of interest.
Experiments on various datasets (SNLI, SQuAD, SRL, Coref, NER, SST) show that the proposed method improve over baseline models.
Ablation analysis demonstrate that using all layers of ELMO is always better than just using only the final layer, and that representations learned by ELMO capture basic notions of word senses and part of speeches.
The paper is well written and I think learning contextualized word representations is an important topic.
However, one thing that I am not sure about from experiments in the paper is whether the improvements come from an increase in model capacity and (unlabeled) data used to train the model, or whether there are more interesting things going on.
- What makes the proposed approach different than just a deeper architecture for each of the considered tasks, where some parts of the network are trained using unlabeled data?
- Is the pretraining with unlabeled data necessary, or can we just have this deep architecture and train everything with the available supervised data?
- An ELMO enhanced model has more parameters than the baseline model for each task. What is the performance of the (non deep) baseline method with comparable number of parameters (bigger hidden size)?
More generally, it is not surprising that given sufficient training data, a deeper model (e.g., ELMO enhanced models) with multiple connections across layers will perform better than shallower models with fewer parameters.
I would like to see more analysis and/or explanations on why the proposed method contributes more beyond this.
The improvements in our paper are not just due to increasing model capacity.  The contextual ELMo representations are a fundamentally different type of word representation then GloVe and word2vec.  The biLM encodes information not available by simply increasing the hidden size of current supervised models and allowing supervised models to preferentially access the internal representations provides the best performance.
We can conclude this because:
1.  Our baseline models are state-of-the-art systems that have been carefully tuned by their respective authors.  If solely increasing model capacity improved performance the original authors would have done so.
2.  Adding ELMo improved performance significantly on every considered NLP task, generally with 10-20% relative error reduction.
3.  In our ablation results (Table 2), the combination of all biLM layers is essential for the highest performance.
4.  Pre-training with unlabeled data is necessary, as shown in previous work (Peters et al 2017 showed that training the language model at a large scale on a large unsupervised corpora was crucial for the NER task).
Certainly, the unsupervised biLM objective allows us to train a high capacity model, similar to traditional word embeddings.
However, we note the pretrained component of the biLM used to compute ELMo representations has about 93 million parameters, whereas the 300 dimensional GloVe vectors have about 650 million parameters.  ELMo’s advantage is in the contextual nature of the representations and the manner in which we integrated it in the downstream task, not from increased capacity over GloVe.
Conceptually, our proposed approach is similar to a deep multitask architecture across tasks, where a piece of the architecture is shared and trained on unlabeled data (the ELMo piece).
However, that still leaves a huge space of possible architectures.  Concretely, one of our paper’s contributions is to propose a particular instantiation of this architecture that works extremely well in practice.
As suggested by the meta reviewer, the empirical results in this paper are better suited for a NLP conference then ICLR.
As a result, we are withdrawing it from ICLR.  Thank you to the Chairs for your consideration and paper acceptance.

Meta-review:
role: PC
data: comment: This is a good paper that presents state-of-the-art results on a number of challenging NLP tasks. The idea is fairly simple and clean, I therefore expect it to get adopted in the community. It also seems to work across several tasks, which is nice. At the same time it is fairly simple (train an LSTM language model and use representations from all levels of the LSTM in the input or output layer of a supervised task of interest) and hence may be more appropriate for an NLP conference than ICLR?

It is a good paper and it will get cited even though the ML contributions are modest. Accept due to the strong empirical results.

Decision: Accept (Poster)