Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper examines ways of producing word embeddings for rare words on demand. The key real-world use case is for domain specific terms, but here the techniques are demonstrated on rarer words in standard data sets. The strength of this paper is that it both gives a more systematic framework for and builds on existing ideas (character-based models, using dictionary definitions) to implement them as part of a model trained on the end task.

The contribution is clear but not huge. In general, for the scope of the paper, it seems like what is here could fairly easily have been made into a short paper for other conferences that have that category. The basic method easily fits within 3 pages, and while the presentation of the experiments would need to be much briefer, this seems quite possible. More things could have been considered. Some appear in the paper, and there are some fairly natural other ones such as mining some use contexts of a word (such as just from Google snippets) rather than only using textual definitions from wordnet. The contributions are showing that existing work using character-level models and definitions can be improved by optimizing representation learning in the context of the final task, and the idea of adding a learned linear transformation matrix inside the mean pooling model (p.3). However, it is not made very clear why this matrix is needed or what the qualitative effect of its addition is.

The paper is clearly written. 

A paper that should be referred to is the (short) paper of Dhingra et al. (2017): A Comparative Study of Word Embeddings
for Reading Comprehension https://arxiv.org/pdf/1703.00993.pdf . While it in no way covers the same ground as this paper it is relevant as follows: This paper assumes a baseline that is also described in that paper of using a fixed vocab and mapping other words to UNK. However, they point out that at least for matching tasks like QA and NLI that one can do better by assigning random vectors on the fly to unknown words. That method could also be considered as a possible approach to compare against here.

Other comments:
 - The paper suggests a couple of times including at the end of the 2nd Intro paragraph that you can't really expect spelling models to perform well in representing the semantics of arbitrary words (which are not morphological derivations, etc.). While this argument has intuitive appeal, it seems to fly in the face of the fact that actually spelling models, including in this paper, seem to do surprisingly well at learning such arbitrary semantics.
 - p.2: You use pretrained GloVe vectors that you do not update. My impression is that people have had mixed results, sometimes better, sometimes worse with updating pretrained vectors or not. Did you try it both ways?
 - fn. 1: Perhaps slightly exaggerates the point being made, since people usually also get good results with the GloVe or word2vec model trained on "only" 6 billion words – 2 orders of magnitude less data.
 - p.4. When no definition is available, is making e_d(w) a zero vector worse than or about the same as using a trained UNK vector?
 - Table 1: The baseline seems reasonable (near enough to the quality of the original Salesforce model from 2016 (66 F1) but well below current best single models of around 76-78 F1. The difference between D1 and D3 does well illustrate that better definition learning is done with backprop from end objective. This model shows the rather strong performance of spelling models – at least on this task – which he again benefit from training in the context of the end objective. 
 - Fig 2: It's weird that only the +dict (left) model learns to connect "In" and "where". The point made in the text between "Where" and "overseas" is perfectly reasonable, but it is a mystery why the base model on the right doesn't learn to associate the common words "where" and "in" both commonly expressing a location.
 - Table 2: These results are interestingly different. Dict is much more useful than spelling here. I guess that is because of the nature of NLI, but it isn't 100% clear why NLI benefits so much more than QA from definitional knowledge.
 - p.7: I was slightly surprised by how small vocabs (3k and 5k words) are said to be optimal for NLI (and similar remarks hold for SQuAD). My impression is that most papers on NLI use much larger vocabs, no?
 - Fig 3: This could really be drawn considerably better: make the dots bigger and their colors more distinct.
 - Table 3: The differences here are quite small and perhaps the least compelling, but the same trends hold.

review_title: Nice examination of learning on-the-fly word embeddings, but a fairly small focused contribution

final_score: rating: 7: Good paper, accept
review_content: 
This paper illustrates a method to compute produce word embeddings on the fly for rare words, using a pragmatic combination of existing ideas:

* Backing off to a separate decoder for rare words a la Luong and Manning (https://arxiv.org/pdf/1604.00788.pdf, should be cited, though the idea might be older).

* Using character-level models a la Ling et al.

* Using dictionary embeddings a la Hill et al.

None of these ideas are new before but I haven’t seen them combined in this way before. This is a very practical idea, well-explained with a thorough set of experiments across three different tasks. The paper is not surprising but this seems like an effective technique for people who want to build effective systems with whatever data they’ve got. 

review_title: Good practical approach for rare (and unseen) word handling

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a method for computing representations for out-of-vocabulary words, e.g. based on their spelling or dictionary definitions. The main difference from previous approaches is that the model is that the embeddings are trained end-to-end for a specific task, rather than trying to produce generically useful embeddings. The method leads to better performance than using no external resources, but not as high performance as using Glove embeddings. The paper is clearly written, and has useful ablation experiments. However, I have a couple of questions/concerns:
- Most of the gains seem to come from using the spelling of the word. As the authors note, this kind of character level modelling has been used in many previous works. 
- I would be slightly surprised if no previous work has used external resources for training word representations using an end-task loss, but I don’t know the area well enough to make specific suggestions 
- I’m a little skeptical about how often this method would really be useful in practice. It seems to assume that you don’t have much unlabelled text (or you’d use Glove), but you probably need a large labelled dataset to learn how to read dictionary definitions well. All the experiments use large tasks - it would be helpful to have an experiment showing an improvement over character-level modelling on a smaller task.
- The results on SQUAD seem pretty weak - 52-64%, compared to the SOTA of 81. It seems like the proposed method is quite generic, so why not apply it to a stronger baseline?

review_title: Review

Meta-review: The pros and cons of the paper can be summarized as follows:

Pros:
* The method of combining together multiple information sources is effective
* Experimental evaluation is thorough

Cons:
* The method is a relatively minor contribution, combining together multiple existing methods to improve word embeddings. This also necessitates the model being at least as complicated as all the constituent models, which might be a barrier to practical applicability

As an auxiliary comment, the title and emphasis on computing embeddings "on the fly" is a bit puzzling. This is certainly not the first paper that is able to calculate word embeddings for unknown words (e.g. all the cited work on character-based or dictionary-based methods can do so as well). If the emphasis is calculating word embeddings just-in-time instead of ahead-of-time, then I would also expect an evaluation of the speed or memory requirements benefits of doing so. Perhaps a better title for the paper would be "integrating multiple information sources in training of word embeddings", or perhaps a more sexy paraphrase of the same.

Overall, the method seems to be solid, but the paper was pushed out by other submissions.

Decision: Reject