Venue: ICLR 2017 conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: this paper proposes to use feed-forward neural networks to learn similarity preserving embeddings. They also use the proposed idea to represent out-of-vocabulary words using the words in given context. 

First, considering the related work [1,2] the proposed approach brings marginal novelty. Especially
Context Encoders is just a small improvement over word2vec. 

Experimental setup should provide more convincing results other than visualizations and non-standard benchmark for NER evaluation with word vectors [3].

[1] http://papers.nips.cc/paper/5477-scalable-non-linear-learning-with-adaptive-polynomial-expansions.pdf
[2] http://deeplearning.cs.cmu.edu/pdfs/OJA.pca.pdf
[3] http://www.anthology.aclweb.org/P/P10/P10-1040.pdf
review_title: marginal novelty

final_score: rating: null
review_content: There are several recent papers by Oren Melamud that deal with embedding contexts and using them to generate context-sensitive representations or similarities:
* Modeling Word Meaning in Context with Substitute Vectors
* A Simple Word Embedding Model for Lexical Substitution
* context2vec: Learning Generic Context Embedding with Bidirectional LSTM
I think it is worth discussing how this work relates to Melamud's, and perhaps even compare the methods empirically.
review_title: Related literature on context embeddings

final_score: rating: 3: Clear rejection
review_content: This paper introduces a similarity encoder based on a standard feed-forward neural network with the aim of generating similarity-preserving embeddings. The approach is utilized to generate a simple extension of the CBOW word2vec model that transforms the learned embeddings by their average context vectors. Experiments are performed on an analogy task and named entity recognition.

While this paper offers some reasonable intuitive arguments for why a feed-forward neural network can generate good similarity-preserving embeddings, the architecture and approach is far from novel. As far as I can tell, the model is nothing more than the most vanilla neural network trained with SGD on similarity signals.

Slightly more original is the idea to use context embeddings to augment the expressive capacity of learned word representations. Of course, using explicit contextual information is not a new idea, especially for tasks like word sense disambiguation (see, e.g., 'Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space' by Neelakantan et al, which should also be cited), but the specific method used here is original, as far as I know.

The evaluation of the method is far from convincing. The corpora used to train the embeddings are far smaller than would ever be used in practice for unsupervised or semi-supervised embedding learning. The performance on the analogy task says little about the benefit of this method for larger corpora, and, as the authors mentioned in the comments, they expect "the gain will be less significant, as the global context statistics brought in by the ConEc can also be picked up by word2vec with more training."

The argument can be made (and the authors do claim) that extrinsic evaluations are more important for real-world applications, so it is good to see experiments on NER. However, again the embeddings were trained on a very small corpus and I am not convinced that the observed benefit will persist when trained on larger corpora.

Overall, I believe this paper offers little novelty and weak experimental evidence supporting its claims. I cannot recommend it for acceptance.
review_title: Standard feed-forward neural net with unconvincing experimental results

final_score: rating: 3: Clear rejection
review_content: This paper presents a method for embedding data instances into a low-dimensional space that preserves some form of similarity.

Although the paper presents this notion as new, basically every pre-trained embedding (be it auto-encoders or word2vec) has been doing the same: representing items in a low-dimensional space that inherently encodes their similarities. Even when looking at the specific case of word/context embeddings, the method is not novel either: this method is almost identical to one of the similarity functions presented in "A Simple Word Embedding Model for Lexical Substitution" (Melamud et al., 2015). The novelty claim must be more accurate and position itself with respect to existing work.

In addition, I think the evaluation could be done better. There are plenty of benchmarks for word embeddings in context, for example: 
* http://veceval.com/ (Nayak et al., RepEval 2016).
* Lexical Substitution in Context
And many higher-level tasks where word similarity in context could be a game-changer:
* Semantic Text Similarity
* Recognizing Textual Entailment / Natural Language Inference
I was disappointed that none of these were even brought up.
review_title: Novelty claim is false, evaluation is partial

final_score: rating: null
review_content: Why did you train your embedding model on such a small corpus (text8)? Most other embedding models are trained on much larger corpora (say, ~1B tokens), so it makes it difficult to interpret your results in the broader context of the literature.

Do you still see substantial gains from the context encoder when training on larger corpora?
review_title: Analogy task

final_score: rating: null
review_content: - why do you enforce the last layer of the first part of the NN be linear?
- I don't understand the argument related to vanishing gradient since your model is not deep or have backprop-through-time. can you elaborate on that?

suggestions:
- you should refer to Oja's work on showing the relationship between neural networks and pca http://deeplearning.cs.cmu.edu/pdfs/OJA.pca.pdf

- I feel like to evaluate representation power of the context or modelling unknown words, an open-vocabulary language modelling on morphologically rich languages or word similarity in context tasks would be better option than an extrinsic task NER.
review_title: questions and suggestions

Meta-review: There is consensus among the reviewers that the novelty of the paper is limited, and that the experimental evaluation of the proposed method needs to be improved.
Decision: Reject