Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Why was Glove not included in the comparison?
The paper could use more content.

also, downstream task experiments like the one in the glove paper would strengthen the paper
review_title: glove

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper investigates the issue of whether and how to use syntactic dependencies in unsupervised word representation learning models like CBOW or Skip-Gram, with a focus one the issue of bound (word+dependency type, 'She-nsubj') vs. unbound (word alone, 'She') representations for context at training time. The empirical results are extremely mixed, and no specific novel method consistently outperforms existing methods.

The paper is systematic and I have no major concerns about its soundness. However, I don't think that this paper is of broad interest to the ICLR community. The paper is focused on a fairly narrow detail of representation learning that is entirely specific to NLP, and its results are primarily negative. A short paper at an ACL conference would be a more reasonable target.
review_title: Solid work, but inconclusive and of narrow interest

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper evaluates how different context types affect the quality of word embeddings on a plethora of benchmarks.

I am ambivalent about this paper. On one hand, it continues an important line of work in decoupling various parameters from the embedding algorithms (this time focusing on context); on the other hand, I am not sure I understand what the conclusion from these experiments is. There does not appear to be a significant and consistent advantage to any one context type. Why is this? Are the benchmarks sensitive enough to detect these differences, if they exist?

While I am OK with this paper being accepted, I would rather see a more elaborate version of it, which tries to answer these more fundamental questions.

review_title: null

final_score: rating: null
review_content: I am not sure I understand what the main finding of this paper is, and why it is novel.
Is it that dependency-based contexts are not necessarily better than linear ones? I don't believe this was ever claimed by anyone. The known result is that dependency-based contexts are more functional while linear contexts are more topical, and it dates back to work on sparse vector representations (Pado 2007 / Baroni 2010 / Turney 2012).

review_title: What is the main claim?

final_score: rating: null
review_content: I quite appreciate systematic comparisons like this, but I'm concerned about how will these results will translate to typical uses of word embeddings, since the intrinsic tasks you used aren't typically especially correlated with downstream accuracy.* Do you have any hypotheses about how these results would translate to application?

*See, for example:
http://www.aclweb.org/anthology/D15-1243
http://www.aclweb.org/anthology/D15-1036
https://fd8c9327-a-62cb3a1a-s-sites.googlegroups.com/site/repevalacl16/26_Paper.pdf
https://fd8c9327-a-62cb3a1a-s-sites.googlegroups.com/site/repevalacl16/22_Paper.pdf

review_title: Extrinsic evaluation?

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper analyzes dependency trees vs standard window contexts for word vector learning.
While that's a good goal I believe the paper falls short of a thorough analysis of the subject matter.
It does not analyze Glove like objective functions which often work better than the algorithms used here.
It doesn't compare in absolute terms to other published vectors or models.

It fails to gain any particularly interesting insights that will modify other people's work.
It fails to push the state of the art or make available new resources for people.


review_title: Belowline

final_score: rating: null
review_content: There are many other types of contexts which should be discussed; see "Open IE as an Intermediate Structure for Semantic Tasks" (Stanovsky et al., ACL 2015).
review_title: Additional Context Types

Meta-review: Reviewers agree that the findings are not clear enough to be of interest, though the effort to do a controlled study is appreciated.
Decision: Reject