- Strengths:  This paper presents a 2 x 2 x 3 x 10 array of accuracy results based on systematically changing the parameters of embeddings models:  (context type, position sensitive, embedding model, task), accuracy  - context type ∈ {Linear, Syntactic} - position sensitive ∈ {True, False} - embedding model ∈ {Skip Gram, BOW, GLOVE} - task ∈ {Word Similarity, Analogies, POS, NER, Chunking, 5 text classific. tasks}  The aim of these experiments was to investigate the variation in performance as these parameters are changed. The goal of the study itself is interesting for the ACL community and similar papers have appeared before as workshop papers and have been well cited, such as Nayak et al.'s paper mentioned below.  - Weaknesses: Since this paper essentially presents the effect of systematically changing the  context types and position sensitivity, I will focus on the execution of the investigation and the analysis of the results, which I am afraid is not  satisfactory.  A) The lack of hyper-parameter tuning is worrisome. E.g.    - 395 Unless otherwise notes, the number of word embedding dimension is set to 500.    - 232 It still enlarges the context vocabulary about 5 times in practice.    - 385 Most hyper-parameters are the same as Levy et al' best configuration.    This is worrisome because lack of hyperparameter tuning makes it difficult to make statements like method A is better than method B. E.g. bound methods may perform better with a lower dimensionality than unbound models, since their effective context vocabulary size is larger.  B) The paper sometimes presents strange explanations for its results. E.g.    - 115 "Experimental results suggest that although it's hard to find any  universal insight, the characteristics of different contexts on different models are concluded according to specific tasks."     What does this sentence even mean?      - 580 Sequence labeling tasks tend to classify words with the same syntax  to the same category. The ignorance of syntax for word embeddings which  are learned by bound representation becomes beneficial.      These two sentences are contradictory, if a sequence labeling task    classified words with "same syntax" to same category then syntx becomes    a ver valuable feature. Bound representation's ignorance of syntax    should cause a drop in performance just like other tasks which does not    happen.  C) It is not enough to merely mention Lai et. al. 2016 who have also done a    systematic study of the word embeddings, and similarly the paper     "Evaluating Word Embeddings Using a Representative Suite of Practical    Tasks", Nayak, Angeli, Manning. appeared at the repeval workshop at     ACL 2016. should have been cited. I understand that the focus of Nayak    et al's paper is not exactly the same as this paper, however they    provide recommendations about hyperparameter tuning and experiment    design and even provide a web interface for automatically running    tagging experiments using neural networks instead of the "simple linear    classifiers" used in the current paper.  D) The paper uses a neural BOW words classifier for the text classification tasks    but a simple linear classifier for the sequence labeling tasks. What is    the justification for this choice of classifiers? Why not use a simple    neural classifier for the tagging tasks as well? I raise this point,    since the tagging task seems to be the only task where bound    representations are consistently beating the unbound representations,    which makes this task the odd one out.   - General Discussion: Finally, I will make one speculative suggestion to the authors regarding the analysis of the data. As I said earlier, this paper's main contribution is an analysis of the following table. (context type, position sensitive, embedding model, task, accuracy) So essentially there are 120 accuracy values that we want to explain in terms of the aspects of the model. It may be beneficial to perform factor analysis or some other pattern mining technique on this 120 sample data.
Output: en