Task: Language Identification
Input (Spanish, English): This paper proposes a framework for evaluation of word embeddings based on data efficiency and simple supervised tasks. The main motivation is that word embeddings are generally used in a transfer learning setting, where evaluation is done based on how faster is to train a target model. The approach uses a set of simple tasks evaluated in a supervised fashion, including common benchmarks such as word similarity and word analogy. Experiments on a broad set of embeddings show that ranks tend to be task-specific and change according to the amount of training data used.  Strengths  - The transfer learning / data efficiency motivation is an interesting one, as it directly relates to the idea of using embeddings as a simple "semi-supervised" approach.  Weaknesses  - A good evaluation approach would be one that propagates to end tasks. Specifically, if the approach gives some rank R for a set of embeddings, I would like it to follow the same rank for an end task like text classification, parsing or machine translation. However, the approach is not assessed in this way so it is difficult to trust the technique is actually more useful than what is traditionally done. - The discussion about injective embeddings seems completely out-of-topic and does not seem to add to the paper's understanding. - The experimental section is very confusing. Section 3.7 points out that the analysis results in answers to questions as "is it worth fitting syntax specific embeddings even when supervised datset is large?" but I fail to understand where in the evaluation the conclusion was made. - Still in Section 3.7, the manuscript says "This hints, that purely unsupervised large scale pretraining might not be suitable for NLP applications". This is a very bold assumption and I again fail to understand how this can be concluded from the proposed evaluation approach. - All embeddings were obtained as off-the-shelf pretrained ones so there is no control over which corpora they were trained on. This limits the validity of the evaluation shown in the paper. - The manuscript needs proofreading, especially in terms of citing figures in the right places (why Figure 1, which is on page 3, is only cited in page 6?).  General Discussion  I think the paper starts with a very interesting motivation but it does not properly evaluate if their approach is good or not. As mentioned above, for any intrinsic evaluation approach I expect to see some study if the conclusions propagate to end tasks and this is not done in the paper. The lack of clarity and proofreading in the manuscript also hinders the understanding. In the future, I think the paper would vastly benefit from some extrinsic studies and a more controlled experimental setting (using the same corpora to train all embeddings, for instance). But in the current state I do not think it is a good addition to the conference.
Output (English): en