Task: Language Identification
Input (Spanish, English): First, let me praise the authors for generating and releasing an NLP data set: a socially useful task.  The authors use an algorithm to generate a 500-cluster-per-language data set in semantic similarity. This brings up a few points.  1. If the point of using the algorithm is to be scalable, why release such a small data set? It's roughly the same order of magnitude as the data sets released in the SemEval tasks over the recent years. I would have expected something orders of magnitude larger.  2. The authors hand checked a small subset of the clusters: they found one where it was ambiguous, and should probably have been removed. Mechanical Turk can scale pretty well -- why not post-facto filter all of the clusters using MT? This is (in effect) how ImageNet was created, and it has millions of items.  3. Evaluating data set papers is an tricky issue. What makes a data set "good" or publishable? There are a number of medium-sized NLP data sets released every year (e.g., through SemEval). Those are designed to address tasks in NLP that people find interesting. I don't know of a data set that exactly addresses the task that the authors propose: the task is trying to address the idea of semantic similarity, which has had multiple data sets thrown at it since SemEval 2012. I wish that the paper had included comparisons to show that the particular task / data combination is better suited for analyzing semantic similarity than other existing data sets.  Two final notes:  A. This paper doesn't seem very well-suited to ICLR.  New NLP data sets may be indirectly useful for evaluating word embeddings (and hence representations). But, I didn't learn much from the paper: GloVe is empirically less good for semantic similarity than other embeddings? If true, why? That would be interesting.  B. The first proposal for the "put a word into a cluster and see if it stands out" task (in the context of human evaluation of topic models), is  Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. Reading Tea Leaves: HowHumans Interpret Topic Models. Neural Information Processing Systems, 2009  which deserves a citation, I think.
Output (English): en