Task: Language Identification
Input (Spanish, English): - Strengths: Zero-shot relation extraction is an interesting problem. The authors have created a large dataset for relation extraction as question answering which would likely be useful to the community.  - Weaknesses: Comparison and credit to existing work is severely lacking. Contributions of the paper don't seen particularly novel.  - General Discussion:  The authors perform relation extraction as reading comprehension. In order to train reading comprehension models to perform relation extraction, they create a large dataset of 30m “querified” (converted to natural language) relations by asking mechanical turk annotators to write natural language queries for relations from a schema. They use the reading comprehension model of Seo et al. 2016, adding the ability to return “no relation,” as the original model must always return an answer. The main motivation/result of the paper appears to be that the authors can perform zero-shot relation extraction, extracting relations only seen at test time.  This paper is well-written and the idea is interesting. However, there are insufficient experiments and comparison to previous work to convince me that the paper’s contributions are novel and impactful.  First, the authors are missing a great deal of related work: Neelakantan at al. 2015 ([Link] perform zero-shot relation extraction using RNNs over KB paths. Verga et al. 2017 ([Link] perform relation extraction on unseen entities. The authors cite Bordes et al. ([Link] who collect a similar dataset and perform relation extraction using memory networks (which are commonly used for reading comprehension). However, they merely note that their data was annotated at the “relation” level rather than at the triple (relation, entity pair) level… but couldn’t Bordes et al. have done the same in their annotation? If there is some significant difference here, it is not made clear in the paper. There is also a NAACL 2016 paper ([Link] which performs relation extraction using a new model based on memory networks… and I’m sure there are more. Your work is so similar to much of this work that you should really cite and establish novelty wrt at least some of them as early as the introduction -- that's how early I was wondering how your work differed, and it was not made clear.  Second, the authors neither 1) evaluate their model on another dataset or 2) evaluate any previously published models on their dataset. This makes their empirical results extremely weak. Given that there is a wealth of existing work that performs the same task and the lack of novelty of this work, the authors need to include experiments that demonstrate that their technique outperforms others on this task, or otherwise show that their dataset is superior to others (e.g. since it is much larger than previous, does it allow for better generalization?)
Output (English): en