In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].

[Q]: - Strengths: Nicely written and understandable. Clearly organized. Targeted answering of research questions, based on  different experiments.  - Weaknesses: Minimal novelty. The "first sentence" heuristic has been in the summarization literature for many years. This work essentially applies this heuristic (evolved) in the keyword extraction setting. This is NOT to say that the work is trivial: it is just not really novel.  Lack of state-of-the-art/very recent methods. The experiment on the system evaluation vs state-of-the-art systems simply uses strong baselines. Even though the experiment answers the question "does it perform better than baselines?", I am not confident it illustrates that the system performs better than the current state-of-the-art. This somewhat reduces the value of the paper.  - General Discussion: Overall the paper is good and I propose that it be published and presented.   On the other hand, I would propose that the authors position themselves (and the system performance) with respect to: Martinez‐Romo, Juan, Lourdes Araujo, and Andres Duque Fernandez. "SemGraph: Extracting keyphrases following a novel semantic graph‐based approach." Journal of the Association for Information Science and Technology 67.1 (2016): 71-82. (with which the work holds remarkable resemblance in some points)  Le, Tho Thi Ngoc, Minh Le Nguyen, and Akira Shimazu. "Unsupervised Keyphrase Extraction: Introducing New Kinds of Words to Keyphrases." Australasian Joint Conference on Artificial Intelligence. Springer International Publishing, 2016.
[A]: en


[Q]: The present submission discusses a "causal regularizer", which promotes the use of causal dependencies (X -> Y, where X is a feature of the learning problem, and Y is the target variable) in predictive models. Similarly, such causal regularizer penalizes the use of non-causal dependencies, which can arise due to reverse causation (Y -> X) or confounding (X <- Z -> Y, where Z is a hidden confounder).  + Overall, this submission tackles one of the most important problems in machine learning, which is to build causal models. The paper discusses and addresses this issue effectively when applied to a dataset in heart disease. In their experiments, the authors correctly identify some of the common causes of heart disease by virtue of their causal regularizer.  - The authors do not discuss the robustness of their approach with respect to choice of hyper-parameters (both describing the neural network architecture and the generative model that synthesizes artificial causal data). This seems like a crucial issue, in particular when dealing with medical data.  - The conclusions of the experimental evaluation should be discussed in greater length. On the one hand, Figure 4.a shows that there are no differences between L1 and causal regularization in terms of predictive performance, but it is difficult to conclude if this result is statistically significant without access to error-bars. On the other hand, Table 3 describes the qualitative differences between L1 and causal regularization. However, this table is hard to read: How were the 30 rows selected? What does the red highlighting mean? Are these red rows some true causal features that were missed? If so, this is related to precision. What about recall? Did the causal regularization pick up many non-causal features as causal?  - Regarding causal classifiers, this paper should do a much better job at reviewing previous work. For instance, the paper "Towards a Learning Theory of Cause-Effect Inference" from Lopez-Paz et al. is missing from the references. However, this prior work studies many of the aspects that are hinted as novel in this submission. In particular, the prior work of Lopez-Paz 1) introduces the concept of Mother distribution (referred as Nature hyper-prior in this submission) which explicitly factorizes the distribution over causes and mechanisms, 2) circumvented intractable likelihoods by synthesizing and training on causal data, 3) tackled the confounding case (compare Figure 1 of this submission and Appendix C of Lopez-Paz), and 4) dealt with discrete data seamlessly (such as the ChaLearn data from Section 5.3 in Lopez-Paz).  On a positive note, this is a well-written paper that addresses the important, under-appreciated problem of incorporating causal reasoning into machine learning. On a negative note, the novelty of the technical contributions is modest and the qualitative evaluation of the results could be greatly extended. In short, I am leaning slightly towards acceptance.
[A]: en


[Q]: - Strengths:  [+] Well motivated, tackles an interesting problem;  [+] Clearly written and structured, accompanied by documented code and dataset;  [+] Encouraging results.  - Weaknesses:  [-] Limited to completely deterministic, hand-engineered minimization rules;  [-] Some relevant literature on OIE neglected;  [-] Sound but not thorough experimental evaluation.  - General Discussion:  This paper tackles a practical issue of most OIE systems, i.e. redundant, uninformative and inaccurate extractions. The proposed approach, dubbed MinOIE, is designed to actually "minimize" extractions by removing overly specific portions and turning them into structured annotations of various types (similarly to OLLIE). The authors put MinIE on top of a state-of-the-art OIE system (ClausIE) and test it on two publicly available datasets, showing that it effectively leads to more concise extractions compared to standard OIE approaches, while at the same time retaining accuracy.  Overall, this work focuses on an interesting (and perhaps underinvestigated) aspect of OIE in a sound and principled way. The paper is clearly written, sufficiently detailed, and accompanied by supplementary material and a neat Java implementation. My main concern is, however, with the entirely static, deterministic and rule-based structure of MinIE. Even though I understand that a handful of manually engineered rules is technically the best strategy when precision is key, these approaches are typically very hard to scale, e.g. in terms of languages (a recent trend of OIE, see Faruqui and Kumar, 2015; Falke et al., 2016). In other words, I think that this contribution somehow falls short of novelty and substance in proposing a pipeline of engineered rules that are mostly inspired by other OIE systems (such as ClausIE or ReVerb); for instance, I would have really appreciated an attempt to learn these minimization rules instead of hard-coding them.  Furthermore, the authors completely ignore a recent research thread on “semantically-informed” OIE (Nakashole et al., 2012; Moro and Navigli, 2012; 2013; Delli Bovi et al., 2015) where traditional extractions are augmented with links to underlying knowledge bases and sense inventories (Wikipedia, Wikidata, Yago, BabelNet). These contributions are not only relevant in terms of related literature: in fact, having text fragments (or constituents) explicitly linked to a knowledge base would reduce the need for ad-hoc minimization rules such as those in Sections 6.1 and 6.2. In the example with "Bill of Rights" provided by the authors (line 554), an OIE pipeline with a proper Entity Linking module would recognize automatically the phrase as mention of a registered entity, regardless of the shape of its subconstituents. Also, an underlying sense inventory would seamlessly incorporate the external information about collocations and multi-word expressions used in Section 6.2: not by chance, the authors rely on WordNet and Wiktionary to compile their dictionary of collocations.  Finally, some remarks on the experimental evaluation:  - Despite the claim of generality of MinIE, the authors choose to experiment only with ClausIE as underlying OIE system (most likely the optimal match). It would have been very interesting to see if the improvement brought by MinIE is consistent also with other OIE systems, in order to actually assess its flexibility as a post-processing tool.  - Among the test datasets used in Section 7, I would have included the recent OIE benchmark of Stanovsky and Dagan (2016), where results are reported also for comparison systems not included in this paper (TextRunner, WOIE, KrakeN).  References:  - Manaal Faruqui and Shankar Kumar. Multilingual Open Relation Extraction using Cross-lingual Projection. NAACL-HLT, 2015.  - Tobias Falke, Gabriel Stanovsky, Iryna Gurevych and Ido Dagan. Porting an Open Information Extraction System from English to German. EMNLP 2016.  - Ndapandula Nakashole, Gerhard Weikum and Fabian Suchanek. PATTY: A Taxonomy of Relational Patterns with Semantic Types. EMNLP 2012.  - Andrea Moro, Roberto Navigli. WiSeNet: Building a Wikipedia-based Semantic Network with Ontologized Relations. CIKM 2012.  - Andrea Moro, Roberto Navigli. Integrating Syntactic and Semantic Analysis into the Open Information Extraction Paradigm. IJCAI 2013.  - Claudio Delli Bovi, Luca Telesca and Roberto Navigli. Large-Scale Information Extraction from Textual Definitions through Deep Syntactic and Semantic Analysis. TACL vol. 3, 2015.  - Gabriel Stanovsky and Ido Dagan. Creating a Large Benchmark for Open Information Extraction. EMNLP 2016.
[A]:
en