TASK DEFINITION: In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
PROBLEM: - Strengths: Zero-shot relation extraction is an interesting problem. The authors have created a large dataset for relation extraction as question answering which would likely be useful to the community.  - Weaknesses: Comparison and credit to existing work is severely lacking. Contributions of the paper don't seen particularly novel.  - General Discussion:  The authors perform relation extraction as reading comprehension. In order to train reading comprehension models to perform relation extraction, they create a large dataset of 30m “querified” (converted to natural language) relations by asking mechanical turk annotators to write natural language queries for relations from a schema. They use the reading comprehension model of Seo et al. 2016, adding the ability to return “no relation,” as the original model must always return an answer. The main motivation/result of the paper appears to be that the authors can perform zero-shot relation extraction, extracting relations only seen at test time.  This paper is well-written and the idea is interesting. However, there are insufficient experiments and comparison to previous work to convince me that the paper’s contributions are novel and impactful.  First, the authors are missing a great deal of related work: Neelakantan at al. 2015 ([Link] perform zero-shot relation extraction using RNNs over KB paths. Verga et al. 2017 ([Link] perform relation extraction on unseen entities. The authors cite Bordes et al. ([Link] who collect a similar dataset and perform relation extraction using memory networks (which are commonly used for reading comprehension). However, they merely note that their data was annotated at the “relation” level rather than at the triple (relation, entity pair) level… but couldn’t Bordes et al. have done the same in their annotation? If there is some significant difference here, it is not made clear in the paper. There is also a NAACL 2016 paper ([Link] which performs relation extraction using a new model based on memory networks… and I’m sure there are more. Your work is so similar to much of this work that you should really cite and establish novelty wrt at least some of them as early as the introduction -- that's how early I was wondering how your work differed, and it was not made clear.  Second, the authors neither 1) evaluate their model on another dataset or 2) evaluate any previously published models on their dataset. This makes their empirical results extremely weak. Given that there is a wealth of existing work that performs the same task and the lack of novelty of this work, the authors need to include experiments that demonstrate that their technique outperforms others on this task, or otherwise show that their dataset is superior to others (e.g. since it is much larger than previous, does it allow for better generalization?)

SOLUTION: en

PROBLEM: I reviewed the manuscript on December 5th.  Summary: The authors investigate the phenomenon of adversarial perturbations and ask whether one may build a system to independently detect an adversarial data point -- if one could detect an adversarial example, then might prevent a machine from automatically processing it. Importantly, the authors investigate whether it is possible to build an adversarial detector which is resilient to adversarial examples built against *both* the classifier and the detector. Their results suggest that training a detector in this more difficult setting still yields gains but does not entirely resolve the problem of detecting adversarial examples.  Major comments:  The authors describe a novel approach for dealing with adversarial examples from a security standpoint -- namely, build an independent system to detect the adversary so a human might intervene in those cases.   A potential confound of this approach is that an adversary might respond by constructing adversarial examples to fool *both* the original classifier and the new detector. If that were possible, then this approach is moot since an attacker could always outwit the original system. To their credit, the authors show that building a 'dynamic' detector to detect adversarial examples but also be resilient to an adversary mitigates this potential escalation (worse case from 55% to 70% detection rate). Even though the 'dynamic' detector  demonstrates positive gains, I am concerned about overall scores. Detecting adversarial examples at this rate would not be a reliable security procedure.  My second comment is about 'model transferability'. My definition of 'model transferability' is different then the one used in the paper. My definition means that one constructs an adversarial example on one network and measures how well the adversarial examples attack a second trained model -- where the second model has been trained with different initial conditions. (The author's definition of 'transferability' is based on seeing how well the detector generalizes across training methods). 'Model transferability' (per my definition) is quite important because it measures how general an adversarial example is across all models -- and not specific to a given trained model. Different methods have different levels of 'model transferability' (Kurakin et al, 2016) and I am concerned how well the detector they built would be able to detect adversarial examples across *all models* and not just the trained model in question. In other words, a good detector would be able to detect adversarial examples from any network and not just one particularly trained network. This question seems largely unaddressed in this paper but perhaps I missed some subtle point in their descriptions.  Minor comments:  If there were any points in the bottom-left of the Figure 2 left, then this would be very important to see -- perhaps move the legend to highlight if the area contains no points.  - X-axis label is wrong in Figure 2 right.  Measure the transferability of the detector?  - How is \sigma labeled on Figure 5?  - Whenever an image is constructed to be an 'adversary', has the image actually been tested to see if it is adversarial? In other words, does the adversarial image actually result in a misclassification by the original network?

SOLUTION: en

PROBLEM: El trabajo presenta una revisión general de la seguridad de software enfocado en las tecnologías del desarrollo de software y cumple función como trabajo recopilatorio. El contenido se encuentra claramente explicado, no se ven problemas de redacción y puntuación. La bibliografía presenta numerosas referencias a artículos en línea y textos de seguridad informática. Sería interesante que se incluyeran más artículos científicos de ser posible.  Si bien el texto establece como trabajo futuro la realización de un estudio empírico para la selección de tecnologías de seguridad, hubiese sido interesante contar con información preliminar de este enfoque para dar mayor valor al trabajo.

SOLUTION:
es