Task: Language Identification
Input (Spanish, English): I reviewed the manuscript on December 5th.  Summary: The authors investigate the phenomenon of adversarial perturbations and ask whether one may build a system to independently detect an adversarial data point -- if one could detect an adversarial example, then might prevent a machine from automatically processing it. Importantly, the authors investigate whether it is possible to build an adversarial detector which is resilient to adversarial examples built against *both* the classifier and the detector. Their results suggest that training a detector in this more difficult setting still yields gains but does not entirely resolve the problem of detecting adversarial examples.  Major comments:  The authors describe a novel approach for dealing with adversarial examples from a security standpoint -- namely, build an independent system to detect the adversary so a human might intervene in those cases.   A potential confound of this approach is that an adversary might respond by constructing adversarial examples to fool *both* the original classifier and the new detector. If that were possible, then this approach is moot since an attacker could always outwit the original system. To their credit, the authors show that building a 'dynamic' detector to detect adversarial examples but also be resilient to an adversary mitigates this potential escalation (worse case from 55% to 70% detection rate). Even though the 'dynamic' detector  demonstrates positive gains, I am concerned about overall scores. Detecting adversarial examples at this rate would not be a reliable security procedure.  My second comment is about 'model transferability'. My definition of 'model transferability' is different then the one used in the paper. My definition means that one constructs an adversarial example on one network and measures how well the adversarial examples attack a second trained model -- where the second model has been trained with different initial conditions. (The author's definition of 'transferability' is based on seeing how well the detector generalizes across training methods). 'Model transferability' (per my definition) is quite important because it measures how general an adversarial example is across all models -- and not specific to a given trained model. Different methods have different levels of 'model transferability' (Kurakin et al, 2016) and I am concerned how well the detector they built would be able to detect adversarial examples across *all models* and not just the trained model in question. In other words, a good detector would be able to detect adversarial examples from any network and not just one particularly trained network. This question seems largely unaddressed in this paper but perhaps I missed some subtle point in their descriptions.  Minor comments:  If there were any points in the bottom-left of the Figure 2 left, then this would be very important to see -- perhaps move the legend to highlight if the area contains no points.  - X-axis label is wrong in Figure 2 right.  Measure the transferability of the detector?  - How is \sigma labeled on Figure 5?  - Whenever an image is constructed to be an 'adversary', has the image actually been tested to see if it is adversarial? In other words, does the adversarial image actually result in a misclassification by the original network?
Output (English): en