Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper is the first paper, in my knowledge, that introduces the problem of identifying anomalous (or corrupted) subset of data input to a neural network. The corrupted inputs are identified vis-a-vis a set of “clean” background set (e.g., the training/ validation data set). Experimental evaluation is performed on the problem of identifying the subset of noisy CIFAR-10 images created by adding targeted adversarial perturbations.

To achieve this, the problem is posed as that of subset detection and subset scanning approaches are adapted for the same. The activation of a node under consideration is converted to a p-value (how extreme the activation is, for that input data, w.r.t. the background set). 

A goodness of fit is then defined for any subset using the Berk-Jones test statistic which is used to create an NPSS scoring function to be maximized over the set of all subsets of the product set – (set of nodes x set of input images). The authors suggest that this combinatorial problem can be addressed efficiently as NPSS has been shown before to have the linear-time subset scanning (LTSS) property (Neill 2012). 

However, since the search is now over a product subspace and the LTSS property holds for individual subspaces (selection of activation nodes or input images while the other is held fixed), the LTSS property doesn’t trivially extend to the product space. The authors suggest an algorithm which alternatively iterates between the two LTSS steps. Since the above algorithm is not guaranteed to obtain the global optimum, multiple random restarts are suggested. This is the weakest part of the paper – neither is a formal proof of convergence provided, nor is the efficiency of the propped algorithm demonstrated empirically. Also, the performance of the algorithm (accuracy) can only be weakly used to judge the goodness of the local optimum obtained as it confounds the power of NPSS with the gap between the local and global optimum.

Contributions

In summary, the main contribution of the paper is the introduction of the problem of the identification of anomalous subsets of adversarially corrupted examples on deep neural networks and the first approach to address the problem via a NPSS scoring function that (partially) satisfies the LTSS property suggesting the existence of efficient (and exact?) algorithms for solving the problem. The work has incremental novelty as it seems to be largely based on prior art (not on DNNs). Initial results look promising.  

Negatives 

However, I have the following concerns:

- (Clarity) Mathematical formulation for the NPSS score function and the optimization problem to be solved should be clearly stated (and not inside paragraphs of text).

- The proposed algorithm is ad hoc - an obvious extension of the standard algorithm that exploits the LTSS property to the product of two set spaces. The algorithm may not find a good optimum. No proof, analysis or study of its properties – optimality gap, convergence, and, efficiency is presented. 

- At deployment, the attacks may be non-targeted or the target may be different for different corrupted inputs. This will dilute the signature as the set of nodes where the ‘extreme values’ appear may not be the same across the corrupted examples. It is not clear if it is the case if the target is the same. In this case, subset scanning may not work. This assumption should’ve been directly tested.

- ‘Berk-Jones can be interpreted as the log-likelihood ratio for testing whether p-values are uniformly distributed on [0,1] vs. … alternate distribution’. Why should this be the test? P-values on clean, background data can be directly obtained and whatever that distribution can be empirically tested or uniformity can be tested against after a whitening transformation?

- Comparison with reasonable ‘baselines’ – for example, with Feinman et al. 2017? Comparison with current state of the art defenses on CIFAR10?

- Why is the BIM attack (Kurakin et al. 2016b) used? There are more powerful attacks available now?

- The NPSS scoring function is supposed to maximize over significance values \alpha. What range was used and how was it decided? In the discussion on Detection Power reported in Table 1, it is suggested that different \alpha_max will trade off precision vs recall. This has not been experimented with. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors apply linear time subset scanning to find groups of anomalous inputs and network activations. They further use this method to detect effects of the same adversarial perturbation algorithm over a set of images.
Major comments:
Overall this is quite interesting work, and seems like a promising method to detect groups of anomalies. However, this work is not complete without comparison to existing methods, the lack of which makes it impossible to evaluate its usefulness. See [1,2].

Minor questions/comments:
The results seem to be applied to ReLU activation networks and anomalous signals are described as having higher activation that the background or the clean images. Could you clear up whether this method works for all activations or just ReLU networks?
I’m envisioning a case where the anomalous images are all anomalous because they are all very similar. If I put 100 of the same (or very similar images) would this be something that this method could detect?
Your results seem to suggest that adversarial perturbation methods produce adversarial images with activations that are more extreme than natural images. This doesn’t seem immediately obvious to me and I would be interested in further exploration of this direction.
A value of \epsilon=0.02 was used in experiments for BIM. While I agree that smaller values of \epsilon are harder to detect it would be useful to have an evaluation of performance over smaller values of \epsilon even if these did not perform as well.
[1] Xiong, L., P ́oczos, B., Schneider, J., Connolly, A., VanderPlas, J.: Hierarchical probabilistic models for group anomaly detection. In: AISTATS 2011 (2011)
[2] Chalapathy R., Toth E., Chawla S. (2019) Group Anomaly Detection Using Deep Generative Models. In: ECML PKDD 2018. Lecture Notes in Computer Science, vol 11051. Springer, Cham
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposed a scheme to detect the presence of anomalous inputs, such as samples designed adversarially for deep learning tasks, that is based on a "subset scanning" approach to detect anomalous activations in the deep learning network. The paper is considering a very interesting problem and provides the suitable application of an approach previously developed for pattern detection. The approach is motivated by p-value statistics of the activation patterns in the deep learning network under the "null hypothesis" of a non-anomalous input.

My rating is "weak reject" because the explanation of the subset scanning approach is not clear. The paper describes two functionals F and G that are defined over subsets of the data and activations. Section 2.2 mentions a method that maximizes F over data and activation subsets by maximizing F over the data subsets under a fixed activation subset and vice versa, and iterating over these two. The section also discusses the function G that measures the priority of an image, but does not describe how the priority is known to provide an optimal solution for the former optimization over F. Another function with the same name is defined to measure the priority of an activation node, and again it is not clear why this will provide an optimal solution for the latter optimization. It would have been helpful to establish (or at least instantiate) the optimality results for this approach; only a citation is provided.

The applicability of the approach is limited to the requirement that multiple adversarial samples be present for accurate detection. It would appear that other approaches to anomaly detection do not require this condition, particularly if they are designed to operate on individual samples. Furthermore, the requirement for the "same system" to design the anomalous samples limits the applicability of the anomaly detection approach to cases like the single adversarial design detection setting studied here.

Some questions for the authors:
Is there a reason why the figures (Fig. 2) show results only for a single class?
Is there a reason why no other anomaly detection algorithms for the activation patterns were used in comparisons? How about other adversarial noise detection algorithms?
Is there intuition behind the difference in performance when all target classes vs. a single target class is used? Does this mean that a multi-class adversary generation would be too diverse for the proposed approach to detect it effectively?

Minor comments
"Weird together" - is too informal, and it's not clear why this phrasing is needed. Consider replacing or explaining.
Typos "anomlaous" multiple times.
review_title: Official Blind Review #1

Meta-review: The paper investigates the use of the subset scanning to the detection of anomalous patterns in the input to a neural network. The paper has received mixed reviews (one positive and two negatives). The reviewers agree that the idea is interesting, has novelty, and is worth investigating. At the same time they raise issues about the clarity and the lack of comparisons with baselines. Despite a very detailed rebuttal, both of the negative reviewers still feel that addressing their concerns through paper revision would be needed for acceptance.
Decision: Reject