Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 

===== Post Rebuttal ===== 

The authors addressed almost all of my comments. I think the revised paper is of higher quality and clarity as a result. Figure 3 is the main remaining concern. However,  given the listed strengths of the paper, I am happy to change my rating to weak accept .

===== Summary ===== 

One strategy for defense against adversarial attacks is to perturb the input sample multiple times and then aggregate their corresponding outputs into a single output. One recent variant of this method (Prakash et al. 2018) shows using this type of defense mechanism degrades the accuracy on clean (unperturbed) inputs. This paper tries to mitigate the accuracy drop on clean inputs. It does so by learning an additional distribution classifier which takes as input the distribution of perturbed samples’ outputs. The method is trained on MNIST, CIFAR10, CIFAR100 and using 4 different adversarial attacks. It shows significant improvement in several cases over standard input-perturbation methods.


===== Strengths and Weaknesses ===== 

+ The paper picks an important problem, proposes a simple technique and achieves significant improvements in certain cases.
+ The experiments are done on 3 datasets and 4 different adversarial attack techniques.
+ The final experiment on end-to-end attack is interesting.

Concerns regarding the main paper’s description
- From Figure 1, it seems that the distribution classifier does a binary classification for the distribution of each class separately. If this is the case, how is the final classification done (using the binomial distribution per class)? The figure suggests that the maximum probability of the binary classifications (bus for yellow class with 0.8 score) is taken. This would be strange since they are not optimized to compete with each other (e.g. as in a more proper cross entropy). In any case, the details of the distribution classifier is missing from the paper, and section 4 refers to the figure. The details should come in the main text using precise language and math formulation. 

- page 4 is very hard to follow since it tries to give a textual description to mathematical concepts. I think it is important to provide the mathematical definition such that 1) it is easier to follow 2) the concepts are unambiguously defined. This includes a definition of softmax distribution, joint distribution of output and transformation, marginalized distributions, clean intra-class distance, adversarial intra-class distance, and clean-adversarial inter-class distance.

- page 4: it seems it is implied in the definition of “clean” images that these images are classified “correctly” by the network. Otherwise, in Figure 2.a, the clean row is uninterpretable not knowing whether the clean image was classified correctly or not.

- page 4,5: what is a “transformation magnitude”? It could refer to both the number of transformations and also the number of perturbed pixels. The behavior in Figure 3 suggests that the latter should be the case, but it’s important to disambiguate.

- figure 3 is counterintuitive. The support (possible distributions) should normally live on a simplex (due the sum-to-one probability constraint). In figure 3, it seems there is support in the full unit hypercube. In particular, we can see density along the horizontal/vertical line of 1-density for one class (i.e. non-zero density for others in this case)

- page 6: “[...] to train a distribution classifier on the distributions obtained from clean images only [...]“. I cannot follow the argument at the end of the first paragraph that led to the proposed *training on clean images only*. I understand that from the qualitative demonstration of 8 different examples in Figure 4 the distributions of perturbed adversarial examples *might be* redundant to the perturbed wrongly-classified clean examples. However, even if it is redundant, why should it be harmful to train the distribution classifier with adversarial examples as well as clean examples?

Regarding Experiments:
- why is the evaluation done only based on accuracy of *correctly-classified* clean images and the recovery of *wrongly-classified* adversarial images? Couldn’t it be that the baseline (majority voting) does a better job than the distribution classifier on the “wrongly-classified” clean images and the “correctly-classified” adversarial images? 
- I think it's still interesting and informative to train the distribution classifier using both clean and perturbed inputs to compare the performance.
- page 6: why is there a need for applying a kernel density estimator followed by a discretization (e.g. for instance instead of a simple histogram)?

===== Final Decision ===== 
My current “weak reject” rating is primarily based on the unclarity of the main paper and secondary regarding the concerns for experiments. 

===== Points of Improvement ===== 
I believe the paper will become stronger if the proposal is described and motivated more formally.


review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The authors analyze the use of image transformation as a defense against adversarial examples, where a challenge is to prevent the deterioration of performance on clean images. To do this, they show that the softmax distributions for clean and adversarial images share similar "features", and therefore one can apply a trained distribution classifier which takes the softmax distribution to return the class label. This is as opposed to original approach of making a prediction for each sample (a random transformation of the input image) followed by majority voting.

I have one major concern about their analysis, which makes me lean weakly towards not accepting the paper. I'm happy to change my score after discussions and further clarifications. In particular, I'm not convinced that the phenomena of softmax distributions appearing similarly for clean and adversarial images is not simply a result of the stochastic transformation converging to a fixed (or possibly neighborhood of fixed) distributions. Does the transformation retain information about the difference between classes? That is, while the divergence of softmax distributions from clean and adversarial images decreases as one applies more transformations, does the divergence of softmax distributions from clean images of two different classes also decrease?

As a non-expert in the field, I found the paper well-written. It motivates the idea well by identifying challenges in a promising technique (random tranfsormations) and provides decent background explanation.

Figure 2 is an interesting example showing the relationship of the softmax distributions for clean and adversarial examples. This raises a few questions which are unclear to me:

1. I'm a bit surprised at how remarkably bad transformation-based defenses seem to be in degrading classifier performance. Is it really the case that the MNIST model only gets 78% accuracy on class label 8 and 36% on class label 6? What about without the transformation?
2. This phenomena of softmax distributions being similar seems to largely depend on the stochastic transformation T and form of attack. This is especially seen in Figure 2b, which suggests that T may be acting like a transition kernel which takes an arbitrary initial state and may have it converge to a fixed stationary distribution. What does the divergence look like for two class distributions of clean images over the number of pixel deflections?

For the defense, I'm also curious what happens if the model used the distribution classifier with random transformations at training time, instead of a separate softmax output layer.

Regarding experiments, I only assessed their sensibility. Unfortunately, I don't know enough about the field to tell how significant the results are. 

1. For the choice of number of transformations and training set size for the distribution classifier, how were they chosen? They seem to vary arbitrarily across the datasets.
2. It seems like the improvements appear only in single-step FGSM, which also tends to be the worst performing, where the other methods are iterative. What's the intuition for this?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents a novel defense method to make the classification more robust. The motivation is based on the observation: the distribution of the soft-max for the cleaned image and its transformed images for one class is similar to the distribution of the soft-max for the adversarial image and its transformed images for the same class, and the distributions of the soft-max for the cleaned image and its transformed images for different classes are different. Then, a distribution based method is proposed to classify the distribution of the soft-max for the cleaned (or adversarial) image and its transformed images. 

After I read the core part several times, finally I understood the paper. Overall I think this paper is well motivated, and the empirical results also support the claims/observations. But I think this paper could be improved by (1) checking the performance over large datasets (such as ImageNet); (2) providing possible analysis on the observation. Otherwise, the readers cannot be fully convinced. 
review_title: Official Blind Review #1

Meta-review: This paper investigates tradeoffs between preserving accuracy on clean samples and increasing robustness on adversarial samples by using transformations and majority votes. Observations on the distribution of the induced softmax show that existing methods could be improved by leveraging information from that distribution to correct predictions, as confirmed by experiments.
The problem space is important and reviewers find the approach interesting. Authors have provided some necessary clarifications during rebuttal and additional experiments. While some reservations remain, this paper's premise and its experimental results appear sufficiently interesting to justify an acceptance recommendation.
Decision: Accept (Poster)