Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a new training method, named Increasing Margin Adversarial (IMA) training, to improve DNN robustness against adversarial noises. The IMA method increases the margins of training samples by moving the decision boundaries of the DNN model far away from the training samples to improve robustness. Under strong 100-PGD whitebox adversarial attacks, the authors evaluated the IMA method on four publicly available datasets.

Overall, I vote for ok but not goor enough - rejection. The proposed strategy sounds reasonable and worked well with simple dataset, the Moons dataset. However, when it was applied to more complicated real dataset such as Fashion-MINST, SVHN, and COVID-19 CT image dataset; there was no significant achievement if compare to the MMA approaches. Thus further investigation is needed to convince benefit of the IMA on real datasets.

In addition, the authors tested only one medical image dataset, COVID-19 CT image dataset. Since there are multiple modalities in the medical field and the diversity among datasets are quite large, it is too early to emphasize the advantage of the proposed method in the medical field in general like the last phrase in the conclusion “We hope our apporach may facilitate the development of robust DNNs, especially in the medical field.”

review_title: Theoretically working but not significant with real data

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
The paper proposes increasing-margin adversarial training (IMA) to improve adversarial robustness of a classifier. IMA works by alternating between two algorithms: Algorithm 1 update the model parameters while Algorithm 2 updates the margin estimate. By iteratively increasing the margins from clean training samples, IMA seeks to make the classifier more robust to L-p adversarial perturbations. The authors conducted experiments on the Moons, Fashion-MNIST, SVHN and a CT image dataset to evaluate IMA’s performances against other baselines and found IMA to outperform or be on par with them.

Pro:
+Improving robustness through the margins from clean samples is an interesting approach.
 
Cons:
-Evaluation on non-standard image datasets used to evaluate adversarial robustness. Lack of evaluation on datasets such as MNIST, CIFAR10/100 or imagenet
-IMA’s assumption that clean samples from different classes are equally spaced from the boundary might not be valid for images. Some classes might require more pixel perturbations to change their ‘ground-truth’ class than others.

Recommendation:
While the idea of improving models’ robustness via increasing margins from clean samples is a refreshing direction to counter adversarial examples, the basis behind the idea of IMA might be flawed. IMA assumes that clean samples from different classes are equally spaced from decision boundaries when in an equilibrium state. However, some classes might require more pixel perturbations to change their ‘ground-truth’ class than others. More discussions and theoretical studies would make IMA more convincing. Another major concern I have is the lack of evaluation on standard image datasets such as MNIST, CIFAR10/100 or imagenet in the paper. Given its current state, I believe the paper is not yet fit for publication.


Comments and Questions:
 
The results in Fig 6 shows that IMA outperforms other methods but drops sharply at 0.3 noise level to almost match TRADES and adv’s performance, what is its performance vs other methods at levels past 0.3?

The statement “a model robust to noises less than the level of 0.2 is good enough for this application“ is not substantiated by any previous work or experiments.

How is the IMA’s performance against black-box attacks?



review_title: Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: In general, the paper has a good quality. The idea is based on a common intuition that adversarial attacks are most influential to the points close to the decision boundary. The proposed algorithm IMA makes effective use of this intuition and adopts an alternating training process. As an experimental work, the experimental performance of IMA is on par with the state of the art in the experimental settings considered in the paper. This work is important to the ML community. It would be interesting to see further exploration of the algorithm in different testing settings. 
The paper is written clearly. There is no difficulty in understanding the content.
Experimental details are provided. 

Detailed comments:
1. In (vanilla) adversarial training, the choice of max perturbation $\epsilon_\max$ is usually crucial to the performance of the classifier on noisy and standard data. Is the performance of IMA also that sensitive to the choice of $\epsilon_\max$? 
And it is briefly mentioned in section 3.3 that IMA might indicate a good $\epsilon$ for vanilla adversarial training. But this does not say anything about the choice of $\epsilon_\max$ for IMA. And this could be very important to its performance (on clean and noisy data).
2. What might happen to the performance of the method under different choices of $\beta$? It might be interesting to see how IMA deals with the well-known trade-off between robust and standard accuracy, which is currently one of the main concerns of adversarial training methods.

Other cons:
1. Figures are not readable when printed. 
 
Given the above concerns, my initial rating is 6. This may change given further detail of the paper.
review_title: The paper proposed an approach to increase the robustness of neural networks for classification tasks. The intuition of the method is to increase the margin of the training samples. The experimental performance of the method is in general on par with the state of the art method.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes to increase the adversarial robustness of a neural net by training the model on both clean and adversarial samples. An adaptive form of the projected gradient descent generates the adversarial samples. Therefore, the noise magnitude is estimated separately for each training sample, such that the decision boundary (suppose a classification problem) of the neural net has maximum distance to each training sample. 

Strengths: 
1.	Appealing idea of having adaptive noise magnitudes.
2.	Relevant experimental section (Covid19).
3.	Illustrative figures, describing the model.

Weaknesses, Suggestions, Questions:
1.	A theoretical discussion about following points will improve the contribution of the paper:
        a.	Why do large margins result in higher adversarial robustness? What happens if I change the attack type? 
        b.	Benefits compared over other adversarial training methods are not clear.
        c.	A more detailed discussion about the equilibrium state is necessary, as currently provided in Sec. 2.3. This is rather an example.
2.	Experimental section:
        a.	Need to report average over multiple runs. Results are very close together and it is hard to favor one method. 
        b.	Sec. 3.1: Since this is the toy-dataset, a discussion why the decision boundaries look as they do, would be interesting. 
        c.	Sec. 3.3: What information is in Fig. 9 middle and right? 
3.	Formatting and writing:
        a.	Detailed proofreading required.  e.g. on p. 3  “using cross-entropy loss and clean data for training”
        b.	Some variables are used but not introduced.  e.g. x_n1, x_n2  in Sec. 2.3.
        c.	Figures are too small and not properly labeled in experimental section.
        d.	References to prior work are missing as e.g. “Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning”
        e.	Algorithms need rework, e.g. information of Alg. 1 can be written in 2,3 lines. 

Though the idea of adaptive adversarial noise magnitude is in general appealing, the paper has some weaknesses: (i) theoretical contribution is relatively minor, (ii) the paper does not present the material sufficiently clearly to the reader, and (iii) experimental evaluation is not sufficiently conclusive in favor of the paper's central hypothesis.
review_title: Minor novelty, not sufficiently clear presentation

Meta-review: The paper proposes a margin-based adversarial training procedure. The paper is lacking in terms of proper dicussion of related literature e.g. similarity and differences to MMA, the "theoretical" discussion on page 5 is incomplete as there is no way how one can estimate the perturbed samples to do the analysis (the authors seem to implicitly already assume that the adversarial samples lie on the decision boundary) and the underlying assumptions are not clearly stated, the reported robust accuracies 
(see https://github.com/fra31/auto-attack for a leaderboard of adversarial defenses) on MNIST and CIFAR10 are worse than that of MMA which are in turn worse than SOTA. Thus this paper is below the bar for ICLR.
Decision: Reject