Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes adversarial interpolation training, which perturbs the images and labels simultaneously. The perturbed image $\tilde x$ is around $x$ and interpolated towards another image $x'$ while the corresponding $\tilde y = (1-\epsilon_y)y +\epsilon_y\frac{1-y'}{C-1}$ is near $y$ but away from $y'$. The distance of interpolating images is L2 distance in the feature space and that for labels is L2 distance in the label space. The paper provides an interpretation of the proposed approach from the perspective of robust and non-robust features. Thorough experiments on different types of attacks and different datasets are performed. Although the results are impressive, I still have some concerns on the method itself:

1. The method seems like a combination of manifold mixup [1] and adversarial training. The interpolation in the feature space is not a new idea and has been explored in Manifold Mixup [1]. The method resembles manifold mixup if we focus on $x$ because $\tilde x$ and $\tilde y$ both retain the original image and target $(x, y)$. The "adversarial" interpolation part is from $(x', y')$ in the sense that $\tilde y$ is away from $y'$.

2. The paper lacks a theoretical explanation, which makes it less convincing how it works so well.

3. I noticed several papers with similar ideas, e.g. [2,3,4]. Could you please discuss the connections with them? I also suggest adding related work on semi-supervised learning in the paper (see [4] for examples). It would be better to compare with Manifold Mixup [1], UAT [4] in the experiments.


Minor:

Page 5, " further break the correlation between $\delta$ and $y'$", what is $\delta$ here? I did not find the definition above the sentence. The notation is directly used without any explanation in advance.


References
[1] Manifold Mixup: Better Representations by Interpolating Hidden States, ICML 2019
[2] MixUp as Directional Adversarial Training
[3] On Adversarial Mixup Resynthesis, NeurIPS 2019
[4] Are Labels Required for Improving Adversarial Robustness?, NeurIPS 2019

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This is an interesting work proposing a new robust training method using the adversarial example generated from adversarial interpolation. The experimental results seem surprisingly promising. The ablation studies show that both image and label interpolating help the robustness improvement. 

I think it is important to provide a running time comparison between the proposed method and SOTA robust training method such as Madry's. Since the feature extractor is implemented by excluding the last layer for the logits, the backprop goes through almost the entire model. It seems that the proposed interpolating method has a similar amount of computation as PGD, so the training should take similar time as Madry's if it can converge quickly. 

Also, there are too many works on robustness defense that have been proven ineffective (consider the works by Carlini). Since this is a new way of robust training and there is no certified guarantee, I would be very conservative and suggest the authors refer the checklist in [1] to evaluate the effectiveness of the defense more thoroughly to convince the readers that it really works. Especially, a robustness evaluation under adaptive attacks is necessary. In other words, if the attacker knows the strategy used by the defender, it may be possible to break the model. PGD and CW are non-adaptive since no defender information is provided. 

[1] Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Contribution: This paper proposed an adversarial interpolation approach for generating adversarial samples and then training robust deep nets by leveraging the generated adversarial samples. However, I have the following concerns:

1. In adversarial interpolation training, why \tilde{y}_i' is set to 1/(C-1)(1-y_{n-i+1})? 

2. This work lack of interpretation of why the proposed method is more effective than PGD adversarial training.

3. How about training deep nets with replicas of the training data but replace the true labels with random labels? I want to see such a result.

4. Can the authors provide the black-box attack results also? I want to see the performance of PGD adversarially trained deep nets on the adversarial images crafted by attacking Adv-Interp trained deep nets, and vice versa.

5. Can the authors provide the visualization of a few adversarial interpolated images?

6. The authors should compare with the existing efforts that using interpolation to improve adversarial robustness. Below are some Related works on using interpolation in deep nets to improve their robustness

1). Bao Wang, Xiyang Luo, Zhen Li, Wei Zhu, Zuoqiang Shi, Stanley J. Osher. Deep Neural Nets with Interpolating Function as Output Activation, NeurIPS, 2018

2). Bao Wang, Alex T. Lin, Zuoqiang Shi, Wei Zhu, Penghang Yin, Andrea L. Bertozzi, Stanley J. Osher. Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization, arXiv:1809.08516, 2018

3). B. Wang, S. Osher. Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning, arXiv:1907.06800, 2019

7. Moreover, the following paper provides a theoretical interpretation of adversarial vulnerability of deep nets, and proposed a nice ensemble of neural SDEs to improve deep nets' robustness.

1). Bao Wang, Binjie Yuan, Zuoqiang Shi, Stanley J. Osher. ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies, arXiv:1811.10745, NeurIPS, 2019


Please address the previously mentioned concerns in rebuttal, and this paper can be acceptable if all my concerns are addressed.
review_title: Official Blind Review #1

Meta-review: Reviewers agree that the proposed method is interesting and achieves impressive results. Clarifications were needed in terms of motivating and situating the work. Thee rebuttal helped, but unfortunately not enough to push the paper above the threshold. We encourage the authors to further improve the presentation of their method and take into accounts the comments in future revisions.
Decision: Reject