Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper first provides explanations to the inherent tradeoff between rotation adversarial attack and sensitivity attacks/spatial transform attacks, through their differences in saliency maps. Further, the authors proposed to utilize pareto training to find the best tradeoff among the four dimensions: natural accuracy, robustness against sensitivity/rotation/spatial transformation attacks. Experimental results show the proposed pareto adversarial training achieves better tradeoff between clean accuracy and adversarial robustness averaged across three types of attacks.

Pros:
1. The idea of using pareto training to tradeoff between different accuracies/robustness is interesting. This reminds me of a very recent paper [1], where the authors considered a relevant (but different) problem: How to achieve in-situ tradeoff between different attacks during inference time.
2. Using shape-bias and saliency maps to explain and demonstrate the inherent difference between rotation and sensitivity attacks is a good point.

Cons:
1. Limited novelty. 
The tradeoff between rotation and sensitivity attacks has already been discussed in many previous works (e.g., [2]); the rotation and spatial translation attacks are defined in previous works [3,4] and the combination of the two is kind of trivial; the authors directly apply pareto training [5] in the adversarial tradeoff task, without any modification to fit the new problem.
2. Missing detailed experimental results.
Only the average robustness scores are provided in Figure 7. Please also consider providing robustness scores on each type of corruptions, so that we see which ones have larger improvements after pareto adversarial training and which ones do not.

[1] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free. NeurIPS, 2020.
[2] Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations. ICML, 2020.
[3] A rotation and a translation suffice: Fooling cnns with simple transformations.
[4] Spatially transformed adversarial examples. ICLR, 2018.
[5] Pareto Multi-Task Learning. NeurIPS, 2019.

review_title: Review

final_score: rating: 3: Clear rejection
review_content: This work aims to study so called “comprehensive” robustness by considering natural accuracy, sensitivity-based robustness and spatial robustness simultaneously. The authors claim that they propose a “Pareto” Adversarial Training strategy to balance the mutual impact within different robustness and the resulting solutions provide the set of optimal balance among accuracy and different adversarial robustness.

Pros:
- The paper investigates comprehensive robustness. For me, the problem itself is interesting.
- This paper provides empirical evidence to show that there exists representation discrepancy for different notions of adversarial robustness. This shape-bias representation viewpoint is appealing.
- The motivation for their training strategy incorporating “Pareto criteria” is good, despite that their method has a few questionable inaccuracies summarized below. 

Cons:
- The proposed “Pareto” Adversarial Training strategy is not well justified. First, the authors misuse Pareto. Pareto optimality (or front) is introduced to describe the trade-off between conflicting objectives from the multi-objective optimization. However, the proposed method is actually solving a series of bi-level optimization problems by varying the hyperparameter $r$. Since each problem is solved independently, it is not a multi-objective optimization, and the solutions obtained are also not Pareto optimal (or front). Second, in the lower-level formulation (12), the authors propose to use weighted quadratic difference to measure the trade-off. However, it is not clear why this makes sense. Is it because the two-moment objective function results in a quadratic programming problem which is easy to solve. In Section 3.2, the authors claim that there exists representation discrepancy for different adversarial robustness. Isn’t it more reasonable to consider the representation discrepancy as a measure? 
- To integrate flow-based and RT-based attacks, the authors propose to use equation (2) instead of the original Flow-based attack equation (1). To justify this, the authors then provide Proposition 1 which proves that the “smooth approximation” of max operation in equation (1) has a parallel updating direction with cross entropy loss in equation (2). However, equation (1) is to minimize the max operation over $\omega_F$, whereas equation (2) maximizes the cross entropy loss over $\omega_F$. Therefore, equation (2) is different from equation (1). Moreover, the integrated spatial attack proposed in Section 2 seems not to appear anymore in their next analysis and training strategy. The readers might think that Section 2 is a little unnecessary as it is unrelated to other parts of the paper.
- In section 4.1, the authors claim that Max AT is closely linked with specific weights of Ave AT by providing Proposition 2. However, I have some concerns about the proof of Proposition 2. First, it assumes that KKT conditions hold. This assumption is strong in the sense that KKT conditions imply zero duality gap which usually does not hold for nonconvex problems. Second, even if KKT conditions hold, we can not derive that $R^*_{max}$ is a first-order stationary point of $\sum \lambda_iR^{S_i}$ because the Lagrangian function might be non-convex with resepct to $f$.

Minor comments:
- It would be better if the authors can give the definition of $x_{\omega_F}$ in (1).
- In equation (9), $clip_\epsilon$ should be used after the update of $w_F^t$.
- In equation (10), what does $n$ represent? The number of training data or the dimension of input?

review_title: Official Blind Review #4

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper first studies the tradeoffs between two forms of spatial robustness, including robustness against Flow-based spatial attack and Rotation-Translation (RT) attack. In particular, it proposes an approach to account for both local and global spatial transformations in an integrated framework. In addition, the paper investigates the relationship between the sensitivity-based (lp-norm based) and spatial robustness, and proposes a training method called ‘Pareto Adversarial Training’ to find optimal combination between natural accuracy, sensitivity-based and spatial robustness.


Pros:

+ Investigating the spatial robustness along with sensitivity based robustness seems to be a natural step towards comprehensive robustness, which is well motivated. Existing works on spatial robustness are well-explained. 

+ Studying the saliency maps of different adversarially-trained models is interesting. Figure 4 visually looks good.
 
+ Introducing the Pareto optimization to account for different robustness objectives is well-motivated.



Cons: 

- The experiments are the major weakness of this paper. In particular, Figure 3 is really hard to parse: (1) legends overlap with the presented curves; (2) Why is the spatial test accuracy against Flow Attack higher than clean accuracy for Caltech-256? (3) How do you set all the hyper-parameters for adversarial training and the attack strength, such as \epsilon_F in Equation 2 and \epsilon_RT in Equation 6?  (4) X-axis label says PGD Trained Models (PGD Robustness), whereas the X-axis tick represents the number of PGD attack iterations. 

- Figure 7 is not explained with enough details. The X-axis label ‘Robustness Score’ is not defined throughout the paper. The meaning of each abbreviated term appeared in the legend needs clarification. Instead of a single Robustness Score, I would recommend to include the sensitivity-based robustness, Flow-based spatial robustness and RT spatial robustness individually for each models.


Other Questions and Comments:

1. From Equation 11 to Equation 12, it is unclear to me why minimizing the two-moment term regarding all losses leads to the optimal \alpha parameter. In addition, why do the constraints in Equation 12 not appear in Equation 11?

2. The presentation of the experimental sections needs to be improved in order to reach the acceptance bar of ICLR. Specifically, the experimental settings of each presented figure should be presented with enough details. Evaluation metrics of the experiments should be defined clearly in the corresponding section. Discussions on the results should be thorough enough such that each figure can be understood clearly.

review_title: Interesting Direction, but experimental results are not clear enough

Meta-review: This paper aims to address the robustness issues by considering natural accuracy, sensitivity-based robustness and spatial robustness at the same. However, the reviewers pointed out that many things, like the expriment, the presentation, the algorithm, are not clear. In addition, the technique part is weak and below the bar of ICLR.
Decision: Reject