Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Intrinsic Adversarial Attack (IAA), a transfer attack method based by jointly matching data distribution. The key assumption of this paper is that the DNN might not be well trained on low-density regions (LDD). Therefore, taking data distribution into consideration during attack could potentially improve the transferability. Empirical evaluation on different model architectures, normal models, robustly trained models, and ensemble-based attack context, demonstrates the advantage of IAA.
main_review: Strength:

- Important topic

- Somehow intuitive and interesting idea

- Sound and feasible solution

- Promising results

Weakness:

- The assumption is intuitive but limited evidence is provided. In particular, the authors only show cases of Gaussian noises, it is unclear to what extent the assumption could be generalized. It would be important to include other noises as well, e.g., corruption cases like CIFAR-10-C.

- Besides robustly trained models, it is unclear how strong the proposed attack could penetrate SOTA defense methods. Please provide more evidence on this.

- It is unclear what is the criterion to decide whether a layer is an early layer or late layer, please justify.
summary_of_the_review: Overall, this paper is well written and works on an important problem. The assumption is simple and intuitive, the proposed method is also feasible and sound.

Even though, there are still quite a few concerns. I am on the borderline of this paper and could be easily flipped to the other side based on the authors' response.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new adversarial attack method which could generate adversarial examples with higher transferability. The proposed method is based on the observation that low-density region of the training data is not well trained. To utilize this, the authors try to align the adversarial direction with the direction to decrease the ground truth density. The proposed method is theoretically support. Their experiments show that the proposed method outperforms multiple adversarial attack method in almost all evaluated scenario. I believe this could be a useful attack method for generating transferable adversarial examples, and providing a strong counterpart for future research on adversarial defense.
main_review: Strengths:
- The proposed method is well-supported by theoretical analysis.
- The experiment is thorough, different scenarios and multiple baseline methods have been covered.
- The results is good. The proposed method outperforms other compared method.

Weaknesses:
- The process of generating adversarial examples includes a bayesian optimization and receptively evaluating the second derivative of the neural network, which could result in high computational cost. What is the computation time of generating adversarial examples? I expect it will slower than normal adversarial attacks but the authors should always report it (including the network structure, hardware information) so that other people could have more understanding on the overall usability of this method.
- I feel there is a disconnection between the motivation and the actual proposed method. In the introduction, the authors show that small Gaussian noise could fool low density data. However, I did not see many connection between this observation and the proposed method. The only transition is that "The most efficient direction towards the low-density region is...". I think the author should elaborate more on this, since this statement is not very obvious and rigorous. Also, what is the formal definition of LDD and HDD?

Minor Comments:
- Since the proposed method can also be used for adversarial attack, I wonder what is the attack success rate of the proposed method? Is it also better than other attacks?
- When citing a paper, one should cite its archival version first instead of non-archival version (e.g., Arxiv, CoRR). For example, citation for (Tramèr et al., 2020) should be "Tramèr, Florian, et al. "Ensemble Adversarial Training: Attacks and Defenses." International Conference on Learning Representations. 2018." instead of "... CoRR, abs/1705.07204, 2020."

summary_of_the_review: Although there are some weaknesses of the paper, based on their experiments, I still believe the proposed method could be useful for generating strong adversarial examples and open the door for future research on defense. I also appreciate their theoretical analysis, although I haven't check the correctness. In general, I am feeling positive on this paper and would like to recommend for acceptance.

==============================
Post rebuttal:

I appreciate the author's comments on my questions. My score will keep the same since it's already reflect my recommendation for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper identifies that adversarial examples in the low-density region of the groud truth distribution have much stronger transferability. This observation leads to the AAI metric which evaluate the alignment of the model’s adversarial attack with intrinsic attack direction. The paper further identifies a set of model hyperparameters that can influence the AAI metric, and find the optimal hyperparameter choices to maximize AAI and generate more transferable adversarial examples.
main_review: ## Strengths
1. The observation on the relationship between low-density data and the high transferability of both random noise and adversarial noise is interesting, likely to inspire future works.
2. Provide theortical analysis on the AAI metric, provide a practical way of computing its value, and emperically shows its high correlation with the attack transferability.
3. Extensive experiments showing the proposed method achieving significantly higher transfer attack performance than SOTA methods, showing the effectiveness of the proposed method. 

## Weaknesses
My major concern on this paper is on the practical significance of the proposed AAI metric and IAA algorithm.

1. The necessity of utilizing the AAI metric to generate highly transferable attack is not justified. The paper introduces model hyperparameters $\beta$ and $\lambda$ utilizing the techniques of smoothing activation gradient and use early layer information, which are motivated by the previous works cited in Sec 3.2.1 and 3.2.2, and are not directly related to the AAI metric. As for the Baysian optimization process, since it's a blackbox optimizer assuming no knowledge of the relationship between the parameter and optimization objective, why not directly using the attack transferability as the optimization objective? An ablation study on this is needed to discuss why AAI is needed for this optimization.
2. The cost of evaluating the AAI objective is not discussed. From the formulation in Equation (4) seems like the computation of AAI requires a second-order derivation with respect to x and a sampling process of Gaussian vector v. How is the computation cost of giving an accurate enough estimation of the AAI value comparing to generating an adversarial example with the model?

Besides, there are some other issues in this paper

3. Some steps in the proof of Theorem 1 are hard to understand. For example, how is the $\log$ function got removed from the $\frac{\nabla_x \log p(y|x)}{|| \nabla_x \log p(y|x) ||_2}$ in the first step of the derivation? As the author mentions the proof is similar to the ones from previous works, to make the paper self-contained I would suggest having some more detailed derivation or explaination in Appendix E so that the proof can be understood without referring to your cited papers.
4. Citation should be provided for the baseline models mentioned in the experiment result tables.
summary_of_the_review: This paper provides interesting theortical insights on how to find highly transferable adversarial examples. However, from the practical side, the proposed training method seems to be inspired from other previous works rather than the observed insight, and whether the discovered AAI metric is practically useful for the optimization process is at doubt. Thus I would suggest a weak reject for now given the potentially low practical significance of the method.

## After rebuttal

After discussing with the author I have a better understanding on the significance of this work. The analysis of AAI motivates the idea of modifying structural hyperparameters of the source model to improve attack transferability, and AAI serves as a good indicator of the transferability towards all model. Although further experiments show it may not be the best objective for choosing the structural hyperparameters, I think it serves as a good theortical strating point for futher analysis of adversarial transferability. Thus I would like to suggest acceptance.
review_title: null

Meta-review: This work studied an important issue, i.e., adversarial transferability, in adversarial examples. It provides a novel perspective that samples in  the low-density region of the ground truth distribution where models are not well trained have stronger transferability across different models. Based on that, it proposed a metric called Alignment between its Adversarial attack and the Intrinsic attack (AAI) to indicate transferability. Inspired by the connection between AAI and transferability, this work further proposed to replace the regular ReLU activation with some smooth activation functions, to enhance the transferability. 

Most reviewers appreciate that the observation is interesting, and the theoretical analysis and the proposed method are intuitive. The reviewers posed some important comments on experiments, and the relationship between the proposed method and the proposed metric. The authors provided satisfied responses to most of these concerns. Although there is one remaining concern that AAI may be not the best metric to choose the structural hyper-parameters, the reviewer still thought it is a good theoretical starting point to further analyze the adversarial transferability. 

After reading the submission, reviewers' comments and the discussions between reviewers and authors, I believe that this work has provided a valuable perspective, a reasonable theoretical analysis and an effective solution for adversarial transferability. It could inspire further studies on adversarial transferability.
Decision: Accept (Poster)