Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This work proposed three different calibration methods, temprature-based, margin-based and angle-based temperature scaling to enlarge the margin between targeted logit and non-target logits to improve transferability of targeted adversarial attacks. This work is highly inspired by the work [1] and perform experiments to show the proposed methods are better than other existing methods.


[1] "On Success and Simplicity: A Second Look at Transferable Targeted Attacks".
Zhengyu Zhao, Zhuoran Liu, Martha Larson. NeurIPS 2021.
strengths_and_weaknesses: First of all, after comparing the related work in [1] and this work, there is a huge amount of overlapping of the equations or rewriting the sentences. This significantly destroys the overall quality of the work.

Second, the improvement of the proposed method over [1] is marginal compared to the improvement of [1] over cross-entropy loss.

Third, the contribution of this work is limited. Although the authors proposed different temperature-scaling based methods to improve transferability of targeted attacks, which only achieve limited experimental gains, this work did not provide extra useful insight to this research area.

[1] "On Success and Simplicity: A Second Look at Transferable Targeted Attacks". Zhengyu Zhao, Zhuoran Liu, Martha Larson. NeurIPS 2021.
questions: In general, it leaves me a poor impression when I realize the great similarity between the Related Works in this work and that in [1]. Although the authors try to reframe the sentences, it's still very unprofessional to structure related works with such a strong similarity with another existing work. 
limitations: See above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a novel and effective method to improve the transferability of adversarial attacks. They increase the logit margins between targeted and non-targeted classes, which can quickly become saturated in cross-entropy loss.
strengths_and_weaknesses: Strengths:
1. The findings are very interesting and the motivation is well-explained.
2. Comprehensive experiments are presented and the combining logit calibrations have significantly better performance than previous methods.

Weaknesses:
1. The proposed method has various settings and hyper-parameters. Compared to the simple Logit method, the proposed method needs more effort for tuning or need combining logit calibrations to achieve better performance. This can make the method less attractive to the community.
3. There is no theoretical analysis to support the empirical findings.
2. The presentation of the results needs to be improved. All tables contain tons of numbers, which makes it hard for the reader to get the point in a short time.
questions: Could authors provide any interpretation of the results in Table 1? For example, why do Margin and Angle have better performance when the surrogate models are VGG17 and Inc-v3, but have lower performance for ResNet50 and Dense121?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper designs a new logit calibration method which is inspired by knowledge distillation. The method uses logit calibrations in the CE loss function so that it can improve the targeted adversarial attack with higher transferability than other attack methods with cross-entropy loss. Except for the primary temperature-based method, this paper designs margin-based and angle-based methods to solve different surrogate models and different norms.
strengths_and_weaknesses: The strengths of this paper are:

This paper designs a new cross-entropy (CE) loss function to improve the targeted adversarial attack, which performs better than Logit (NIPS21).

Except for the temperature-based method, this paper designs margin-based and angle-based methods to solve different surrogate models and norms.

The weakness of this paper are:

This paper follows `Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On success and simplicity: A second look at transferable targeted attacks. NeurIPS, 34, 2021.' from academic writing skills and code in specific. However, instead of Zhao et al. designing the Logit loss and using it to generate universal adversarial perturbations, this paper's method does not have any additional functions such as UAP. 

Moreover, this paper's method only exceeds the Logit loss by around 10%, which is not a significant improvement. Therefore, this paper lacks novelty.

The equation 14 seems to have some mistakes. On the left of the equation, would $z_{i}$ be $\tilde{z}_{i}$?

A few grammar problems in this paper should be improved. For example, in line 275, it should be "be similar to"; in line 23, it should be "Following many approaches"; in line 27, it should be "it is vital to explore."

questions: Firstly, for the influence of different $T$ in CE, this paper claims that when the surrogate model is VGG16 and Inc-V3, a larger $T$ obtains better transferability. However, I am curious about is there a limitation of $T$ on VGG16 and IncV3. For example, after the ASR achieves 600, the performance of the targeted attack will decrease when $T$ continually increase. And then, although this method is based on CE, it would be better if the authors designed a new name to describe it.
limitations: The relationship between temperature-based, margin-based, and angle-based logit calibration is unclear. This paper claims that the margin-based one is designed to face different surrogate models, and the angle-based one is designed to solve the influence of various norms. However, in the experiments, the performance on T=5, T=10, Margin, and Angle does not prove the relation between them. This paper does not evaluate which method is the best for the targeted attack, or in other words, which option should I choose if I need to achieve the highest attack success rate?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper targets improving the transferability of adversarial attack using the logit calibration. Despite the recent success in untargeted black-box attacks, the targeted transferability of adversarial attacks remains challenging. The paper takes a closer look at the vanishing gradient issue in the CE loss function which is commonly used to learn transferable adversarial samples and suggests that the logit margin between the targeted and non-targeted classes quickly gets saturated during the optimization process. So, to improve transferability they aim to enlarging logit margins which consequently reduce saturation and  enable longer optimization and iterations. The paper investigates three different types of logit calibrations including temperature-based, angle-based and margin-based inspired by previous studies and techniques.  Experiment conducted using ImageNet dataset and different methods including ResNet50, DenseNet-121, VGG-16 and Inception-v3. Results are compared to SOTA methods including Po+Trip, Logit, and TTP. 
strengths_and_weaknesses: Strength:
- Quality: The writing quality is very good, very easy to follow, there are areas that can be improved but nothing major.
- Clarity: Very clear. Easy to understand the motivation and the thought process of different method’s component. 
- Significance and novelty: Novelty is a bit limited and built up mostly on top of previous methods,  but it is also not a weakness because this work attempts to solve an interesting problem and the analysis and results are valuable. The results are somewhat important. Mostly inspired by Logit [30], future researchers might use the suggested logit calibration and increase the number of iterations for targeted attacks.

Weaknesses:
- The quality of results and presentation of it could be improved significantly. (see below)
- The distinction between [30] and this paper should be clarified and the introduction in page 2 [line 36-56] can benefit from a re-writing. (see below)

questions: - There is a large novelty overlap between [30] and the current method. It is proper to get the similarity and distinction discussed upfront. Specifically,  discussion around CE starting line 36 is getting very confusing and blurry going through line 56. Vanishing gradient and the use of Logit loss has been discussed and proposed in previous arts, for example [30]. This gets discussed later at line 145+ and in method, however, I feel the contribution of the current work can get discussed in a more clear way and upfront in introduction. 

Experiments are limited and can get improved:
- First the organization of the results is not optimized or ideal. (1) Following Table 1, 2 and 3 is very hard and replacing this with line graphs that capture progress through iterations would be very beneficial. (2) Overall collective of Table 1, 2, 3 seems to be more exploratory and ablation tables rather than the main results. The main message that I read from these two tables are T=10, 20 and a combination of Margin + Angle or T + Angle can result in the best outcome. So, why not introduce a single best receipt and present everything else as ablations? (3) Also I would suggest sticking with conventional methods such as heat-map to summarized heavy tables, as it is a norm in adversarial attack literature.  [21] have good examples of result presentations.

- One relevant question, is also as Table 2 suggests the best outcome is coming from T=10 or T=20 so why in Table 3 we analyze the effect of combining logits as T=5 +Margin or Angle? If any underlying study suggests this, results should be provided.  

The results could be strengthened by:
- Providing experiments on another dataset (e.g., CIFAR-10, MNIST, SVHN) Since the proposed method works well on ImageNet, it could only be a minor concern.
- Incorporating study of a real-world attack for example the Google Cloud Vision API.
- Providing targeted success rates for transfer with varied targets.

limitations: I cannot find any specific discussion around the potential negative social impact of this work. Also, the limitation of the method was not addressed in the paper.

To improve this part, discussion around the benefits of adversarial attack research can get discussed. Potentially this can motivate the AI community to design stronger defenses against transferable attacks, and  in the long run such results can be directly used for social good applications, such as protecting privacy. On the contrary, there are applications that can benefit from transferable attacks in a harmful manner to damage the outcome of any AI system, e.g. imaging a scenario that someone uses such attacks to intrude with the outcome of a medical AI device.  

I also suggest that authors discuss the limitation of the work, failure cases and processing time.

Overall, I enjoy reviewing this paper and looking forward to reading the authors' responses.
review_title: null

Meta-review: metareview: In this paper, the authors propose novel method to improve transferability of targeted adversarial attacks by enlarging the margin between targeted logit and non-target logits.  Experiments on ImageNet with different methods demonstrated the effectiveness of the method. However, as is pointed out by the reviewers that there exist high overlap between the paper and the existing works, which significantly hinders the novelties of the paper. The paper are expected to clarify the novelty and provide more comprehensive evaluations. 



confidence: Certain
Decision: Reject