Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studied the problem of transferability of adversarial examples. Specifically, it considered the setting of an input sample and two independent networks that well-behave on the sample, if the adversarial example derived from a specific neural network and the sample can fool the other network.
Building upon previous work that practically prevents the adversarial example overfitting on a specific network by introducing input transformations and momentum strategy into the optimization procedures of generating adversarial examples, this paper proposes to accelerate the optimization procedures by setting an initial value for the momentum item for the iterative updating. The initial value can be considered a good estimation of the adversarial perturbation, so it is generated by a procedure similar to common adversarial attack methods but involving more randomness to improve its adaption.
Considering the way of getting the initial value of momentum, the method proposed in this work actually involves more updating steps compared with other baselines.
In the experiment, this work compared the attacking performance of the proposed method with methods applying momentum, and the result shows that the adversarial examples generated by the proposed methods have better attacking performance on new models with varying network architectures.
strength_and_weaknesses: Strength
This paper did completely present the idea and is easy to follow.

Weaknesses
One of my biggest concerns is the unfair comparison in the experiment. The improved attacking performance of the generated adversarial examples on the target models is more likely caused by a stronger attack induced by “more iteration steps” (Steps 2 to 9 in algorithm 1). To me, it is not the reflection of the transferability of the generated adversarial examples. I suggest the experiment should be conducted under saturation attack and equally computational costs. Specifically, the computational costs of steps 2~9 in Algorithm 1 should be considered in other baselines, and a greater number of iterations should be used to validate the root cause of the performance improvement, attack strength, or attack transferability.
Another aspect of unfair comparison is the selected baselines. The selected baselines in the experiment seem to be a weakened version of the proposed method. Some parallel work should be used for a fair comparison. Besides, the work is claimed to be a work in the line of black-box attack, then some work in this line should be considered too [4].

My second concern is about the careless statements, some are listed as (1), (2).
(1)	In Paragraph 2 of the Introduction, “Iterative gradient-based and optimization-based attacks have high white-box but low black-box attack success rates, which means that such two attacks are impracticable in the real world”. Impracticable is not a reasonable conclusion for white-box attack. Please refer [3].
(2)	2.2 in the Related Work – the well-known fact about adversarial training is that insufficient inner maximization steps don’t induce enough robustness. However, adversarial training and its variants [1,2] with sufficient inner steps are still the golden baseline to defend against adversarial attacks. The statements about adversarial training are inaccurate and incomplete.

Another concern is the mathematical foundation of the work. The adversarial perturbation implies the local incremental of loss landscape over a specific point w.r.t a specific function. Setting initial values of perturbation to accelerate optimization is hard to corelated with the transferability of adversarial attacks.
Specifically, the statements “leading to better acceleration in the first few iterations”, and “accelerate optimization effectively during the early iterations to improv transferability”, do not make too much sense to me.

My last concern is about the novelty of the proposed method. The $g^{exp}$ in line 10, which is derived from line 2 to line 9 is the core idea of this paper, i.e., setting an initial value for the adversarial perturbation, which is not technically novel to me.

[1] Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks[J]. arXiv preprint arXiv:1706.06083, 2017.
[2] Zhang H, Yu Y, Jiao J, et al. Theoretically principled trade-off between robustness and accuracy[C]//International conference on machine learning. PMLR, 2019: 7472-7482.
[3] Carlini N, Athalye A, Papernot N, et al. On evaluating adversarial robustness[J]. arXiv preprint arXiv:1902.06705, 2019.
[4] Shukla S N, Sahu A K, Willmott D, et al. Simple and efficient hard label black-box adversarial attacks in low query budget regimes[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 1461-1469.
clarity,_quality,_novelty_and_reproducibility: Overall, the quality and originality of this paper are somehow not good. Please refer to my concerns in the *Weaknesses*. But it is easy to follow.
summary_of_the_review: After reviewing the paper, this work cannot persuade me that the proposed method can potentially improve the transferability of adversarial attacks under a fair comparison. Secondly, the proposed method is not technically novel to me. Besides, the introduction,  background, and terminology are less well-written.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors modify the momentum term by two novel methods to improve transferability of adversarial examples. 
strength_and_weaknesses: + the performance in experiments are quite good compared to existing  MI-FGSM.

- the necessity of predicting of accurate momentum is lacking. Accurate momentum is helping optimization, but why it can lead better adversarial transferbility?

- the proposed method is claimed to have advantages over standard  Polyak momentum and Nestrov momentum, then I am wondering how about the performance on DNN training tasks?

- momentum is an estimation for second-order information, without comparing with Hessian, it is hard to say which estimation is more accurate, i.e., I do not like the title saying precise momentum. 

- following the idea that "accurate" momentum can help, then can we expect that some second-order method can further improve the transferbility?
clarity,_quality,_novelty_and_reproducibility: - the paper is clearly written but it is suggested to give more explanation on the method: why it is more accurate and why accurate second order information helps adversarial transferbility.

- the method looks novel but I do not know whether there is similar idea in training (not adversarial attack) and optimization, for which there are many modification on momentum. 


summary_of_the_review: The performance of this work is quite promising. The main weakness is the lacking of explanation and at least I am not convinced why those modification can help adversarial transferbility. Thus, I will give a borderline score and am willing to change it during rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two methods, Experienced Momentum (EM) and Precise Nesterov momentum (PN), to improve the transferability of adversarial attacks. Specifically, the proposed method starts with a few iterations to accumulate the gradient, and uses it as the initial momentum. PN considers the gradient of the current data point in the pre-update of Nesterov momentum to make the pre-update precise. The combination of EM and PN boosts the transferability of conventional momentum based adversarial attacks.
strength_and_weaknesses: Strengths: 
-	Extensive experiments show that EPN significantly improves the transferability of various momentum based adversarial attacks. 
Weaknesses:
-	The approach is not tested on ensembles of models. Neither does it be tested on advanced defense models other than AT, such as Randomized Smoothing (RS), Neural Representation Purifier (NRP).
-	Ablation experiments to verify the effectiveness of RCS are missing in the paper.
-	EM requires a few iterations first. NI doubles the time it takes to compute the gradient. The efficiency of the algorithm needs to be discussed more in this paper.

clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is clear and the paper is easy to follow.
Quality: The experimental results are impressive, but lack of strong theoretical support.
Novelty: An increment on existing methods.
Reproducibility: Not applicable.

summary_of_the_review: The experimental results are impressive, but lack of strong theoretical support.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the transferability of black-box adversarial attacks. As a benchmark method, the introduction of momentum in iterative attacks helps the attack to get out of the local optimal solution for better transfer attack performance. The authors argue that the initial setting of zero in momentum inhibits the effect of momentum so propose Experienced Momentum (EM) with Random Channels Swapping (RCS) for accelerating optimization effectively. On the other hand, for problems of imprecise pre-update of the traditional Nesterov momentum, authors propose Precise Nesterov momentum (PN), which pre-updates by considering the gradient of the current data point. Finally, combining the above two methods to obtain EPN and achieve state-of-the-art results on some methods and models.
strength_and_weaknesses: Strengths:
1. EPN focuses on better optimization methods and random channel data enhancement to improve transferability.
2. This work proposes Experienced Momentum (EM) with Random Channels Swapping (RCS) for accelerating optimization effectively.
3. This work proposes Precise Nesterov momentum (PN) to complete a more precise pre-attack to help momentum out of the local optimal solution.

Weaknesses:
1. The overall motivational analysis part of the paper is lacking. The authors do not adequately explain the motivation for the pre-trained momentum and the PN. Furthermore, there is also no relevant citations or empirical experiments as support. 
2. The experiments are done only for some basic models and defense models, and do not consider the performance of attacks under ensemble models and advanced defense methods (HGD, R&P, NIPS-R3, etc.), resulting in a lack of experimental completeness. In model selection, homologous models are simply stacked too much. Ensemble models and advanced defense methods are necessary for previous work on transfer attacks [1][2][3].
3. The benchmark method uses for the experiments do not consider methods with better performance, such as VT-SITIDI-MIFGSM [3]. Only a relatively single VT method is considered, and the performance in the paper does not exceed the current SOTA method [3]. 
4. The pre-trained momentum introduces additional attack iterations. Can a more iterative attack with a common method achieve a similar effect?

[1] Dong, Yinpeng, Tianyu Pang, Hang Su, and Jun Zhu. "Evading defenses to transferable adversarial examples by translation-invariant attacks." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4312-4321. 2019.
[2] Gao, Lianli, Qilong Zhang, Jingkuan Song, Xianglong Liu, and Heng Tao Shen. "Patch-wise attack for fooling deep neural network." In European Conference on Computer Vision, pp. 307-322. Springer, Cham, 2020.
[3] Wang, Xiaosen, and Kun He. "Enhancing the transferability of adversarial attacks through variance tuning." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1924-1933. 2021.

clarity,_quality,_novelty_and_reproducibility: The expression of this paper is relatively clear but the novelty is limited. It should be able to reproduce well.
summary_of_the_review: We tend to reject this paper due to lack of motivational analysis and necessary experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposed a new initialization of the momentum based adversarial attack, to improve the transferability to target models. The reported results look good, but there are several important concerns from the reviewers, mainly including:
1. Lack of motivation: the authors didn't provide convincing explanation about why the proposed initialization could improve the adversarial transferability. 
2. Incremental technical novelty.
3. Careless statement.
4. Inadequate experiments. 

Although the authors made some efforts in the rebuttal, most above concerns were not well addressed. 
justification_for_why_not_higher_score: Poor motivation, inadequate experiments.
Decision: Reject