Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to attack a Bayesian model for improving the transferability of black-box adversarial attacks. Specifically, the authors employ gradient-based attack algorithms on their constructed Bayesian models and expect the generated adversarial attacks can better fool other unseen models. Extensive experiments show that the proposed method surpassed baseline methods in terms of attack success rate.
strength_and_weaknesses: Strength

1. In general the paper is well-written.
2.  The authors conduct extensive experiments to validate the transferability of the proposed method by concerning many different neural architectures and multiple baselines.
3. The proposed way to employ Bayesian estimation for improving the transferability of attacks looks novel to me.

Weaknesses

1. When both the substitute model and the victim model are adversarially trained (Table 6), the improvement is quite marginal. Therefore, the proposed method may only be employed to improve the transferability of attacks on nonrobust models, which weakens the empirical significance.
2.  In Eq. 6, ∆w is sampled from the gaussian prior. Then why do we need p(∆w) ≥ ε? Plus, if p(∆w) ≥ ε is important, then ε should be a very important hyper-parameter but there is no experiment showing the sensitivity,

clarity,_quality,_novelty_and_reproducibility: The Clarity is good. The proposed way to employ Bayesian estimation for attack transferability looks novel. The authors claimed that the code will be available.
summary_of_the_review: Given the strength and weaknesses, I tend to rate the paper as marginally above the acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to optimize for diversity in substitute models and advocate attacking a Bayesian model for improving the transferability of adversarial examples.  The author also developed a Bayesian formulation for performing attacks and advocated possible finetuning for improving the Bayesian model.  Extensive experimental results have demonstrated that the proposed method can enhance the transferability of adversarial examples, yielding better attack success rates.
strength_and_weaknesses: Strengths:
1.	Utilizing Bayesian models to increase generative adversarial sample mobility is instructive, and the study of ensemble attacks is interesting. The attack method can be equivalently regarded as performing adversarial attack on a set of infinitely many substitute models.
2.	The baselines are advanced, and extensive experiments demonstrate the effectiveness of the proposed method.

Weaknesses:
1.	Some experimental settings are not very reasonable. For example, in Table 4, the authors adopt ResNet-50, Inception v3, MobileNet, and MNASNet altogether as substitute architectures of baselines while adopting ResNet-50 and MobileNet as substitute architectures when combining their method with the ensemble methods. Why not experiment with the same surrogate model? This comparison may be unfair when most of the black box models use the ResNet structure. Moreover, it will be more convincing if the author can add some experiments on advanced defense models, such as adversarial training models, Randomized Smoothing (RS)[1], and Neural Representation Purifier (NRP)[2].
2.	There are some severe spelling mistakes in the article that the author should check carefully, such as “Aidmix” should be “Admix”.
3.	The authors could try to add some analysis on why this method works.
[1]Cohen J, Rosenfeld E, Kolter Z. Certified adversarial robustness via randomized smoothing[C]//International Conference on Machine Learning. PMLR, 2019: 1310-1320.Naseer M, Khan S, Hayat M, et al.
[2] A self-supervised approach for adversarial robustness[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 262-271.

clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is clear and the paper is easy to follow.
Quality: The experimental results are impressive.
Novelty: Somewhat novel.
Reproducibility: Not applicable.

summary_of_the_review: The experimental results are impressive and the idea is interesting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of crafting adversarial attacks that are transferable across different deep neural networks architectures (DNNs). In order to solve this problem the authors propose to craft the attacks on Bayesian neural networks (BNNs) trained with Gaussian posterior approximations. On a set of experiments on Cifar-10 and ImageNet the authors show that their approach outperforms state-of-the-art methods.
strength_and_weaknesses: Strengths

- The main idea of the paper, which is to use BNNs to obtain transferable adversarial attacks, is clear and intuitive

- Experimental results are convincing and show that the proposed method outperforms competitive algorithms

Weaknesses

- A recent paper [Gubri et al (2022a)] also explored BNNs to obtain transferable attacks. While the methods in this paper seem to obtain better empirical results, the results of this paper feel incremental compared to [Gubri et al (2022a)]

- Section 3 and 4 are often confusing and contain some inaccuracies. In particular:

-- After Eqn 1, p(y | x,w) is the likelihood and not the predictive distribution.

-- In Section 3, experiments are mixed with the explanation of the methods and this creates confusion. Furthermore, in this Section the authors present adversarial attacks for BNNs assuming that the posterior is an isotropic Gaussian. However, for basically all approximate posterior inference methods commonly employed for BNNs, the posterior will in general never be an isotropic Gaussian. In view of this, I would re-organise the Section presenting the method for a general Gaussian posterior.

-- L is Eqn 3 is not defined (I guess it is the loss, but please say it explicitly)

-- In Section 4 the description of the experimental setting takes almost one page. To improve readability I would move some of the details (especially those about the details of the methods the authors compare against) in the appendix. 

- Something that is not clear to me is why the authors need to develop a new method to attack BNNs, instead of using methods already available in the literature, see e.g. [1,2,3]. This should be at least discussed and motivated. Also, it is not clear how the authors are attacking BNNs trained with SWAG. Please, clarify

- An important aspect that the authors should keep in mind (and discuss) is that BNNs are known to be more robust to adversarial attacks compared to standard neural networks [4] (depending on the approximate posterior method employed). As a consequence, the results of this paper seems to hint on the fact that while it may be harder to find adversarial attacks for BNNs, these may transfer more easily across different architectures.

- Why in the experiments do you need to use a bigger $\epsilon$ for ImageNet than for Cifar-10? This is counterintuitive for me  


[1] Liu, Xuanqing, et al. "Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network." International Conference on Learning Representations. 2018.

[2] Yuan, Matthew, et al. "Gradient-Free Adversarial Attacks for Bayesian Neural Networks." Third Symposium on Advances in Approximate Bayesian Inference. 2020.

[3] Carbone, Ginevra, et al. "Robustness of bayesian neural networks to gradient-based attacks." Advances in Neural Information Processing Systems 33 (2020): 15602-15613.



clarity,_quality,_novelty_and_reproducibility: The idea of using BNNs to craft transferable adversarial attacks is not completely original and novel. The quality and clarity of the paper could be improved (see Strength and Weaknesses Section). Enough details are given in the paper to reproduce the results.
summary_of_the_review: The paper presents an interesting empirical idea with encouraging experimental results. The quality of the paper is hindered by the clarity of the writing, some inaccuracies, and missing discussion and justification about the methods used to train and attack a BNN.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to attack a Bayesian model for improving the transferability of black-box adversarial attacks. A gradient-based attack is constructed where the generated adversarial attacks can better fool other unseen models. Extensive experimental results have demonstrated that the proposed method can enhance the transferability of adversarial examples, yielding better attack success rates.

Strength:
1. Well-written presentation
2. A novel Bayesian attack method
3. Extensive experiments on adversarial transferability

Weakness:
1. Need more discussions on most recent works
2. Some of the experimental settings are unclear
justification_for_why_not_higher_score: This is a good paper overall, with solid technical contributions and extensive experiments. However, AC thinks that the paper doesn't bring new insights into adv. attack and adv. sample generation. 
justification_for_why_not_lower_score: The contributions of the paper are above the bar of ICLR, and the paper should be of broad interest of ICLR audience.
Decision: Accept: poster