Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a game-theoretic analysis of adversarial robustness. In the adversarial examples game the authors propose, attackers choose an attack method (including both the optimization method and the defense to attack) while defenders choose a defense. This game is connected to the study of transferability of adversarial examples across diverse attacks. With respect to transferability, often attacks which are strong against one attack produce adversarial examples that do not transfer well to another defense. The authors then show that one can determine an optimal randomized strategy for attacker and defender by considering a matrix game where the payoffs are the success rates of how well each attack transfers to each defense (the values called $r_{d, a}$ in the paper). An optimal defense strategy is computed for CIFAR-10 and Tiny-ImageNet given several defenses and it is shown that it has higher robust accuracy than any single one of the defenses.
strength_and_weaknesses: In general, I think the idea of the paper is quite interesting and the theory and experiments are explained clearly. As far as I know, nobody has explored an optimal strategy for randomizing over multiple defenses by using game theory. The experiments seem to support the theoretical analysis and there is a nice result that the randomized strategy is more robust than any single of its members. The connection between the game theoretic analysis and transferability of adversarial examples is also quite interesting.

I think that the main weakness of the paper is that the presentation could be improved. I found that a lot of the paper seemed to focus too much on just the methods, mathematical analysis, and algorithms, and not enough on the motivation for them. This meant that I didn't appreciate the ideas behind the paper until I had more fully internalized the methodology sections. I worry that readers will just read the title, abstract, or introduction and not appreciate all the contributions.

In particular, I think there might be too much emphasis on the parts of the paper dealing with transferability of adversarial examples. This idea has been explored a lot in the past and a casual reader might assume that the paper is just another study on transferability. Instead, I think the paper might be more impactful if the authors emphasize the game-theoretic analysis and the idea of coming up with better randomized defenses. A title that might be more attention-grabbing could be something like "Optimal Randomized Adversarial Defenses via Game-Theoretic Analysis of Attack Transferability." If the authors focus on the goal of picking a good randomized defense and use the analysis of transferability more as an argument to justify that goal rather than as a primary contribution, I think more people will find the paper novel and interesting. If more space is needed to add more motivation, I think some of the details of the new attacks could be moved to the appendix since I think the game-theory ideas and results are more interesting than the attacks. But I'm just one person so maybe others will think the attack/transfer part of the paper is more interesting!

Besides the presentation, one objection to the method in the paper (and randomized defenses in general) is that a defender could just re-submit the same adversarial example until the defender randomly picks the vulnerable model. For instance, say an attacker is trying to post offensive pictures by fooling a content-filtering system on social media. Then, they could just resubmit the pictures until the content-filtering system randomly picks a vulnerable classifier. It would be good to discuss this limitation somewhere in the paper.

Smaller issues/suggestions
 * On page 9, you say "For CIFAR-10 instance r* = .573, meaning the worst expected performance of the ensemble against
these attacks is to get a robust accuracy of 57.5%." Shouldn't it be 57.3%? Also, seems like there is a typo and it should be "For CIFAR-10 for instance" or just "For CIFAR-10".
 * This seems like a relevant paper to cite, although your approach is definitely quite different: Balcan et al. Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games. (Also I realize that it came out after you submitted, so use your judgement as to whether you think it should be cited.)
clarity,_quality,_novelty_and_reproducibility: The descriptions of the method are clear and in general the math and experiments seem correct. I did not closely inspect all the experimental details. While the attacks presented are only slight modifications of existing attacks, the game-theoretical analysis seems quite original.
summary_of_the_review: Overall, I found the paper to be an interesting analysis of adversarial transferability and its connection to randomized defenses via game theory. I would encourage the authors to work on the presentation of their ideas so that others can more easily appreciate their results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how adversarial examples crafted for one defensive model transfer on other models, and correspondingly how random defensive models help adversarial robustness.
strength_and_weaknesses: # Strength 
The studies topic is an interesting and practical one. According to my knowledge, such topic has not been discussed in the past. The theoretical results seems sufficient to me. However, I have not checked its every detail. 

# Weakness
The empirical verification is not very convincing nor sufficient. According to the theory established by this work, I expect the author to construct a state-of-the-art defense model by randomly choosing the prediction model from an ensemble of models.
clarity,_quality,_novelty_and_reproducibility: This paper is clear. The quality is relatively good.
summary_of_the_review: As is.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a finite set of attackers and a finite set of defenders and views them in a game-theoretic framework to choose the optimal choices for attacker and defender.

Specifically, it solves a min-max problem to find the optimal weights for attackers and defender.



strength_and_weaknesses: Strength
-  The paper mentions to many methods in adversarial machine learning.
-  Some experiments about the behaviors are pretty interesting though not new.

Weaknesses
-  The paper is not well-organized and well-written. The sections are not really logically linked and coherent each other.
-  Section 5 is messy with inconsistent notions.
   - $\theta(x + a(\theta, x ,y))$: $\theta$ is generally for a model parameter and it is strange to write $\theta(x + a(\theta, x ,y))$.
   - The notations $p_a$ and $a$ are confusing because it is easy to think $p_a$ depends on $a$. Similarly for $p_d$ and $d$.
   - No clear definition for $a_i(U,x,y)$. Here $U$ is a set of defenders, but why $a_i(U,x,y)$ returns a common perturbation for every defender in $U$.  
-  The assumption about finite number of defenders is not feasible. We might interpret $d_i$ as a defender type (e.g., CNNs, ViTs). However, it seems not sufficiently rich to represent the model parameters. 
- The main experiment in Section 6 is very humble.

clarity,_quality,_novelty_and_reproducibility: - The idea of min-max problem in attack and defense is not new  (e.g. [1]) and other relevant works.
- There is no literature for other works investigating game-theoretic perspective of attacks and defenses (e.g., [2], [3], and so on). 
- The writing needs a significant improvement. 
- The contribution of the paper is not clear. 

[1] Jingkang Wang, Tianyun Zhang, Sijia Liu, Pin-Yu Chen, Jiacen Xu, Makan Fardad, and Bo Li. Adversarial attack generation empowered by min-max optimization. Advances in Neural Information Processing Systems.

[2]  Le, T., Tuan Bui, A., Minh Tri Tue, L., Zhao, H., Montague, P., Tran, Q. &amp; Phung, D.. (2022).  On Global-view Based Defense via Adversarial Attack and Defense Risk Guaranteed Bounds . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics.

[3] Cranko, Z., Menon, A., Nock, R., Ong, C. S., Shi, Z., and Walder, C. (2019). Monge blunts bayes: Hardness results for adversarial training. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research



summary_of_the_review: - The paper is not well-organized and well-written. The notions used are confusing. 
- The theoretical setting is not feasible and practical.
- The contribution is not clear and significant.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers a set of recent defenses against adversarial attacks on the image domain and uses existing attacks or develops them to capture a game-theoretic interaction between the attacks and defenses. To obtain the utility functions, they evaluate the effectiveness of different attacks on different defenses, thereby conducting a transferability study. Eventually, they consider a mixed strategy over a single-classifier or a less-than-n-ensemble selection for the defender by formulating the game as a zero-sum game and solving for a Nash-equilibria in this game. They show that this is more effective than using single defenses.
strength_and_weaknesses: ### Things I liked

- I like how the authors concisely described many of the recent defenses, the attacks and highlighted their relevance for this work.

- The authors propose new attacks (eg. MIME) by composing existing attacks, situate its need well in the landscape of existing attacks, and showcase its effectiveness.

### Things that need clarification / improvement

- The study of transferability of attacks on defenses, the formulation of interaction between attacks & defenses, and using game-theoretic equilibrium to come up with a mixed-strategy over defenses is not novel. [1] does all this-- given the white-box threat model, their game-theoretic formulation considers that the attacker may even know the defender's strategy and thus, considers a Stackelberg Equilibria as opposed to a Nash Eq. This paper does the study on more recent attacks and defenses and consider a more relaxed threat-model. Thus, all the 3 novel questions the paper presents on page 1 already have answers. Having said that, I do appreciate the renewed study on recent attacks and defenses but cannot support the novelty claim.

- While the reward function mostly concentrates on the loss function value for the adversarially perturbed examples, it doesn't account for the loss on the non-perturbed examples. I would surely suggest the authors to consider the loss on the latter examples in the reward function, at least for the defender, similar to [1]. Otherwise, the strategy found may over optimize for accuracy/loss on the adversarial examples while losing out on accuracy for the true test distribution.

- Given this is a zero-sum game, the Nash Eq. generates a min-max payoff and can be calculated in polynomial time. Hence the statement "These linear programs can be solved using weakly polynomial time algorithms like the interior point method Karmarkar (1984), or those developed in Vaidya (1989)" makes me wonder if the authors are aware of the subtle aspects of their formulation.

- Game_n considers a larger space (that explodes exponentially $~2^n$) of defense actions where the defender can use up to n-classifiers for ensembling. Note that this incurs much larger computational cost as one needs to estimate the utility values for each of the attack-defense pairs using $N$ samples. Hence beyond simply reporting accuracy on the adversarial examples and the test set, the authors should report the time taken to form the game-matrix. Also, the authors should discuss how effective the final strategy becomes as the number of samples used for the utility estimation ($N$), and therefore the cost, increases or decreases.

- Some study on if an attacker can attack the game-theoretic defense strategy is necessary to understand if this too is an effective defense. Based on results in [1], I would assume black-box distillation would not be as effective. Would be interesting to see how this holds for different values of $n$ $GAME_n$ and $N$ in estimating utilities.

[1] Sengupta, S., Chakraborti, T., & Kambhampati, S. (2019). Mtdeep: Moving target defense to boost the security of deep neural nets against adversarial attacks. In International Conference on Decision and Game Theory for Security.
clarity,_quality,_novelty_and_reproducibility: The paper's novelty is limited in terms of technical ideas but empirically, it does evaluate a game-theoretic setup on recent defenses and attacks. Given the pace of development in this direction, it is definitely worth-while. I would definitely ask the authors to refactor their contribution statement in this way.

The lack of mention on what $N$ is and a more though study as how $N$ affects the utility estimates and thus the overall game-theoretic effectiveness, and computation cost is important to give an overall picture.
summary_of_the_review: See above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers a set of recent defenses against adversarial attacks on the image domain and uses existing attacks or develops them to capture a game-theoretic interaction between the attacks and defenses. To obtain the utility functions, they evaluate the effectiveness of different attacks on different defenses, thereby conducting a transferability study. Eventually, they consider a mixed strategy over a single-classifier or a less-than-n-ensemble selection for the defender by formulating the game as a zero-sum game and solving for a Nash-equilibria in this game. They show that this is more effective than using single defenses.

All reviewers appreciated the game-theoretic perspective on multiple attacks and defenses in adversarial robustness. The original recommendations had high variance, ZYXb and hEcL being positive; while rrfN and once recommending rejection. The main concerns expressed by rrfN and once were related to the novelty of the work (in particular the authors missing an important citation mentioned by once), as well as some aspects of the writing of the paper as raised by rrfN. Through discussion with the authors as well as the revision of the paper, some of these concerns were addressed, and rrfN increased their recommendation from 3 to 5. On the other hand, ZYXb agreed with the novelty concerns and decreased their recommendation from 8 to 6.

While this paper is quite promising in its contributions, this AC judges that the support for this paper was too low to recommend acceptance at ICLR. The extensive discussions can yield to significant improvement in the writeup, and the authors are encouraged to take them in consideration and resubmit. In particular, the revision should more properly contrast its contributions with the citation [1] from reviewer "once".
justification_for_why_not_higher_score: Not enough support from the reviewers.
Decision: Reject