Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper study the adversarial robustness in transductive learning, where the defenders can update the models during test time by interacting with the adversary in multiple rounds. The authors propose "Greedy Model Space Attack" (GSMA), where the key idea is to general adversarial examples considering all the history models in the past rounds. The result show that the proposed attack can decrease robust accuracy more than other baseline (the static AA, DENT AA).  
main_review: ## Strength
1. Experimental results seem to suggest the proposed GMSA is effective (though there are some places not clear and require clarification, see below questions)
2. The proposed idea (greedy model space attack) is simple yet seemingly effective

## Weakness
1. The writing needs to be improved. The current draft is written in a complicated way, which is hard to parse, e.g. the notations, the problem setting explanation etc.
2. The are some questions/places not clear and require clarifications from the authors, see below.

### Questions
Q1: How practical is the problem setting of adversarial robustness in transductive learning? Also, it's a bit not clear about how exactly the adversary and the defender interact. I assume it works as follows:
adversary generate a batch of adv examples (adv_0) based on model_0 (original model) --> defender get adv_0 and update the model to model_1 --> adversary generate a new batch of adv example (adv_1) based on model_1 --> defender get adv_1 and update the model_1 to model_2 --> repeat to the end

- Are my above understanding correct? If so, then is it practical to assume that the adversary and defender will interact in this way? 

- Perhaps a more difficult setting is when the adversary have no ideas of when the defender actually update the model? In this case, would the proposed method work (it seems that the proposed method is a white-box attack setting)? 

Q2: it's a bit not clear to me regarding the "private randomness". 
- What happens if the adversary also know about the randomness in $\Gamma$? The attacker should be able to develop a stronger attack?
- The authors mention that the $\Gamma$ can be randomized smoothing type of defense, but I didn't see the experiment results compared with randomized smoothing? Can the authors show some experiments for this? 

Q3: Table 1, Table 2
- Table 1: It's not clear to me how exactly is the difference between the AA under static and AA under RMC? Using my understanding in Q1, does it mean that, AA (static) refers to attacking model_0, while AA (RMC) refers to attack model_i, while i = {1, 2, 3, ...} ? Then why AA (RMC) does not work? 
- Table 2: can the authors explain what's the difference between DENT-AA and AA under DENT? why they have a large gap (~20%) in robust accuracy? 
- Can the authors explain again what's the difference between DENT and RMC? It looks like the proposed GSMA is more effective on RMC compared to AA (RMC) while not so effective compared to AA (DENT)? 

Q4: the authors mention the setting of adversarially ordered and naturally ordered data. But from the Algorithm 1 and 2, the data order doesn't seem to matter in GMSA? Why DENT and RMC need the naturally ordered data? 


summary_of_the_review: In summary, this paper propose a simple yet effective GSMA attack to evaluate adversarial robustness of models in the transductive learning setting. Some results are promising but there are a few places require further explanation/clarification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes adaptive attacks against transductive robust learning methods. These methods aim to improve the robustness of models to adversarial examples by updating the model using unlabeled test data. However, this paper notes that previous evaluation of transductive robust learning has not considered attacks that are truly adaptive, i.e. the attacker should be able to utilize its knowledge of the transductive learning algorithm to craft better examples. Two types of adaptive attacks against transductive learning algorithms are proposed, a single step method and a greedy method taking into account all previous instances of the model learned during transductive robust learning. The experimental results on two existing training methods and two newly proposed ones, one of which is a strawman, demonstrate that the proposed adaptive attacks render transductive robust training to be no more robust than standard robust training. The paper's conclusion is that the previously claimed improvements in robustness were an artifact of the use of non-adaptive attacks.
main_review: **Strengths**
- The paper is generally well-written and clear. It clearly articulates issues with the previous evaluation of transductive learning algorithms for learning and lays out the threat model that should be used in the future when evaluating these algorithms. It also delves into the various aspects of the multi-round game that results from an attack on transductive robust learning.
- The empirical study is fairly thorough, tackling 4 different types of robust learning algorithms across two standard datasets, using the two different attacks proposed in the paper.
**Weaknesses**
- While the paper does a good job of articulating the challenges for an adaptive attacker on transductive algorithms, including that the problem results in a bilevel optimization problem for the attacker, it does not use particularly novel methods to tackle this problem. Both proposed methods just turn the problem into a static optimization problem which just use PGD to solve it. There is a plethora of literature on end-to-end poisoning attacks (see [1], [2]) that uses methods such as gradient unrolling or replacing the model with an approximation derived from the Neural Tangent Kernel. The GSMA attack that is proposed merely uses the history of all the trained models, and is essentially still a static attack. I would urge the authors to understand the methods used in the poisoning literature for circumventing the challenges of bilevel optimization.
- Some aspects of the manner in which the final robustness numbers are determined for the proposed attacks are unclear. Specifically, when the transductive model is run for T iterations, is the robustness reported using the best adversarial example set over all iterations, but evaluated with respect to the final model with all test data? Further, for DENT, the value of T is set to 2, so does that mean the numbers reported are only with respect to two samples from the test set? I read through the paper and appendix multiple times for clarity on this point, but it did not seem to be addressed.
- The strawman TADV approach is not evaluated with sufficient rigor. The private randomness can be circumvented by the attacker simulating adversarial training themselves with multiple instantiations of the initialization and adversarial example generation, and then generating universal adversarial perturbations to be effective across all these locally generated models. It would be interesting to see how the defense performs under that setting.

**References**
[1] Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization, https://arxiv.org/abs/1708.08689
[2] Neural Tangent Generalization Attacks , https://proceedings.mlr.press/v139/yuan21b.html

summary_of_the_review: Overall, while I found the paper to elucidate an important shortcoming in previous work on evaluating the robustness of transductive robust learning, the attack methods used lacked novelty. In fact, in a number of cases the simple FPA algorithm had the best performance. I urge the authors to consider improved methods to circumvent the bilevel optimization problem.

+++++++++++++++++++

The clarifications have improved the paper. While I still believe more innovation is possible for the attack, I have decided to raise my score to a 6 since the paper is likely to lead to interesting follow-up work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new attack strategy for adversarial examples in the case where transductive defense strategies are employed. The proposed attack is based on projected gradient descent and manages to break existing transductive defenses that were thought to perform well. A new strategy for adversarial training is also proposed yielding more robust models in the transductive setting than existing defenses. Experiments are performed on MNIST, CIFAR-10 and GTSRB, against both naturally trained and robust models.
main_review: Strengths:
- I appreciate the formalism of the adversarial game for transductive learning introduced by the paper, which also seems novel.
- The paper covers a wide range of topics: formalizing the transductive attacks and defenses, breaking transductive defenses thought to be strong, experimenting with domain adaptation as defense against the attack proposed in the paper.
- The proposed attack seems to perform well, defeating existing defenses. The experimental section seems to follow standard practices for robustness evaluation.
- The implementation is provided.
- The paper is well-written.

Weaknesses:
- The topic of the computation cost of the proposed attack is not discussed.
- The proposed method seems to be a variation of PGD (limited novelty), but considering that the paper has other contributions and GMSA works well in practice, this work is arguably worth publishing.

Other comments / questions:
- The paper could benefit from additional proofreading.
- The transferred adversarial samples produced with PGD and AutoAttack might be weaker than some generated with the proposed attack, thus not resulting in a proper evaluation.
summary_of_the_review: Good formalism for adversarial machine learning in transductive setting, plus other smaller contributions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper intends to break a specific type of adversarial defenses. These defenses choose to dynamically modify the model for the purpose of robustness and defenses. This paper proposes to counter such defenses with a new optimization algorithm.
main_review: My biggest concern is about the importance of the studied topic in this paper. Specifically, the so-called transductive defenses are not very popular and have not been fully recognized by the community so far. In fact, I happen to have read the paper of RMC when it was published in ICML and thought that the authors of RMC should have tested some adaptive defense-aware attacks in their work, which they did not. After all, breaking random or dynamic models for attacking purpose has been investigated by BPDA and EOT. Meanwhile, I also served as a reviewer for Dent, which was rejected at the time and has not been accepted by other conferences according to my knowledge. In the review opinions for Dent, many reviewers also questioned that it failed to conduct some defense-aware and adaptive evaluations. Thus, to me, it seems that the proposed attack GMSA origins from the obfuscated gradient work and only breaks a few defenses that are somewhat not canonical. 
summary_of_the_review: I give a weak reject score for now. I may change my score after I exchange my opinions with other reviewers.
review_title: null

Meta-review: The paper formalizes the adversarial attack problem for transductive defenses, where the model is sequentially updated with a batch of (adversarial) test inputs. The paper comes up with a quite generic attack scheme and their instantiation of this scheme shows that RMC and DENT are not robust respectively not more robust than the underlying adversarially robust base model.
  
Positive 
- formal treatment of attacks on transductive defenses including discussion about different types of attacker knowledge
- the attack model is quite generic and could work for future transductive defenses and thus is a useful baseline attack which could be suggested to be used by future transductive defenses for robustness evaluation. In particular, as the standard AutoAttack is not designed for transductive defenses and thus can overestimate adversarial robustness

Negative
- the description is sometimes overly technical and some (important) details had to be clarified
- the technical novelty of the attack is limited
- the overall accuracy but also robust accuracy depends on the chosen batch. Therefore the authors should report mean and standard deviation over several different random draws of batches 
- the Transductive Adversarial Training Defense seems to consist of adversarial retraining from scratch after each incoming batch. This is excessively costly and not practical.

Minor:
- The batch size is an important parameter which apparently is assumed to be known in this work

The paper is borderline. Two reviewers argue for rejection, two for acceptance. Only one reviewer engaged in the discussion. 
In my point of view the positive point of having a reference for correct evaluation of adversarial robustness of transductive defenses weighs more than the raised negative points which can be fixed (at least partially). Thus I think that this paper is a valuable contribution to the field of adversarial robustness.
Decision: Accept (Poster)