Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an evaluation protocol to study robustness of image classification models. The protocol essentially creates a new test set depending on the model being evaluated and the dataset it was trained on. This, the authors argue, would reduce dependence on static benchmarks and create a “dynamic evaluation protocol”. However, this creates an issue where robustness of models cannot be directly compared since they no longer have the same test set. An oracle (CLIP) is introduced which helps us compare robustness fairly.

The authors provide an algorithm to generate new, ‘perturbed’ images. We use a generic generator model (VQGAN in the paper) to perturb the clean image, then check whether this perturbation has the same label as the original label (using the oracle). This step is repeated several times to stay within a budget.

The above algorithm applied to images in the clean dataset produces perturbed images which act as a test set. This test set depends on the choice of the model being evaluated, oracle and generator. Therefore, it is a general, dynamic way of evaluating robustness.
The authors also introduce a new metric called oracle-oriented robustness (OOR) = (Counterfactual acc)/(standard acc). This is used to measure the robustness gap to the oracle.

MNIST, CIFAR-10 and ImageNet are used for experiments with several vanilla image classification models (ResNet etc) and some models with claimed robustness properties (AugMix, DeepAugment etc).
strength_and_weaknesses: Strengths:
1. Overfitting benchmarks and evaluating robustness are important problems to the community.
2.  The authors have used a large selection of classification models to test their hypothesis.
3. Comprehensive ablation studies on the oracle and the generator.
4. The paper is fairly simple to understand and concepts are explained well.

Weaknesses:
1. The authors introduce a scoring function $\alpha$ in their image generation algorithm which guides the generation process but there is no concrete example given about what it is. I’m assuming this scoring function (at least the one used in the paper) essentially maximizes the loss of a given image in the direction of a different class. This is the same as Santurkar et al. where they use the inner maximization problem to generate images using a classifier.
2. I was under the impression that OOR values would be <1 since standard accuracy would usually be higher than counterfactual accuracy but all values in the paper seem to be > 10 at least. Is there a ‘*100’ missing? 
3. The authors claim that images generated using their method are diverse. However, no diversity metric are given. Since there is a computational budget, a lack of diversity of images could artificially inflate the robustness of a model.
4. Is OOR supposed to give a ranking of robust models? I ask because in Table 6 in Appendix D, a vanilla model has higher OOR than PGD (so more robust than PGD). I find this hard to believe. This must mean that OOR specifically depend on the threat model being used. The authors should be more explicit about this limitation.
5. It is unclear how OOR behaves when you have different generators but the same oracle. 
6. Since we are using the oracle, any gaps in the robustness of the oracle will also show up in the model being evaluated. The authors, to their credit, mention this in Section 5 but it is still a major limitation.
clarity,_quality,_novelty_and_reproducibility: This work is original to my knowledge and there are no major issues with the writing.

Typo: See section 4.4, bullet 3 “robsutness”

summary_of_the_review: Overall, I think the authors have proposed a fairly simple and solid idea with good evaluation, however, there seem to major limitations about its applicability which need to be clarified. I think this is just below the publication threshold for me.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper questions the validity of fixed benchmarks in evaluating model robustness for real-world applications where the i.i.d. assumption may not hold. Instead, the authors propose to use dynamically perturbed test examples to evaluate each model separately. The perturbed examples must meet two requirements. First, they can still be correctly classified by an oracle model so that most of the underlying causal structure of the original examples are preserved. Second, the examples are perturbed in an iterative manner that maximizes the loss induced by these examples for each given model. A new metric, namely oracle-oriented robustness (OOR), is also proposed for this new evaluation protocol. The OOR of a model is defined as the accuracy of that model on the perturbed examples over the accuracy on the original test examples. With this new evaluation protocol, experiments on ImageNet suggest that strong data augmentation is a key indicator of high OOR despite the model architecture (CNN or ViT). In particular, dynamic data augmentation usually leads to higher OOR than pre-augmented data. Finally, another main empirical finding reported in the paper is that the choice of the generative model used for image perturbation does not affect OOR very much.
strength_and_weaknesses: # Strengths
- The paper studies an important open problem: how to measure model robustness in a general out-of-distribution setting.
- The idea of generating hard test examples for each individual model during evaluation and measuring the robustness of a model with respect to an oracle model is novel.
- The paper is overall well-written and easy-to-understand.

# Weaknesses
- In section 2.2, the authors suggest that the dynamic generation process helps avoid the selection bias of the models. However, how much of the selection bias is avoided is unclear. Actually, it is arguable that current models are affected by the selection bias to a degree that requires any special treatment (see [1]). Besides, the choice of the generative model and the oracle model can also introduce selection bias, albeit less direct than fixed benchmarks. So, the proposed protocol is at best “mitigating” the issue rather than “avoiding” as stated in the paper. This weakens the significance of the paper.
- Throughout all the experiments on ImageNet, the validation rate (VR) is almost the same for models of different architectures and data augmentation techniques. This begs the question whether the generated examples are sufficiently diverse so that they really expose different problems of each model. The provided examples show some visual variations but are not conclusive. Imagine that a model is much more robust than the other, then intuitively, the VR of the robust model should be lower than the VR of the other model because the generated examples for the robust model should be harder for the oracle model. But this is not the case for the models considered in the paper. Moreover, if the oracle model (e.g., CLIP) itself has some weaknesses, then a model that has the same weaknesses would probably have higher OOR than another model that does not share those weaknesses, given that their VR is the same. The paper does not mention if the OOR is sensitive to the choice of the oracle model. All these points are currently unclear and require further investigation.
- Regarding the sparse submodel of VQGAN, the authors first use VQGAN to generate a style-transferred dataset (the first sentence of the third paragraph of section 3.2.1) which is then utilized to find out the feature dimensions related with style. This appears to be circular reasoning. If the style dimensions are unknown and need to be find out, how did the VQGAN generate a style-transferred dataset in the first place? What if there are also significant semantic changes? In that case, the pruned dimensions may also contain many relevent style dimensions. The paper reports that 99.31% of the dimensions are pruned on average, but it only leads to 28.5% runtime reduction on ImageNet. It is not clear if the sparse submodel is efficient enough considering the expense of possible reduction in the diversity of the generated examples.
[1] Recht et al. Do ImageNet Classifiers Generalize to ImageNet?

clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is mostly clear, although there are still some minor points that need to be improved:
- The text of Fig. 1 is too small.
- In section 3.2.1, the authors use lowercase “l” to denote label, which is easy to be confused with the number 1 and the loss “l” (which is also in lowercase).
- Typo in the last paragraph of section 3.2: missing a double quote after “an image of {class label}”.
- Typo in the last line of section 3.2.1: redundant word “the”.
- Typo in the first sentence of the third paragraph of section 4.2: “vanillar”.

# Quality
The quality of the paper has room for improvements since some parts of the paper seem to be written in a hurry.
For example,
- The quality of Fig. 1 is not very good.
- The choice of math notations and the formatting are somewhat casual.
- As an empirical paper, the experiments can be more comprehensive.

# Novelty
The paper presents several novel ideas which are interesting and worth exploring.

summary_of_the_review: The paper studies an important and general problem, and the approach is novel.
Nevertheless, there are some weaknesses to the central arguments of the paper, and the overall quality of the paper just barely meets ICLR standards (which is quite high, in my opinion).
At least some of the issues need to be addressed before I can recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an adversarial attack benchmark where the attacked samples are generated by high-quality generative models filtered by a "surrogated oracle" (a model trained with large-scale extra data points, such as CLIP). More specifically, the proposed method generates adversarial samples by maximizing classification loss (i.e., cross-entropy loss) using VQGAN. The generated image is validated by the CLIP model, and if the CLIP model predicts the generated image with a wrong label, then the generated image is set to the original image. With the generated adversarial samples, this paper shows that the existing vision models are not robust to the proposed attack method.
strength_and_weaknesses: ## Strength

- This paper tackles a very important problem in the robustness benchmark: we need a dynamic benchmark (based on optimization) rather than a manually collected dataset. Also, the generated dataset should be realistic (not based on l2 or l-infinity ball).
- This paper proposes a sparse submodel of VQGAN for reducing the computational cost

## Weakness

- I feel this paper has a limited novelty. The main modules are from the other works (VQGAN, CLIP). The idea of "adversarial attack by generative model" is an old and popular idea [R1-5]. The novel part of this paper could be (1) using a novel generative model rather than GAN, and (2) filtering based on the CLIP model, but I think these contributions are very limited. Furthermore, using VQGAN and CLIP models could be problematic as my next comment.
    - [R1] Song, Yang, et al. "Constructing unrestricted adversarial examples with generative models." Advances in Neural Information Processing Systems 31 (2018).
    - [R2] Xiao, Chaowei, et al. "Generating adversarial examples with adversarial networks." arXiv preprint arXiv:1801.02610 (2018).
    - [R3] Poursaeed, Omid, et al. "Generative adversarial perturbations." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
    - [R4] Qiu, Haonan, et al. "Semanticadv: Generating adversarial examples via attribute-conditioned image editing." European Conference on Computer Vision. Springer, Cham, 2020.
    - [R5] Jang, Yunseok, et al. "Adversarial defense via learning to generate diverse attacks." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
- I feel that the quality of the proposed generation process is still limited. Furthermore, I feel that the proposed method is highly biased toward VQGAN and CLIP, leading to biased evaluation results.
    - The proposed method highly depends on extra modules, such as VQGAN and CLIP. Namely, the quality or diversity of the generated images will be bounded by the VQGAN performance. Similarly, the quality of the generated images is bounded by the CLIP zero-shot generalizability to the generated images; if CLIP zero-shot performs worse for specific types of perturbation, then the proposed framework cannot cover such perturbation. As shown in the "Potential limitation" paragraph, the CLIP zero-shot performance is not consistent across the classes. It could cause a biased benchmark toward the selected models (i.e., CLIP).
    - As shown in Table 1-2, only about one-third of images are changed by VQGAN for ImageNet. It means that VQGAN and CLIP models cannot generate a proper image, or classify the generated images correctly. I think the quality of the proposed generation process is still not effective for measuring robustness.
    - Not only the generated samples will be biased toward CLIP, but also the metric will be biased toward CLIP as well. Because the generated samples are biased to the CLIP zero-shot performance, "CA" score will also be biased.
- Also, this benchmark is only available when a strong and generalizable generative model and a strong and generalizable "surrogated oracle" models are available. Thus, this benchmark is only limited to natural image benchmarks, such as ImageNet. This is not a strong weakness, but it is worth to be discussed.

## Questions and minor comments

- There is no detailed explanation of how VQGAN is guided by classification loss. I presume that the guidance for VQGAN is done by directly optimizing VQGAN encoder space (as the CLIP-guided VQGAN), but it would be great if the authors will provide more details for this.
- I feel that the concepts of "counterfactual" and "confounder" are used incorrectly in Section 3.3.
    - What does "counterfactual" mean here? For example, in the field of causality, the terminology "counterfactual explanation" describes the causal relationship by "If X has not occurred then Y does not occur". Similarly, in machine learning, a counterfactual example means a modified example with the minimum changes (to see what affects the prediction at most). However, the terminology "counterfactual accuracy" is weird. How accuracy becomes "counterfactual"?
    - Why the standard accuracy acts as a confounder of "counterfactual accuracy"? How standard accuracy and counterfactual accuracy are related in terms of causality? I cannot find any relationship between them, as well as, it is not trivial to treat "accuracies" as random variables. Also, to define a confounder (as far as the reviewer understood) we need to define "Do operation". What is "do operation" for accuracy scores?
- The formulation for $\hat x$ seems weird. I recommend avoiding using the same notation for the optimized results ($\hat x$ in LHS) variable ($\hat x$ in RHS, below $\arg\max$).
- Some citations are missing. For example, Section 4.2. missed citations for Mixup, CutMix, and random erasing (Cutout and RandAug are worth being cited as well).
clarity,_quality,_novelty_and_reproducibility: - I feel that the quality of this paper has a large room for improvement. For example, this paper borrows some terminologies from the field of causality (e.g., "causal structure", "counterfactual", "confounder"), but they are not well-defined in this paper. Also, I feel this paper is not self-contained. For example, there is no detailed explanation of how the generative model is guided by classification loss. I also recommend making Section 4.4. self-contained in the main text.
- In terms of novelty and originality, I think this paper has limited novelty and contribution as my previous comment.
summary_of_the_review: - I think this paper has very limited novelty and the quality of this paper has a large room for improvement.
- Furthermore, the proposed method highly depends on VQGAN and CLIP models. I think it will cause serious problems: (1) the generation quality itself is limited by VQGAN and CLIP performances (2) the metric is biased toward the CLIP zero-shot performance. Hence, measuring the proposed benchmark will measure how a model is similar to the CLIP model, not measure how a model is robust.
- Overall, I recommend "reject".

---

Post-rebuttal comment: As my last comment, my concerns are somewhat addressed by the response, but I feel that my main concerns still remain after reading the response, the other reviews, and the revised paper. Hence, I will maintain my initial recommentation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Several works stress test ImageNet classifiers on out-of-distribution benchmarks, which are static, defined either through pre-defined perturbations or a new test-set collection procedure. The paper proposes measuring out-of-distribution robustness through dynamically generated samples using VQGAN for each model instead, constrained to maintain semantic information . The oracle in this case is CLIP pretrained on a large number of text-image pairs.
strength_and_weaknesses: Strengths:

The paper leverages large pretrained models to generate per-model out-of-distribution datasets dynamically which is new.

Weakness:

The paper is unclear on several details which are important to clarify before I can assess the experiments. Please see below.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

Several key details are unclear or missing in Section 3.1.

* What exactly is the scoring function $\alpha$ ? Only the inputs to the functions are provided but the exact formualation is missing.
* What is the optimization objective with respect to the VQ-GAN latent space? How is the latent space perturbed?
* In Section 3.2.1, the paper describes a “style-transfer” process to sparsify the VQGAN latent space. Can the authors describe the process in detail?
* VR: Validation Rate: Why is the validation rate 100 or not close to 100, given that the algorithm introduces the constraint that the perturbed image has to be correctly classified in Section 3.1?
* I had to read until the methods section to understand that the oracle is a pre trained CLIP model. It would be nice if this is introduced as soon as possible in the introduction.




Some typos and rephrasing:

* as high as a human can reach -> as high as a human
* while push the study of robustness evaluation further -> while pushing the study
* More details of these lines
* oracle-parallel performance -> performance comparable to the oracle
* by assuming an oracle -> with respect to an oracle
* instead of indirectly compare models’ robustness -> instead of indirectly comparing models robustness	
* in either machine learning context or causality context -> in either context of machine learning or causality.
* the accuracy on the images our generation process successfully produces a counterfactual image -> the accuracy on the counterfactual * images, that our generative model successfully produces
* s when tested by data from different distributions -> tested on data from different

summary_of_the_review: See strengths and weaknesses above.

I think the paper could be of interest to the community but more work is required to provide all the adequate details necessary.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an adversarial attack benchmark, where the attacked samples are generated by high-quality generative models filtered by a "surrogated oracle" (a model trained with large-scale extra data points, such as CLIP). More specifically, the proposed method generates adversarial samples by maximizing classification loss (i.e., cross-entropy loss) using VQGAN. The generated image is validated by the CLIP model, and if the CLIP model predicts the generated image with a wrong label, then the generated image is set to the original image. With the generated adversarial samples, this paper shows that the existing vision models are not robust to the proposed attack method.

However, there are several obvious weakness: 1) This paper has very limited novelty and the quality of this paper has a large room for improvement. 2) The proposed method highly depends on VQGAN and CLIP models. It will cause serious problems: namely, the generation quality itself is limited by VQGAN and CLIP performances, and the metric is biased toward the CLIP zero-shot performance. Hence, measuring the proposed benchmark will measure how a model is similar to the CLIP model, not measure how a model is robust. 3) Whether the proposed evaluation method is currently a better alternative to fixed benchmarks. More experiments are needed to clearly show this point. For example, experiments showing that fixed benchmarks indeed do not reflect the OOD robustness of models in general, and experiments on different oracle models. Overall, this paper may not be ready for publication at ICLR. The next version must be a strong paper if authors can take comments into consideration.
justification_for_why_not_higher_score: According to my expertise and reviewing process, this paper should belong to a Reject.
justification_for_why_not_lower_score: According to my expertise and reviewing process, this paper should belong to a Reject.
Decision: Reject