Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on utilizing synthetic images generated by a generative model for the task of achieving robustness to adversarial attacks. Towards this, the paper aims to assess the suitability of the proxy distribution defined by the generative model for the underlying task. The paper shows that conditional Wasserstein distance serves as a selection criterion for the proxy distribution. Acknowledging the intractability of estimating the conditional Wasserstein distance, the paper then proposed a novel metric based on robust discrimination between the proxy distribution and the real distribution. Finally, the paper empirically demonstrates that using synthetic images from a suitable generative model can indeed improve the robust and clean accuracy of the model over existing baselines.
main_review: Strengths:

1. The paper captures the utility of a proxy distribution for ensuring robustness via conditional Wasserstein distance between the proxy distribution and real distribution.

2. Given that estimating the conditional Wasserstein distance is prohibitive, the paper proposed a tractable metric based on robust discrimination between proxy and real distribution to measure the utility of a proxy distribution for the underlying task.

3. Extensive experiments on five standard image benchmarks demonstrate the utility of the proposed method in improving model robustness by utilizing synthetic images produced by generative models.

4. The experimental results in the paper also show that widely used quality metrics for the generative models (e.g., FID) fail to capture their utility to general synthetic images that can ensure robustness for real images.

Weaknesses/questions:

1. What does PORT stand for?

2. What is the motivation for working 64x64 size images for ImageNet as opposed to utilizing usual 256x256 or 224x224 image sizes? Would the performance improvements attained by the proposed method scale to larger images?

3. In last line of page 5, '...our trained discriminator...' --> '...our trained *robust discriminator...'?  Also, should $f(x)$ be $\sigma(x)$?

4. What does ARC stand for?

5. While comparing with RST in Section 3.1.2, do you use $L_{smooth}$ in the definition of $L_{adv}$ in Section 2.3?
summary_of_the_review: The paper studies an important problem of improving the model robustness against adversarial attacks. Towards this paper aims to leverage the recent advancements in the area of generative models. The paper shows that utilizing the synthetic images generated from suitably chosen generative models can boost model robustness. The paper both theoretical and empirically studies various metrics to select the correct generative model for the underlying task.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates if generative models can be used to improve the robust accuracy of image models. The authors show that the current SOTA across multiple attack models and commonly used datasets can be significantly improved by using generative models. They show that the conditional Wasserstein difference between the test distribution and the one learned by the generative model is an upper bound for the difference in robustness achieved by models trained on either distribution. 
They investigate what generative models provide good proxy distributions by robustly training discriminators and using their accuracy at different attack strengths as a metric. The discriminators are also shown to be helpful to order generated samples by their effectiveness for adversarial training. 
main_review: The paper is a very good contribution to the research on adversarial robustness. The authors manage to push the SOTA, explain why certain generative models work better at improving the robustness and extensively evaluate their approach across lots of datasets, attack models and hyperparameters. Related work is described and contrasted. Using generative models for robustness certainly is a very important direction for research.

ARC could also be an interesting metric for generative models in general. 

The paper is clearly written but sometimes a little bit cramped as the authors do a lot of things, especially in section 3. 

Small comments and questions:


- You write that you do a hyperparameter search on γ. Did you find that different values work better for different generative models?
- Have you tried different ratios of real and synthetic images in the batch? How does it interact with γ?
- The improvement on especially imagenet is rather small. Do you have any intuition why?
- On the top of page 8 you talk about outperforming Zhang et al. 2020 but I don't see it listed in table 4.
- While you can infer it from the rankings it would be helpful to remind readers what metrics in table 5 are increasing or decreasing with quality (just a small arrow pointing up or down next to the metric would help). 

- The epsilon ball is sometimes written as Ball_e(x) and sometimes as Ball_x(e) (bottom of page 4 and bottom of page 5)
- Figure 1: The caption starts with "Why robust discrimination is effective?" Either this should be "Why is robust discrimination effective" or shouldn't end in a question mark. Same for page 5 "Why non-robust discrimination...?". 

summary_of_the_review: The authors provide an in depth evaluation of using generative models to augment adversarial training, showing large improvements and interesting insights both empirical and analytical. I think the paper should be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new metric for how well a synthetic distribution can help to improve the adversarial robustness. This metric is designed based on measuring conditional Wasserstein distance between the two distributions.
main_review: The paper studies an important problem for adversarial robustness, and I like the theoretical analysis part. My concern is mainly on the experiments:
* For CelebA, CIFAR-100, ImageNet, why is there only one baseline while for Table 1, there are several other baselines.
* Could you provide some comparisons between the adversarial training on normal synthetic distribution and robust synthetic distribution?

And I have an additional questions:
* It's quite interesting that the performance of DDPM is much better than other generative models, is there any explanations for this? Could you provide an ablation study where a) use DDPM and standard synthetic metric b) use DDPM and robust synthetic metric?
summary_of_the_review: The paper studies an important problem for adversarial robustness, and I like the theoretical analysis part. My concern is mainly on the experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates how adding additional synthetic data can help with robust training.
While the basic idea is very simple, the paper adds an interesting theoretical analysis that allows us to estimate what generative distribution would work best for robust training.
main_review: Strengths:
- The ARC metric is interesting and novel. The authors also show that has concrete benefits in allowing us to estimate which generative model would help in robust training.
- Empirical results are good, good number of datasets.

Weakness:
- One issue is a missing part of the proof of theorem 1. In the proof, the authors define a distribution D' by a mapping of average distance epsilon from D. This is clearly an upper bound on the Wasserstein distance, but to show that it is the Wasserstein distance, i.e. the minimal coupling is missing. Without this correction, I cannot take the theoretical part (which is the major novelty) into account.
- While the ARC metric is interesting, to compute it we need to compute several robust classifiers. This makes it pointless in practice as we could, in less computation time, directly measure the robust accuracy that we are interested in.
- Missing ablation study - in C.3 the authors show that if they generate 15M and then sample 10M to do robust training, their sampling is better than randomly taking 10M. What is missing is the results of training on the whole 15M. As the whole 15M are already generated, what would be the benefit of sampling 10M if it doesn't perform better?
summary_of_the_review: The paper offers some interesting theoretical and practical results, but without the hole in the proof fixed I do not think it should be accepted. I will gladly raise my review if this is adressed.
review_title: null

Meta-review: In this work, authors use proxy distributions learned by advanced generative models to improve adversarial robustness. In the discussion period, authors did a good job in addressing reviewers' questions and comments. All reviewers think the paper is above the accept threshold, so do I.
Decision: Accept (Poster)