Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes to certify for a network the robustness to change in the input distribution for bounded Wasserstein shift.
strength_and_weaknesses: Strength:
- The paper is well-written and mostly easy to follow
- The problem of provable-robustness guarantees is important, especially for DNN.
- Giving a bound using Wasserstein distance can be stronger as it handles shifts that change the support

Weaknesses:
-  The work, more or less, reduces the Wasserstein distance to total variation (whose robustness was already explored previously by Ben-David et al) by smoothing. One issue is that smoothing can reduce performance, thus the robustness is on the smoothed classifier and not on the original classifier.
- The paper talks about adversarial attacks but does not show these results in the main paper. I think these results are much more important than the results presented in the paper on color etc. and must be in the main paper.
- All results should be compared to the empirical robustness (although maybe with fewer plots at once as it is too cluttered to understand). 
- I found sec. 6 trivial and misleading. If a classifier is guaranteed to be robust, then of course trying to fool it will fail. It is also misleading as hardness to learn normally is distributions over $(x,y)$ that learning algorithms fail to learn from which isn't the case. 
clarity,_quality,_novelty_and_reproducibility: The work is somewhat novel and reproducible. 
The paper is clear, but some of the important figures (that should be in the paper but are in the supplementary) are unclear

summary_of_the_review: The idea isn't groundbreaking but important and worthy of publication. Unfortunately, the authors use most of the paper on less important parts of the work and skip the important parts.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends the recent certified robustness theory with a simple procedure that randomizes the input of the model within a transformation space. It abandons the  $l_p$ distance change assumption, and does not need any restrictive assumptions or global Lipshitz. With the above theoretical analysis, this paper can provide performance guaranteed on the lower bound on the performance with distribution shift (Both OOD problem and adversarial attack). Experiments give specific guarantees for three natural shifts. Further experiments on the poisoned dataset also show empirical success.
strength_and_weaknesses: Strength:
- The paper extends the certified robustness into the certified distribution shift robustness.
- The paper proposes a very novel new scenario for the guarantee on the natural distribution shift.
- The method is simple and the theory is solid which can even work on the unlearnable scenario.
​
Weakness:
- The paper majorly focuses on how to extend the certified robustness into natural transformation. ​However, more discussions about what properties of models can impact the OOD robustness is also appreciated.
- The experiment on the OOD data can be more comprehensive. Typically, many results are all drawn with the theoretical proof. The empirical analysis is not sufficient. With only CIFAR10 and SVHN datasets, it is not convincing enough to say whether those problems are a practical OOD problem. We can not say that the model trained with a larger dataset, imagenet, really suffers from this kind of natural shift problem. It is possible that the empirical performance can always have higher performance that the theoretical lower bound. In that way, the provable robustness may not be so important.

clarity,_quality,_novelty_and_reproducibility: ​The major novelty comes from the no free lunch theory. We can say that, there is no model that can work well in all conditions. Most existing methods may work well on some specific dataset with specific distribution shifts. However, there is only a few understanding on what exactly the distribution shift is and how can the model generalize on the shifted distribution. From the no free lunch theory, we can say that the OOD method can work well on some distribution shifts, but not for all of them. To give theoretical proof on this, what ood the model can generalize is definitely a good direction for the development of trustworthy AI and how to develop a good OOD method.
summary_of_the_review: The work quality is good. In terms of methodology, this paper is an extension from the robustness domain. The theoretical results are interesting. Some experiments should be extended correspondingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: From a high level viewpoint, the main result of this paper is a robustness guarantees on models under "smoothing": the paper makes the claim that by smoothing inputs, the difference between smoothed expectations under D and D' with a Wasserstein distance within epsilon is bounded by some function psi of epsilon satisfying a TV-distance "lipshitz type bound" for the smoothing operation. Some applications and discussions on color shifts, hue shifts, brightness shifts, etc., are provided in the paper. 
strength_and_weaknesses: Overall, the paper is adequately clear. 

Some comments on strengths are:
1. The paper discusses the importance of robustness in machine learning with many examples and references to literature. 
2. The problem in question itself is an important topic and relevant to ICLR. The paper further contributes to the field by providing interesting insights and results.

Some comments on weaknesses:
1. Why is a bound between the difference of the expectations of smoothed versions meaningful? From a practical view, would it be more reasonable and interesting to have a result that somehow bounds a smoothed expectation with the original, unsmoothed expectation?
2. Taking a brief look at the proof of Theorem 1, it appears that a concave increasing psi is required as an artifact of the proof; is there any additional insight/comments on other reasons why a concave increasing psi is necessary for the definition of (3) and its impact on Theorem 1?
3. More insights can be added to the experiments. In particular, the paper provides calculations of certified accuracy for various examples, but comparisons are lacking. It is difficult to see if the "certifications" are tight. The practicality of these certifications also can use some more explanation.
4. It would be helpful to include discussions on novelties in proof techniques, if any, within the main text.
clarity,_quality,_novelty_and_reproducibility: Overall, the writing quality and clarity is adequate. Please find some comments on quality and clarity in the above section. The work is built on past work within the robustness literature, with limited novelty in proof techniques.
summary_of_the_review: Overall, I think the paper is well written but with several concerns as listed above. I would be happy to adjust my score after discussions with other reviewers and the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors extend the randomized smoothing certified defense to bounding the change in the performance of a model under distribution shifts bounded in the Wasserstein distance metric. 

Strengths:
1. Novel certification procedure that applies to distribution shifts not handled by prior work.
2. Experiments demonstrating the efficacy of the developed approach.
3. Applying the technique to obtain fundamental limitations on learning from a poisoned dataset.

Weaknesses:
1. Experiments are only done on easier datasets and the scalability of the technique to more complex datasets or prediction tasks is not studied.
justification_for_why_not_higher_score: The authors propose an interesting novel certification technique. However, the experiments in the paper do not clearly demonstrate the value of this technique for realistic distribution shifts.
justification_for_why_not_lower_score: The authors develop a novel certification technique for certifying robustness under distribution shifts bounded under the Wasserstein distance metric.
Decision: Accept: poster