Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes certified fairness as the worst case prediction loss on a distribution Q such that d(P,Q) \leq \rho, where P is the training distribution. The proposed definition is analyzed in two different cases (i) general shifting (ii) sensitive shifting

strengths_and_weaknesses: 
S1 The problem of studying fairness under distribution shift is an important topic.

S2 Paper presents interesting ideas to bound loss for shifted distribution.


W1 Theorem 1 is the main result, and it lacks any justification in the main paper. Theorem 2 uses thm 1 and its proof is not that interesting.

W2 Please add more insights for the proof in the main paper and appendix. 

W3 Experiments: The process used to generate distribution shift is synthetic and highly specific. Please explain a high level summary of the steps taken and how it impacts the results.

What is the solved used for the optimization problem? How much is the runtime?

Comparison with baselines: Please add a detailed comparison of the proposed approach with baselines and discuss advantages. Fig 3c is the only comparison it does not explain the reasons for difference in quality of different techniques.

Typos:
line 359: following following
line 361: one dimensio 

Overall, the paper presents many interesting theoretical ideas but the empirical evidence is weak and presentation can be improved. 
questions: Please explain key steps of the proof and add crucial details of experiments to the main paper.
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose fairness certification on end-to-end model performance based on a fairness constrained distribution. 
strengths_and_weaknesses: Strength: 
- The paper seems solid. 

Weakness: 
- The paper is too dense with notations. An illustrative example (perhaps, earlier in the paper) would improve readability.
- Experimental results such as figures are narrowly explained.
questions: - How do you train a classifier on a distribution?
- How do you get $ \rho $ the distance between two distributions in practical datasets?
- How do you differentiate between a fair and a non-fair distribution?
limitations: In my opinion, there is no negative societal impact of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose the certified fairness problem under two kinds of distribution shifts, i.e., the sensitive shifting and the general shifting. They provide the fairness certification framework to deal with the problems. Theoretically, they achieve the upper bound for the two settings and the bound is tight for the sensitive shift setting. Finally, they conduct experiments on six real-world datasets to prove the effectiveness of the method.
strengths_and_weaknesses: **Originality**

\+ The authors propose a new problem setting, i.e., the certified fairness problem, and develop non-trivial methods to deal with it.

**Quality**

\+ The method provided in this paper is technically sound with detailed proof. The experiments are comprehensive.

\- The relationship between traditional fairness notions (e.g., DP and EO) and the proposed setting should be further discussed. It is well-known that common fairness notions can not be satisfied at the same time in non-trivial settings [1]. As a result, it is not clear why Proposition 1 could guarantee DP and EO simultaneously. More discussions of the tightness of the bounds in Proposition 1 are encouraged.

**Clarity**

\+ The paper is generally well-written. Several small typos exist. For example
 - Line 40: "other other" --> "other"
 - Line 361: "dimensio" --> "dimension"

**Significance**

\- My major concern is about the problem setting proposed in this paper. Fairness problems generally require that several specific statistics match among different subpopulations (e.g., DP and EO) or individuals, which is not the case proposed in this paper. As a result, I am not clear whether the proposed problem setting is practical and sound in real-world scenarios. It would be better if the authors provide real-world examples to demonstrate the soundness of the proposed problem setting.

[1] Kleinberg, Jon, Sendhil Mullainathan, and Manish Raghavan. "Inherent trade-offs in the fair determination of risk scores." arXiv preprint arXiv:1609.05807 (2016).
questions: See details in the Strengths and Weaknesses section, especially the points in the Quality, and Significance parts.
limitations: More limitations are demonstrated in the Strengths and Weaknesses section.
review_title: null

Meta-review: metareview: The paper considers an important problem of certifying fairness of trained classifiers based on its performance on some set of fairness constrained distributions. They show the framework for two types of shifts -sensitive shifts and general shifts.

Reviewers are supportive of acceptance. There were many concerns raised by reviewers and reviewers have promised changes and have made them in their revisions. I hope they can incorporate it in their camera ready.
confidence: Certain
Decision: Accept