Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper discusses how to estimate fairness measures in the presence of missing values. Especially, it compares the fairness measure for the complete data domain (data distribution assuming no missing values) and the fairness measure estimated with reweighing on the complete case domain (data distribution after dropping all the data points that have missing values). They provide an upper bound and a lower bound on the difference of the two. They show experimental results of estimating the fairness measures on synthetic data and two real-world datasets to evaluate.  
main_review: While fairness in machine learning and handling missing values in the machine learning pipeline have been active areas of research, the intersection of them — ensuring/assessing fairness in the presence of missing values — is a relatively new and unexplored area. Existing works have mostly focused on empirical studies and there has been little to no theoretical study on the topic. 

This paper provides a theoretical study on estimating group fairness metrics, such as accuracy parity gap, in the complete case domain. This problem can be thought of as statistical estimation under a domain shift, and hence share the similarity with the results in [37], which studies fairness in transfer learning. However, transfer learning setting specifically for the missing values problem is new. Also proof-technique-wise, the paper claims to make some advances from the existing works in domain adaptation, but I did not check the proofs. 

As the authors acknowledge, the paper considers only the simplest setup where we drop a row if it has any missing values, which might not be the most practical scenario. Furthermore, the paper only evaluates the bias on two generic datasets which do not have missing values and they had to generate artificial missing values. The lack of real-world evidence and the simple problem setup limits the practical value of this work. However, I believe that it is an important first step to establishing a theoretical understanding of fairness in the presence of missing values. 

The paper is very well-written and clear.
limitations_and_societal_impact: I did not find any statement regarding this in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors tackle the problem of ensuring fairness between subgroups in the presence of missing data, as the missingness can be biased between classes. The authors propose upper bound as well as lower bounds on the generalization error when propensity weights have to be estimated. The former is novel but classical, the latter is novel but also greatly original.
main_review: The authors place themselves in the setting where some data can be censored at random and fairness between classes needs to be enforced.
The authors correctly give a parallel to the domain adaptation setting but the same setting can be found in the survival analysis literature where similar upper (not lower) bounds exist [1]

[1]G. Ausset, S. Clémençon, and F. Portier, “Empirical Risk Minimization under Random Censorship: Theory and Practice,” Available: http://arxiv.org/abs/1906.01908

The main result, in Theorem 1, provides an upper bound of the disparity in fairness while Theorem 2 provides a lower bound. Both results are theoretically very interesting and the proofs bring valuable methods.

I have, however, some remarks about the experiments:
1/ First the results for the true weights should not be in bold as those performances are impossible to achieve.
2/ Following 1/, this means that in 3 out of 6 cases the best estimator is the unweighted one. This needs to be mentioned and analysed. Why is that?
3/ Why do the SVM w perform so badly? Any insights?

Other than those small concerns and the review of literature that needs to be improved, this paper succinctly studies and solve a very interesting theoretical problem. I think it is a great fit for NeurIPS. I do not have any additional remarks, but given the limited time to review the proofs I can have missed errors.
limitations_and_societal_impact: Yes the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides theoretical guarantees (upper and lower bounds) on the estimation error of fairness, in the setting where missing data is present and only the complete rows are used for estimation. In particular, the authors consider the “accuracy parity gap” as the fairness metric, which can be applied to both classification and regression tasks. The paper also includes some numerical experiments to validate the theoretical findings.
main_review: Theoretical analysis of fairness estimation with incomplete data is largely missing in the algorithmic fairness literature and can be a valuable contribution. The results also apply to both classification and regression. The authors claim that the results can be generalized to other group fairness notions, but this claim is not supported with details. 

While the theorems for upper and lower bounds can be useful in understanding the behavior of fairness estimation under different settings, I am not sure if they will have much practical impact. To obtain intervals of fairness guarantees using the theorems, it requires quantities that are not available in practical settings such as the total variation distance with the target distribution. Also, the lower bound only holds with a very low probability.

The authors also empirically study the effects of multiple factors on fairness estimation which is a nice contribution, but the results are somewhat to be expected from studies in data domain shift and propensity score models. I would have liked to see more experiments on how these factors affect the bounds, to see how useful the bounds would be in different scenarios.

Overall, the paper is well-written, and the contributions and limitations are clearly described. While the theoretical perspective for fairness estimation with missing data is novel, I am leaning towards reject due to concerns about practical impact, as mentioned above.

Minor comments:
- I found the terms “complete case domain” and “complete data domain” to be not very distinctive and confusing at times. However, I do understand these are terms often used in the literature.
- It seems the estimation approach here can be applied to other domain shifts such as under-representation of samples for certain groups. More explicit and detailed discussion on this would be interesting.
- Why was the upper bound assessed only on classification tasks and the lower bound only on regression tasks?
- Figure 2 caption: typo (b) -> (c)
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents  theoretical results about fairness and missing data. The authors show theoretical bounds for the fairness metric "accuracy parity gap" considering weights. The weights are estimated from the data based on correctly or incorrectly specified propensity score models under a given missing data mechanism.

Some experiments on synthetic and real data (where some parameters are controlled) are useful to demonstrate the validity of  the included theoretical.
main_review: The paper is easy to read and follow. The authors motivate properly the impact  of unfair machine learning models. They also introduce the common problem of having missing data in training instances. Some works have studied the combination of missing data with fairness issues, however not significant theoretical results related to fairness metrics have been developed. In this aspect, the authors are able to propose a  framework  where they formalise the problem and the analysed   metric  ("accuracy parity gap" ), as well as three missing data contexts: missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR). In this framework, they include a weight w used to express the fairness difference between D_t (complete data domain) and D_s (complete data domain). Considering this estimated value w, the authors develop two results about the limits in APG. This is the main result of the paper.

After that, the authors include a set of synthetic experiments where they study the bias considering different values of w and different imbalance factors. The experiments show the Assessment of the upper and lower  bound in classification and regression. Additionally, they add experiments with real data, however in a controlled environment since  they generate missing values under the three settings, namely, MCAR, MAR and MNAR.

weak points of the paper:

* I do not see clearly the applicability of the proposed results. Although the author comment some ideas, I miss a clear example where they can show the benefits of the study.

* In this sense, I also miss experiments with real data with actual missing values.  Some studies have shown that there exist some patterns with missing data and fairness. The application of the proposed framework over one real case could be useful.

* Finally, it could be also interesting in see how the proposed result could be employed to define imputation methods that could mitigate the fairness inequalities.
limitations_and_societal_impact: No negative impact.
review_title: null

Meta-review: The reviewers agreed that this is a well-written paper that addresses a relatively unexplored facet of fair ML: the impact of missing data on fairness. The setup considered in the paper is simple, yet amenable towards an interesting theoretical analysis. However, the paper also has limitations, particularly in terms of how the bounds can translate to "fair imputation methods," as noted by reviewer ykYW. I agree with this concern, and wished that the authors had gone further than the results in the paper and discussed methods to *ensure* fairness in the presence of missing data. The reviewers also noted that the experimental results are on datasets where entries are artificially missing, raising concerns about the practical impact of the paper in setting where data missingness may be correlated with group attributes.

I also found the claim "Our work provides the first known results on fairness guarantee in analysis of incomplete data" to be a bit misleading, since there have been a few papers that at least discuss the topic (e.g., "Missing the missing values: The ugly duckling of fairness in machine learning" by Fernando et. al and https://arxiv.org/abs/1911.12587), albeit to a less theoretical extent.

Overall, the merits of this paper outweigh is limitations, and its publication will encourage more discussions on the impact of missing data on fairness.

Decision: Accept (Poster)