Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: review
Main review: The paper proposes an analysis of learning with fairness constraints in the presence of label noise. The difficulty of the framework is that the authors study the general problem of instance-dependent noise, while previous work focused on simpler notions of noise such as class-conditional noise.

The authors first study the setup of loss minimization under equalized odds constraints. essentially assuming that the noise model is known. Then the authors study a special case of counterfactual fairness (based on the model in Figure 1 and instanciated on the Adult dataset), and propose a variant of the algorithm of Kusner et al. to perform counterfactually fair prediction when the ``true'' label is not observed.

Overall the paper is well written, even though the assumptions are not entirely clear (see below). The novelty and originality is limited because there is a wide literature on both fair ML with label noise and counterfactual fairness, and the paper stays close to existing approaches. Also, a large part of the paper is devoted to calculations that could be deferred to an Appendix. Experiments are carried out thoroughly with relevant baselines. 

pros: compared to the counterfactual fairness paper (by Kusner et al.), I like the idea of distinguishing the observed biased label, a ground truth label Y and a counterfactually fair label that is learnt. This allows to avoid a problem of the original paper that it was impossible to evaluate whether the algorithm was close to an underlying truth or not, since it was not clear what the "ground truth" was. 

some comments:
* The value of Section 3 to the paper is unclear to me, since my understanding was that the intention was to treat the case where the noise model is unknown. 
* It would be better to spell out from the start of section 3 that the assumption is that the transition matrices are unknown
* In Eq 7 I don't understand what T_{-1} is. The dependency of T on X seems to have magically disappeared, and I do not see how that is possible without making additional assumptions (e.g., that the noise only depends on the protected group, but not on arbitrary features of X
* The model used in the section on counterfactual fairness is clear but not clearly motivated. Since U only depends on Z, I don't understand the value of having functions of U for W and Y -> why not simply functions of Z directly? 
* the motivation underlying mixup in that setting is not clear to me, in particular it is introduced when "f is miled by tilde{Y}" (what does "mislead means") and then the justification is that "mixup increases the sample complexity of the training set" (what is the sample complexity of a dataset?)
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes a novel problem that learning a fair classifier with instance-dependent label noise. The authors provide a general framework to deal with this problem for both statistical fairness notions and causality-based fairness notions. Experimental results show the effectiveness of the proposed methods.
Main review: Pros:
1. This paper is well motivated. Learning a fair classifier with instance-dependent label noise is a novel, interesting and significant problem. 
2. This paper is presented clearly and organized logically.
3. This paper proposes a general framework and considers fairness including both statistical fairness notions and causality-based fairness notions. In addition, the methods are well supported by theoretic analysis.
4. Extensive experimental results are reported on two real datasets and compared with six representative methods.

Cons:
1. My main concern is the rationality of the constructed causal graph. Although the authors take ADULT dataset as an example to demonstrate the proposed causal graph interprets the data, it is uncertain whether the construction of the causal graph is complete. For example, what happens when the parents of A exist?
2. Table 1 shows that the label noise makes fairness-aware algorithms even more prejudiced than fairness-unaware ones, and the instance-dependent label noise is a more challenging setting compared with class-dependent label noise. It would be better if the authors can provide a deeper analysis of this phenomenon.
3. It would be better to add some discussion about the relationship and distinction between this paper and related work.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper considered an important machine learning direction, the fairness issue, in the context when there is labeling noise in the outcome. The paper showed the existing method may suffer from the existence of certain types of label noises and therefore the authors developed robust method for both statistical fairness and counterfactual fairness to account for those. The proposed methods are considered together with other state-of-the-art in two real datasets.
Main review: Originality: The paper considered an interesting situation in which there are label noises on the outcome which renders most state of the art fairness-aware methods collapsing. This is a new problem to consider in the fairness literature and the authors provided an initial attempt to solve the issue.

Significance: The paper addressed an issue in the fairness background which is clearly relevant to the CLeaR community. The method did not significantly advance the state-of-the-art but rather I feel it is an immature attempt to solve the problem, but the results in the paper indeed provided new insights to the research problem.

Technical quality: The proposed approach appears reasonable and has been substantiated by empirical results but not theoretical results.

Clarity: The draft is badly written, at least in the current forms, especially the experiments section. Table 2 and table 4 have exactly the same caption, which did not provide enough information for the readers. I also spent some time finding where the R-p-fair is defined. I believe the draft needs rewriting, otherwise, as a reader, I have to take a lot of guesses.
review_title: null

Meta-review: The reviewers generally agreed that this paper makes a contribution to fair learning. There were some questions and confusions that were largely ameliorated in the authors' response. (One reviewer did not respond to the response, but I read both the review and response and felt that the authors cleared up the confusion.) This paper is above the threshold for acceptance.
Decision: Accept (Poster)