Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed a method to preprocess the dataset, so a machine learning classifier learned on the pre-processed data would be counterfactually fair.

Strong points:
+ The studied fairness problem is very important, and the causality-base fairness is a very challenging question
+ The proposed method is very intuitive and easy to follow

Points for Improvements:
- On the experiments part, I was wondering why the paper didn't compare to the original Counterfactual fairness paper's algorithms by Kusner et al. 2017. They have proposed three levels of algorithms to study the counterfactual fairness.
- On the experiment evaluation part, I was wondering why the paper chose MAE as the performance metrics, because we are talking about classification problems here. Are the ground truth scores just 1, and 0, and we are comparing our prediction probability to these 1 and 0's?
- Sorry if I missed the point, but I would appreciate the authors kind clarification: the dataset is preprocessed using a function we proposed, and the counterfactual metric (CF-metric) is also evaluated using this exact preprocessing function, while all other methods have no access to this function. Would this appear to be unfair to other baselines? What if say the EO, AA algorithm was also trained on the pre-processed dataset?
- The paper assumes a causal model as in Fig. 1. Does this model always hold in practice? If it is not, what should we do?
- Minor: Page 4 Section 3, Line 1, $\mathcal{S} \times \mathcal{A} \rightarrow \mathcal{A}'$, missing a '
- Minor: Above Page 3 Definition 1., U is used as unit and also previously as noise variable. Would probably be less confusing to choose another character. 
review_title: A paper on an important topics, but need improvements on execution

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper addresses the problem of preprocessing the data in a way that the predictions of a learning task will be counterfactually fair. The counterfactual fairness definition is borrowed from that of (Kusner et al., 2017). The authors propose ortogonaliza tion and marginal distribution mapping so as to achieve counterfactual fairness. They test their proposed approach on synthetic and real data.

The math of Definition 1, Counterfactual Fairness of (Kusner et al., 2017) is wrongly stated. Counterfactual fairness is defined over the predicted $\hat{Y}$. I suppose (although am not sure) that it’s a typo because it’s correct in the proof of the theorem in the appendix.

The MAE of FLAP-1 and FLAP-2 are higher than other methods in Figure 2, although are lower in the loan application experiment. The authors can elaborate on why this is the case and provide some discussion. This is also the trend in examples 2 and 3 in Figure 4. The authors discuss the DF-metric which has shown promising results in these figures. More discussion on the MAE would be appreciated.

I would encourage the authors to run more experiments with more real-world datasets (possibly ones that are publicly available) so that the readers can get a more comprehensive comparison of the methods. 

It is not clear what the difference is between the FTU predictor and the AML predictor? The text says, in Section 3.1., that “A Fairness-Through Unawareness (FTU) predictor $f_{FTU}(a)$ only uses the non-sensitive attributes” and that “An Averaged Machine Learning (AML) predictor [the mathematical expression] only needs non-sensitive attributes a as its input”. These descriptions are confusing. 

review_title: Review of Counterfactual Fairness through Data Preprocessing

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a preprocessing method that eases fair learning called FLAP.
They focus on counterfactual fairness introduced by Kusner et al. 2017. Under certain conditions, the preprocessing allow to biased the data such that traditional learning becomes fair.
This is a very interesting idea that is novel to me.

The paper is well written, and the related work is coherent with the work done. However, I think a few new paper are missing, for example Counterfactual Fairness: Unidentification, Bound and Algorithm from Wu et al 2019 that bounded the reachable counterfactual fairness. It would have been interesting to see how the results related to these bounds.

I think that the statement:  “We prove that CF is equivalent to the conditional independence of the decision and the sensitive attributes given the processed non-sensitive attributes” is a bit misleading. As far as I understand, this is only true given that either condition 1 is satisfied or conditions 2 and 3 are satisfied. There exists statistical test for conditional independence but checking for condition 1, 2, 3 seems more complicated.

On the real data, I would like to see a discussion on the meaning of CF-metric and FLAP method given that the condition 1, 2,3 are not verified.

I think that theses clarifications are needed for a clear accept.

----- edit after the authors' answers -----
I'm not convinced by the authors' answer on the meaning of the CF-metric when condition 2 is not verified, which is the case on real data. "... but we would expect fairer decisions to have smaller CF-metric in practice" doesn't seem enough to claim that FLAP is better because its CF-metric is lower. In addition we don't have confidence interval to assess if FLAP is really better on MAE.

Finally the review of reviewer1 made me realize that CF-metric and FLAP are based on the same function. I partly agree with the authors' answer, i.e., we need a counterfactual function to construct a good metric and then it is optimal to use it as well in the algorithm. However it is not fair to use that metric to claim that the algorithm is fairer. 

Therefore I lower my rating to 5. 
review_title: An interesting paper but some poitn need to be clarified

Meta-review: The paper introduces an approach to counterfactual fairness based on data pre-processing, and compare it to other two counterfactual fairness approaches on the Adult and COMPAS datasets.

The reviewers are in agreement that, in its current state, the paper should not be accepted for publication at the venue. Their main concerns are around the metric used to measure fairness, and these were not resolved during the discussion. The reviewers would have also appreciated more experiments on real-world datasets to get a more comprehensive comparison of the methods. Finally,  discussion and comparison with other methods to achieve counterfactual fairness from the literature were limited. 
Decision: Reject