Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: General:
The paper proposed to use a causal fairness metric, then tries to identify the Pareto optimal front for the vectorized output, [accuracy, fairness]. While the proposed method makes sense, I am not sure what exactly their contribution is. It is kind of clear that Pareto optimal exists, and what they did is to run the experiments multiple times with multiple \lambda values for the Chebyshev method and plot the Pareto optimal front. 

Pro:
Ran multiple experiments and drew the Pareto optimal front for the considered dataset. 

Con & Question:
The so-called causal fairness metric does not seem to be any more fundametal than the other proposed metrics. It seems like they worked with another metric. 
After defining the fairness metric, everything else seems straightforward. Just use test (validation) set to estimate the accuracy & fairness, then plot the results on the 2d plane. 
Can we identify the Pareto optimal front without running all 1500 experiments? What happens when running a model takes long to train? Then, the proposed method cannot be practical. 
Figure fonts are very small and hard to see. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors propose a novel joint optimisation framework that attempts to optimally trade-off between accuracy and fairness objectives, since in its general formal counterfactual fairness is at odds with classical accuracy objective. To this end, the authors propose optimising a Pareto front of the fairness-accuracy trade-off space and show that their resulting method outperforms an adversarial approach in finding non-dominated pairs.

As main contributions, the paper provides:
* A Pareto objective formulation of the accuracy fairness trade-off
* A new causal fairness objective based on the existing Weighted Average Treatment Effect (WATE) and Average Treatment Effect for the Overlap Population (ATO)

Overall, I think the paper makes an interesting contribution to the field of fairness and that the resulting method seems quite attractive for a real-world practitioners. However, I found the writing / notation imprecise at times and the experimental section too small (lacking an extensive set of baselines, and only on two datasets). For these reasons, I give it a Weak Accept.

Some feedback on notation / writing:
* Typo on page 2, the loss L should be defined on X x Y and not Y x Y
* In page 5, h is being used without being introduced first 
* the justification for using ATO in the internal layers of the network is a bit insufficient

In terms of suggestions, I think the experimental section needs to be extended and that the various modelling choices need to be explored and/or be further justified.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a method to approximate the "fairness-accuracy landscape" of classifiers based on neural networks.
The key idea is to set up a multi-dimensional objective, where one dimension is about prediction accuracy and another 
about fairness. The fairness component relies on a  definition of fairness based on causal inference, relying on the 
idea that a sensitive attribute should not causally affect model predictions.

I found the causal idea intriguing, since it makes sense that we don't want a sensitive attribute to have a causal effect.
However, there may be several problems with this approach:

1) For a causal estimate to be valid we need several assumptions. For example, we need A (the sensitive attribute) 
to be independent of potential outcomes conditional on X --- the so-called "unconfoundedness assumption" in causal inference.
We also need "positivity", i.e., that 0< P(A=1|X) <1. 
These assumptions are not discussed in the paper. Furthermore, the particular context of the paper, where the treatment is actually an immutable characteristic, makes such discussion much more subtle. 
What will we do, for instance, if there are no A=1 in the sample when X = ...?


2) The authors seem to assume that the propensity score model is well specified. This can be tested, e.g., using [1].
What do we do when this fails? 


3) Why do we want U to be small, i.e., why do we want the causal effect of A to be small, is never justified.
In particular, its relation to "fairness" is never fleshed out, but just assumed to be so.
This can be problematic when, say, we are missing certain important X that are important for A. 
Then, there will be a measurable causal effect of A on h(). 

Some other problems:
- What is the reason for focusing on 'neural classifiers'? There is nothing specific in the method or analysis 
that relates to neural networks, except for the use of the causal estimand in a 'hidden layer'.

- In the Introduction, the authors could cite the works of Amartya Sea, etc., on fairness. 
Certainly the study of fairness problems did not start in 2016.

- What exactly is a "sensitive attribute"? If we don't want to bias our predictions, then why include it in the analysis?

- It is unclear what is new and what is related work in page 3.

- Sec. 4: The claim that "causal inference is about situations where manipulation is impossible" discards 
voluminous work in causal inference through randomized experiments. In fact, many scientists would 
agree that causal inference is impossible without manipulation.

- As mentioned above, why this particular estimand leads to more "fairness" is never explained.

- Do we need a square or abs value in Eq (5)?

- The experimental section is weak and does not illustrate the claims well. 
   It would be important to explain the choice of the particular causal estimand, the choice of the hidden layer to put the estimand in, to explore the choice of the objective, and so on. Currently, none of these choices/design aspects are being investigated.



[1] "A specification test for the propensity score using its distribution conditional
on participation" (Shaikh et al, 2009)
review_title: Official Blind Review #3

Meta-review: This manuscript investigates and characterizes the tradeoff between fairness and accuracy in neural network models. The primary empirical contribution is to investigate this tradeoff for a variety of datasets.

The reviewers and AC agree that the problem studied is timely and interesting. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the novelty of the results. IN particular, it is not clear that the idea of a fairness/performance tradeoff is a new one. In reviews and discussion, the reviewers also noted issues with clarity of the presentation. In the opinion of the AC, the manuscript is not appropriate for publication in its current state. 
Decision: Reject