Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: review
Main review: The paper studies an approach to simplify the computation of prediction functions under constraints on path-specific effects. For the case where the the direct effect of the treatment to the outcome variable is in the paths that must be controlled, they present a reparametrization of the likelihood function that makes explicit the total path-specific effect that should be controlled. The advantage of this reformulation is to allow for simpler algorithm for the optimization of the likelihood under fairness constraints. The authors then discuss an extension of their model to learn the full distribution (including P(x))

pros: the paper presents the problem and approach thoroughly. Theorem 2 seems non-trivial and rather strong.

cons: overall I found the paper hard to follow and I had difficulties understanding the conclusions of the experiments:
* As for many papers on controlling path-specific effects, the notation and the formulas are quite heavy, but I guess this is somewhat unavoidable. Also, and similarly to other papers on controlling path-specific effects, the paper discusses only a few very simple causal models. While the generality of the theoretical results is still there, it is unclear how practical that is.
* I did not understand the take-away message of Simulation 1 (Actually, I am not sure to understand what part of the paper is assessed in Simulation 1 and how it relates to simulation 2). 
* The authors consider a transductive setting (we know the points to be labeled), and there is no reason for transductive learning to be better than inductive learning in general. It is unclear to me why the estimation of  P(X) turns out to improve predictive performances on this problem.
* In general, I found it unclear what was the aim of the authors -- improving predictive performance given a fairness constraint, or finding a better approximation to the true distribution that respects the fairness constraints. From the experimental results, there is no algorithm that performs better on the two criteria, and it is unclear to me whether this was expected or whether the experimental results are inconclusive.
review_title: null

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: A very strong piece of work!
Main review: In this paper, the authors proposed a new method to train the fair predictive models. The new approach is guaranteed to achieve more accurate estimate and is easier to optimize than the one from (Nabi and Shpitser 2018).

1. The paper is very well written and easy to understand. I did not check the proof in details, but as far as I am aware, the proofs are correct and technically sound;

2. The simulations are solid and very comprehensive. It would be nicer if the authors can provide a easier to use computational toolbox from their proposed method. This can create a broader impact of the proposed work;

3. The authors can consider citing the other results aiming for fairness in computer assisted decision making, such as the one from
https://arxiv.org/abs/2110.15310 and the references therein. In particular, it would also be interesting to compare the criteria of fairness used in the two papers.

Overall, as I am not an expert of fairness, and I did not keep a close track of the most recent publications in fairness, my comments might be biased. Still I believe this is a good paper!
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: In this work, based on Nabi and Shpitser (2018), the authors introduce new theoretical results and optimization techniques to make model training easier and more accurate. Experiments on simulations show the effectiveness of this method.
Main review: The topic of this work is significant. How to impose hard fairness constraints on predictive models and optimize them efficiently is a bottleneck in fairness learning. Below are some concerns: 

Concerns:
1.	The problem this work want to address is not clearly defined. In the second paragraph of Section 3, the authors aim to predict the outcome labels for new instances, subject to fairness constraints. Are new instances the test data? Then, the authors train on the combination of historical data and new instances. Does this mean the test data is used in the training phase?
2.	The motivation of this work is confusing. What is the distribution $p^{*}(Y, Z)$ of the fair world? What is the distribution $p(Y, Z)$ of the unfair world? Z is the features. Why would Z change in the two worlds?
3.	What is the definition of $g(p(Y, Z))$? Why in practice, it should be strictly bounded as 0? What is the variables that appear in the estimator $\widehat{g}(\mathcal{D})$? Doesn’t $\mathcal{D}$ contain the whole Z (Z1+Z2)?
4.	In your Theorem, the observed data distribution $p(Y, Z)$ is induced by a causal model. Is there any requirements for the causal model? Has it to be in the form of Figure 1?
5.	This paper is not well organized, e.g., the Fig 1 is referred to in page 2 but located in page 5. The logic and connection among those sections is ambiguous.
6.	Some grammatical issues, e.g., `… pathways are … context-specific, hence require input ...’ and typos, e.g., $R_i$ in the formula at the bottom of page 5 is out of the bracket, should be fixed.
7.	How will this method perform on real-world dataset?
review_title: null

Meta-review: The paper proposes an approach to facilitate the learning of predictive models satisfying certain (path specific) fairness constraints, by introducing a reparameterization of the likelihood to reduce the computational complexity of the associated optimisation problem, in combination with empirical likelihood techniques to improve the predictive performance. Key theorem 2 behind the method is also relevant to mediation analysis in general.

All reviewers were clearly in favour of acceptance, albeit with moderate confidence. The latter is likely primarily due to the technically and conceptually quite challenging ideas behind the notion of path-specific fairness and the resulting optimisation problem to learn the target fair prediction model. In that sense the main aim of the paper - reducing the complexity of tackling this problem in practice - is a strong argument for its relevance. Whether or not it is also sufficient to bring ‘path-specific fair prediction models’ to the masses is doubtful, as it still remains a hard problem, but it is definitely a step in the right direction. Nevertheless it is a pity that the apparent intuitive simplicity of ‘fairness’ does not lead to an equally transparent solution that would reassure non-experts as well.

Pros:
- tackles important problem in a highly topical subject
- good technical quality 
- non-trivial results backed up by experimental support
Cons:
- clarity remains the main issue: despite attempts by the authors the paper will still be hard to access for most researchers not familiar with the subject of path-specific fairness (although the ideas should translate to other notions of fairness as well)
- experimental results are limited and hard to generalise 

On the whole the benefits clearly outweigh the drawbacks, and therefore I recommend accept, with slightly lowered confidence to reflect the lingering uncertainty among the reviewers. Given the amount of general interest in fairness and related aspects I also recommend an oral presentation to give the authors an opportunity to present their work in a more easily accessible format.

Decision: Accept (Oral)