Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Authors: Haoxuan Li
Zeyu Tang
Zhichao Jiang
Zhuangyan Fang
Yue Liu
Zhi Geng
Kun Zhang
Venue: ICML 2025 poster (2025)
Keywords: Counterfactual fairness
Principal strata
Primary area: social_aspects->fairness

Abstract: Fairness in human and algorithmic decision-making is crucial in areas such as criminal justice, education, and social welfare. Recently, counterfactual fairness has drawn increasing research interest, suggesting that decision-making for individuals should remain the same when intervening with different values on protected attributes. Nevertheless, the question of "which attributes and individuals should be protected" is rarely discussed in the existing counterfactual fairness literature. For example, when considering leg disability as a protected attribute, the algorithms should not treat individuals with leg disabilities differently in college admissions, but one may naturally consider this factor when selecting runner athletes. In other words, when and how to enforce fairness is expected to depend on the causal relation between the protected attribute and the outcome of interest. Formally, this paper proposes principal counterfactual fairness using the concept of principal stratification from the causal inference literature, focusing on whether an algorithm is counterfactually fair for individuals whose protected attribute has no individual causal effect on the outcome of interest. To examine whether an algorithm satisfies principal counterfactual fairness, we derive the statistical bounds and propose a post-processing approach to achieving principal counterfactual fairness with minimal individual decision changes. Experiments are conducted using synthetic and real-world datasets to verify the effectiveness of our methods.

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Haoxuan Li Zeyu Tang? Zhichao Jiang? Zhuangyan Fang Yue Liu Zhi Geng Kun Zhang 27
Abstract
1. Introduction
Addressing the fairness of automated algorithms is critical to making safe decisions in areas such as criminal justice (Bren- nan al., 2009; Dieterich et al., 2016), education (Reardon & Owens, 2014), and social welfare (Chouldechova et al., 2018). To achieve fair machine learning, many association- based fairness notions have been proposed to constrain the statistical independence between protected attributes and de- cisions, e.g., statistical parity (Dwork et al., 2012), equalized odds (Hardt et al., 2016), and predictive parity (Choulde- chova, 2017). In addition, the algorithmic fairness can also be approached from a causal perspective (Kusner al., 2017; Zhang et al., 201 201 8a:;; Zhang & Bareinboim, 2018; Nabi & Shpitser, 2018; Wu et al., 20;9;;;; Chiappa,
Fairness in human and algorithmic decision- making is crucial in areas such as criminal justice, education, and social welfare. Recently, coun- terfactual fairness has drawn increasing research interest, suggesting that decision-making for indi- viduals should remain the same when intervening with different values protected attributes. Nev- ertheless, the question of "which attributes and in- dividuals should be protected" is rarely discussed in the existing counterfactual fairness literature. For example, when considering leg disability as protected attribute, the algorithms should not treat individuals with leg disabilities differently in college admissions, but one may naturally con- sider this factor when selecting runner athletes. In other words, when and how to enforce fair- ness is expected to depend on the causal relation between the protected attribute and the outcome of interest. Formally, this paper proposes princi- pal counterfactual fairness using the concept of principal stratification from the causal inference literature, focusing on whether an algorithm is counterfactually fair for individuals whose pro- tected attribute has no individual causal effect on the outcome of interest. To examine whether an algorithm satisfies principal counterfactual fair- ness, we derive the statistical bounds and propose post-processing approach achieving principal counterfactual fairness with minimal individual decision changes. Experiments are conducted us- ing synthetic and real-world datasets to verify the effectiivnees of our methods.
2019; Imai & Jiang, 2020; Mishler et al., 2021; Zuo et al.,
2022). Among them, counterfactual fairness (Kusner et al.,
2017) has garnered considerable attention recently. This criterion demands that any alterations made the values of protected attributes do not result in changes to individual decision-making
Nevertheless, as Chouldechova & Roth (2020) pointed out, the question of "which attributes and individuals should be protected" is rarely discussed in the existing counterfac- tual fairness literature. Should counterfactual fairness hold for all sensitive attributes on all individuals? For example, when considering leg disability reasonable require that the algorithms should not treat in- dividuals with disabilities differently in college admissions, but should the algorithms also be required to make the same decisions for individuals with disabilities when selecting runner athletes? In such cases, clear that it is not ap- propriate to select individuals with disabilities as running athletes. But what if the sensitive attribute is gender or race instead? How to reflect the differences between disability and gender as sensitive attributes?
protected attribute, it is
To tackle the above issues, we summarize relevant studies in Table 1, which can broadly divided into two branches. On one hand, instead of requiring fairness hold on all individuals as in demographic parity (Darlington, 1971), equalized odds (Hardt al., 2016) constrains the exami- nation of demographic parity to subgroups with the same observed outcome. By considering the effect of decision- making on the observed outcomes, counterfactual equalized
'Peking University 'Carnegie Mellon University "Sun Yat- sen University "Xiaomi 'Remmin University of China Beijjng Technology and Business University 'Mohamed bin Zayed Uni- versity of Artificial Intelligence. Correspondence to: Yue Liu cliuyue_stats @ rer.ccccco www.erreeuc.ooo.eo edu.en>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Table A summary of the proposed principal counterfactual fairness and related concepts.
Fairness Definition
Formulation (A: protected attribule: D: decision: Y: outcome; X: covariate)
Demographic Parity (Darlington, 1971)
A IL
Equalized Odds (Hardt et al., 2016)
A IL D
Y
Equality of Opportunily (Hardi el al., 2016)
A IL D
Y
Counterfactual Equalized Odds (Mishler et al., 2021)
A D =(D=0)= =1
Principal Fairness (Imai& & Jiang. 2020)
ALD (LDD(Y(D=0))(D)
Counterfactual Parity (Mitchell et al., 2021)
P(D(0) 1) =P(D(1) 1))
Conditional Counterfactual Fairness (Mitchell et al.. 2021)
P(D(0) = 1) = =[([(1) -00)=110 =0)==1]]] 1) |X
Principal Counterfactual Parity (ours)
P(D(0) 1) ((((()) = 1) Y(A = 0) Y(A=
Principal Conditional Counterfactual Fairness (ours)
P(D(0) = 1) = P(D(1) = 1) YY(A- = 0) = Y(A= =))
Principal Counterfactual Equalized Odds (ours)
P(D(O) 1) = P(D() = 1) (A = 0) Y(A = 1)= y,
Counterfactual Fairness (Kusner al., 2017)
Di(A, 0) D1(A 1)
Path-Specific Counterfactual Fairness (Chiappa, 2019)
D,(A, = 0) = D,(A, = 1, M,(A.) = M.
Principal Counterfactual Fairness (ours)
D,(A, = 0) D, = 1) holds for Y1(A, = 0) = Y,(4 = 1)
odds (Mishler et al., 2021) generalizes the above concepts to make fair decisions on individuals with the same value the potential outcome under control. Principal fairness (Imai & Jiang, 2020) further uses the concept of principal strati- fication from the causal inference literature to consider the joint potential outcomes of the decision on outcome, How- ever, despite considering specific subgroups defined from counterfactual view, these faimess notions still use the statistical independence of sensitive attributes and decision- making that subgroup, which is not sufficient to guaran- tee individual counterfactual fairness (Kusner et al., 2017: Mitchell et al., 2021). On the other hand, path-specific counterfactual fairness (Chiappa, 2019) promotes the no- tion of counterfactual fairness to restricted on unfair paths, rather than considering the total effect of sensitive attributes on decision- decision-making. Despite partially answering the ques- tion of "which attributes should be protected", similar to counterfactual fairness, path-specific counterfactual fairness requires fairness on all individuals. This motivates us to rethink counterfactual fairness to better answer that "which attributes and individuals should be protected".
dividuals satisfy counterfactual fairness. In summary, the proposed principal counterfactual fairness further considers the effect of protected attributes on outcomes of interest from a counterfactual perspective, and we show the prin- cipal counterfactual fairness would degenerate to standard counterfactual fairness when the protected attributes have no individual causal effect on outcomes for all individuals.
To examine whether an algorithm satisfies principal coun- terfactual fairness, we first derive the necessary conditions! for an algorithm to satisfy principal counterfactual fair- ness based on statistical bounds. Then we propose an optimization-basee evaluation method to test whether an algorithm satisfies principal counterfactual fairness. Specifi- cally, the algorithm does not satisfy the principal counter- factual faimesss if the feasible region under partieular con- straints is the empty set, or if there exists a principal stratum with the optimized maximum probability value less than zero. We further propose principled post-processing ap- proach to achieve principal counterfactual fairness with min- imal individual decision changes, and theoretically prove the optimality of the post-processing approach using doubly robust estimation. We conduct extensive experiments on synthetic and real-world datasets to verify the effectiveness of the proposed algorithm.
In this paper, instead of forcing decisions to remain the same for all individuals when the protected attribute changes in counterfactual fairness, we propose principal counter- factual fairmess using the concept of principal stratification from the causal inference literature (Frangakis & Rubin, 2002; Pearl, 2011), focusing on whether the counterfac- tual fairness holds for individuals whose protected attribute has no individual causal effect on the outcome of interest. For the aforementioned example, since leg disability (as sensitive attribute) may affect athlete performance (as outcome), we only require that decisions remain similar for those individuals with disabilities that do not affect athlete performance. In contrast, since disability and gender (as sen- sitive attributes) do not have a causal effect on the exam pass (as an outcome), we might expect decision Decision-making for all in-
The main contributions of this paper are:
We propose novel fairnsss notion using the concept of principal stratification, called principal counterfactual fairness, which requires the counterfactual fairness to hold only when the protected attribute has no individual causal effect on the outcome of interest.
We derive the necessary conditions for an algorithm
'Due to partial identifiability, it is difficult to find necessary and sufficient conditions for principal counterfactual fairesss, similar problems also exist in the counterfactual faimess literature (Kusner et al., 2017).
2

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
to satisfy principal counterfactual faimess based on statistical bounds, and propose an optimization-baset evaluation method to test whether an algorithm satisfies principal counterfactual fairness.
statistical fairness notions have been proposed, such as de- mographic parity (Darlington, 1971), i.e., A IL. D, equal- ized odds (Hardt et al., 2016), i.e., A IL D Y, and equality of opportumity (Hardt et al., 2016), i.e., A TL D Y = 1. By noting the causal effect of decision D on the observed outcomes Y, counterfactual equalized odds generalizes the above concepts to make fair decisions on individuals with counterfactual advantaged outcomes (Mishler et al., 2021), i.e., A T1 D Y 0) 1. Principal fairesss fur- ther uses the concept of principal stratification from the causal inference literature to consider the joint potential out- come of decisions on outcomes (Imai & Jiang, 2020), i.c., A TL D | (Y = 0), YD = 1)). Nevertheless, despite considering a specific counterfactual stratum, these fairness notions still use the statistical independence of sensitive at- tributes and decisions on that stratum, which is not sufficient to guarantee causal effect-basee fairness notions (Kusner et al., 2017; Mitchell et al., 2021).
We further propose principled post-processing ap- proach to achieving principal counterfactual faireess with minimal individual decision changes, and theo- retically prove the optimality of the post-processing approach using doubly robust estimation.
We conduct experiments on both synthetic and real- world datasets to verify the effectiveness of the proposed optimization-base evaluation and post- processing approach.
2. Preliminaries
We first formalize the issue of fairness in decision making, as well as summarize the related statistical and counterfac- tual fairess notions that have been widely studied. Suppose a simple random sample of n units from a super population P. for each unit i, the covariate (e.g., age or income) and the binary protected attribute (e.g., gender or disability) are denoted as X, E X' and A, E {0, 1}, respectively. Let Y; E y = {0, be the binary outcome variable of interest and D; {0, 1} be the binary decision variable. For the sim- plicity of exposition, we assume the protected attribute, de- cision variable, and outcome variable are all binary, and co- variates are discrete, but these variables can all be extended to other variable types in our work. To study the counterfac- tual fairness problem, we adopt the potential outcome frame- work (Rubin, 1974; Neyman, 1 1990). Specifically, let (0))) and Y1)1) be the outcome of the unit i had this unit have the protected attribute A,- 0 and A, 1, respectively. Since each unit can only have one particular value of protected attribute, we always observe the corresponding outcome be either (0) or (1(),, but not both. This is also known as the fundamental problem of causal inference (Holland, 1986; Morgan & Winship, 2015). Formally, the observed outcome for unit is Y other words, the observed outcome is the potential outcome corresponding to the protected attribute value, which is also known as the consistency assumption in the causal inference literature (Hernán& & Robins, 2020).
Instead of considering the statistical (conditional) indepen- dence between the protected attribute A and decision D, causality-basee fairness considers the causal effect of the protected attribute A on decision D. Among them, cown- terfactual parity in Definition requires that there is no average causal effect of the protected attribute A on deci- sion D over the population (Mitchell et al., 2021).
Definition 1 (Counterfactual parity (Mitchell et al.. 2021)). An algorithm D for decision decision-making satisfies counterfactual parity, if under any value o and a' attainable by A,
P((((a) = P(D('' =
By incorporating the covariate X, conditional counterfac- tual fairness in Definition 2 requires that there is no condi- tional average causal effect of the protected attribute A on the decision D over subpopulations under context X x for all X.
Definition 2 (Conditional counterfactual fairness (Mitchell et al., 2021)), An algorithm D for decision--mking con- ditional counterfactually fair, if under any context X and any value and a' atlainable by A.
(,) + A,Y, In
(P(((a) = x=))= = 1 x))
Different from counterfactual parity in Definition I which constrains on the total population and conditional coun- terfactual fairness in Definition 2 which constrains on the subpopulations determined by the covariates, individual counterfuctual fairness in Definition further requires that there is no individual causal effect of the protected attribute A on the decision D over all the individuals.
Based on the observed protected attributes, covariates, and outcomes of interest, i.e., (A,,X, Y a machine learning algorithm (-) for decision-making is obtained. Specifically, let (0) and (1) be the potential algorith- mie decisions for the unit had this unit have the protected attribute A, = 0 and A, = 1, respectively. By the con- sistency assumption again, the algorithmic decision for in- dividual in the factual world would be D,. In order for algorithms to make fair decisions, as shown in Table 1, many
Definition 3 (Counterfactual fairness (Kusner et al., 2017)). An algorithm D) for Doecision-making is individual counter-
3

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
factually fair, if under any context X x: and any value a and a' attainable by A,
stratum (in blue). Unlike the observed outcome Y,, however, the potential outcomes, and hence principal strata, not affected by the sensitive attribute value. Moreover, since we only observe one potential outcome for any individual, prin- cipal strata are not directly observable and be distinguished, as shown in Table 2.
= P(Di(a)=
1.
Counterfactual fairness states that A should not be cause of decision D in any individual instance, with many follow- up studies (Zhang& & Bareinboim, 2018; Chiappa, 2019). As in Table 1, one representative variant is terfactual fairness, which requires counterfactual fairesss to hold only on unfair paths (Chiappa, 2019).
In the disabled athlete selection example, the principal strata are defined by the athlete performance (A4) under each of the two scenarios disabled A, 1 not disabled A; = 0. Then it is fair to let the algorithmic decision D be unaffected by whether those individuals are disabled A = 1 or not A = 0, because the disability of those individuals has no individual causal effect on the athlete's performance, i.e., (0()) (1))). The following Definition formally states the principal counterfactual parity, which requires counterfactual parity to hold on that particular stratum.
coun-
Despite partially answering the question of "which attributes should be protected", similar to counterfactual fairness, path- specific counterfactual fairness also requires fairness on all individuals. This motivates us to rethink these counterfac- tual faimess notions to better answer the question of "which and how to decide the attributes and individuals that should be protected".
Definition 4 (Principal counterfactual parity). An algorithm D for decision decision-making satisfies principal counterfactual parity, if under any value o and a' attainable by A,
3. Principal Counterfactual Fairness
P(D(a)) = ( = Y(d)) = P(((dd) = ((a) = Yad)))
In this section, we first propose the notions of principal counterfactual fairness using the concept of principal strati- fication from the causal inference literature. Ordered from weakest to strongest, we propose principal counterfactual parity in Definition 4, principal conditional counterfactual fairness in Definition 5, principal counterfactual equalized odds in Definition 6, and principal conditional counterfac- tual fairness in Definition 7, respectively. We also derive the necessary conditions for an algorithmic decision to satisfy principal counterfactual fairness based on statistical bounds.
By conditional on covariate X, Definition 5 states the prin- cipal conditional counterfactual fairnsss.
Definition 5 (Principal conditional counterfactual fairness).
An algorithm D for decision decision-makinng is principal conditional counterfactually fair, if under any context X = =z and any value and a' attainable by A,
P(D(a) - 1 ((o) - Y(d'). X - -P(D()) - 1 ( = Y(d',. =x).
Specifically, the principal strata are defined as the joint potential outcome values (Frangakis & Rubin, 2002), i.e., (a)()), (aa)), where and a' are sensitive attribute values attainable by A, and each principal stratum represents how an individual would be affected by the protected attribute on the outcome of interest. In the proposed principal counter- factual fairness, we focus on whether counterfactual fairness notions hold on individuals whose protected attribute has no individual causal effect on the outcome of interest, i.e., (a(a)= Ya(a)) for all a and a' attainable by A.
We now descrihe the potential limitations of using principal conditional counterlactual fairness in Definition 5. Recall the disabled athlete selection example and let Y denote athlete performance. Although the protected attribute has no individual causal effect on the outcome for both ((a)= = Y (a') = 0 and Y(a) = Y (a') = 1 individuals. However, since individuals at Y()) = Y (a') = 1 have better athlete performance compared with individuals at ((a) Y(a')
O, it is natural to allow high probability of being selected as an athlete for individuals at the stratum Y(a) = Y(a)) = 1.
That is, P (()( =1 (a)) = =Yaa') = 1) > P(D(a) =
Compared with the previous counterfactual fairess notions, Table 2 shows the difference: the proposed principal coun- terfactual fairness notions focus only on those individu- als in "principal fairness" " stratum (in red), while previous counterfactual fairness notions focus on individuals in both "principal faimess" stratum (in red) and "auxiliary fairesss"
(a) = Y(a') = 0). This motivates us to further divide stratum (Y = (()) into multiple strata (Y( ( = 3) for all y E 3, and propose the corresponding principal counterfactual equalized odds in Definition 6.
Detinition 6 (Principal counterfactual equalized odds). An algorithm D for dacisiionmmkinng satisfies principal coun- terfactual equalized odds, if under any context X = x and any value a and a' attainable by A, for all y E y,
*Counterfactual fairness in (Kusner et al.. 2017) refers to individual counterfactual fairness with the definition that P (DD
= y X x, == P (U) = y X = v, A = a). This is equivalent to ((D,(a) Da(a')) = using potential outcomes formula-
P(D(a)= = Y ((a)) = v. = x)
tion (Mitchell et al., 2021).

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Table The principal counterfactual fairness considers units in the principal fairness strata (in red), whereas counterfactual faimess
considers all units including in the auxiliary fairness strata (in blue).
Observed data
(4 0,Y 0) (A O. 1) (4 (A- 1,-1,--0 0)
1,-1YY- 1)
Principal faimess
(Y(0) Y() = 0) (Y(0) 1,Y = (Y10) = =0 0
1)
Auxiliary fairness
(Y(0) - (1) 1) (Y(0)- - LY))- (Y(0) - -. - (Y - - 0,
For the case of binary variables, it is equivalent to ro(x))
Similarly, the principal counterfactual equalized odds in Definition under stratum Y (0) = (1) (1)==1 is violated if either of the following two inequalities holds:
Ti (3r) = O, where
(2) (P)(1)) = 1 ( = (() = y,
P(D(0)- = ((0) = ((1) = y, x= =x),
4o1 (2) (x) (1 - 4n (1) (1 201 (3)) )ol (x) < Poi(z)) P1o(2))
< Poi Pio(z),
for y = 0, Denote (zy(2) = =
z) and Qay(x) = P(D = | a y. y. x). which can be calculated from the observed data. Under the ignorability assumption, the following lemma provides the sharp bounds on (x) and T1 (z).
Further, instead focusing on specific subgroup, an exten- sion of individual counterfactual fairesss in (Kusner et al., 2017), we define principal counterfactual fairness to achieve strict individual fair.
Assumption 1 (Ignorability).
A IL (Y(1), Y(O), D(1), D(O)) X.
Lemma 1. Under Assumption I, the sharp upper and lower bounds on TZ) are
Definition (Principal counterfactual fairness). An algo- rithm D for decision decision-making principal counterfactually fair with respect to outcome Y, if under any value and a! attainable by A.
0, (1- (10(()) (no(())pPul())) Pro(z))
} } P1o(r) Pon (x)
Lower(ro(<)) = max
1, 900(<)) (x)
min
(1)(2)) poi (x:)
P(D,(o)= = Di(a') (V(a) = (((())={(())) =1
Q10(1 (210 (x)
Upper(ro(x)) = min
(1) - Pol (x:)
(1 900 (2) Puo (oo (2)
We finally point out that the principal counterfactual fair- ness would degenerate to counterfactual fairness when Yi(a) (aaa)) holds on all individuals for any value and a' attainable by A.
min
0,
Pio(z) pot (x)
The sharp upper and lower bounds on Ti (x) are
0, (1 - 911 (z) (2))())))(()) (p1(()) (r) P1o(z)
Lower(n) (x)) = =max
Corollary 1 (Relation to counterfactual faimess)). The prin- cipal counterfactual fairness is equivalent to counterfactual fairness in (Kusner et al., 2017), when the protected at- tributes have no individual causal effect on outcomes for all individuals.
1, goi (2))p11 (x)
min
(z) - Pio(x))
1, 911 (I))11 (x)
Upper(n((x)) = -min
Pui (z) - Pio (2r)
(1 901 (x) poi (x)
4. Implementing Principal Counterfactual Fairness
min
0,
Pon (z) (1)
The Theorem below gives necessary inequality conditions show if the algorithm satisfies principal counterfactual equalized odds based on statistical bounds in Lemma 1.
4.1. Optimization- Evaluation
We started with an optimization-based evaluation method for principal counterfactual fairness in Definition 7, while other principal counterfactual fairness notions can be eval- uated by similar arguments. Denote Wdg.d1 vo- (x) P(D(O) do, D(1) d,, Y()) you Y(1) y1 X 2). then principal counterfactual fairness is equivalent to Wo, (z) = (1,000.0(z) = (z) = (1)
Theorem 1. Under Assumption 1, the principal coun- terfactual equalized odds in Definition 6 under stratum (0) = (1)) = 0 is violated if either of the following two inequalities holds:
(00(z)poo(z) + (1 (11(()))P1o0() Pro(z) Poi
0. The proposed optimization constraints for evaluating
(10(z)p1o(z) + (1 -2oo(x)) (x) P1o(x) poi (x).
whether the algorithmic decisions satisfy the principal coun-
5

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
terfactual fairness are given as
and the doubly robust (DR) estimator is given as
= d, (a) | = = B (X)+ I(A=a) a) (II) d, =y) - -x)x (xa) x x}
(X)+
wo.1..0()) = (1.0..00.))) = (2) = (1,0,1,1(x) = 0,
v00.01 (1) 20 0 for all dg, d1, No. 1/1 E {0,1),
Ead a.b Wdo, (x)) (0) = do, (0) yo | X x)
Theorem 2. Suppose that | - Ta [T( 22 1 (x) p ( 0r on ) (1 for all x E X and attainable by A,
for all do yo E 10,
then PD (D()) = (a) = y X = x) is asymptotically
E., a. Wa.dibm (x) = P(D(1)) - y = 91 X = x)
normal
for all d1.31 E 1),
vallPR( d. d (a) - y x
-P(D(a) = dd)(a) - X = =))} - o( (1)3),
for all x X, where the first equation is the equivalent condition of principal counterfactual fairesss, the second equation comes from the positivity of the probabilities, and the last two equations come from the definition of w(r). Notably, under Assumption I, the terms on the right-hand side of the last two equations can be identified and esti- mated using the observed data (see Section 4.2). Therefore, with these constraints on w(x) imposed by the above equa- tions set, we can determine that the algorithmic decision does not satisfy principal counterfactual fairness, if there exists x E X' such that the feasible domain of w(xx satisfy- ing these constraints is the empty set. In practice, we can also take one of (x)),, 4h, (z))), 1,, (Ir), and 11.0,1.1 (z), denoted as ((r). as the objective function, and let the remaining three terms equal to zero as the optimiza- tion constraints, then solve the minimum and maximum values of (z)), and obtain its value interval by solving this optimization problem. The algorithm should be considered as a violation of principal counterfactual fairness, when the minimum value of ((x) is greater than 0 or the maximum value of (xx) is less than 0.
where 01 (z))
v [x) I(A a) (I(D d,Y y) P2 -x)(
X
(x)
4.3. Post-processing approach
In applications, we first use the DR (or OR, IPS) estimators in Section 4.2 and plug them into the last two constraints of the optimization problem in Section 4.1. As discussed in Section 4.1, if the feasible domain is the empty set, or if there exists x E X' such that the interval of values of (ιx) does not contain 0, then the decision should be consid- ered to violate principal counterfactual faimesss Inspired by (Mishler et al., 2021), we further propose a post-processing method to adjust previously unfair decisions by minimal individual decision changes so that it no longer violates optimization- -aseed fairness evaluation in Section 4.1. The advantage of the post-pree post-processing approach is the applicabil- ity to any already-in-used models but evaluated unfair (Lo- hia et al., 2019).
4.2. Estimation
Specifically, consider a set of non-negative parameters e(x) = f=)o <01 (I)), e10(x), (x)( for all x E X. where each parameter <aa(x) denotes the probability of forcing the decision D = on the individuals with A a and X = I. Without loss of generality, let (ad()) fa(1-d) (I) I 1, and let D' be the final decision after the post-processing, then have
Let (x) = = - a x) and Ta(z) =
=
a X= = z). with (x) and (x) be the estimated conditional-mean and propensity, respectively. To estimate the right-hand side of the last two equations in the above optimization problem, without loss of generality, one needs to estimate P(D(a) = d, d,)(a) = y X x)) for a, d, y E 10. Let P and E. be the estimated probabil- ity and expectation that can be obtained via regression or subclassification. The outcome regression (OR) estimator is
PPA ==)
(a(D'(a)= (,))) X
= P.(D' = d, = a x)
= Ead(z) -
a. = 2)+
foR ( = X = x) =PPD = Y = 4 = a, X = x) p (x).
(1 - <ao(s) - PPD]
y A= = a, X = x)
(P)(x))P(((()) P(Y(a)=y| xX
x (1 (x) (au(=))-P((D)) = d, (a)) = X = z)
The inverse propensity scoring (IPS) estimator is given as
= ((D(() = -1_,y((a) = X = x)+
(1 - (1-cx(-a((t)) P(D(a) = d, (a)= - X= = x)
()r((D(a)) = d, y x = x) -
In order to obtain a fair decision D' while minimally chang- ing the original decision D, we obtain the estimated ((x: for
-ė - -I(D d.Y = =)) = =]
6

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Table Synthetic experiment results for varying models and estimators. The intervals under W0100s W10000 W0111 and N1011 show the
minimum and maximum values when the other three are set to O.
Method
M0100
600
100000
610
W0111
601
W1011
611
CF1
PCF f
LR+ OR
[-0.636, -0.083]
0
[-0.774, -0.083]
[0.083, 0.223]
0
[-0.672, -0.083)
0.109
+3.60%
+6.68%
LR+ IPS
[0.153, 0.231]
0
[-0.772, -0.153]
1-0.666, -0.153]
0.166
[-0.715, -0.153]
0
+4.71%
+6.89%
LR+ DR
[-0.631, -0.024]
0
[-0.710, -0.024]
[0.024, 0.196]
0
[-0.683, -0.024]
0.127
+2.97%
+6.09%
SVM+ + OR
[0.104, 0.260]
0
[-0.741, 10.10
0.124
[-0.743, -0.104]
0
[-0.619,
+1.47%
+5.15%
SVM + IPS
[0.192, 0.227]
0
[-0.720, 10.192]
0
[-0.739,
-0.192]
0.199
[-0.732, -0.192]
0
+4.56%
+8.98%
SVM+ + DR
[-0.662, -0.034]
0
[-0.763, -0.0344
[0.034, 0.182]
0.186
10.607,00.034
+5.77% +8.30%
RF OR
[-0.706, -0.110]
0.120
[-0.713, -0.1
1-0.689, -0.110]
0
[0.110, 0.195]
+2.18%
+6.67%
RF + IPS
[0.163, 0.211]
0
[-0.727, -0.163]
[-0.680, -0.163]
0.171
[-0.755, -0.163]
0
+6.42%
+8.35%
RF+ DR
[-0.713, -0.051]
0
--0.57
[0.051, 0.253]
0.203
[-0.661,
+6.49%
+9.02%
NB + OR
[-0.674, -0.161]
0
[-0.744, 10.161
[0.161, 0.232]
0.173
[-0.742, -0.16]]
0
+5.04%
48.31%
NB + IPS
[-0.821, -0.1751
0
10.175, 0.19
1-0.775, -0.175]
0
[-0.577, -0.175]
0.181
+8.88%
+9.31% %
NB + DR
[-0.793, -0.1891
0
[-0.688, -0.1891
[-0.707, -0.1891
0
[0.189, 0.2081
0.192
+4.83%
+9.16%
all x E X by solving the following optimization problem.
by considering the following programming problem
a = arg min 1
{ 1, + w (X,,))
1
E' = arg
arg min
= Σ
(X,) + (Xi,
n
+w, 1, (X,) w i,
1.1 (X.))
s.t. 10. = (1))0000))) (x) (1) 00
s.t. d1..d1.80.31 (z) 2 for all do, d1, yo, 81 E {0.1),
20 (x) 2 0 for all do. d1 y/, /n E (0,1),
Σ a. ado. (0) do. Y(0) yo X = x)
(ad(z))a(1-a)) a1 (x) 2 0 and (x)( + Ea(1 (2) <
for all do y0 E (0, 1}.
for all a, {0, 1},
Σ ba (z) = Pe (1) = d1. Y(1) = y1 x
E at (do,a..o. (x) (D) do, (())= yo X =)) x)
for all d1, E 10,
for all do, yo E {0, 1}. E.p Wa,di (x) P. (D) d1, (() y1 X x)
for all X, where Pa(D Y(a) Xx=)) is the joint distribution of the potential outcomes (D'(a), Y(a)) for the post-processed decision function D' using é(x) obtained in Section 4.3. We use the mean sum of squares as the metric for evaluating principal counterfactual fairness. Theorem proves the consistency results of a* to O, validating the effectiveness of the post-processing approach.
for all d1 E {0. 1}.
for all x E X. In practice, we use the DR estimate in Section 4.2 and then plug it into the last two constraints to obtain the DR estimates for P. (D(((()) = d, Y (a) = y X x), and estimate ((x) for all æ E X. Theorem proves the consistency of the estimated ((x) to the optimal cxx)).
Theorem 4. Suppose that [1 (=(=))12-11 ng
for all x E X and attainable by A, then ||= Op(1/Vn).
Theorem 3. Suppose that (x) - Ja [=(x))ll=llldd ue. (x)]l=====>) on for all a E X and a attainable by
5. Empirical Investigation
To verify the effectiveness of the post-processing approach, we conduct experiments on both synthetic and real-world datasets. The performance is evaluated by two metrics: counterfactual faimess (CF): P(D(O) = D(1)) and prin- cipal counterfactual faimess (PCF): P(D(O) = D(1) | (0) = (())). For all experiments, we calculate the val- ues of CF and PCF before and after the post-processing operation and report the percentage change of each metric.
A, then [|ā(x) (x)]==
4.4. Theoretical Analysis
After obtaining the optimization solution ê(x) in Section 4.3, the adjusted decision function D' is determined by combining original decision D and ē(z). Next, we prove the adjusted decision D' tends to be fair as sample size n + oc
7

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
5.1. Synthetic Experiment
Table 4 shows the real-world experiment results. When solv- ing the optimization problem in Section 4.1 on the whole population, we find that the interval of all four w covers O, i.e., the current algorithm already satisfies the principal counterfactual fairness. Therefore, post-processing is un- necessary at this point, so there is no corresponding change in CF and PCF, When using X, to divide the population, the optimization equation for the subgroup of X1 2 120 has no solution, and when grouping according X2, the optimization equation for the subgroup of X2 > 0 has no solution. When we divide the whole population into four subgroups, we find that the optimization equations for three of subgroups are unsolvable. Compared to the case of two subgroups, for the unsolvable subgroups, the distance be- tween zero and the interval of the four w and the value of the non-zero (ar( is significantly larger when the popula- tion is divided into four subgroups. This indicates that the constraint is violated to a stronger extent when the popu- lation is divided into more subgroups. Meanwhile, for the case of four subgroups, the CF and PCF of each unsolvable subgroup change more due to the larger Ead(z)) In addition, for solvable subgroups, the interval of Wo100 and W0111 are very close to exclude 0 when the population is divided into four subgroups. which further indicates that as the number of population group increases, each subgroup becomes more difficult to satisfy the optimization equation. Finally, the growth of PCF is larger than that of CF for all unsolvable subgroups, which shows that our approach is more effective on the population with ( = -11).
Synthetic data are generated from a structural equation model based on a random DAG with 10 nodes and 40 di- rected edges according to the Erdös-Rényi (ER) model, where four different models: Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF) and Naive Bayes (NB) respectively to obtain the estimations of P(Y = 1 A = a, X = z) as the decisions ((a, x). Specifically, following the previous study (Zuo et al., 2022), the path coefficients By* of directed edges X, + Xx are sampled from Uniform([-22 2, -0.5] U [0.5, 2)) distribution (see Appendix C for more details). Then we check whether the optimization equation in Section 4.1 is solvable. If the feasible domain is empty, we further use the post-processing method in Section 4.3 to obtain (z).
Table 3 shows the synthetic experiment results. First, if the intervals of the four w do not contain O, implying that the optimization equation has no solution. At this point, we found that there will be one (1)( nonzero, while the other three (z) are O, which is may due to the linearity of the constraints. Second, after the flip based on the (z), there are positive changes in both PCF and CF, which shows the effectiveness of the post-processing method. In addition, the increase in PCF is more pronounced than CF for all models, which is because our approach focuses only on the population with Y (0) Y1)).
5.2.
Experiment
The STUDENTINFO file in the Open University Learning Analytics Dataset (OULAD) dataset (Kuzilek et al., 2017) is used for the real-world experiment. The data attributes include demographic information about the students such as gender, age, education level, disability, and other attributes as well as their final grades. This dataset contains 32,593 students and 11 attributes. We treat disability as the sensi- tive attribute and binarize the final grades as the outcome of interest. First we learn a CPDAG from the raw data us- ing the PC algorithm in the causal-learn package. We find studied_credits: the total number of credits for the mod- ules the student is currently studying (denoted as X1) and prm__e__ree_alllemps:: the number of how many times the student has attempted this module (denoted as X2) with an undirected edge between it and the disability. Therefore we sample four DAGs from the learned CPDAG corresponding to the four cases of no subgroup. X1 as subgroups, X₂ as subgroups, and both X, and X₂ as subgroups, respectively. For each DAG, we determine the path coefficients based on linear regression and treat the residual of the regression as noise. For each subgroup, the subsequent steps are the same as in the simulation experiments. The LR model is used to obtain the decision D and the DR estimator used to estimate (aa(a) = d, (a) = y X = z).
6. Conclusion
This paper studies the question of "which attributes and individuals should be protected" in the context of counter- factual fairness. Motivated by the example that disability serves as a sensitive attribute for different outcomes of inter- est (e.g., college admissions, athlete selections), we suggest that when and how to enforce fairness expected to depend whether the protected attribute has no individual causal effect on the outcome of interest. Formally, we propose principal counterfactual fairness, and theoretically derive the necessary conditions for an algorithm to satisfy principal counterfactual fairness based on statistical bounds. Based on this, we further propose principled post-processing approach to achieve principal counterfactual fairness with minimal individual decision changes. A limitation of this work is that the principal counterfactual fairness is partially identified, i.e., we cannot give unbiased point estimates from the data, but can give its statistical bounds and a falsification method. We leave it to future work about how to develop new identification and estimation strategies under practical assumptions. In addition, combining causal discovery to achieve decisions that satisfy the principal counterfactual fairesss also serves as interesting future topic.
8

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Table Real-world experiment results for different subgroups. The intervals under Wo100, W1000. W0111 and W1011 show the minimum and maximum values when the other three are O.
Subgroup
(X)
(X)
c10(X)
(X)
(X)
(X)
ci(())
|
CFf
PCF
None
[-0.716, 0.024]
0
[-0.747, 0.020]
0
[-0.457, 0.078]
0
[-0.078, 0.268]
0
X; 2 120
[-0.131, 0.274]
[-0.713, 0.059]
0
[-0.778, 0.093]
[-0.377, 0.131]
0
X; < 120
[-0.580, -0.030]
[0.030, 0.239)
[-0.747, -0.030)
0.040
[-0.702, -0.030]
0
+1.35%
+1.79%
X2 >
[0.039, 0.242]
0
[-0.675, -0.039]
0.049
[-0.658, -0.039]
[-0.705, -0.039]
0
+3.52%
+3.97%
X2 =
[-0.715. 0.007]
0
[-0.575, 0.007]
0
[-0.332. 0.092]
1-0.376, 0.213]
0
Xi 2 120, X2 >0
[0.173, 0.287]
0
[-0.754, -0.173]
[-0.715, -0.173)
0.196
[-0.704, -0.173]
0
+3.80%
+8.51%
X1 2 120, X2
[-0.749, 0.000]
0
[-0.444, 0.057]
0
[-0.748, 0.001]
[-0.057, 0.246]
0
X1 < 120, X2 >0
[0.176, 0.220]
0
[-0.738, -0.176]
0
[-0.731, 176]
0.184
[-0.706, -0.176]
0
+5.22%
+9.59%
X1 < 120, X2
[-0.822, 00.231]
0
[-0.666, -0.231]
0
1-0.742, -0.231]
0
[0.231. 0.304)
0.124
+3.10%
+5.02%
Note: In world experiments, X1 studied and X2 nam. www_pree_tttengaee
Acknowledgments
References
This work was supported in part by National Natural Sci- ence Foundation of China (62382002, (2201629). YL. was supported by the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJDI 10001). We would also like acknowledge the support from NSF Award No. 2229881, AI Institute for Societal Decision Making (AI- SDM), the National Institutes of Health (NIH) under Con- tract RO1 R01HL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program. ZT is supported by the National Institute of Justice (NIJ) Graduate Research Fellowship, Award No. 15PNIJ-24-GG-0156--
Brennan, T., Dieterich, W. and Ehret, Evaluating the pre- dictive validity of the compas risk and needs assessment system. Criminal Justice and behavior, 36(1):21-40 2009.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W. and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21:1-68, 2018.
Chiappa, S. Path-specific counterfactual faimess. In Thirty- Third AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 7801-7808, 2019.
Impact Statement
Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2): 2017.
This paper introduces a novel Principal Counterfactual Fair- ness framework that addresses key challenges in algorithmic faimess by incorporating the principal of causal inference. Unlike existing counterfactual fairness approaches, our ap- proach selectively protects individuals whose protected at- tributes (e.g., disability status) do not causally affect relevant outcomes (e.g.. academic performance). By utilizing prin- cipal stratification, we achieve context-aware fairness: the algorithm can adaptively apply fairness constraints based on whether the protected attribute causally affects the outcome in a given scenario (c.g.. a disability may receive fair consid- eration in athlete selection, but not in academic admissions). Our work provides feasible method, including statistical bounds and post-proccee proscssccssinng algorithms, to audit and enforce faimesss with minimal disruption to existing systems. This advances the use of Al in high-stakes areas such as criminal justice and social welfare, where nuanced causality-driven fairness is critical to avoid over- or under-protectio of dis- advantaged groups. The framework bridges causal inference and fairness research, providing a principled way to align algorithmic decision-make making with societal values.
Chouldechova, A. and Roth, A. A snapshot of the frontiers of fairness in machine learning. volume 63, pp. 82-89. ACM New York, NY, USA, 2020.
Chouldechova, A., Benavides- Denavides-Prado, D., Fialko, O., and Vaithianathan, R. A case study of algorithm- assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency, pp. 134-148. PMLR, 2018
Darlington, R. B. Another look at "cultural fairness" 1. Journal of educational measurement, 8(2):71-22, 1971.
Dieterich, W., Mendoza, C., and Brennan, T. Compas risk scales: Demonstrating accuracy equity and predictive parity. Northpointe Inc, 7(7.4):1, 2016.
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214-226. 2012.
Frangakis, C. and Rubin, D. B. Principal stratification in causal inference. Biometrics, 58(1):21-29 2002.
9

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Rubin, D. B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of edu- cational psychology, 66:688-701, 1974.
Hernán, M. A. and Robins, J. M. Causal Inference: What If. Chapman & Hall/CRC., 2020.
Shapiro, Asymptotic analysis of stochastic programs. Annals of Operations Research, 30:169-186, 1991.
Holland, P. W. Statistics and causal inference. Journal of the American Statistical Association, 81:945-960, 1986.
wager, S. and Athey, S. Estimation and inference of hetero- geneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):122-- 1242, 2018
Imai, K. and Jiang, Z. Principal fairness for hu- man and algorithmic decision-making. arXiv preprint 1rXiv:2005.10400 2020.
Wu, Y., Zhang, L., and Wu, X. Counterfactual faireess: Unidentification, bound and algorithm. In Proceedings of the Twenty-Eights International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 10-16, 2019a.
Kusner, M. J., Loftus, J., Russell, C., and Silva, R. Coun- terfactual fairness. In Advances in Neural Information Processing Systems, pp- 4066-4076, 2017.
Wu, Y.. Zhang, L., Wu, X., and Tong, H. PC-Fairness: A unified framework for measuring causality- based fairness. Advances in Neural Information Processing Systems, pp. 3399-3409, 2019b.
Kuzilek, J., Hlosta, M., and Zdrahal, 7. Open university learning analytics dataset. Scientific data, 4(1):1-8, 2017.
Lohia, P. K., Ramamurthy, K. N., Bhide, M., Saha, D., Varshney, K. R., and Puri, R. Bias mitigation post- processing for individual and group fairness. In Ieassp 2019-2019 ieee international conference on acoustics, speech and signal processing (icassp), pp. 2847-2851. IEEE, 2019.
Zhang, J. and Bareinboim, E. Fairness in decision meaiiig the causal explanation formula. In "hirty-Second AAAI Conference on Artificial Intelligence, 2018.
Zhang. L., Wu, Y., and Wu, X. Achieving non- discrimination in data release. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1335-1344, 2017a.
Mishler, A., Kennedy, E. H., and Chouldechova, A. Fair- ness in risk assessment instruments: Post-processing to achieve counterfactual equalized odds. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 386 400, 2021.
Zhang, L.,, Wu, Y., and Wu, X. A causal framework for dis- covering and removing direct and indirect discrimination. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 9929-39355 2017b.
Mitchell, S., Potash, E.,, Barocas, S., D' Amour, A., and Lum, K. Algorithmic fairness: Choices, assumptions, and defi- nitions. Annual Review of Statistics and Its Application, 8:141-163, 2021.
Zhang. L., Wu, Y., and Wu, X. Achieving non- discrimination in prediction. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018a.
pp.
Morgan, S. L. and Winship, C. Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge University Press, second edition, 2015.
Zhang, L., Wu, Y., and Wu, X. Causal modeling- discrimination discovery and removal: criteria, bounds, and algorithms. IEEE Transactions on Knowledge and Data Engineering, 31(11):2033 2050,, 2018b.
Nabi, R. and Shpitser, I. Fair inference on outcomes. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Zuo, A., Wei, S., Liu, T., Han, B., Zhang, K., and Gong, M. Counterfactual fairness with partially known causal graph. In Advances in Neural Information Processing Systems, 2022.
Neyman, J. S. On the application of probability theory to agrieultural experiments. essay on principles. section 9. Statistical Science, 5:465-472, 1990.
Pearl, J. Principal stratification a goal or a tool? The International Journal of Biostatistics, 111):1-13,20 2011.
Reardon, S. F. and Owens, A. 60 years after brown: Trends and consequences of school segregation. Annual Review of Sociology, 40:199-218. 2014.
10

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
A. Proofs
Proof of Theorem In this proof, for the simplicity exposition, we only consider a special case that the conditional set empty
Let
=. -))x)x())///(x
And
(D(a) d,
- = v) Ain + A2n
Ain
Pady X, (),, P(D(a) = d,Y(a) = n)}
A2n
Pad (A, X, D, At Pa.dy (A, X, D, D,Y; Y; p)]
n
By the central limit theorem, VnAI is asymptotic normal. For Ain. Define the Gateaux derivative of the generic function g in the direction (+ n. f - μ4 by 01a - n19. By a Taylor expansion for [[A2a] yields that
B[A2n] El (1 X, D,Y; aa pd Pa,d, (A, X, D,;; Na. 1/ A2))
a (tn Ror - Bl (A, X, D, Y; Tai [1
1
+ = 02
x 2 fr -.. - (A, D,Y; Jay d.y) +...
The first-order term
na." 2d.0 -yy (4 AA ( = 0,
For the second-ordes term, get
02
F - k2
I(A = a){I d, = v) - (X)) (
E
- (X)33
(Ta(X))
[(A a)
(Ta(X)) (X) Fa (x))(i()(X)) - (X))
E
(x)( (ft (X) To. (X)) (x)3
C || Ta (X)
p/a (X)Il2
where C' is a finite constant. All higher-order terms can be shown to be dominated by the second-ordd order term. Therefore,
[[Aan] op(n 1/22..
This completes the proof.
We first introduce a lemma from (Shapiro, 1991), and use it in the proof of Theorem and 3.
Lemma 2 Let θ be a compact subset of R Let C(0) denote the set of continuous real-valued functions on θ, with C(0) ... C(O) the r-dimensional Cartesian product. Let (0) (vo...., E be vector of convex

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
functions. Consider the quantity a* defined the solution to the following convex optimization program:
a" min (0)
#eB
subject to 3 (0) - 0. = 1.j.......
Assume that Slater's condition holds, so that there some 0 E θ for which the inequalities are satisfied and non-affine inequalities are strictly satisfied, i.c. (0) < 0 if w non-affine, Now consider a sequence of approximating programs, for n = 1, 2, .....
an = min o
subject to 2) (0) - 0, = ,,...,r
with Vi (0) = Vonn Ven E C. Assume that [(n)((nn - v)) converges in distribution to a random element W E L for
some real-valued function (n) Then:
[(n) (ān - 00) ~4 L
for particular random variable L. It follows that o, 0g- Or(1/f(n))-
Proof of Theorem 3. According to Theorem 2, the constraint part of the empirical optimization problem is asymptotic to the constraint part of the oracle optimization problem. This can be proved directly from Lemma 2.
Proof of Theorem 4. Because c" is solved by the programing question, it means,
min n Σ The (X,)) (..o..(( w ( wf.o,,, 0,
s.t. Wdo (x) 20 0 for all do. d1 you y1 E {0,1}
Σ a.b "do. P.- (D) do Y()) yo X x)
for all do yo {0, 1)
E Wardiben (I) (D) = d, Y() 91 X x) for all d, yr (0, 1).
In the same way, we can prove that, for the following programing.
a" = min Σ
I wi,oo..(X.)) + 1x,, + w wi. (X.),
1=1
s.t.
d1 (x) 20 for all do. d1, Vo. 11 E {0,
Σ a.6 W (do.a.o.6 = (0) = dg Y(0) = y/ X = x)) for all do yo E {0.
Σ Waddh (x) = (1) = d1, Y(1) = = x) for all di, E (0, 1}.
According to the proof of Theorem 2. V c ~ W. W is random element, the constraint part convergence at rate /n. Using Lemma 2, Vna" converges to random element.
B. Estimation of Nuisance Parameters with Sample Splitting in Theorem 2
both t. (X) and ndw are not assumed to sufficiently "well-behaved," i.e. if they not belong to Donsker classes, we need estimate them by a data-splitting method (Chernozhukov et al.. 2018; Mishler al., 2021; wager Athey, 2018).
Let K be small positive integer, and (for simplicity) suppose that n/K is also integer. Let I,, In be random partition of the index set I (1,...,n) n so that # T = for & = 1,... K. Denote g as the complement of Tg.
Step 1. Nuisance parameter training for each sum-sample.
for 1: to K do
12

---

Fairness on Principal Stratum: A New Perspective on Counterfactual Fairness
Construct estimates (x) and (X) using the sample with Ig
(2) Obtain the predicted values of at (X), and (X) for E Ln.
end
Step 2. All the predicted values aa(X)) and (X.) for i E consist of the estimates of n. (X;) and u denoted as ng (X,), and (X,). respectively.
C. Experimental Details of Synthetic Experiment
We first randomly generate a DAG with 10 nodes and 40 directed edges according to the Erdös-Rényi (ER) model. Following the previous study (Zuo et al., 2022), the path coefficients Здж of directed edges X, - Xx are sampled from a Uniform([-2, -0.5] u [0.5, 2]) distribution. The data are generated according to the following equation:
X = Exxepxx)) 3zk X, + 6,, i= 1,....74,
where pa (Xx) represents the parent nodes of Xs, noise E; ~ 20,2. and is the sample size, which is 1,000 in the experiment. Then randomly select two nodes the outcome Y and the sensitive attribute A. respectively. A is drawn from a (in, I)) with probability (EEx (xpa((() A) Aa4 X; + G)), where a-)) denotes the sigmoid function. To evaluate our post-processing approach, need have Y (0). (1). ((0) and D(1) for each sample. Therefore, generate (a()) from Binomial((0, 1) with probability of xxxx and use four different models: Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF) and Naive Bayes (NB) respectively to obtain the estimations of PPY = == x = x) as the decisions D x). After obtaining potential outcomes for each individual, we estimate (() = (())
3,y Xj(a) + +z), where X(a) means the value of X when A = a,
({(a=)1) y X = x)) using OR, IPS and DR estimators, respectively, where d and y E {0, 1}. Then we check whether the optimization equation in Section 4.1 is solvable. If the feasible domain is empty, we further use the post- post-processing method in Section 4.3 to obtain (ad(x). Finally, we flip D(0) and (1) using (x)(2) as the corresponding probabilities and compute the percentage change of CF and PCF.
13


Reviews:
[{"review_id": "3x3AHuNVYJ", "reviewer": "Reviewer_GxND", "summary": "In this paper, the authors propose a new fairness notion called principle Counterfactual Fairness (PCF). The motivation behind this notion is that algorithms only need to be fair to individuals whose protected attribute has no individual effect on the outcome of interest. The authors derive necessary conditions to assess whether an algorithm satisfies principle CF and propose a corresponding optimization-based evaluation method. They also introduce a post-processing algorithm to adjust unfair decisions. The effectiveness of the algorithm is validated on synthetic datasets and one real-world dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and the time dedicated to reviewing our work. We address your concerns and questions as follows.\n\n> **Comparison between CF and PCF**\n- We can define PCF within the SCM framework, which is **equivalent** to the potential outcome framework used in our original manuscript.\n- For **notations**, $A$, $X$, and $Y$ are defined the same as in CF [1], and $D$ denotes the decision-making from $A$, $X$, and $Y$, which broadens the $\\hat Y$ in CF [1]. In SCM, $Y=f_Y(A, X, \\epsilon_Y)$ and $D=f_D(A, X, Y, \\epsilon_D)$, so as $Y$ and $\\hat Y$ in CF [1].\n- For **fairness metrics**, PCF requires $P(D_{A←a}(U)=y\\mid X=x, A=a)=P(D_{A←a'}(U)=y\\mid X=x, A=a)$ **only for individuals with $Y_{A←a}(U)=Y_{A←a'}(U)$**, i.e., $A$ has no effect on $Y$, whereas **CF requires to be satisfied for all individuals.**\n- The **main challenges** are how to identify individuals with $Y_{A←a}(U)=Y_{A←a'}(U)$ (see Judea Pearl’s \"Principal Stratification – A Goal or a Tool?\" for more details within SCM, especially Fig. 1 and Table 1), and how to enforce CF on these individuals, instead of all individuals.\n\n> **More experiments on common datasets**\n- Motivated by the reviewer, we add experiments comparing more CF methods on the suggested Law and UCI Adult datasets.\n\n|Law|PCF ↑ on $Y(0)≠Y(1)$ (\\%)|CF ↑ on all individuals (\\%)|\n|-|-|-|\n|CF [1]|3.15 ± 0.80|5.13 ± 0.74|\n|CF Rep. [2]|1.71 ± 0.51|1.18 ± 0.47|\n|PSCF|1.84 ± 0.42|1.21 ± 0.41|\n|Principle Fairness|2.60 ± 0.39|4.37 ± 0.65|\n|Quantile CF|2.34 ± 0.20|2.64 ± 0.31|\n|DCEVAE [3]|4.01 ± 1.16|**5.58 ± 0.87**|\n|Ours|**5.54 ± 1.19**|3.85 ± 0.90|\n||\n\n|UCI Adult|PCF ↑ on $Y(0)≠Y(1)$ (\\%)|CF ↑ on all individuals (\\%)|\n|-|-|-|\n|CF [1]|2.89 ± 0.86|4.42 ± 1.10|\n|CF Rep. [2]|2.30 ± 1.14|0.67 ± 0.82|\n|PSCF|1.61 ± 1.22|1.31 ± 0.87|\n|Principle Fairness|2.62 ± 1.28|3.12 ± 0.94|\n|Quantile CF|1.79 ± 0.40|1.56 ± 0.43|\n|DCEVAE [3]|3.34 ± 1.07|**4.67 ± 1.25**|\n|Ours|**4.45 ± 1.36**|3.38 ± 0.93|\n||\n\n- Key observation 1: **Our method improves PCF more significantly compared to the original CF metric**, in which PCF only focuses $Y(0)≠Y(1)$ but CF focuses on all individuals.\n\n- Key observation 2: **Existing CF methods wouldn’t perform better than the proposed approach on the PCF metric.**\n\n> **Causal discovery in experiment**\n- **Our method for imposing PCF does not require causal discovery.** Considering there is no DAG in real-word datasets, we implement causal discovery to first obtain CPDAG, then sample a DAG as the ground truth for simulating the counterfactuals.\n\n> **Necessary condition for testing PCF**\n- Thank you for your insightful question! Below we supplement both theory and experiments to analyze the power of our proposed test—what is the likelihood that an algorithm violating PCF can pass this test (also known as sensitivity)?\n- Theoretically, the power of this test depends on the 8 identifiable probabilities\n$$\\\\{P(D(0)=d_0, Y(0)=y_0 \\mid X=x), P(D(1)=d_1, Y(1)=y_1 \\mid X=x) \\text{ for }d_0, y_0, d_1, y_1=0, 1\\\\}.$$\nRecap that $P(D(0)=a, D(1)=b, Y(0)=c, Y(1)=d \\mid X=x):=w_{a,b,c,d}(x)$, by setting $w_{0100}(x), w_{1000}(x), w_{0111}(x), w_{1011}(x)=0$, let\n$$\\mathcal U=\\\\{(w_{0000}(x), w_{1100}(x), w_{0011}(x), w_{1111}(x), w_{ab10}(x), w_{ab01}(x)\\text{ for }a, b=0, 1)\\mid w_{0000}(x)+w_{1100}(x)+w_{0011}(x)+w_{1111}(x)+\\sum_{a, b}w_{ab10}(x)+\\sum_{a, b}w_{ab01}(x)=1\\\\}$$\nbe the unit polyhedron in the 12-dim space with total edge length 1, and denote the linear transformation in Sec. 4.1 from the 12-dim non-zero $w_{a,b,c,d}(x)$ to the above 8-dim identifiable probabilities as $B$. Then the power of this test is $\\int_{B(\\mathcal U)} dP$.\n- In fact, it be proved that such an optimization-based approach can obtain the tightest upper and lower bounds of $P(D(0), D(1), Y(0), Y(1) \\mid X=x)$, indicating the optimality of this test.\n- Empirically, we add experiments reporting the sensitivity and specificity of our PCF test using various estimators.\n\n|Law|Sensitivity ↑|Specificity ↑|\n|-|-|-|\n|OR|0.67 ± 0.12|**1.00 ± 0.00**|\n|IPS|**0.72 ± 0.10**|**1.00 ± 0.00**|\n|DR|0.71 ± 0.10|**1.00 ± 0.00**|\n||\n\n|UCI Adult|Sensitivity ↑|Specificity ↑|\n|-|-|-|\n|OR|0.79 ± 0.12|**1.00 ± 0.00**|\n|IPS|0.77 ± 0.12|**1.00 ± 0.00**|\n|DR|**0.81 ± 0.12**|**1.00 ± 0.00**|\n||\n\n> **Causal discovery might be helpful for PCF**\n- PCF enforce CF, **it would be also interesting to enforce path-specific CF (Chiappa, 2019) on some individuals rather than all.** To achieve this, the learned causal diagram via causal discovery can help to estimate the principal strata direct effect (PSDE) to identify these individuals (see also Sec. 4.1 of Pearl’s \"Principal Stratification – A Goal or a Tool?\").\n\n> **Interpretation of Theorem 2**\n- Theorem 2 shows the proposed DR estimator can unbiasedly estimate $P(D(a)=d, Y(a)=y \\mid X=x)$ in Sec. 4.1 with large samples.\n***\nWe are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed -- thank you so much!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The theorem seems correct to me, but I didn't check all the proofs.", "experimental_designs_or_analyses": "The experiment setup makes sense to me.", "relation_to_broader_scientific_literature": "This work proposed a novel definition of Counterfactual Fairness.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "My major concerns are as below\n\n**Relationship to CF** \n\n1. Overall, I find it challenging to compare the proposed PCF with the original CF in [1]. Is it possible to define principle CF within the SCM framework? More specifically, what would the causal relationships between $A$, $D$, and $Y$ look like in a causal graph?\n\n**Experiment**\n\n1. My primary concern is about what the authors aim to justify through the empirical study and whether they achieve that goal. Regarding the first question, would it be useful to test previous CF methods to assess if they are indeed too restrictive for PCF? Regarding the second question, in the current draft, the authors show that their post-processing algorithm can improve CF and PCF. How can we be sure that existing fairness or CF methods wouldn’t perform better than the proposed approach?\n2. Could the authors provide justification for their choice of dataset? For instance, why not use datasets like Law or UCI Adult, which are more commonly used in CF literature [1][2][3][4]?\n3. What is the motivation behind using a causal discovery algorithm and creating subgroups based on its results? This part is unclear to me, particularly since the definition of PCF is not based on an SCM.\n\n**Significance**\n\nI cannot fully acknowledge the significance of the proposed framework due to the following reasons:\n1. As mentioned above, there is confusion about the comparison between CF and PCF.\n2. Concerns about the experimental design and its ability to justify the proposed framework’s or method’s contributions.\n3. It appears that the authors only provide a necessary condition for an algorithm to satisfy PCF, which relies on the minimum and maximum values of the solution to an optimization problem. I’m uncertain about the effectiveness of this criterion—what is the likelihood that an algorithm could pass this test without truly satisfying PCF? This aspect is not discussed, either theoretically or empirically.\n\n[1] Kusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual fairness. Advances in neural information processing systems, 30.\n\n[2] Zuo, Z., Khalili, M., & Zhang, X. (2023). Counterfactually fair representation. Advances in Neural Information Processing Systems, 36, 12124-12140.\n\n[3] Kim, H., Shin, S., Jang, J., Song, K., Joo, W., Kang, W., & Moon, I. C. (2021, May). Counterfactual fairness with disentangled causal effect variational autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 9, pp. 8128-8136).\n\n[4] Rosenblatt, L., & Witter, R. T. (2023, June). Counterfactual fairness is basically demographic parity. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 12, pp. 14461-14469).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What is the relationship between the proposed notion and Individual Fairness?\n2. In the conclusion section, the authors suggest that causal discovery might be helpful for PCF. Could the authors elaborate a bit more on this?\n3. Could the authors provide some interpretation of Theorem 2, which seems to be missing in the current draft?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bMidNJdTQB", "reviewer": "Reviewer_acEP", "summary": "This paper introduces Principal Counterfactual Fairness (PCF) and proposes to unify two approaches,\n* Principal Stratification : \nFrangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58(1), 21-29.\nIn their 2002 paper \"Principal Stratification in Causal Inference,\" Frangakis and Rubin introduce a framework to address the challenges of adjusting for posttreatment variables in causal studies. They propose the concept of principal stratification, which involves classifying subjects based on the joint potential values of a posttreatment variable under each treatment being compared. This classification creates principal strata that are unaffected by treatment assignment, allowing for the estimation of causal effects within these strata, termed principal effects. (see also Judea Pearl’s 2011 \"Principal Stratification – A Goal or a Tool?\") \n* Counterfactual Fairness : \nKusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual fairness. Advances in neural information processing systems, 30.\nIn their 2017 paper, \"Counterfactual Fairness\", Kusner et al. introduce a formal framework for evaluating fairness in machine learning models by using the concept of counterfactuals. The central idea is that a model is fair if its predictions do not depend on sensitive attributes, like race or gender, in a way that would change under hypothetical counterfactual scenarios.\n\nHere, the authors introduce some new fairness criterions (Principal Counterfactual Parity, Principal Counterfactual Equalized Odds, and Principal Conditional Counterfactual Fairness) based on principal stratification from causal inference. It refines counterfactual fairness by ensuring fairness only for individuals whose protected attributes have no individual causal effect on the outcome.\nThey derive statistical bounds to assess whether an algorithm satisfies Principal Counterfactual Fairness, they propose an optimization-based evaluation method that detects fairness violations by solving feasibility constraints, they develop a post-processing approach that minimally adjusts algorithmic decisions to enforce fairness while preserving accuracy and finally, they use doubly robust estimation techniques to ensure efficient estimation of fairness constraints.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and valuable feedback! Below, we address your questions and indicate the changes we’ve made thanks to your suggestion.\n\n> **Unfortunately, no comparison to alternative fairness definitions are considered. Only one real-world dataset (OULAD) is used. Lack of reporting accuracy metrics before and after fairness adjustments (trade-offs).**\n\nThank you for pointing this issue! We follow your suggestion to add extensive experiments comparing our method to more baselines on two new datasets: Law and UCI Adult.\n\n|Law|PCF ↑ on $Y(0)≠Y(1)$ (\\%)|CF ↑ on all individuals (\\%)| Accuracy ↓ on all individuals (\\%)|\n|-|-|-|-|\n|CF|3.15 ± 0.80|5.13 ± 0.74|3.28 ± 0.85|\n|CF Rep.|1.71 ± 0.51|1.18 ± 0.47|**1.89 ± 0.32**|\n|PSCF|1.84 ± 0.42|1.21 ± 0.41|2.07 ± 0.48|\n|Principle Fairness|2.60 ± 0.39|4.37 ± 0.65|2.05 ± 0.21|\n|Quantile CF|2.34 ± 0.20|2.64 ± 0.31|2.19 ± 0.23|\n|DCEVAE|4.01 ± 1.16|**5.58 ± 0.87**|2.81 ± 0.53|\n|Ours|**5.54 ± 1.19**|3.85 ± 0.90|1.97 ± 0.38|\n||\n\n|UCI Adult|PCF ↑ on $Y(0)≠Y(1)$ (\\%)|CF ↑ on all individuals (\\%)| Accuracy ↓ on all individuals (\\%)|\n|-|-|-|-|\n|CF|2.89 ± 0.86|4.42 ± 1.10|2.60 ± 0.70|\n|CF Rep.|2.30 ± 1.14|0.67 ± 0.82|1.64 ± 1.00|\n|PSCF|1.61 ± 1.22|1.31 ± 0.87|**1.13 ± 0.54**|\n|Principle Fairness|2.62 ± 1.28|3.12 ± 0.94|2.12 ± 0.63|\n|Quantile CF|1.79 ± 0.40|1.56 ± 0.43|2.24 ± 1.22|\n|DCEVAE|3.34 ± 1.07|**4.67 ± 1.25**|3.23 ± 1.66|\n|Ours|**4.45 ± 1.36**|3.38 ± 0.93|1.85 ± 0.78|\n||\n\n- Key observation 1: **Our method improves PCF more significantly compared to the original CF metric**, in which PCF only focuses $Y(0)≠Y(1)$ but CF focuses on all individuals.\n- Key observation 2: **Existing CF methods wouldn’t perform better than the proposed approach on the PCF metric.**\n- Key observation 3: **Our post-processing approach exhibits very competitive performance in terms of the trade-off between fairness and accuracy** -- our PCF results are the best with only a slight decrease in accuracy.\n\n> **In Section 4.1, the fairness violations may not always be detected accurately (false positives/negatives possible).**\n\n- Theoretically, the power of this test depends on the 8 identifiable probabilities\n$$\\\\{P(D(0)=d_0, Y(0)=y_0 \\mid X=x), P(D(1)=d_1, Y(1)=y_1 \\mid X=x) \\text{ for }d_0, y_0, d_1, y_1=0, 1\\\\}.$$\nRecap that $P(D(0)=a, D(1)=b, Y(0)=c, Y(1)=d \\mid X=x):=w_{a,b,c,d}(x)$, by setting $w_{0100}(x), w_{1000}(x), w_{0111}(x), w_{1011}(x)=0$, let\n$$\\mathcal U=\\\\{(w_{0000}(x), w_{1100}(x), w_{0011}(x), w_{1111}(x), w_{ab10}(x), w_{ab01}(x)\\text{ for }a, b=0, 1)\\mid w_{0000}(x)+w_{1100}(x)+w_{0011}(x)+w_{1111}(x)+\\sum_{a, b}w_{ab10}(x)+\\sum_{a, b}w_{ab01}(x)=1\\\\}$$\nbe the unit polyhedron in the 12-dim space with total edge length 1, and denote the linear transformation in Sec. 4.1 from the 12-dim non-zero $w_{a,b,c,d}(x)$ to the above 8-dim identifiable probabilities as $B$. Then the power of this test is $\\int_{B(\\mathcal U)} dP$.\n- Empirically, we add experiments reporting the sensitivity and specificity of our PCF test using various estimators.\n\n|Law|Sensitivity ↑|Specificity ↑|\n|-|-|-|\n|OR|0.67 ± 0.12|**1.00 ± 0.00**|\n|IPS|**0.72 ± 0.10**|**1.00 ± 0.00**|\n|DR|0.71 ± 0.10|**1.00 ± 0.00**|\n||\n\n|UCI Adult|Sensitivity ↑|Specificity ↑|\n|-|-|-|\n|OR|0.79 ± 0.12|**1.00 ± 0.00**|\n|IPS|0.77 ± 0.12|**1.00 ± 0.00**|\n|DR|**0.81 ± 0.12**|**1.00 ± 0.00**|\n||\n\nThe above experimental results align with our theoretical claims for our proposed PCF test in Sec. 4.1, i.e., our PCF test is necessary so that the false positive rate is 0. We also empirically show that our PCF test has a relatively low false negative rate.\n\n> **More discussion on the two important references**\n\nWe appreciate your insightful comments!\n\n- Compared with Imai & Jiang (2023), we highlight the following differences.\n  - For estimands, they focuses on $(Y(D=0), Y(D=1))$, while we focused on $P(D(A=0), D(A=1)\\mid Y(A=0), Y(A=1))$, resulting in different assumptions and techniques.\n  - For assumptions, they further assume monotonicity holds, i.e., $Y(D=1)\\leq Y(D=0)$, while we only assumes ignorability, which can be relaxed by leveraging the sensitivity analysis such as (Fawkes, Jake, et al., NeurIPS 24).\n  - For techniques, their framework is more like instrumental variable or mediation analysis (see Fig. 1 in Imai & Jiang), while we take advantage from linear programming used by (A. Li and J. Pearl, AAAI 22 & 24).\n\n- For Kilbertus et al. (2020), we remark that\n  - Principle stratification can be regarded as an unmeasured confounder (see J. Pearl's “Principal Stratification - A Goal or a Tool?” for details), and Kilbertus et al. (2020) studies how fairness constraints are affected by unmeasured confounders;\n  - Our paper proposes PCF built on this unmeasured confounder, thus we can compare the difference in performance between PCF and original CF benefiting from Kilbertus et al. (2020).\n***\nPlease let us know if you have further questions -- thank you so much!", "claims_and_evidence": "The paper clearly defines PCF and shows how it extends existing fairness definitions using principal stratification.\nThe paper derives statistical bounds for verifying fairness, ensuring a solid mathematical foundation, it provides necessary conditions for fairness violations are rigorously formulated using probability constraints and finally, the doubly robust estimation approach ensures reliable estimation under specific assumptions.\n\nThe authors present an optimization-based approach that adjusts decisions with minimal changes to ensure fairness, and theoretical proofs confirm the optimality of the post-processing adjustments. Furthermore, the study includes both synthetic and real-world datasets (OULAD dataset), improving credibility. In the last section, some performance metrics (Counterfactual Fairness and Principal Counterfactual Fairness) show measurable improvements post-adjustment.\n\nNevertheless, other claims are not based on convincing evidences. For instance, \"Principal Counterfactual Fairness is the best way to define fairness in causal settings\" : if the paper makes a strong case for PCF, it does not compare its approach against alternative fairness frameworks, such as path-specific counterfactual fairness (Chiappa, 2019), fairadapt based on quantile regressions (Plečko, Bennett & Meinshausen, 2024), sequential transport on graphs (Fernandes Machado, Gallic & Charpentier, 2025) or principal fairness (Imai & Jiang, 2023).\n\nFernandes Machado, A., Charpentier, A., & Gallic, E. (2024). Sequential conditional transport on probabilistic graphs for interpretable counterfactual fairness. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37).\n\nImai, K., & Jiang, Z. (2023). Principal fairness for human and algorithmic decision-making. Statistical Science, 38(2), 317-328.\n\nPlečko, D., Bennett, N., & Meinshausen, N. (2024). fairadapt: Causal reasoning for fair data preprocessing. Journal of Statistical Software, 110, 1-35.\n\nThe claim \"The proposed optimization method reliably detects fairness violations\" is not clear. The paper proves necessary conditions for fairness violations but does not establish their sufficiency due to partial identifiability issues. Thus, even if a violation is detected, it is unclear whether the algorithm is truly unfair or if the bound is too loose.\n\nAlso, \"Post-processing ensures fairness with minimal impact on accuracy\", but  the paper does not report accuracy metrics before and after fairness adjustments (there are accuracy trade-offs, which are not discussed in details in the paper).", "methods_and_evaluation_criteria": "Yes.\nThe paper derives statistical bounds to evaluate fairness violations, ensuring a rigorous methodology.\nSynthetic experiments allow the authors to validate fairness constraints in controlled settings.\nThe real-world dataset (OULAD – Open University Learning Analytics Dataset) provides a practical benchmark for fairness in education, aligning well with fairness applications in admissions and grading.\nAnd the use of doubly robust (DR) estimation improves the reliability of fairness assessments (this method is commonly used in causal inference and helps handle estimation biases)\n\nBut as mentioned above, the study does not compare PCF against other causal fairness definitions, and the study does not report accuracy trade-offs after fairness corrections. Finally, the fairness constraints rely on statistical bounds, meaning they cannot fully determine whether an algorithm is truly unfair. I have reviewed the key proofs supporting the theoretical claimsThis limitation is acknowledged in the paper, but further discussion on how to improve identifiability would be valuable.", "theoretical_claims": "I have reviewed the key proofs supporting the theoretical claims.\n\nThe ignorability assumption (Assumption 1) is a classical assumption, usefull to derive theoretical results, but it is very strong and may not always hold in real-world scenarios. The sufficiency of these bounds for detecting unfairness is not guaranteed, meaning fairness violations could still occur without detection.\n\nIn Section 4.1, the authors claim that the fairness condition is violated if the feasible domain of optimization constraints is empty or if a principal stratum’s probability is negative. The proof relies on the correct estimation of potential outcomes, which is partially identifiable from data, meaning that fairness violations may not always be detected accurately (false positives/negatives possible).\n\nAnd finally, in the doubly robust estimation part (Theorem 2 & 3), the doubly robust estimator provides asymptotically consistent estimates of fairness violations. The proof is correct, but real-world applications may suffer from model misspecification issues.", "experimental_designs_or_analyses": "Yes.\nAs already discussed, there are no baseline fairness measures (e.g., standard Counterfactual Fairness) reported for comparisons.\nAnd there are no statistical robustness checks (confidence intervals, significance tests), which makes it hard to assess reliability.", "relation_to_broader_scientific_literature": "The paper brings Principal Stratification into Fairness Research, which could be seen as a novel contribution, it refines Counterfactual Fairness by applying fairness constraints only where appropriate, and it develops an optimization-based post-processing fairness intervention.", "essential_references_not_discussed": "As mentioned in the introduction, the most import related references are Frangakis & Rubin (2002) abd Kusner et al. (2017)\n\nBut (at least) two important references are missing :\n\nImai, K., & Jiang, Z. (2023). Principal fairness for human and algorithmic decision-making. Statistical Science, 38(2), 317-328.\n\nKilbertus, N., Ball, P. J., Kusner, M. J., Weller, A., & Silva, R. (2020). The sensitivity of counterfactual fairness to unmeasured confounding. In Uncertainty in artificial intelligence (pp. 616-626). PMLR.\n\nimai & Jiang (2023) is one of the first to use principal stratification in algorithmic fairness, it would be nice to explain differences between the two approaches\n\nKilbertus et al. (2020) studies how fairness constraints are affected by unmeasured confounders. Since principal stratification accounts for hidden heterogeneity, so citing this work would clarify the link between PCF and confounders.\n\nThere might also be connexions with \n\nZuo, Z., Khalili, M., & Zhang, X. (2023). Counterfactually fair representation. Advances in Neural Information Processing Systems, 36, 12124-12140.\n\nRosenblatt, L., & Witter, R. T. (2023, June). Counterfactual fairness is basically demographic parity. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 12, pp. 14461-14469).", "other_strengths_and_weaknesses": "Unlike standard Counterfactual Fairness (Kusner et al., 2017), this method ensures fairness only in relevant subgroups, making it more context-sensitive.\n\nUnfortunately, no comparison to alternative fairness definitions are considered. Only one real-world dataset (OULAD) is used, and having other popular datasets (adult, law) in the Supplementary material could have been interesting", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pciI6aBar5", "reviewer": "Reviewer_H5CQ", "summary": "This paper introduces principal counterfactual fairness (PCF), a novel measure of fairness which enforces (to my understanding) that; if a sensitive attribute A did not have a causal effect on an outcome Y for an individual, then our prediction of Y should likewise not be causally influenced by that sensitive attribute. The reason this is important, is that there are cases where we want our decisions to be dependent on a sensitive attribute (such as when predicting an ability score, we would want to use data on that persons disabilities), but dependent in the right way. For example, if the disability did not affect the ability we are measuring for this person, then we shouldnt penalise them for this. This is in contrast to traditional counterfactual fairness, which demands that the prediction is not caused by the sensitive attribute, regardless of if that attribute causes the outcome Y or not. The authors present a formal definition of PCF and provide statistical bounds along with an optimization-based evaluation framework to verify fairness conditions. They provide a theoretical analysis and empirical validation through experiments with synthetic and real data.\n\n## update after rebuttal\n\nFollowing rebuttal, my concerns have been resolved by the clarifications proposed by the authors (especially those that make it clear, in a graphical sense, when their proposed measure is non-trivial or doesnt reduce to counterfactual fairness), and I am recommending acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Can you come up with a simple SCM describing a scenario where PCF and CF give different answers? Ideally, where PCF gives the more intuitive result.**\n\nThank you for the constructive suggestion to help us improve the readability of our paper!\n\n**First, we define PCF within the SCM framework**, which is equivalent to the potential outcome framework used in our original manuscript.\n - For notations, $A$, $X$, and $Y$ are defined the same as in CF, and $D$ denotes the decision-making from $A$, $X$, and $Y$, which broadens the $\\hat Y$ in CF;\n- In SCM, $Y=f_Y(A, X, \\epsilon_Y)$ and $D=f_D(A, X, Y, \\epsilon_D)$, same as $Y$ and $\\hat Y$ in CF;\n- PCF requires $P(D_{A←a}(U)=y\\mid X=x, A=a)=P(D_{A←a'}(U)=y\\mid X=x, A=a)$ **only for individuals with $Y_{A←a}(U)=Y_{A←a'}(U)$**, i.e., $A$ has no effect on $Y$, whereas **CF requires to be satisfied for all individuals**;\n- The **main challenges** are how to identify individuals with $Y_{A←a}(U)=Y_{A←a'}(U)$ (see Judea Pearl’s \"Principal Stratification – A Goal or a Tool?\" for more details within SCM, especially Fig. 1 and Table 1), and how to enforce CF on these individuals, instead of all individuals.\n\n**Next, we follow the reviewer's suggestion to provide a very simple SCM satisfying PCF but violating original CF.**\n- Consider $\\epsilon_Y\\sim \\operatorname{Bern}(0.5), Y=A+(1-A)\\epsilon_Y$, and $D=Y$ ($D$ is perfect prediction of $Y$);\n- In this way, when setting $A←1$, we always have $D_{A←1}(U)=1$ and $Y_{A←1}(U)=1$, that is, $P(D(1)=1, Y(1)=1)=1$;\n- When setting $A←0$, we have $D=Y=\\epsilon_Y\\sim \\operatorname{Bern}(0.5)$, that is, $P(D(0)=0, Y(0)=0)=0.5$ and $P(D(0)=1, Y(0)=1)=0.5$;\n- For the joint distribution, we thus have $P(D(0)=0, D(1)=1, Y(0)=0, Y(1)=1)=0.5$ (violating CF) and $P(D(0)=1, D(1)=1, Y(0)=1, Y(1)=1)=0.5$ (satisfying CF);\n- The former half violates CF, due to $D(0)=0\\neq D(1)=1$, but PCF doesn't care these individuals due to $Y(0)=0\\neq Y(1)=1$. While the latter half satisfies CF, due to $D(0)=D(1)=1$;\n- As a result, this SCM satisfies PCF defined on individuals $Y(0)=Y(1)$, but violates the original CF defined on all individuals;\n- To further enforce CF, the learned predictor $D$ need to change its predictions on the $(D(0)=0, D(1)=1, Y(0)=0, Y(1)=1)$ individuals, which will inevitably sacrifice the accuracy of the predictor due to the updated CF predictions $D'\\neq Y$ on these individuals (recap that $D=Y$ meets PCF).\n\n> **Assume we can exclude exogenous confounders from the set of endogenous variables {A, D, Y, X}. Can you give examples what are the general graphical conditions for which PCF is distinct from CF? Ideally, specfiying a DAG.**\n\n- The above specified DAG provides a valid example for which PCF is distinct from CF, then we discuss the general graphical conditions for which PCF is distinct from CF.\n- Intuitively, as the reviewer noted, the most interesting cases (that do not reduce to standard counterfactual fairness) are where for some sub-population A does not cause Y and for the rest sub-population A does cause Y.\n- As an extreme case, if A never cause Y, then we always have $Y_{A←a}(U)=Y_{A←a'}(U)$, making PCF degenerates to CF. Instead, if A cause Y for all individuals, then PCF degenerates to no fairness constraint.\n\n\n\n> **More discussion on the ignorability assumption**\n\n- The reviewer is correct that we assume X contains all confounders between A and {D, Y}, but we don't assume that D is conditioned on all X.\n- The ignorability assumption only assumes that all confounders are observed, instead of assuming all confounders are conditioned by D and making the backdoor path be blocked.\n- **We would like to kindly remark that it's natural to extend our approach to avoid the usage of the ignorability assumption, such as using sensitivity analysis [1] in causal inference literature.** We leave this for future work, considering the orthogonality of these two issues.\n- We thank the reviewer for pointing out this issue and referring us many insightful references, we will definitely cite and discuss them in our final version.\n\n> **Interpretation of Theorem 2**\n- Theorem 2 shows the proposed DR estimator can unbiasedly estimate $P(D(a)=d, Y(a)=y \\mid X=x)$ in Sec. 4.1 with large samples.\n***\nWe are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.\n\n> **Validation is limited to the OULAD dataset**\n- We kindly ask the reviewer to refer to the rebuttal we provide to Reviewer M8WL, in which:\n  - We add extensive experiments comparing our method to more baselines on two new datasets: Law and UCI Adult;\n  - We also add experiments to analyze the power of our proposed test -- \"what is the likelihood that an algorithm violating PCF can pass this test (also known as sensitivity)?\" on the above two new datasets.\n\n***\n**Reference**\n\n[1] Fawkes, Jake, et al. The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning. NeurIPS, 2024.", "claims_and_evidence": "PCF is a compelling idea, and their definition is sound and captures what they intended. The result could be more clearly explained and motivated however. For example, when first introducing the athlete / disability example, the authors could clearly state what the desired outcome is. Some athletes may have a disability A = 1, but this does not necessarily always cause them to be below the threshold performance (Y = 0). For example, the athlete may have found ways to overcome their disability with specific training (observed in X). If they would have the same Y regardless of A, then A shouldnt influence our prediction.\n\nThe general claim, that “‘if some factor didnt influence my outcome then it shouldnt influence your prediction”, feels quite general, and the paper could be improved by more motivating examples beyond the disability example. \n\nThe main issue with the paper is that ignorability (a standard assumption) is not discussed at all. There should be a proper discussion of what it means in this context, and references to papers discussing the assumption and giving it context (e.g. [1]). \n\nThe implications of ignorability for the applicability of the result should also be discussed. As I understand it, you are assuming that X contains all confounders between A and {D, Y}. Are you assuming that D is conditioned on all X? If so, then this assumption restricts the result a lot, as the most interesting cases (that do not reduce to standard counterfactual fairness) are where for some sub-population A does not cause Y, but A and Y are correlated via a confounder W. \nThe issue is that conditioning on W breaks this backdoor path, and excluding any W removes the novelty of the result. So it appears the result is interesting in cases where there are endogenous confounders W between A and Y (noting that in most settings the inputs to the algorithmic decision D are fully observed, in which case assumption 1 reduces to there being no unobserved confounders between A and Y). \n\n[1] Fawkes, Jake, Robin Evans, and Dino Sejdinovic. \"Selection, ignorability and challenges with causal fairness.\" Conference on Causal Learning and Reasoning. PMLR, 2022.", "methods_and_evaluation_criteria": "The experimental evaluation seems thorough. I would encourage the authors to also present their results in the SCM formalism. It would not require much effort, and in settings where you have knowledge of the underlying structual equations, you can directly evaluate PCF without having to rely on bounds. Even a toy example with and SCM would improve the paper, especially if it could be use to highlight the kinds of settings for which PCF differs from CF.", "theoretical_claims": "The theoretical results appear sound, though I have not checked the appendices in depth.", "experimental_designs_or_analyses": "The authors show their post-processing approach effectively improves PCF, demonstrating practical applicability. The subgroup analyses highlight how fairness violations vary depending on contextual covariates. While the validation is limited to the OULAD dataset, I think this is reasonable given that primary contribution of the paper which is theoretical.", "relation_to_broader_scientific_literature": "The authors provide a thorough review of related fairness measures which they use to situate and motivate their results.", "essential_references_not_discussed": "Ignorability, and its application to causal fairness. \n\n[1] Rosenbaum, Paul R., and Donald B. Rubin. \"The central role of the propensity score in observational studies for causal effects.\" Biometrika 70.1 (1983): 41-55.\n[2] Pearl, Judea. \"Generalizing experimental findings.\" Journal of Causal Inference 3.2 (2015): 259-266.\n[3] Fawkes, Jake, Robin Evans, and Dino Sejdinovic. \"Selection, ignorability and challenges with causal fairness.\" Conference on Causal Learning and Reasoning. PMLR, 2022.", "other_strengths_and_weaknesses": "The paper is clearly written, and after some thinking the fairness measure the authors are proposing is appealing, but needs to be better explained and motivated, and the impact of this result will be more clear to the reader once the affect of the ignorbility assumption is properly discussed. But ultimately, I dont think ignorability is necessary for PCF to be applicable.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Is the above interpretation of PCF accurate?\n\n2. Assume we can exclude exogenous confounders from the set of endogenous variables {A, D, Y, X}. Can you give examples what are the general graphical conditions for which PCF is distinct from CF? Ideally, specfiying a DAG. \n\n3. Can you come up with a simple SCM describing a scenario where PCF and CF give different answers? Ideally, where PCF gives the more intuitive result.  \n\n4. Can you provide some exposition on theorem 2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xf3jnJ7Euk", "reviewer": "Reviewer_M8WL", "summary": "This study addresses an important question about which attributes and individuals should be protected. It proposes principal counterfactual fairness based on the concepts of principal stratification and counterfactual fairness. Theoretical analysis of principal counterfactual fairness is provided. In practice, a CPDAG is learnt from data using the PC algorithm in the causal-learn package. Experiments were conducted on synthetic data and one real dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and the time dedicated to reviewing our work. We address your concerns and questions as follows.\n\n> **Can the authors demonstrate the proposed ideas on more datasets?**\n\nThank you for pointing this issue! We follow your suggestion to add extensive experiments comparing our method to more baselines on two new datasets: Law and UCI Adult.\n\n|Law|PCF ↑ on $Y(0)≠Y(1)$ (\\%)|CF ↑ on all individuals (\\%)| Accuracy ↓ on all individuals (\\%)|\n|-|-|-|-|\n|CF [1]|3.15 ± 0.80|5.13 ± 0.74|3.28 ± 0.85|\n|CF Rep. [2]|1.71 ± 0.51|1.18 ± 0.47|**1.89 ± 0.32**|\n|PSCF [3]|1.84 ± 0.42|1.21 ± 0.41|2.07 ± 0.48|\n|Principle Fairness [4]|2.60 ± 0.39|4.37 ± 0.65|2.05 ± 0.21|\n|Quantile CF [5]|2.34 ± 0.20|2.64 ± 0.31|2.19 ± 0.23|\n|DCEVAE [6]|4.01 ± 1.16|**5.58 ± 0.87**|2.81 ± 0.53|\n|Ours|**5.54 ± 1.19**|3.85 ± 0.90|1.97 ± 0.38|\n||\n\n|UCI Adult|PCF ↑ on $Y(0)≠Y(1)$ (\\%)|CF ↑ on all individuals (\\%)| Accuracy ↓ on all individuals (\\%)|\n|-|-|-|-|\n|CF [1]|2.89 ± 0.86|4.42 ± 1.10|2.60 ± 0.70|\n|CF Rep. [2]|2.30 ± 1.14|0.67 ± 0.82|1.64 ± 1.00|\n|PSCF [3]|1.61 ± 1.22|1.31 ± 0.87|**1.13 ± 0.54**|\n|Principle Fairness [4]|2.62 ± 1.28|3.12 ± 0.94|2.12 ± 0.63|\n|Quantile CF [5]|1.79 ± 0.40|1.56 ± 0.43|2.24 ± 1.22|\n|DCEVAE [6]|3.34 ± 1.07|**4.67 ± 1.25**|3.23 ± 1.66|\n|Ours|**4.45 ± 1.36**|3.38 ± 0.93|1.85 ± 0.78|\n||\n\n- Key observation 1: **Our method improves PCF more significantly compared to the original CF metric**, in which PCF only focuses $Y(0)≠Y(1)$ but CF focuses on all individuals.\n- Key observation 2: **Existing CF methods wouldn’t perform better than the proposed approach on the PCF metric.**\n- Key observation 3: **Our post-processing approach exhibits very competitive performance in terms of the trade-off between fairness and accuracy** -- our PCF results are the best with only a slight decrease in accuracy.\n\nIn addition, we find it's meaningful to add experiments to analyze the power of our proposed test -- \"what is the likelihood that an algorithm violating PCF can pass this test (also known as sensitivity)?\" on the above two new datasets. The results are shown as below.\n\n|Law|Sensitivity ↑|Specificity ↑|\n|-|-|-|\n|OR|0.67 ± 0.12|**1.00 ± 0.00**|\n|IPS|**0.72 ± 0.10**|**1.00 ± 0.00**|\n|DR|0.71 ± 0.10|**1.00 ± 0.00**|\n||\n\n|UCI Adult|Sensitivity ↑|Specificity ↑|\n|-|-|-|\n|OR|0.79 ± 0.12|**1.00 ± 0.00**|\n|IPS|0.77 ± 0.12|**1.00 ± 0.00**|\n|DR|**0.81 ± 0.12**|**1.00 ± 0.00**|\n||\n\nThe above experimental results align with our theoretical claims for our proposed PCF test in Sec. 4.1, i.e., our PCF test is necessary so that the false positive rate is 0. We also empirically show that our PCF test has a relatively low false negative rate.\n\n> **In practice, it can be challenging to correctly infer causal relationships from data.**\n\nThank you for raising this concern. We would like to clarify that **our method for imposing PCF does not require causal discovery.** Considering there is no DAG in real-word datasets, we implement causal discovery to first obtain CPDAG, then sample a DAG as the ground truth for simulating the counterfactuals.\n\nNote that our proposed method does not require a known DAG (or even a CPDAG). Instead, **the only assumption we make is the ignorability assumption in line 232**, i.e., $A \\perp(Y(1), Y(0), D(1), D(0)) \\mid X$, meaning that there is no unobserved confounders. We would also like to remark that **it is natural to extend our approach to further relax this assumption**, such as using sensitivity analysis [7] in causal inference literature. We leave this for future work, considering the orthogonality of these two issues.\n\nLastly, motivated by the reviewer that obtaining an accurate DAG from observational data is challenging, we note that **recent studies have been focused on how to achieve CF with partial DAGs [8, 9]**. It would be useful to incorporate these works, but it should still be noted that our approach does not try to infer causal relationships from data.\n\n***\nPlease let us know if you have further questions -- thank you so much!\n***\n**References**\n\n[1] Kusner, Matt J., et al. Counterfactual fairness. NeurIPS, 2017.\n\n[2] Zuo, Zhiqun, et al. Counterfactually fair representation. NeurIPS, 2023.\n\n[3] Chiappa, Silvia. Path-specific counterfactual fairness. AAAI, 2019.\n\n[4] Imai, Kosuke, and Zhichao Jiang. Principal fairness for human and algorithmic decision-making. Statistical Science, 2023.\n\n[5] Plečko, Drago, et al. fairadapt: Causal reasoning for fair data preprocessing. Journal of Statistical Software, 2024.\n\n[6] Kim, Hyemi, et al. Counterfactual fairness with disentangled causal effect variational autoencoder. AAAI, 2021.\n\n[7] Fawkes, Jake, et al. The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning. NeurIPS, 2024.\n\n[8] Zuo, Aoqi, et al. Counterfactual fairness with partially known causal graph. NeurIPS, 2022.\n\n[9] Li, Haoxuan, et al. A Local Method for Satisfying Interventional Fairness with Partially Known Causal Graphs. NeurIPS, 2024.", "claims_and_evidence": "The results on the synthetic data is good. The evidence from the real dataset could be stronger.", "methods_and_evaluation_criteria": "The real dataset used in this study lacks ground truth, making evaluation and judgment challenging.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Can the authors demonstrate the proposed ideas on more datasets, such as German Credit, Adult, and COMPAS available at https://ashryaagr.github.io/Fairness.jl/dev/datasets/. Feel free to choose other datasets if the above ones are not appropriate.", "relation_to_broader_scientific_literature": "This study contributes to algorithmic fairness, especially it enchances counterfactual fairness.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No", "questions_for_authors": "Algorithmic fairness is a complex topic. Counterfactual fairness and its variants are excellent ideas. However, in practice, it can be challenging to correctly infer causal relationships from data. In addition, our knowledge about specific applications is often incomplete. How does the proposed approach handle this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce Principal Counterfactual Fairness (PCF) and unify two existing problems in the literature. The empirical results are mostly compelling (the extra results in the rebuttal helped considerably), and the theoretical results seem correct. The authors also plan to add missing citations in a future version. However, there is one concern: the authors must be careful when mixing between "observed outcomes" and "ground-truth outcomes", which could be an issue with a biased dataset.
Decision: Accept (poster)