Group Fairness in Reinforcement Learning via Multi-Objective Rewards
Authors: Jack Blandin
Ian A. Kash
Venue: Accepted by TMLR ()

Abstract: Recent works extend classification group fairness measures to sequential decision processes such as reinforcement learning (RL) by measuring fairness as the difference in decision-maker utility (e.g. accuracy) of each group. This approach suffers when decision-maker utility is not perfectly aligned with group utility, such as in repeat loan applications where a false positive (loan default) impacts the groups (applicants) and decision-maker (lender) by different magnitudes. Some works remedy this by measuring fairness in terms of group utility, typically referred to as their "qualification", but few works offer solutions that yield group qualification equality. Those that do are prone to violating the "no-harm" principle where one or more groups' qualifications are lowered in order to achieve equality. In this work, we characterize this problem space as having three implicit objectives: maximizing decision-maker utility, maximizing group qualification, and minimizing the difference in qualification between groups. We provide a RL policy learning technique that optimizes for these objectives directly by constructing a multi-objective reward function that encodes these objectives as distinct reward signals. Under suitable parameterizations our approach is guaranteed to respect the "no-harm" principle.

Published in Transactions on Machine Learning Research (05/2024)
Group Fairness in Reinforcement Learning via Multi- Objective Rewards
Jack Blandin
blandin @ruic. edu
Deportment of Computer Science University of Illmois Chicago
Ian Kash
iankash@rric.c edu
Department Computer Science
University of Blinois Chicago
Reviewed on OpenReview: openrevveee.r..ee.....m. net/ net/forum? forum? id= cueEUSC7LE
Abstract
Recent works extend classification group fairness measures to sequential decision processes such as reinforcement. learning (RI) by measuring fairness as the difference in decision- maker utility (e.g. accuracy) of each group. This approach suffers when decision-maker utility is not perfectly aligned with group utility, such as in repeat loan applications where false positive (loan default) impacts the groups (applicants) and decision-maker (lender) by different magnitudes. Some works remedy this by measuring fairness in terms of group utility, typically referred to as their 'qualification', but few works offer solutions that yield group qualification equality. Those that do are prone to violating the 'no-harm' principle where one or more groups' qualifications are lowered in order to achieve equality. this work, we characterize this problem space as having three implicit objectives: maximizing decision-maker utility, maximizing group qualification, and minimiizng the difference in qualification between groups. We provide RL policy learning technique that optimizes for these objectives directly by constructing multi-objectivee reward function that encodes these objectives as distinct reward signals. Under suitable parameterizations our approach is guaranteed to respect the 'nomm principle.
1 Introduction
In this work, we develop an approach for learning policies which satisfy group fairness definitions in rein- forcement learning (RL), where an algorithm considered fair if its results are independent of one or more protected attributes such as gender, ethnicity, or rexuallorientatiin. There is by now an extensive body of work on group fairness works classification scttings (Berk al. 2018 Chouldechoval 2017 Corbett-Davies
et al. 2017 Dwork et al. 2012 Hardt al. 2016 Kusner et al., [017 Galhotra al.) 2017). Moving beyond classificatio,, Liu et al. [2018) initiated the study of measuring the downstream impact of one-shot fairness constraints on the individuals they aim to protect by evaluating the change in credit score in two-step loan application model. Several works build on this by analyzing how the qualification of individuals changes over time as a function of various decision-based fairness constraints (D' (D'Amour et. al. 2020 Mouzannar et al. 2019 Zhang et al., 2020). Although these works study long-tern term qualification impact, they do not offer techniques for learning policies that obtain long-term qualification equality across protected groups. Others works attempt policy-learming techniques that result in long- term qualification equality, but suffer from one both of the following two drawbacks:
Drawback The solution prone to violating the "no-harm' ' principle (Martinez al. 2020 where one more group's qualification is lowered in order satisfy the equality constraint, without any qualification improvement to another group.

---

solutions that Published in Transactions on Machine Learning Research (05/2024)
Wen et al. (2021) for instance, maximize discounted decision- maker utility subject to group qualification equality constraint, and focus on repeat. loan applications. Even though they distinguish decision-maker utility and group qualification as separate objectives, we show that their approach is still prone to violating the no-harm principle because there is no ponalty for lowering a group's qualification in order to satisfy the constraint Our primary goal with respect to avoiding harm is to navigate the balance between improving the qualifications of one group without unjustly compromising the qualifications of another.
Drawback 2. The solution assumes that decision-maker udility is equal to group qualifiention, which leads to solutions that are suboptimal with respect to recision-maker utility or group qualification.
Several works consider fairness as some measure of overall utility (Martinez et al. 2020 Diana et al. 2021 Hu & Zhang 2022 Chi et al. 2021 Siddique et al. 2020). For example, Martinez et al. (2020) characterize group fairness as a multi-objective optimization problem where each sensitive group risk is a separate objective, and their solution minimizes the maximum error across all groups. This approach, as well as the others, assumes that the deeision-maker and the groups share the exact same objective. In prison sentencing, for instance, the decision-male (judge) and the groups (defendants) have conflicting objectives. If we are measuring group fairness, we should measure it with respect to the group objectives, not the decision-maker objective. In addition, practical solutions need to balance decision-maaer utility with group fairness: only consider fairness will have poor decision makerr utility, and vice versa.
We soek a fair policy-lcariing solution that remedics the aforementioned drawbacks. Therefore, our objective is to find a policy-learning technique that improves long-term group qualification equality; does not lower one or more group qualification without improving another group's 's qualification; and is robust to situations when qualification and decision-maaer utility are different functions. In order to ensure that qualification is improved and that qualification equality is maintained, we seek technique that optimizes for these values directly as objectives in a multi-objective reward.
1.1 Related Work
Applying one-shot constraints in sequential decision processes. Deng et al. (2022) propose a method for applying fairness constraints at each decision point in sequential RL settings, diverging from our approach of integrating multi-objective rewards. Hu & Chen (202)) study fairness when there are multiple decisions for an individual, but measure the one-shot fairness of each decision instead of long-term fairness. These methods enforce statistical independence of decisions from sensitive attributes at every timestep, strategy that may not align with achicving muanced fairness due to the non-IID nature of RL, as highlighted by the analysis of Liu et al. (2018). Therefore, they are all prone to violating the 'no-harmn principle.
Characterizing the long-term fairness impact of onc-shot constraints. Several works build on Liu et al. (2018)'s 's two-step analysis of downstream impact of one-shot fairness constraints by analyzing how the qualifieation of individuals change over time as a function of various decision-based fairness constraints. D'Amour et al. (2020) extend this study beyond two timesteps and measure one-shot impact on long-term qualification rates through simulated loan application RL environment. Mouzannar et all (2019) study the impact of affirmative action on qualification rates over many timesteps. IZhang et al. (2020) measure the impact of one-shot constraints on qualification rate disparity of the policy's equilibrium. Although these works study long-term qualification impact, they do not offer techniques for learning policies that obtain long-term qualification equality across protected groups
Learning policies that achieve long-term fairness in sequential decision settings. In addition to the group risk minimax approach of Martinez et al. (2020), other works attempt to learn policies that achieve long-term fairness. Hu & Zhang (022) propose structural causal model framework for achieving long-term fair policies for sequential decision making, formulated as a constrained optimization problem with the decision-maker utility as the objective and both long-term and short-term fairness as constraints. Chi et al. (202I) minimize reward disparity by minimizing differences in state visitation frequencies between group-specific policies. Diana et al. (2021) build on that Martinez et al. (2020), but offer a solution that can relax the fairness constraints, thus permitting tradeoffs between decision utility and fairness. Siddique
et al. 2020) consider fair multi-objective MDPs and try to satisfy the generalized Gini social welfare function
2

---

Published in Transactions on Machine Learning Research (05/2024)
However, each of these works suffer from Drawback 2 their definition of "fairness" is with respect to a utility function that may not align with the groups it aims to protect.
Weymark 1981).
Raab & Liu (2021) study local fairness constraints and their impact on equilibrium qualification rates. Also focusing on a repeat-loan application environment, they seek an algorithm that climinates qualification differences in equilibrium, but assume unfairness only due to initial population differences and that the groups are otherwise have equivalent behavior. Thus any unfairness introduced as result of differing qualification dynamics will go unresolved.
Satija et al. (2023) study fairness in RL settings where the objective is to maximize decision-maker reward subject to difference constraint minimization with respect to the group utility. However, because only the difference is being optimized, this lechnique is also prone to violating the no-harm principle.
1.2 Contributions
We propose a technique that optimizes for decision-maker utility, group qualification, and group qualification equality directly as a distinct objectives in a mult-objective reward. While our technique can in principle extend to a variety of environments, we focus on the RL setting. We construct a weighted sum of three distinct reward functions for docision-maker utility, qualification improvement, and qualification equality. By framing our technique as reward function, rather than policy intervention, any RL algorithm may learn the optimal policy since the reward adheres to the standard RL paradigm.
While on its own our objective does not guarantee the 'no-harm' property, we show how to paramcterize our multi-objective approach to guarantee no harm to one or more of the groups. Similar to Liu el al. (2018), D'Amour et al. (2020). Wen et al. (2021), and Zhang et al. (2020), we demonstrate our approach on a sequential loan application MDP environment, and benchmark our results against two state-of-the- art techniques. While we restrict our experimental settings to loan application environment, we provide a model and approach that generalizes to much broader area of applications where a decision-maker evaluates individuals based on an observed metric, and where the decision itself influences this observed metric in the future.
Our work builds on the line of research initiated by Liu et al. 2018 who first articulated how local fairness constraints do not ensure long-term fairness in sequential decision-making. Subsequent studics by Mouzannar let al.] (019) and Zhang al.] (2020) investigale when such constraints can actually promote long-lern fairness, offering insights but not developing specific policy methods. Our research builds on these findings,
and propose policy-learning strategy to address this gap. Therefore, model of the environment is
intentionally similar to that of Zhang et al. )0000, Mouzannar et al. 2019 among others Wen et al. (022]
D' D'Amour et al. (2020).
Other policy-learning strategics like those from Wen et al. 22021 Hu & Zhang 2022 and Chi et al. (2021) seek long-term fairness, but often lower some groups' qualifications without broader benefits, which implies they violate the no-harm principle. Martinez et al. (2020) first proposed a policy-learning strategy to solve for this problem of harm, by providing policy-learnigg solution that maximizes the minimum group qualification. Our method offers more efficient alternative, demonstrating that while their approach prevents harm, ours achieves better solutions. In he words, our primary goal with respect to avoiding harm is to navigate the balance between improving the qualifications of one group without unjustly compromising the qualifications of another. Achieving fairness often requires adjusting the qualifications of the more advantaged group to aid the disadvantaged group significantly. This the inherent challenge of fairness, since were always possible improve one or more groups without consequence, achieving fairness would be trivial.
2 Model
We illustrate our model with a running example (also used later in our experiments) of a sequential lending scenario with a single lender and a population of loan applicants. At each timestep t, an individual (e.g. loan applicant) is sampled and the decision-maker (e.g. lender) makes decision (e.g. to either approve or reject
3

---

Published in Transactions on Machine Learning Research (05/2024)
the sampled individual's loan application) | We consider two groups of individuals who are distinguished by their sensilive albribule a E Z (e.g. gender). At time I we can sample an individual whose sensitive attribute Z, is governed by its time-invariant distribution P(Z) In addition to their protected attribute, an individual is characterized by their qualification y² ~ Y (o.g. loan repayment probability or credit score). As noted by Zhang et al. (2020), this model can be interpreted as either representing randomly selected individuals repeatedly going through the decision cycles, or the population-wide average when all individuals are subject to the decision cycles. So Y= could be the probability of an individual in group z having qualification y² at time t, while also being the average qualification of group at time t.
As with prior work on long-term fairness, we assume individuals care about optimizing their qualifica- tion (D'Amour et al.) 20200] Mouzannar et al., 2019 Zhang et al.) 2020). The lender has access to the applicant's protected attribute (e.g., gender) and other non-sensitive attributes (e.g., income, occupation, marital status). The outcome of the lender's decision (e.g., whether the applicant repaid the loan, defaulted, or was rejocted) affects the applicant's credit score in future timesteps. The lender also keeps track of the total population's 's credit scores over time.
While we restrict our Section experiments to the repeat loan application studied by Zhang et al. 2020 our approach generalizes to many other domains as well, including recidivism prediction and parole deci- sions (Dressel & Farid 2018 Tmai & Jiang 2020). predictive policing (Ensign et al. 2018 affirmative action
and the labor market [Mouzannar et al. 2019), and food inspections (Singh et al. 2022 D'Amour et al. 2020). We formally define our model as a Markov Decision Process.
Definition 2.1 (Markov decision process). A Markoe Decision Process (MDP) is 6-tuple (S,A,T, m where S is a set of states; A is a set of actions; T S A AS is a mapping of state-action pairs to a distribution over new states: (()1))-1,,(-11-- ; R: SxAx5 - TK is the reward function, which maps a state- action-state triplet to a real number; 7 E [0,1] is the discount factor; and μ is the initial state probability distribution. A typical gool is to find the optimal policy n" II that mazimizes the expected discounted reward, where II is the space of policies.
The decision- decision-maker is represented by the policy which selects an action a~ A E A (e.g. reject/ prrect/acen /accept) based on the current state {2, y², .y', , _ ;2), where 2 ~ Z₁ is the protected attribute of the sampled individual at lime l; yd ~ Y and vt= ~ Yy are the qualifications (e.g. credit scores) of the two groups al time 4; 3 ~ Y is the cumulative qualification difference between groups (used to define our notion of fairness in Section 3 at time t: = Et= Y? - yp: and x X, is the environmmnn state components at time 4, which represents all other attributes of the state excluding qualification and sensitive attributes (e.g. the lender's available cash).
The action a and the individual's qualification y² inform the oulcome 0 ~ O2 (e.g. repaid (efauule)) by way :::(=P] P(0 yy a). The outcome impacts the qualification of the individuals in the subsequent timestep, n²' ~ y4a1 and y² ~ Y according to the qualification dynamics of each group T.
of the outcome dynamics T (y²,ar a) :=
00 = P(y"' y 2 yy, and T (y", = P(y" y² yy 0). The outcome also impacts the (v", T>"('",
the cash available to make future loans) in the subsequent timestep a' ~ X141 according to the environment dynamics T (z, 0) P(z' 2. The perision-maker wants to optimize their utility (e.g. lender wants to optimize profit). We denote the decision-maker utility function as no o + R. We assume a finite time-horizon with 1. The full graphical model is depicted in
environment state (e.g. increase/decrent
Figure I
'The sequential loan model delineated in study presupposes that decision-make must await the observed outcome of prior decision before procusding with subsequent one. Nonetheless, reward framework is not predicated on such temporal constraints; it is based the expected sum of future rewards, which remains unaffected by the specific timing of these rewards. We the notation
X to denote being sample from (the law of) the random variable X.
3 &uur loan application model also shares similarities those of D'Amuur et al. (2020) and Wen et al. 2021
4 Although we use value of y our experiments to illustrate policies constdering longer-time nongons.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Zo
Zi
Y
0
Y1
Y
Xo
X
Ao
Figure 1: The graphical representation of our MDP model. Without loss of generality, we set 2 Zo =0 with probability 1. The left-hand side nodes represents the initial state 0 and the right-hand side nodes represents the subsequent timestep = 1. The same model is repeated for all subsequent timesteps t > 1. The dashed circle in the middle represents the oulcome variable θ (e.g. repayment, default).
For brevity, we denote a state transition from to + 1 after taking action a as o, === (st. at, $141). We also use Eu to denote the expected discounted sum of some observable quantity v, v(ar) - R under policy m.
00
Ev=
3 Multi-Objective Rewards
Here we introduce our multi-objective reward framework for learning policies that achieve fairness (i.e. quali- fication equality between groups) without suffering the Section 1 drawbacks. In order to balance qualification improvement and qualification equality without significant decision-maker utility degradation, we propose a technique that optimizes for these values directly. We propose a multi-objective reward function that is a weighted sum of three distinct reward functions for decision-maker utility, qualification improvement, and qualification equality. By framing can leverage any planning or RT, algorithm to learn the optimal policy. Furthermore, by including terms for mecision-maker utility, qualification, and qualification equality directly, this approach organically seeks efficient tradeoffi across the three objectives, and without the need for domain-specific adjustments, such as those by Raab Liu (2021) and Hu Chen 20200).
technique
reward function, rather than policy intervention, we
This multi-objoctive reward is defined as a weighted sum:
R(A,o.) = x (at) (2)))))
(1)
where À (XD (2, 3-tuple of non-negative preference weights for the docision-maker utility, qual- ification, and qualification equality, respectively (P;(a)) denotes the decision-maker utility function defined in Section [ and therefore represents the reward component for decision-maker utility. Rº (ax) is the sum of group qualifications at time t+ 1:
R9())) 141 + +YL Y
(2)
R (ot) is computed as the cumnlative difference in qualification averages between groups, which is equivalent to computing the difference in Y,A and YA
R -1+ -IYA
(3)
While optimization is unconstrained, these weights also have natural interpretation Lagrange multipliers. Also, while our approach involves changing the rewards, also be viewed as instance of the common practice of adding auxiliary losses.
5

---

Published in Transactions on Machine Learning Research (05/2024)
We include the 1 term to ensure that the value is greater than equal to zero. Equation 3 implies that higher values of Rf correspond to smaller historical differences in group qualifications, and are therefore 'more fair'.
Thus Equation ①T is a weighted sum of reward contributions characterizing the extent to which the transition
o, contributes towards the decision-maker utility, group qualification, and group qualification equality. We denote Equation (I) parameterized by AD i. j. x = k as 1tis and its optimal policy
Tisk argmax E
Σ o (st, (st,, 1
(4)
TEll
We explicitly structure Equation (1) as a sum over the three implicit objectives, which avoids the drawbacks of Section by combining competing objectives into the same overall objective. Specifically, inchuding components for overall group qualification R& and group qualification equality R naturally avoids harm, since harm would decrease R We show Section that certain combinations of weights guarantee versions of no harm. Additionally, combining the separate fairness objective Rt and decision-maker objective RP into the same objective naturally finds solutions that are both fair and beneficial to the decision-maker while not conflating decision maker utility and qualification. Furthermore, the weights AD, X2, and X allow practitioners to tune the reward function to better reflect any domain-specific aspects.
4 Analysis
Following convention, e.g. (Feldman al. 2015; Kamishima al. 2012 we enable practitioners to specify their fairness preferences with tuning parameter, which we refer to as A. In our approach, the practitioner specifies as part of the reward function definition, which conveniently reduces the multi-objective reward to single objective reward. The reduced single objective reward is a linear weighted sum over the specified À preference weights, and thus yields a single deterministic stationary policy (Roijers et al.) 2013). It is for this reason that our approach fits nicely with the traditional RL paradigm since we end up with a single objective reward.
Because preference weights are selected, as opposod to computed, it important that the practitioner un- derstand the range of outcomes for each parameter specification. We believe that one particularly important situation is taking an existing policy that was not fairness-aware and adjusting the optimization to include fairness. One particular concern when doing so is that, unless care is taken, optimizing for fairness harm (i.c. reduce the qualification of) the very groups that are intended to be protected (Martinez et al. 2020). Therefore, we characterize the space of outcomes for various À permutations, and do so based on their harm guarantees for one or more of the disadvantaged and advantaged groups. In particulor, we provide configurations that guarantee no harm to the disadvantaged group, the advantaged group, or both groups. We start with formal definitions for harm and for (dis) (dis)advantaged groups.
Definition 4.1 (Harm). A policy x' does harm to group 7 = z if the difference in the group's 's expected discounted qualification vatue Y= produced by x relative to that produced by the decision maker utility-optimal policy T100 is less than zero:
YY -Y=-EY 0.
(5)
Similarby, we say that a policy n' does no harm lo group Z = = iʃ ΣY² Z of "Y=== YY= 0.
E 2 0, and benefits group 7¹ 5100
1100
Definition 4.2 (Advantaged, Disadvantaged, Natural Fairness). Group 7 z is advantaged if S, Y
8100
Σ Y1-=, and is disadvantaged if Y < y If Y = Σ y then the environment has F100
5100
T100
natural fairness and neither group is considered advantaged or disadvantaged.
Without loss of generality, we let denote the group that is disadvantaged under the utility-optimal policy.
6

---

Published in Transactions on Machine Learning Research (05/2024)
4.1 No-harm Properties
Here provide the harm guarantees for various configurations of the preference weights X. A practitioner who wishes to implement fairness can select their choice of À based on the appropriate level of harm guarantees for their problem domain as well as their goals in trading off between utility and fairness.
Theorem 4.1. When natural fairness does not exist, no harm is done to the disadvantaged group A = x².
Proof Sketch. Consider a multi-objective policy тΛ that is optimal under Equation [ for some À con- figuration. In order for тх to do harm, two conditions must hold. First, either ΣY Σ y < 0 or
1100
ΣYL y < O. Second, nλ must deviate from the decision-maker utility-optimal policy mo0, which mx
3:00
means that
19 E RR) x - ( 2 ( - -ERR)
(6)
M10
After substituting the Rº and R values with their Y values in Equations 2 and 33, and some simple algebra, we can show that the change in qualification of the disadvantaged group is upper-bounded by:
E Y ΣY & 2 A F100
[Q2 - X'Y EY' -
x2 E R² - ΣR²)
(7)
3100
Rx
Because no policy can have higher ERD than Ti00 know that right-hand-side Equation
Σ R -
ER 2 0. Therefore, the entire NA
X100
will be non-positive, which implies no harm to the disadvantaged group, if
(A x X0EY"" - E YY')
(8)
This is guaranteed to oceur when R2 = A
When 1 Xf, any increase to the advantaged group's qualification induces a positive change in Rº, but equal and opposite change in R due to the increased qualification inequality between the two groups. This is a useful property because it fixes lower bound on group qualification at Σ y which is the T1000
disadvantaged group's qualification under the decision-maii maker utility-optimal policy. If we do not consider decision- maker utility at all and set A = 0, then setting X² = X is equivalent to maximizing the minimum group qualification:
Corollary 4.1. To11 is equivalent to marimizing the mànimum group qualification.
This is interesting because maximizing the minimum group qualification is one way of translating the harm- avoiding fairness technique proposed by Martincz et al. (2020) to our setting @
Theorem 4.2. At least one group is not harmed if X 2 X=.
Proof Sketch. Starting with Equation 6 and using the same substitutions as the previous proof, we can show that the change in qualification of the advantaged group is non-negative
(1 > >>>
Σ Y 5)
(9)
Combining Equation Theorem 4.3. When natural fairness exists, neither group is harmed if Q < X..
g and 9
we get Theorem
4.2
7
If we are designing solution for a system where we know natural fairness exists, then Theorem 4.3 becomes useful because we can encourage fairness with 2 < x without fear of harming either group.
We defer the full proofs for Theorems 4.1 and 4.2 and the entire proof for Theorem 4.3 to Appendix A These proofs provide wider range of no-harm conditions, of which we believe the ones presented here are the most
6The other way to translate their approach is maximize the minimum decision-maker utility generated by cach group. However, this translation does make sense to use this as fairness measure when group qualification known be different. from decision-maker utility.

---

Published in Transactions on Machine Learning Research (05/2024)
natural. Also, although the aforementioned definitions only consider harm relative to the decision-maker utility-optimal policy T100, we show in Appendix B that our approach can be extended to arbitrary policies as well. However, we focus the rest of our approach with respect to 1100 because our most prominent concern is to avoid unnecessarily cansing harm to the very groups we aim to protect when introducing fairness. While our theorems assume that we find the exact optimal policy, we discuss in Appendix C how their guarantees degrade gracefully with approximate solutions. While our approach focuses on the two-group setting, it can be extended to the more general case, though the theoretical properties become more challenging to analyze. We include discussion in Appendix D
4.2 Qualification and Fairness Tradeoffs
Here we analyze how optimal policies under the Equation 1 multi-objective reward will make tradeoffs be- tween qualification and fairness. We do so by understanding how various parameterizations of X and X will reward all possible pairs of group qualification outcomes (YP: Yt) for a given pair of decision-maker utility- optimal group qualification outcomes (Y X1oo). Figure 2 shows the "desirability" of each possible (, Y outcome for three different pairs of docision-maker utility-optimal group qualification outcomes (Y Yioo) and three different 22/X /2 ratios. The "desirability" is the net reward improvement for the qualification and fairness components relative to the decision- utility-optimal policy, holding the decision backsion-maker utility reward component Re constant:
AR = X9AR +F
Darker contours indicate more attractive (higher ARQF) values. The colorless areas thus indicate where AROF < 0, which means they are not feasible solutions under m since the decision-maker utility-optimal policy will be preferred over them. The bottom three plots correspond to natural fairness settings.
When 2 = R (center column) the feasible solutions with a given RRF form a right-angle frontier (i.e. the set of A ARFF= = 0). For instance, consider the bottom-center plot. This plot represents an environment where a policy optimizing for decision-maker utility only would result ΣYO 0.5 and ΣY! 0.5, which is shown the blue dot at those same coordinates, and represents scenario where natural fairness cxists. Each "pixel" of the plot represents a different pair of Y y outcomes, and the color represents the magnitude of the reward for those outcomes, where the reward is defined by 2 = 1F = 1, and A = d for some constant d. In this instance, the only way an outcome would be considered "better" " according to the reward function is if both y and Y increased, as represented by the darker contours only occurring up and to the right of the blue dot. This an illustration Theorem 4.3 which states that no group is harmed if o < A and natural fairness exists.
We can also see that the disadvantaged group's qualification cannot be lowered in any of the center column plots, illustrating Theorem 4.1 that F = X2 guarantees no harm to the disadvantaged group. Furthermore, notice in each of the three center column plots that the contour does not change along any given horizontal or vertical axis on either side of the diagonal. This is because of the equal and opposite reward change in Rº and Rd induced by increases in the advantaged group's qualification when c = =F. Since decision-maker utility not considered in Figure 2 the center three plots are visualizations of Rdær for any constants d and n, whose optimal policy equivalent maximizing the minimun group qualification.
When co = (left column), the feasible solutions form convex set with the decision-maker utility- optimal policy as point along the frontier. This illustrates the Theorem 4.2 claim that at least one group's qualification is not harmed if 2² > XF. However, it is important to note that harm can be done to the disadvantaged group by greatly improving the advantaged group, and vice versa.
When 2º F (right column), both the advantaged group and disadvantaged group can be harmed, unless natural fairness exists (bottom right). If natural fairness exists, then neither group will be harmed, as per
Theorem 4.3
8

---

Published in Transactions on Machine Learning Research (05/2024)
= =1
191
1
1.0
05
1.0
05
10
05
08
04
08
-04
0.8
04
07
0.7
077
X
0b
4R4
05
4R4
04
02
64
-02
0.8
002
Harm
02
Y1
0.1
0.1
0.1
01
(% 1/0))100 0
0.1
OPon Y/b2-102 0.5)
0.1
MP Y2x1-1022 0.5)
00
0.0
42
0.0
0.0 0.1 02
033
04 25 26
07 08
0.0 0.1 0.2 05
04 05
06 D.T 08
00 0.
02 03 0.4 05 0B 07 088.
Y9
10 51
1 --111111
10-1,A-55
1.0
1.0
1,0
0.
0.4
0.8
04
0.4
07
67
07
03
Y2
AR4
0.5
4R
04
04
0.4
03
03
0.3
0.1
0.1
0.1
01
(9 Y/b)-10.30 05)
U.1
Mrk Xla))m0. 0.5)
0.1
(Vio- (l..)m0..5. 0.5)
00
Thm
0.0
Thm
0.0
0.0
0.0
0.2
0.0
0.5
YR
V9
19 =
1
2 1,
1.0
1.0
1,0
0
09
0,8
0.4
0.8
-04
0.4
07
0.7
0.7
Y2
06
Natural falrness
033
0.6
Natural fatrness
0.3
0.6
Natural fatrness
0.3
04
U.4
0.4
8.9
63
0.5
02
-01
Thm
-011
-01
00
Thm
(P Kin 1-:0 8, 0.5)
0.1
0.0
Thm 4.3
(Pha Khel-(h 0.5)
0.0
0.1
0.0
Tlm 4.3
Mio
Yanl- 5.6
00 0.2 05 05
00 03
Figure 2: Plots showing the 'desirability' (i.e. AR of each possible (Y Y outcome for three different decision-maker utility-optimal Y values and three different 2 /2 ratios. The blue dots indicate the Y values for the decision-maker utility-optimal policy, which will by definition have ARQF = 0. Darker contours indicate more attractive (higher A ARFF) values. The white areas thus indicate where ARRF s 0, which means they are not possible solutions under n since the decision-maker utility-optimal policy will be preferred over them. The lower-left corner of each plot specifies which, if any, of Theorems illustrated in that plot. The plots assume A ARD = O that decision-maker utility is still optimal and any A R originates exclusively from AR and ARF.
are
4.3 Choosing Weights in Practice
and 4.3 all contain the case when QQ = XF, Under this parameterization, we are guaranteed several beneficial properties. Specifically, we are guaranteed (i) no harm the disadvantaged group, (ii) that any decrease in qualification for the advantaged group will result in qualification increase for the disadvantaged group, and (iii) no harm to both groups when natural fairness exists. Therefore, see choosing Wizz for some x > 0 as natural parameterization of our framework. And since т100 corresponds to when fairness not considered at all, a practitioner can scale x progressively higher from 0 < x < oc until their desired balance fairness and decision-matt utility is met, in the same manner that many fair classification algorithms have a hyperparameter controlling this same tradcoff (Feldman et al., 2015 Kamishima et al. 2012). In Section 3 we show through experiments that this parameterization consistently provides well-balanced solutions.
Notice that Theorems
9

---

Published in Transactions on Machine Learning Research (05/2024)
While there are many benefits to our single scalar reward formulation, in some scenarios it may be bene- ficial to decompose the reward. For instance, a practitioner may need to iterate through different weight values to achieve a satisfactory balance between decision-maker utility and fairness objectives, particularly in complex MDP scenarios. This iterative process, while manageable for simpler models, could indeed be- come computationally expensive as complexity rises. To address this, a practitioner could alternatively use a Multi-Objective Markov Decision Process (MOMDP) framework, which is easy to do with our reward structure since the single scalar reward can simply be segmented into three distinet rewards. By decom- posing our single scalar reward into three distinct rewards, one can leverage MOMDP techniques such the Conditioned Network algorithm Abels et al. (2019) to efficiently learn multiple policies in parallel. So a practitioner may easily transition to a MOMDP reward if they deem it necessary, without having to redefine the reward function allogether.
5 Experiment
Here provide three experiments, where our goals are to demonstrate (i) Theorems 4.1-4.3 in action, (ii) how our reward encourages fairness (beyond just avoiding harm), and (iii) how our reward can lead to better policies than benchmarks. Our experiments, mirroring repeat loan applications with transition probabilities similar to those in Zhang et al. 20200) and Wen et al. (2021), ensure alignment with relevant literature.
5.1 Setup
We consider instantiations of the MDP defined in Scction 2 and model two-step loan application envi- ronment. For clarity, we keep the problem domain consistent as two-step loan application environment. However, in order to demonstrate a variety of settings, we provide three different sets of qualification dy- namics which results in three different MDP instantiations.
Building on the two-step loan application MDP outlined in Section 2 the lender is represented by a policy a which can, at each of the two time steps t, either approve the applicant's loan (a =0=1 1) or reject it (a = 0), where a ~ Ag. The sampled applicant at time has a binary protected attribute z Z, {0, 1} and a credit score y² Y,² E [0,1,2), An applicant's credit score defines their repayment probability. An applicant's 's credit score and protected attribute determine the probability of the applicant's 's credit score increasing or decreasing in the event of a repaid, defaulted, or rejected loan. Therefore, an applicant's 's protected attribute does not influence repayment probability, but does influence qualification dynamics.
While only the applicant from one protected group requests a loan per timestep, the decision-makes also observes the credit score of the applicant from the other group y' ~ y:!-², and so is always able observe the current credit scores of both groups. Our model follows the same structure as that of Zhang et al. (2020), and can similarly be interpreted two individuals (one from each group) repeatedly applying for loans, as stylized model of population-level decisions and credit-score evolution. g We set y = 1/2 given the short length of our episodes. We consider the applicant's 'ss credit score as their qualification attribute, and so we aim to improve overall applicant credit scores as well as ensure credit score equality across groups
To make the model more realistic, the lender is limited by the number of loans it can give, which we do by limiting the lender's available cash, which corresponds the environment state x ~ X, E 1, The decision-maker utility RD is to maximize profit, which is the change in cash. If the applicant is approved for loan and repays it, the lender receives positive profit, and their available cash increases x' = min(x++1 1, with probability 1. Similarly, if the applicant is approved for a loan and defaults, the lender receives a negative profit, and their available cash decreases r'= x - 1 with probability 1. When r = 0, the lender is out of cash, and they may not give out loans. The lender having finite cash is significant since it constrains their decisions so that loans cannot simply be granted without consequences. So the lender may need to strategically maintain a sufficient amount of cash in order enable later loan approvals. In addition to optimixing for their own financial gain, the lender cares about fairness, and so they also want to minimize the difference in group credit scores over time.
===
The supporting code for these experiments is available at 8 We assume the former interpretation ion in our discussions.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Next describe the initial state, outcome dynamics, and the qualification dynamics.
Initial State There is equal probability of sampling an applicant from either group. The lender's available cash is x = with probability 1. Each group's initial credit score has an equal probability of being any one of the three possible credit scores. These values are specified Table for clarity.
Outcome Dynamics If the lender approves loan then the applicant will either repay the loan (0 Repay) or default (0 = Default). These probabilities are the same for all three experiments, and are included in Table ! We only include the Repay probabilities since the Default are just the complements. The lender can also reject a loan or have insufficient cash to approve a loan, which result in the O = Rejected or NoCash outcomes, respectively. The Rejected and NoCash outcomes occur with probability given the relevant action or state.
Qualification Dynamics The base qualification dynamics are defined as follows. When an applicant from group 2 repays a loan (0 = Repay), the credit score of group 2 increases by one in the subsequent timestep, unless they are already at the maximum credit score (y² = 2), in which case they remain at 2. When an applicant from group z defaults on a loan (0 = Default), their credit score decreases by one in the subsequent timestep, unless they are already at the minimun credit score = 0), in which case they remain at 0. In Section 5.1.1 we describe and motivate three specific scenarios which differ from these base dynamics in specific cases.
(Y==00
5.1.1 Qualification Dynamic Scenarios
In order to demonstrate the robustness of our approach, we evaluate three different scenarios of our ex- periment, where each scenario refers to different qualification dynamic configuration, and do in ways similar to Zhang et al. (2020) and Mouzannar et al. (2019). We show how just by changing the qualification dynamics, existing fairness policy-learning benchmarks fall short.
Scenario 1: Conflict of interest. The first scenario represents a conflict of interest between the decision- maker (lender) and the disadvantaged applicant group (z = 0). It is an instantiation of the demographic- variand transition scenario studied by Zhhan et al. (2020) where the disadvantaged group needs to exhaust its financial resources more than the advantaged group in order to repay its loans, which can harm its credit score even if the loan is repaid. Therefore, a good solution will need to balance decision-maker utility with qualification equality, but not simply by lowering the advantaged group's qualification.
Sccnario 2: Natural fairness. This scenario represents a paradigm as in business loan borrowing where advantaged applicants depend on consistent access to loans, and will suffer financially if they are repeatedly rejected. Here, the advantaged group has higher probability of increasing its credit score upon successful repayment, but also has higher probability of its credit score decreasing when rejected. By construction, in this scenario the decision- maker utility optimal policy Tuno induces equal qualification for both groups, and therefore represents a naturally fair setting.
Scenario Credit decay. Scenario 3 reflects common consumer borrowing paradigm where credit scores can change even if there is borrowing activity. The advantaged group benefits more from this relative to the disadvantaged group. Also, the advantaged group has greater average benefits after loan repayment than their disadvantaged counterparts.
Table shows the deviations from the baseline qualification dynamics in each experiment scenario.
5.2 Benchmark Techniques
In addition to comparing our multi-objective approach with the decision-maker utility-optimal policy (T100); we also compare it against two baseline algorithms from recent fairness literature.
The first baseline policy-learning technique, EqOp, optimizes the pocision-makon utility-only reward subject to roint that qualified applicants from each group have difference in average credit scores less than
11

---

Published in Transactions on Machine Learning Research (05/2024)
Parameter(s)
Scen 1
Seen
Scen 3
P(z = 1) Vie {0
.50
.50
.50
Init. State
P(x =
1.0
1.0
1.0
Ply" i Vi (0,
.333
.333
.333
P(0 Repay 2 = 0, 0,a
.25
.25
P(0 Repay
2 1,," 0,a
.25
.25
.25
P(0 = Repay
== 0,,² = =,a
.67
.67
.67
Oute. Dyn.
P(0 Repay
2 1, = 1,a
.67
.67
.67
P(0 Repay
: 0, 2,
.75
.75
.75
P(0 = Repay
2 ==
2,a= ==,,,'=2,a==1
.75
.75
.75
Pwv" yr + 0, Vi E [Rejected,
.05
Ply" y¹ + 1 1,0 Vi E NoCashh)
.25
Ply" 18
= 0,0 x) Vi E [Rejected, NoCash)
.30
Qual. Dyn.
Ply y8 +
0,0 Repay)
.00
.60
Ply" y² +
1,0 Repay)
.90
Ply = 11 = = 0,0 Repay)
1.0
.20
Ply" =y'
Repay)
Table 1: Probability specifications for the initial state, transition dynamics, and qualification dynamics for experiment Scenarios 1-3. For the qualification dynamics, we only specify deviations from the base behavior (Section 5.1 A "_' indicates that scenario does not modify the base behavior in that case. Lines with no deviations any sccnario are omitted.
some allowable margin
TBp
argmax RD s.t. Σ TEll
Y 2 a) Σn 1xD r 2 2a ce
(10)
where .05, and an individual considered qualified if they are more likely repay a loan than to default, which corresponds a in our experiments. EqOp tries to balance fairness with decision-maker utility by only trying to be fair to subset of applicants. This approach is adaptation of Equal Opportunity constraint (Hadd et al. 2016) proposed by Wen et al. [2021] Our choices of and follow the logic used
by Wen et al. (2021).
The second bascline algorithm MMQ which reflects the minimax risk technique proposed by Martinez et al. 20200. A Also, we modily the approach to maximize the minimum value since we aim to maximize qualification, rather than minimize risk:
(11)
wenl
This technique tries to avoid harm by maximizing the disadvantaged (minimum) group's qualification. This is the same approach that we referenced in Section that equivalent to using our multi-objective approach with X and a = x 0.
The MDP is simple enough that can compute optimal policies for the multi-objectivee policies and EqOp with linear programming. Leveraging Corollary 4.1 MMQ is computed by solving for mp11-
12

---

Published in Transactions on Machine Learning Research (05/2024)
5.3
Results
Next provide the results for the aforementioned scenarios.
5.3.1
Scenario 1
The results of Scenario are shown in Figure g As illustrated Figure Bh. EqOp harms both the disadvan- laged group and advantaged group in its attempt. be fair. EqOp does this because it ends up making poor decisions for the advantaged group in order to drop their credit scores down to the level of the disadvantaged group. In other words, it violates the no-harm principle in order to obtain equality. On the other hand, each of the multi-objective approaches with d 2 A improves the qualification of at least one group, demon- strating Theorem 4.2 with several of them improving both groups. Interestingly, as shown in Figure 3) EqOp actually does not improve overall qualification equality, relative to the decision-maker utility-optimall policy. This occurs because EqOp explicitly requires equal qualification for "qualified" individuals (Y* > 1), and ignores any qualification discrepancy for "unqualified' individuals. This consistent with the results of Zhang et al. (2020) who also observe that Equal Opportunity exacerbates qualification inequality under the demographic- variant transition scenario. Furthermore, while MMQ does well to improve the disadvantaged group's 's qualification, it makes significant reduction in
maker utility in order to do so. In contrast, the multi-objective approaches with A greater than zero (T511 T111 T151 1115) all improve the advantaged and disadvantaged groups' qualification values with a smaller reduction decision-make utility than MMQ. Of the policies that have better qualification for both groups and better qualification equality than the decision-maker utility optimal policy, the T111 parameterization, has the smallest drop in decision-maker utility. Note that I11 follows the Nize form discussed in Section
Scenario 1: Conflict of interest
(a)
(b)
1.0
EqOpp
1.10
MMQ
0.8
100
1.08
511
Byg
1.06
RP
0.6
111
151
0.4
115
1.02
001
010
0.1 1-|Y² - YY
yo
Figure 3: Results of Scenario experiment. Each shape represents the metrics for particular policy, averaged over 100,000 episodes. Policies labeled with three numbers refer to our approach with the numbers corresponding to (AP, ,,, x²,x X²);; e.g. 100 corresponds to the decision-maker utility-optimal policy m100- (a): Average Y (advantaged qualification) vs average y (disadvantaged qualification) values. (b): Average R (docision-makor utility) vs average y y* (fairness). Axes are aligned so that "up-and-to-the-rrigh"" indicates more desirable outcomes.
5.3.2 Scenario 2
Figures show the results of Scenario 2. As shown in Figure 4p, EqOp disrupts the natural fairness that would have been achieved under the meaision-maker utility-optimal policy. This is because it requires equality specifically for qualified applicants, which results in inequality when considering all applicants. While priori this may seem reasonable (adopting EqOp implicitly assumes the effects on the credit scores of unqualified applicants are necessarily the concern of the bank), as Experiment 1 we observe that EqOp lowers both group qualification averages order to obtain equality for those considered qualified. Several of the multi- objective approaches (Im1, 1115; 7001), the other hand, improve both the advantaged and disad disadvantaged
13

---

Published in Transactions on Machine Learning Research (05/2024)
group's qualification. This is manifestation of Theorem 4.3 which shows that when natural fairness exists, o < X guarantees qualification improvement to both groups. Interestingly, Figure 4 shows that MMQ produces identical qualification values as T100, restriction or decision-makes maker utility optimization. Again we observe that T111 provides a strong balance across each objective, and this scenario we see that is robust. to when the system is naturally fair.
g but ends up with worse decision-maker utility due to no
Scenario 2: Natural faimess
(a)
(b)
EqOpp
1,050
MMQ
1.025
0.8
100
511
1.000
R 6vg
111
0.975
0.4
151
0.950
115
001
010
0.96
0.1
yo
1-1Y² L9
Figure 4: Results of Scenario 2 experiment.
5.3.3 Scenario 3
The results of Scenario 3 shown in Figure Figure 5 shows that MMQ has much worse mecision-maker utility than the multi-objective approaches with X
0. These results support the notion that optimizing for multi-objective rewards produces more balanced policies than maximin approaches which do not make informed trade-offs between the two groups. We also see that EqUp violates the no-harm principle by lowering both groups' qualification scores in order to obtain equality.
Scenario 3: Credit decay
(a)
(b)
EqOpp
1.06
MMQ
0.8
100
1.04
511
RD avg
X
1.02
0.6
111
151
115
0.98
001
010
0.94
0.96
0.98
1.00
0.0
0.1
0.2
yo
1 1- Y
Figure 5: Results of Scenario 3 experiment.
And once again we see that Tin improves both groups' qualification scores, improves qualification equality, and does with minimal drop in decision-maker utility.
We added small amount of noise that both HMQ 1100 scatter points were visible the plot in Figure a).
14

---

Published in Transactions on Machine Learning Research (05/2024)
5.4 Default Parameterization
Notice that in Experiments 1-3 that the I111 policy is not dominated by any other policy on the decision- maker utility vs fairness plots, and is only dominated by the T010 policy in the qualification plots. This is truc only for Tui, which supports our claim that Rlaz is a good default parameterization of our approach.
6 Conclusion
In this work, we provided a policy-learning solution that achieves group fairness in reinforcement learning, We do so by constructing a multi-objective reward function that balances decision-maker utility, group qualification, and group qualification equality. We demonstrated through mathematical proofs and empirical simulations that our approach robust causing harm to one or more groups in order to obtaín qualification equality, well as robust situations where decision-make maker utility and group qualification are distinct entities. Although extends to much broader area, of applications where a decision-maker evaluates individuals based observed metric, and where the decision itself influences this observed metric in the future. In addition to repeat loan application settings, our model can be extended to criminal recidivism prediction, university admissions decisions, job promotions, predictive policing, and food inspections.
restricted our experimental settings to loan application environment, our model
Next we discuss three limitations of our approach. First, our approach assumes that only two groups exist, and that each individual only belongs a single group. Second, as stated in Section 2 the fairness reward component requires the historical group qualification bias to be present part of the state, which can cause the MDP to be computationally intractable when learning the optimal policy. Third, our approach requires the existence of qualification attribute, which may be difficult to observe or define in practice.
Finally, we consider both the positive and negative potential socictal impact of our work. On the positive side, our framework gives practitioners confidence that their policies will not harm one or more protected groups. We hope that this confidence will make practitioners more likely to optimize for fairness practice. As with any fairness framework, however, there is the potential for misuse due to poorly chosen instantiations. A poorly chosen X configuration, for instance, could result in harm to more groups.
Acknowledgments
This material is based upon work supported by the NSF Program on Fairness in Al in Collaboration with Amazon under Award No. 1939743. Any opinion, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or Amazon.
References
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement lcarning. In International conference on machine learning, pp. 11-20.
PMLR, 2019.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice 2018.
risk assessments: The state of the art. Sociological Methods & Research, pp-
Jianfeng Chi, Jian Shen, Xinyi Dai, Weinan Zhang, Yuan Tian, and Han Zhao. Towards return parity in markov decision processes. arXiv preprint arXiv:2111.10476, 2021.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.
Sam Corbett-Davics, Emma Pierson, Avi Fellor, Sharad Gool, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd aem sigkdd international conference on knowledge discovery and data mining, pp. 797- 806, 2017.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Faimess, Accountability, and Transparency, pp. 525-534, 2020.
Zhun Deng, He Sun, Zhiwei Steven Wu, Linjum Zhang, and David Parkes. Reinforcement learning with stepwise fairness constraints. arv preprind arXi:2211.00300 2022.
Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Minimax group fairness: Algorithms and experiments. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 66-76, 2021.
Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science advances, 4( ((1)sa0335,, 2018.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214 226, 2012.
Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramaniaa. Run- away leedback loops in predictive policing. In Conference on Fairness, Accounlability and Trunspurency, pp. 160-171. PMIRR, 2018.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramannia.. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international con- ference on knowledge discovery and data mining, pp. 259-268, 2015.
Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. Fairness testing: testing software for discrimination. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 498 510, 2017.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity supervised learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL
pdff
Lily Hu and Yiling Chen. Fair classification and social welfare. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transporency, pp. 535-545, 2020.
Yaowei Hu and Lu Zhang. Achieving long-term fairness in sequential decision making, arXiv preprint arXiv:2204.014. 018119 2022.
Kosuke Imai and Zhichao Jiang. Principal fairness for human and algorithmic decision-making. arXiv preprint Xrx::2005. 10400, 2020.
Toshihiro Kamishima, Shotaro Akaho, Hidcki Asoh, and Jun Sakuma. Fairness-aware classifier with projudice remover regularizer. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, Seplember 24-28, 2012. Proceedings, Part II 23, pp. 35-50 50. Springer, 2012.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In 1. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss//pproeeddng.. Paper .dd.
neurips.cc
Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pp. 3150 3158. PMLR, 2018
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective PMLR, 2020.
perspective. In International Conference Machine Learning, pp.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Hussein Mouzannar, Mesrob I Ohannessian, and Nathan Srebro. From fair decision making to social equality, In Proceedings of the Conference on Pairness, Accounlability, and Trunsparency, pp. 359-368, 2019.
Reilly Raab and Yang Liu. Unintended selection: Persistent qualification rate disparities and interventions. Advances Neural Information Processing Systems, 34, 2021.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48:75, 2013.
Harsh Satija, Alessandro Lazaric, Matteo Pirotta, and Joelle Pincau. Group fairness reinforcement learn- ing. Thans. Mach. Learn. Res., 2023, 2023.
Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforce- ment learning with average and discounted rewards. In International Conference on Machine Learning, pp. 8905-8915. PMLR, 2020.
Shubham Singh, Bhuvni Shah, Chris Kanich, and Ian A Kash. Fair decision-making for food inspections. Equity and Access in Algoritlmss, Mechanisms, and Oplimization, pp. 1-11. 2022.
Min Wen, Osbert Bastani, and Ufuk Topcu. Algorithms for fairness in sequential decision making. In International Conference on Artificial Intelligence and Statistics, pp. 1144-1152. PMLR, 2021.
John A Weymark. Generalized gini inequality indices. Mathematical Social Sciences, (4)):099433) 1981.
Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang, How do fair decisions fare in long-term qualification? Advances in Newral Information Processing Systems, 33: 18457-18469, 2020.
A Full Proofs for Theorems 4. 4.3
They show wider range of conditions suffice than presented in the main text where we focused on the simplest, most natural cases.
Here we provide full proofs for the theorems in Section 4.1
Proof of Theorems configuration. In order for Ta to do harm, two conditions must hold. First, either Σ y
Consider a multi-objective policy म that is optimal under Equation
for some
1100
ΣY - Σ y 0. Second, Ta must deviate from the decision-maat maker utility-optimal policy T100. which nл
4100
means that.
e (E E + R
2
E R ER
(12)
Next substitute Rº and Rf values with their Y= values in Equations
22 and
3
++1 >
H+1
19(0)) = xm + Y H (ai) = 1+ 1+| pr = =1+
Σ
Y
x
Y
Σ
x
(13)
K=
<=0
1=0
0
to get their summations entire episode:
(R.) = ΣYH ΣY1
ER = YY!! ΣY, - - ΣY:
When summing over the episode and substituting into Equation 12 the t terms end up canceling since they are present in both the Σ R and Σ R terms. Also, the Y, terms in Equation (13 telescope when summed Fx
2100
over the episode. We are left with:
( y CEP+YY' -
12 1-
- --1
E - Σ Y
> X Σ - -RR) (14) F100
FA
A
F100
F1no
7x
KA
7100
1100
17

---

Published in Transactions on Machine Learning Research (05/2024)
There are two sets of absolute values. By Definition 4.2 and without loss of generality, we know that Σ - - y 2 0, so we can drop the second set of absolute values. To drop the first set of absolute #100
values, we split the rest of the proofs into two parts, where we prove each of the theorems for the case when the absolute value expression is positive and when it is negative.
A.1 Part
Here we prove Theorems 1.1 4.3 under the assumption that the first absolute value expression in Equation 14 is positive. That is, we assume:
n 2 Er
(15)
"A
which allows us to simply drop the absolute values:
( ΣY-) ( + m r\) +
[[- ( (-( -1 E
rO)]
2 X2 ( -ERR) (16)
T100
1100
F4
x2
1100
T100
X100
53
After some simple algebra, we get
(10 (10 X-EX"
E r - x
E RD EER $100 x2
E r ΣY 1100
RA
#100
(17)
X
which serves as an upper bound on the harm the disadvantaged group. Because no policy can have higher
ER than Ti00, know that Σ RO - ERD 2 0. Therefore, the entire right-hand-side of Equation (17]
T100
will be non-positive, which implies no harm to the disadvantaged group, if any of the following conditions are true:
=x=²; Σx¹ yy,;
(18a)
(18b)
" 5100
(19 A > Σ Y");
(18c)
5100
or (19 <19<)0) A < Σ Yl)
(18d)
$100
A
A I y - EYY < 19 - X
E R² ΣNd]
(18e)
1100
1100
or (19 > X A < Σ Y');
(18f)
or (19 x A (ΣY²
Σ Y')
(18g)
53
3100
X
A Ey - E Σ <
E R² Σπ³)
(18h)
5100
From Equation 18a) we get Theorem
If assume that 2º XF, we can rearrange the terms in Equation advantaged group to get:
17 to isolate the harm the
(F Y ΣY") + X RD ERD
SY--EYL 1100
Because no policy can have higher ERD than Ti00, we know that Σ - ERD 2 0. Therefore, the entire
8100
NA
right-hand-side Equation A.1 will be non-positive, which implics no harm to the advantaged group,
(2 ) ( ( rd 2 ΣY").
(19)
1200
Combining Equations 19 18f and 18a we see that if 2² X then either the advantaged group or the disadvantaged group not harmed. This gives us Theorem
4.2
18

---

Published in Transactions on Machine Learning Research (05/2024)
Under natural fairness, Σ y E EY, we can replace both values with the same variable E y
1100
$100
1100
Σ r
Σ Y.. Applying this to Equation 16
yields
#100
#100
12 TA
+ - - 2( 2 ro.1)] +
2 x2 M Rº ERR)
1100
TA
TA
F100
5x
or,
1 + ) 2 D y > ( -
-EER-RRR + A (y - E
By our Equation Similarly, we can simplify Equation
15 assumption, the right had side is positive and so Ey - Σ yo.1 is positive as well.
100
(19
(EE
EY
(E ER²
Σ you
R
$100
3100
100
As before the RD term is negative so we can drop it and rearrange terms to get:
E E
1² x 22 + A
100
EYl
Σ Lo,
8 x100
or:
y Σ 10u
X - 22 2º F
3A
$00
Σyl
Σ Lo,
$100
If 2 X2, then the left-hand side must be non-negative. This implies that Σy
Σ y0,1 is also non-
52 1100
negative, which gives us Theorem 4.3
A.2
Part II
Here we prove Theorems 4. 4.3 under the assumption that the first absolute value expression in Equation
14 is negative. That is, we assume:
Σy"; 2 ΣY
(20)
which gives us:
2 ( + EY-) (E y + E vz)] + ">
(-(EY y-_ E rン)] 3100
x Σ
R
-P-RRR (21)
110
1100
100
1100
x2
After some simple algebra we get:
(1
M r 10-1r--rr 5(x" ='''''''''''-
-
x2 n²
MR
(22)
F100
32
F100
5100
As before, we know that Σ RDD ΣRD' 2 0, so
1
T
(X + 1
Σ y0_ ΣY¹) - -19 ΣYD)
(23)
100
1100 A
Tf Q = FF, then
(X +
(EY" EX-) 0
(24)
80 ΣY1> > Σ Y Combining this with the Equation
20
gives us
FA 1100
Σ > ΣY 0
This is Theorem 11
19

---

Published in Transactions on Machine Learning Research (05/2024)
If > X then the right-hand-side of Equation 23 is negative if E E > EY is negative, then the sett-hand-side must also be negative, which implies that ΣY the Equation 20 assumption, we get
When the right-hand-side
#100
y From this and
RA
1100
Σr
EY'
y
(25)
MA
"100
Alternatively, if the right-hand-side of Equation
23
is positive, then ΣY" > Σ yl. Combining this with
1100
Definition 4.2 we have
ΣY
E y Σ Y
(26)
Y
1100
1100
which gives us M > M yo. Combining this with Equation
25
gives us Theorem 4.2
Under natural fairness,
E Y Σ Y E y Applying this to Equation 21 yields
1100
9100
1100
(MEE"++EY") 42
Σ 10. Σ yo.1) +X [[
(0EE1-EE)))) -EY you E yo
X Σ
RP-ER' ERD) 35
1100
9100
n1
X200
8100
110
Simplifying yields
1 [ + ( E you + E youl
Er EY
nx
X100
$100
82
Suppose for contradiction group 1 is harmed. Then
2 [ Σ par > X - A [ E y
R100
RA
$100
If F 2 Q this is contradiction, which gives us Theorem 4.3
B Relative Harm Properties
In Section 4 we provided definitions and theorems based on harm, where harm is defined with the decision- maker utility-optimal policy T100 as the baseline policy. We can, however, generalize our definition of harm as well as our theorems by defining relotive harm. Instead of using 7100 as the baseline, relative harm uses some multi-objective policy л as the baseline policy.
Definition B.1 (Relative Harm). With respect to some baseline policy n', a policy x does relative harm to group Z z if the group's 's expected cumulative qualification value is lower under a than under n':
ΣY=>
(27)
Similarly, we say that policy does relative harm to group Z if EY² S Ey= and relatively benefits group 7 Σ ΣY=
We can reconstruct our Section properties with respect to relative harm by adding the condition that there is no improvement. to decision-maker utility and by defining the advantaged/ hawnlaageel/issdvantaaaeee with respect to the baseline multi-objective policy σx Without loss of generality, we set the group that is disadvantaged under the bascline policy тX as = O. We can guarantec the following propertics for multi-objective policies Ta that have the specified changes in À relative to their X' values. The proofs are identical to their Section 4.1 counterparts after replacing 1100 with Tx and adding the constraint that there is no improvement to decision-maker utility.
groups
_2D)), and when natural fairness does not Thcorem B.1. Relative some policy Ty where X' = 12 .F'
exist, no relative harm is done to the disadvanlaged group is Q2_ "0" X - =F" and Σ RR > ΣRD
This is the relative form of Theorem 4.1 group if the change in X² is equal to the change in X and there is improvement to decision-maker utility.
Intuitively, says that no relative harm is done to the disadvantaged
20

---

Published in Transactions on Machine Learning Research (05/2024)
Theorem B.2. Relative to some policy nT, at least group is not relatively hormed if 2 2 and E RDD 2 ERD
AN
This is the relative form of Theorem 4.2
Xx Theorem B.3. Relative to some policy ax, when E = Ey 2 ER..
neither group is relatively harmed if
Rr - 2 S -F= and E RR- Xx
This the relative form of Theorem 4.3 The E
Σy1-= constraint is similar to natural fairness
fly "
except that it is for some baseline rather than T100-
C Harm Properties with Approximate Solutions
In this appendix we explain how our results guarantccing no harm apply in settings where we find only approximately optimal policies. In particular, we show how additive approximation error can be carried through our analysis, providing bound harm that is linear in the approximation error.
Our analysis for the case of exact optimization came from the observation that
( - E n2 x
E RR) 2
A'EERREER)
(28)
1100
100
110
This came from na being optimal, meaning in particular it must (weakly) improve on m100- Tf instead it is optimized to within an additive error of we would have
R
R
+ (ER
E R.) +c2
X2 RR
-RRR)
(29)
N100
XA
5100
5100
3A
After some simple algebra, instead of 17 we get
(A - X (EY
Y X R - ERR)+E
Σ Y ΣY Furthermore, since T100 may also be optimal only up an additive factor of e, the term Σ R
#100
#100
(30)
2 + AF
TA
-P--PP ERD 2x
T100
may be negative by up e. e. Thus, rather than coucluding that the harm is non-positive we can only conclude it is at most [1+x"). X Thus, the possible harm grows linearly in the approximation error. Furthermore, we can see that increasing the weight on fairness (i.e., taking 2 XF as discussed Section 4.3 and increasing this hyperparameter) decreases the possible harm.
Note that this bound goes to infinity we take the weight on fairness to 0. This is intuitive. If we were not interested in fairness and simply optimized w twice, while the objective (decision-maker utility) can only be harmed by due to approximation error, this could cause arbitrary changes to the qualification of different groups Putting weight on fairness what allows us to translate bounds on approximation error into bounds on harm.
D Extension to More than Two Groups
Our main focus is the two-group setting, |2| = 2, which reflects the most common scenario in group fairness. We can, however, generalize approach to more than two groups as follows.
We keep the overall reward structure consistent with Equation which we repeat here for convenience:
R(), = XD (at) + 19 (o.) + FF (o.).
but define Rº and R as functions of |Z|:
RO())) = |z|
Y
(31)
0=0
21

---

Published in Transactions on Machine Learning Research (05/2024)
2
2 Σ
1
Rº
(ot)
+
Y
avg
|
Y(+1
(32)
1=0
where
2
Y
Y
(33)
Z
0
Under this scenario, our theoretical properties need be adjusted in order to consider that there no longer clear "disadvantaged" and "advantaged" group, A full theoretical analysis of this new case likely needs new ideas, as the two cases we consider in our proof become twelve cases, even with only three protected groups. However, provide numerical illustrations in Figure show how our approach is still relevant in this more general setting. As we can see, similar to the two-group setting, X² < X narrows the band of qualification values to only consider qualification combinations more optimal they are all close together. Similarly, a = x creates a reward where higher Y values are considered better, but only if it not at the expense of lowering another group's qualification, as characterized by the box-like contours upward and rightward from the bascline. Finally, X X still encourages higher qualification values, even if it is at the expense of lowering another group's qualification.
22

---

Published in Transactions on Machine Learning Research (05/2024)
50=5,1=1
10=1,
19=1,2==5
(o Moo. (0..2 0.2, 0.2)
(Pon. Kmo (0.22 0.5)
(Yie Ylew 1-002- 0.5)
1.0
1.0
1.0
0.8
0.8
0.8
0.6 X-
_
0.4
0.4
0.2
0.2
0.0
0.0
1.0
0.8
0.8
0.8
0.0
0.2
0.4
0.6
-17
0.0
0.2
0.4
0.4
0.8
-++
0.0
022
04
0.4 4%
0.6
yf
0.6
1.0
0.0
0.2
yp
066
1.0
0.0
0.2
>p
0.6
0.8
1.0
0.2
10 5,
A 1, 1,
A 1, -
foo. Vioe 1o (0 2, 0. 35, 0.5)
(D Yoo 1o0 (00.2 0.35, 0.5)
(Y Vi (C..2, 0.35, 0.5)
1.0
1.0
1.0
0.8
0.6 X
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
1.0
1.0
1.0
0.8
0.8
0.8
0.0
0.6
0.0
0.6
0.0
0.6
0.4
>p
0.6
0.8
1.0
0.0
0.2
0.4
0.4
kp
0.6
0.8
1.0
0.0
0.2
0.4
0.2
0.4
0.6
0.8
0.0
0.2
47
0
A
1
1, A'
A
1, 1'
(Y Yioo Yio (0.55 0.5. 0.5)
(Y Yino Yio (- 5, 0.5 055
(Y Yiea. Yoo 1-10. D.5, 3.5)
1.0
0.8
0.8
0.6 $
0.6 $
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.4
02
0.4
02
0.4
4%
0.4 1p
0.6
0.4
04
0.8
0.0
088
0.0
0.8
10
0.0
Figure 6: Plots showing the 'desirability' (i.e. ARQ of each possible (Y,, Y outcome for three different decision-maker utility-optimal Y values and three different X²/XF ratios. The blue cubes indicate the Y values for the decision-maaer utility-optimal policy, which will by definition have ARQF = 0. Darker contours indicate more attractive (higher ARFF) values. The colorless areas thus indicate where ARQP < 0, which means they are not possible solutions under म since the decision-maker utility-optimal policy will be preferred over them. The dotted area indicates the scenarios with natural fairness. The plots assume A AR 0 so that decision-maker utility is still optimal and any AR originates exclusively from AR& and ARE
23


Reviews:
[{"review_id": "I0Mau9ctFa", "reviewer": "Reviewer_HwGU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-- The paper studies a really important problem. \n\n-- The \"no harm principle\" is especially important in resource allocation problems at have been largely overlooked by previous work.\n\nWeaknesses:\n\n-- The setting is almost identical to Zhang et al so in that sense the novelty of the formulation is limited.\n\n-- There is no discussion of how the optimal policy can be computed. I assume this can be done in a brute-force dynamic programming style since there are only two time steps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies group fairness in reinforcement learning. The paper's stylized model is inspired by Liu et al and Zhang et al in which loan applicants from two groups arrive at each timestep and the decision maker has to make a loaning decision which in turn can change the distribution of (qualification) of applicants in the next time step. The paper aims to study the long-term effects of these decisions in particular the relationship between the decision maker's utility and the disparity between the qualification status of applicants from the two groups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-- I would like to see more justification for the transition probabilities provided in Table 1 or at least a more detailed exploration of the parameter space.\n\n-- Is it clear when natural fairness exists? In particular, can we quantify when the \"no harm principle\" can be achieved? The authors explore this empirically but can we say more?\n\n-- Can the authors provide intuition on why the proposed solution is different compared to the previous solutions? In particular, previous work aimed at maximizing utility subject to fairness constraints. This can be written as an unconstrained problem by using Lagrangian multiplier. The current work looks at an optimization problem which is the weighted sum of utility and degree of fairness violation. To me it is unclear why this formulation can satisfy the two desiderata that the paper is after while the previous formulation cannot. \n\n-- Intuitively why do we need two time steps? Can the analysis be done using one time step similar to Liu et al and Zhang et al.\n\n-- Some recent work in the area such as https://openreview.net/forum?id=JkIH4MeOc3 are not cited.", "broader_impact_concerns": "Broader impacts is sufficiently addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L0crUWJwoO", "reviewer": "Reviewer_ytNM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper proposes a framework based on reinforcement learning with a multi-objective reward function, which balances three objectives: (1) maximizing the decision-maker's utility; 2) maximizing group qualification; 3) minimizing group qualification disparity. \n\n2. By adjusting the weights of each reward term, the authors can show theoretically that the policy doesn’t harm one or both groups (i.e., group qualification does not deteriorate). \n\n3. The paper studies an important topic that is relevant to the audience of TMLR. It is well-written and organized. \n\n\n\nWeaknesses:\n\n1. While the authors claim the proposed method satisfies the \"no-harm\" principle, it seems that \"no harm\" is mostly guaranteed for one group (Thm 4.1 & 4.2), i.e., one group's qualification doesn't deteriorate. Only when natural fairness exists, can harm be avoided for both groups (Thm 4.3). \n\n2. My major concern of the paper is its novelty. Using MDP to formulate the interactions between individuals and policy is standard and has been widely used in prior works. There are also extensive studies that simultaneously consider the welfare of both decision-maker and individuals (including qualification and group equality), e.g., [1]. To balance multiple objectives, the paper simply defines a reward function as the weighted sum of three rewards.  Since the algorithm for finding the optimal policy seems standard, I am uncertain if defining a new reward function is sufficiently novel.\n\n3. The experiments are conducted on simulated dynamics, which may limit the impacts of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The study proposes a reinforcement learning method for long-term group fairness; it defines a multi-objective reward and directly optimizes three objectives: maximizing decision-maker utility, maximizing group qualification, and minimizing the difference in qualification between groups. Importantly, the proposed approach aims to achieve equality without harm, i.e., avoiding reductions in qualifications for specific groups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Why is the qualification disparity defined as the difference between the cumulative qualification of two groups, but not the difference between instantaneous qualification, i.e., $Y_t^1-Y_t^0$? Also, many related works are aiming at reducing disparity between two groups which were not discussed in the paper, e.g., [2]\n\n2. After defining MDP with multi-objective reward, how to find the optimal policy? \n\n3. It is helpful to add more related works and discuss differences in more detail. For example, the authors mentioned two drawbacks that existing works suffer but only introduced one work Wen et al. (2021) that violates the \"no-harm\" principle.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HsexgLOpdu", "reviewer": "Reviewer_fmFj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n\n- The problem of fairness in RL is an important one. The authors provide an alternate formulation of this problem that includes the utilities of the groups and differences between them in the modelling itself (graphical model in Fig 1). \n- The paper is written clearly and easy to read. \n\n## Weaknesses \n- The methodology is limited to only two groups. It is not clear how the approach would scale with number of groups. For instance, I believe there will be combinatorial hyperparameters or weights for including each pairwise term. \n- The core contribution of the paper is the weighted-sum objective (Eq 1). Similar objectives have considered in the literature [1, 2] where the weighted-sum objective can be considered to the lagrangian versions of the objectives studied in the related papers. The main claims of the work is how the authors claim their proposed objective can address the \"no-harm\" drawback. However, this claim is not supported in the analysis or the results. For instance, when no natural fairness exists, i.e., there is discrepancy between groups, Theorem 4.1 is only provide guarantees for the disadvantaged group (but not for advantageous group). Similarly, Thm 4.2 just provides results that at least one group (out of 2) is not harmed. In both the cases, there is case for harm for one out of the two groups, so it is not clear how the proposed approach is better than the existing work in terms of drawbacks. \n- There is little discussion on how the weights should be set, especially for the non-natural fairness case. If the author's are proposing to always increase while satisfying $\\lambda^Q = \\lambda^F$, then shouldn't the analysis be done w.r.t. this objective. \n- There is limited empirical evidence of the claims. All the experiments are done a small synthetic lending environment where the behaviour seems contrived. For instance, in Sec 5.3.1, when making comparison to EqOp baseline, all the proposed weights/approaches that do not violate the no-harm principle achieve a lower $R^D$ than the EqOp baseline. There is clear trade-off here which is not discussed in the paper. \n\n\n\n### References:\n- [1] Algorithms for fairness in sequential decision making, Wen et al, AISTATS 2021.\n- [2] Group fairness in RL, Satija et al, TMLR 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks at the problem of group fairness in the RL setting. They approach this problem via proposing three implicit objectives which are then converted to a single scalar objective that can be solved with any traditional scalar RL methods. The three objectives are based on: maximizing the system-designer/decision-maker's utility, maximizing the group's utilities (qualification) and minimizing the difference between groups. These objectives are then scaled via some hyper-parameters and then combined together. The authors claim that the proposed method addresses the drawbacks of other approaches, i.e., they can explicitly model the decision-maker's utility and ensure the \"no-harm\" property. Here no-harm property refers that the method prevents drop in utility of a group on introducing fairness. They provide empirical evidence of their method on a small synthetic MDP lending task.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Notation: Is the paper in the context of infinite-horizon discounted MDPs or finite horizon. Both the notations are used throughout the paper which can be confusing at times. \n- Clarity: Both Fig 2 and 3 pack too much information and are tough to read. I would encourage the authors to only keep the most representative figures for readability. For Fig 3, the authors can also use different markers along with different colors. \n- Empirical studies: The paper needs more evidence to support their claims, particularly of those regarding the drawbacks. The authors can  use other studies like [3] or some DeepRL experiments. \n- Analysis: There should be some discussion of the trade-offs with $R^D$ (w.r.t. $\\pi_{100}$?). Additionally, more discussion should be on how to find the weights so that there is \"no-harm\" to both the groups. If this can not be achieved, then the authors should mention that too. \n- Limitations: More emphasis should be placed the the method is limited to only two groups. Additionally, when the MDP is not known (RL setting), even setting the correct weights might lead to harm when the learning algorithm is still getting information about the environment [1,2].\n\n\n### References:\n- [3] Dynamic Positive Reinforcement For Long-Term Fairness, Puranik et al, ICLR 2022 Workshop on Socially Responsible Machine Learning.", "broader_impact_concerns": "Please see my comment about limitations in the above section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ysu2VT3Jzh", "reviewer": "Reviewer_hMiN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important problem of ensuring that the policies followed by reinforcement learning systems account for fairness requirements rather than simply maximising decision-maker utility. The authors have identified significant limitations in prior approaches to fair RL, and have developed an approach which addresses those limitations.\n\nThe approach itself is elegantly simple – three rewards addressing different objectives which we wish to optimise, which are combined together via a weighted scalarisation to produce a scalar reward. As a multiobjective RL researcher, I was initially concerned by the use of a linear weighted sum of objectives, as it’s well known in MORL that this may be incapable of finding some Pareto-optimal solutions. But the authors do a good job of demonstrating that given the interconnected nature of the second and third objectives, a linear weighting makes sense. It simplifies analysis of the effect of changing the weighting parameters, and also means that this technique can be implemented using any existing single-objective RL method, which is practically important given that these algorithms are generally more advanced and better understood than their MORL counterparts.\n\nThe formal analysis of the effect of the weights and the empirical demonstration in Section 5 do a good job of highlighting the benefits of this new approach compared to the previous fair RL methods which are used as baselines. (I should note that my mathematical expertise is not high, so I can’t vouch for the correctness of the proofs, but I studied them extensively and did not see any flaws).\nOverall I thought the paper was well-written and clearly presented. In particular the first 3 sections really clearly defined the problem area and the proposed approach. I did find the Analysis section much harder to follow – I had to re-read this multiple times. It took me a while to comprehend Fig 2, but once I did I found this assisted my understanding of the earlier material, so I think it might be beneficial to provide some worked numeric examples in Section 4.1 as well. \n\nThe main weakness of the proposed approach is that it is restricted to problems involving just two groups, whereas in practice we would usually require policies to be fair across multiple protected attributes. The authors do acknowledge this limitation in the conclusion, and an approach which works for the special case of two groups is still of value, but I would have expected to see some discussion about whether this approach might be extended to the case of >2 groups, or whether an entirely different method will be required.\n\nIn the conclusion the authors also note a limitation in that the inclusion of the historical group qualification bias as part of the state may make learning the optimal policy from the MDP computationally intractable. In the empirical study, the authors have used linear programming to find the optimal policy (which to my understanding requires a complete model of the MDP, rather than learning from experience like most RL methods – please correct me if my assumption here is incorrect). Given that the paper is framed in terms of RL, I would have preferred to see results based on a model-free method, or at least some discussion of whether standard model-free RL could be applied.\n\nI also had some difficulty in understanding the nature of the MDP used in Section 5, and how it was executed in the context of the empirical evaluation. The cash attribute of the lender is in the range (0,1,2) starting with a value of 1, and increases or decreases by 1 (capped to that range) whenever a loan is paid back or defaulted. For even the most qualified lendees, the default rate is 25%. So it seems to me that even following an optimal policy, executing this MDP will almost inevitably result in entering the ‘no cash’ state (only 2 defaults in a row are required to go from the maximum cash value of 2 to the minimum of 0), and once this state is entered it can not be exited as the lender can not make any further loans. The results in Fig 3 are reported as being over 10,000 episodes – it seems inevitable that the ‘no cash’ situation arises during that time-period, so how is this handled?\n\nLooking more closely at the results in Fig 3, it is surprising that policy 511 outperforms 100 on the decision-maker utility in Scenario 1, and dominates it in Scenario 3. 100 focuses exclusively on the decision-maker utility, whereas 511 applies at least some weighting to the fairness objectives, so it seems unusual that it does better with regards to the decision-maker utility? Is this just due to the stochasticity of the MDP (in which case perhaps more than 10K episodes are required), or is there some other algorithmic factor at play here?\n\nFinally, while I earlier identified the choice to scalarise the objectives into a single reward as a strength (as it allows standard single-objective RL methods to be used), this is also a potential limitation of the approach, as it eliminates the possibility of carrying out multi-policy learning. Even if we limit the weight-space to the 1xx option recommended by the authors, we may need to execute the learning process multiple times with different values of x in order to find a policy which provides a suitable trade-off between decision-maker utility and the fairness objectives (as is discussed in Section 4.3). For a simple MDP and the linear programming method used here, the cost of finding several policies is not excessive, but for more complex MDPs and learning methods which actually require sampled experience from the environment this could become costly. If the problem was treated as a MOMDP, then multi-policy MORL methods could be used to learn multiple policies in parallel (for example, the Conditioned Network algorithm of Abels et al, 2019 can learn multiple linear-weighted policies, so could be a good fit for this task).\n\nReference\nAbels, A., Roijers, D., Lenaerts, T., Nowé, A., & Steckelmacher, D. (2019, May). Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning (pp. 11-20). PMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the issue of ensuring that the sequential decision-making policy learned by a reinforcement learning agent satisfies fairness requirements, specifically in terms of fairness between groups. It does this via a multi-objective formulation of the reward, with separate terms accounting corresponding to decision maker-utility, group qualification, and group qualification equality. These are combined via a weighted sum process to form a scalar reward (which can therefore be optimised for using standard RL methods). The authors show both theoretically and via a small empirical example that appropriately chosen weights can achieve suitable trade-offs between the different objectives. In particular this approach avoids violations of the “no harm” principle (ie it does not allow one group’s qualification to be lowered in order to improve group qualification equality), and allows for a suitable balancing of decision-maker utility and group qualification – previous approaches to fair RL have not provided this functionality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A clearer explanation of the MDP’s ‘no cash’ situation and of the process used to produce the results in Fig 3 is required, and would be important for me in recommending acceptance as the results are possibly open to misinterpretation without that.\n\nI would also like to see the authors address the other issues I have identified above (briefly summarised below), but these re less critical – more a matter of making the paper easier to read and more impactful rather than of meeting the criteria for acceptance\n-\tAdd worked examples in section 4.1\n-\tDiscussion of the results of policies 511 vs 100\n-\tExtended discussion of limitations and possible extensions in the Conclusion\n\nThere are also some smaller presentation issues which should be addressed in the final copy:\n-\tOn p5 the statement is made that “R^D is the decision-maker utility function defined in Section 2”. But Section 2 doesn’t actually define R^D – it just says the lender wants to optimize profit).\n-\tAlso on p5 R^Q is described as the *average* of group qualifications, whereas it is actually the *sum* of group qualifications.\n-\tP6 starts with a duplicated a: “a a single objective”\n-\tFig 2: The labels for the Y100 values superimposed on each graph are very hard to read – small font on a grey background. I had to zoom in a long way on the pdf before these were legible, and this contributed to the time it took me to understand this figure. \n-\tP8: “thee” should be “the”", "broader_impact_concerns": "I don’t have any further concerns beyond those which the authors have already addressed in their Conclusion. Including fairness objectives in addition to decision-maker utility clearly offers the potential for more ethical development and deployment or RL systems than is the case for standard RL methods that focus exclusively on decision-maker utility.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree the paper makes interesting contributions in an important question. The revised version addressed issues raised in initial reviews, such as clarity of contributions and experiments, and discussions of limitations. While there remain limitations, most notably the restriction to the two-group setting, the existing contributions are useful to be published.

Other comments:
* Is the condition needed in Theorem 1 that "natural fairness does not exist"? Should the conclusion still hold when natural fairness exists (Thm 3)?
* Fig 1: the diagram suggests outcomes are observed immediately at t=1. Is it a valid assumption for problems like loan decisions?
Decision: Accept (poster)