Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems

Abstract: Long-term fairness is an important factor of consideration in designing and deploying learning-based decision systems in high-stake decision-making contexts. Recent work has proposed the use of Markov Decision Processes (MDPs) to formulate decision-making with long-term fairness requirements in dynamically changing environments, and demonstrated major challenges in directly deploying heuristic and rule-based policies that worked well in static environments. We show that policy optimization methods from deep reinforcement learning can be used to find strictly better decision policies that can often achieve both higher overall utility and less violation of the fairness requirements, compared to previously-known strategies. In particular, we propose new methods for imposing fairness requirements in policy optimization by regularizing the advantage evaluation of different actions. Our proposed methods make it easy to impose fairness constraints without reward engineering or sacrificing training efficiency. We perform detailed analyses in three established case studies, including attention allocation in incident monitoring, bank loan approval, and vaccine distribution in population networks. 

Reviews and author responses:

 Thanks for adding some theoretical justification. I have updated my score based on the author rebuttal.

 I would like to thank authors for the detailed response. I would like to encourage authors to incorporate the proposed clarifications in the manuscript, so that the contribution and takeaway msgs can be clearer and more approachable.

Post-rebuttal: I have updated my score accordingly.

 Thanks for your response. The related works are well compared! 

 > Thanks for the response. I'm still a little unclear on the distinction between the hyperparameters in your method and a method where we weight the auxiliary objective - if your method tells us to take actions when fairness is worse than some threshold, then isn't the setting of that threshold equivalent to setting the weight on an auxiliary objective?

Answer: Thank you and yes we should further clarify. There are two parts in any learning problem: one is the definition of the objectives, and the other is the computation procedures for reaching the objective. In other words, one is what do we optimize, and secondly how do we optimize. The major difference between our methods and standard reward engineering is that our hyperparameters are only used to improve the second step of how to optimize, without affecting the objective and the problem definition itself, while reward engineering focuses on defining the problem itself in the first step. We will elaborate on these below. 

First of all, the threshold parameter is part of the objective definition, not what we need to choose in the algorithm. The threshold declares the amount of tolerance of “unfairness” that a problem allows, and both our approach and reward engineering with an auxiliary objective need to use it. In the reward function form, it would become an additional loss term of “unfairness value minus that threshold”, and then choosing a weight for this term. In our case, we interpret this given threshold as a region for “stabilizing” the unfairness value. In both cases, the threshold can be arbitrarily chosen by the problem formulation, and it can be zero, in which case it specifies that the unfairness should be minimized to as small as possible without tolerance. Our algorithms work for any choice of the threshold, and it is not a hyperparameter that we need to choose in the algorithm itself. 

But perhaps the reviewer is also asking about the distinction between our advantage regularization hyperparameters $\beta$ and auxiliary objective weights $\zeta$, and what the difference is between reward engineering $\zeta$ and tuning $\beta$. This is a great point that we should address in the paper, because the difference is crucial. On one hand, the $\beta$ we are using is in the gradient update and it does not affect the reward definition and does not hold one to commit to specific weights of the different objectives in the first place. Tuning $\beta$ improves the "how to learn" aspect of optimization similar to how a learning rate would in standard deep learning procedures: the advantage regularization terms help with better convergence on the objective. Our procedure is orthogonal to and does not change the objective itself (i.e. "what to learn"). On the other hand, reward engineering is defining the objective in the first place, and after that, the underlying hyperparameters, such as learning rate, still need to be chosen, which is at the level of what our proposed methods are focusing on. In short, we only control how to optimize (learning procedures), not what to optimize (objective). 


 Thanks for the response. I'm still a little unclear on the distinction between the hyperparameters in your method and a method where we weight the auxiliary objective - if your method tells us to take actions when fairness is worse than some threshold, then isn't the setting of that threshold equivalent to setting the weight on an auxiliary objective?

 We thank the reviewer for the detailed comments and questions. 

We did not include a detailed mathematical analysis of our advantage regularization terms and focused on showing the empirical effectiveness of the methods, because we worried that the background theory may distract the focus and clarity of the paper from the long-term fairness context. For your reference, here we give a short introduction to the theoretical justification of our proposed methods, and we will provide more details along the following lines in the paper or the appendix. 

First, we consider the problem of long-term fairness as a control problem: the goal is to obtain a control policy to regulate the entire system to satisfy the fairness properties. Consequently, we draw inspiration from control-theoretic methods in the Lyapunov stability framework [1-4], a tool for estimating and improving stability in a nonlinear dynamical system. Note that long-term fairness is essentially a property about stability: we wish to control the system such that the violation of fairness reduces over time, and ideally gets controlled to be under a certain threshold. This connection makes it possible to use Lyapunov methods to derive the regularization terms. 

Before deriving our advantage regularization terms, let us first set up some definitions for Lyapunov stability. Before continuing, please refer to Section A: “An Introduction to Lyapunov Stability” of the appendix for a short introduction to Lyapunov stability.

Next, with the Lyapunov stability idea in mind, we come back to our derivations of the advantage regularization terms, in particular the second one. Recall in Proximal Policy Optimization (PPO), the advantage $\hat{A}_t$ describes the difference between the Q-value of an action and the expected value of a state at time $t$, to indicate whether an action should be taken more often in the future. Let $\Delta : S \rightarrow \mathbb{R}^\ge$ be a function of the state and a fairness constraint to minimize. We make it our task to transform the landscape of $\hat{A}_t$ into a Lyapunov-like landscape with respect to $\Delta$.

First, we wish to enforce the decrease of $\Delta$ in the landscape of $\hat{A}_t$. We add a value-thresholding term $\text{min}(0, -\Delta(s_t) + \omega)$, where $\omega$ specifies a tolerable threshold of $\Delta$. This term penalizes the advantage evaluation for $(s_t, a_t)$ when $\Delta(s_t)$ is greater than $\omega$, and remains inactive otherwise. This term transforms the advantage landscape by adding a region of attraction around some equilibrium point, which in this case is the origin.

Next, we take inspiration from the sample-based Lie derivative to design the decrease-in-violation term $\text{min}(0, \Delta (s_{t+1}) - \Delta(s_t))$, where $\Delta(s_{t+1}) - \Delta(s_t)$ measures the dynamics of $\Delta$ over time. Like the value-thresholding term, this term only penalizes the advantage term if $\Delta(s_t)$ exceeds some tolerable threshold $\omega$, and remains inactive otherwise. While similar to a sample-based Lie derivative, the decrease-in-violation term is not exactly a derivative. Instead, it imposes the soft requirement of negative-definiteness of Lie derivatives in Lyapunov conditions.

[1] Wassim Haddad and Vijaysekhar Chellaboina. Nonlinear dynamical systems and control: A lyapunov-based approach. Nonlinear Dynamical Systems and Control: A Lyapunov-Based Approach, 01 2008.

[2] S. M. Richards, F. Berkenkamp, and A. Krause, “The lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems,” in 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, 2018, pp. 466–476.

[3] S. Liu, D. Liberzon, and V. Zharnitsky, “Almost lyapunov functions for nonlinear systems,” Automatica, vol. 113, p. 108758, 2020.

[4] R. Bobiti and M. Lazar, “Automated-sampling-based stability verification and doa estimation for nonlinear systems,” IEEE Transactions on Automatic Control, vol. 63, no. 11, pp. 3659–3674, 2018.


 We thank the reviewer for the detailed comments and questions. Our responses are listed below (we refer citations to the original paper):

> Would be useful to clarify (to me) what the novelty is in this paper: is it the method for optimizing and auxiliary objective in the RL setting, the exploration of fairness, or something else?

Answer: Our novelty largely stems from the way we impose fairness requirements in standard policy optimization procedures for long-term fairness. We enforce fairness by regularizing the advantage evaluation during the policy gradient step, rather than incorporating the fairness requirement as part of the reward function. We aim to address the reward-hacking issue specified by [12] in the future directions, by imposing fairness requirements without introducing additional reward engineering. We compare our approach with existing methods, and show it is a practical and more efficient algorithm that can be of use to the long-term fairness community.

> How general is the proposed method? If can be used beyond fairness but you're just choosing to explore fairness here for some reasons, would be nice to walk through that

Answer: The reviewer brings up a good point. Technically, if $\Delta$ can be formulated as a generic constraint to be minimized in the context of our RL setup, our proposed method still holds. However, we find this method to best fit the use case in addressing the problem of long-term fairness, and thus that is what we focus on. We will mention this point after defining our method.

> Can you clarify more how to interpret the difference between A-PPO and R-PPO?

Answer: The Reward-Only Fairness Constrained PPO (R-PPO) is Proximal Policy Optimization (PPO) with fairness constraints applied only at the reward level as penalty reward terms, whereas the Advantage Regularization PPO (A-PPO) is PPO with fairness constraints applied only at the policy gradient level as advantage regularization terms. The only difference between R-PPO and A-PPO is how and where we enforce the fairness constraints during training. 

> line 142: Q value is never defined.

Answer: We will add a brief definition for Q value in the preliminaries.

> line 168: there are a few hyperparameters to balance this function (the \beta's). I'm not sure how this relates to the critique of existing methods on line 45, where the authors say that the hyperparameters weighting fairness and accuracy constraints may result in reward hacking. 


Answer: In multi-objective reward functions, each objective is represented as a distinct term and weighted by importance. These objectives are linearly combined to form a scalar reward. However, when the objective weights are poorly selected, an agent can reward hack by maximizing for the wrong tradeoff between different objectives, and adopt undesirable behaviors.  

In our case, our $\beta$ hyperparameters exist at the policy gradient level in the advantage evaluation to enforce a specific property into the value function, and does not exist at the reward level which defines the objective. This specific property is a Lyapunov property, which on a high level enforces the policy to take actions that are more “fair” if the fairness constraint $\Delta$ is violated beyond a certain threshold. The $\beta$ terms are necessary because the magnitude of $\Delta$ and its derivative are highly dependent on the training environment setup. These $\beta$ terms provide a way to balance the advantage regularization terms across various scenarios.


> line 168: I may just not be familiar enough to the relevant literature but I don't understand why the reference to Lyapunov stability is helpful here

Answer: Lyapunov stability is a framework in control theory for estimating and improving stability in a dynamical system by reducing some error signal, which in our case can be modeled as the violation of the fairness requirement over time. On a high level, Lyapunov stability characterizes the state of being stable in a dynamical system through the enforcement of certain properties in the Lyapunov landscape. We can use these same Lyapunov stability techniques to enforce fairness constraints in the context of RL by transforming the value landscape to a Lyapunov-like landscape through advantage regularization.

> Line 303: what is the "min-max normalization" and what is it for?


Answer: Min-max normalization is a type of normalization that transforms each number in $x \in \mathbb{R}^n$ to between 0 and 1, inclusive. More specifically, for each real number in $x$, the minimum number becomes a 0, the maximum number becomes a 1, and all other numbers get scaled to a value between 0 and 1.

We use min-max normalization as a trick during training to achieve higher numerical stability by adjusting the scales of each term in our proposed advantage function $\hat{A}_\beta$ before performing their weighted sum. In general, normalization is not required for our methodology to work.



 We thank the reviewer for providing additional related works. Below, we discuss how these works are related to our work in detail.

Paper [1] proposes a CMDP-based solution on the problem of long-term fairness in recommendation. They claim that prior strategies solve for fairness in recommendation in a one-shot context, and fails to consider the dynamic nature of the system where item popularities change over time. Thus, [1] studies how to maintain long-term fairness on item exposure for the task of splitting items into groups by recommendation, using a modified Constrained Policy Optimization (CPO) procedure. [1] and our work both solve long-term fairness problems in nonlinear dynamical systems. [1] models the problem as a CMDP, while our work models the problem as an MDP and uses advantage regularization techniques to enforce fairness. Our proposed method is much more cost efficient than CMDP-based methods at training time and better satisfies the task of achieving high overall utility while minimizing violations of the fairness requirements.

[2] aims to address the long-term fairness problem in 3 parts. First, they define the fairness notion of return parity, which requires MDPs sharing the same state-action space across different demographic groups to achieve similar expected returns. They then provide a decomposition theorem for return disparity that partitions it into the disparities between group reward functions, group policies, and the group policies' state visitation distributions. Finally, they provide algorithms to decrease the return disparity by tasking policies to maximize their expected returns while maintaining similar state visitation distributions. Like us, [2] models their long-term fairness problem as an MDP, but they set up their constraint as the difference in the reward gaps across different demographic groups rather than defining it as a deterministic function of the state space. This requires that they design their fairness requirement directly into the reward function, which can be inflexible in that multi-task objectives bring up the issue of establishing trade-offs between different objectives and satisfaction of the fairness requirement. Our approach specifically aims to address this issue by enforcing fairness constraints at the policy gradient level, not the reward level. 

We will add these works to the revised paper.

Finally, we thank the reviewer for explaining the concern for the clarity around the fairness notion. We will make the definition more explicit in Section 3.

[1] Ge, Yingqiang & Liu, Shuchang & Gao, Ruoyuan & Xian, Yikun & Li, Yunqi & Zhao, Xiangyu & Pei, Changhua & Sun, Fei & Ge, Junfeng & Ou, Wenwu & Zhang, Yongfeng. (2021). Towards Long-term Fairness in Recommendation. 445-453. 10.1145/3437963.3441824. 

[2] Jianfeng Chi, Jian Shen, Xinyi Dai, Weinan Zhang, Yuan Tian, and Han Zhao. Towards return parity in markov decision processes. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors,  International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Learning Research, pages 1161–1178. PMLR, 2022.


 We thank the reviewer for the detailed comments and questions. Our responses are listed below:

> What does the term "human-designed policies" mean in the summary of experimental results?

Answer: A human-designed policy is a handcrafted strategy designed by a human for a specific scenario that does not involve any learning-based algorithmic design. For instance, the greedy agent from the attention allocation scenario and the equality of opportunity (EO) agent in the loan scenario is a human-designed policy. We will further elaborate on what a human-designed policy is in the experiments setup section.


> What is the contribution of the paper, is it in terms of technical modification of the PPO algorithm, or there are additional takeaway messages?

Answer: Our contribution comes in 2 parts.

First, we demonstrated that RL approaches are effective for designing policies that can achieve long-term fairness. Note that in the work of [1] only human-designed heuristic policies are evaluated, and algorithmic approaches have not been proposed for policy design. Through these scenarios, we demonstrate that policy optimization methods in deep RL can find strictly better decision policies such that they achieve higher overall utility and violate less of the fairness requirements than previously known strategies.

Second, we propose a novel method for effectively imposing fairness requirements in standard policy optimization procedures by regularizing the advantage evaluation during the policy gradient step. This approach specifically addresses the reward-hacking issue, as discussed in the future directions in [1] by enforcing fairness requirements without additional reward engineering. We compare this approach with existing methods, and show that it is a simple, more efficient, and practical algorithm that can benefit the long-term fairness community.

We will emphasize the contributions listed above in the introduction.

[1] Alexander D’Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D. Sculley, and Yoni Halpern. Fairness is not static: deeper understanding of long term fairness via simulation studies. In Mireille Hildebrandt, Carlos Castillo, L. Elisa Celis, Salvatore Ruggieri, Linnet Taylor, and Gabriela Zanfir-Fortuna, editors, FAT* ’20: Conference on Fairness, Accountability, and Transparency, Barcelona, Spain, January 27-30, 2020, pages 525–534. ACM, 2020.


> How does the proposed modification of PPO algorithm overcome the drawbacks/difficulties of current RL approaches (to algorithmic fairness) outlined in the introduction?

Answer: One drawback of standard policy optimization approaches is that the use of monolithic reward functions in the standard RL framework requires that they optimize for policies with respect to a single numerical value. Because the goal of decision systems is inherently multi-objective, the tradeoff between maximizing overall utility and minimizing violations of the fairness requirement becomes nontrivial at the reward level and may incentivize the learning agent to reward hack. 

In our approach, we avoid adding the fairness constraint as yet another objective in the reward function, but instead enforce it at the policy gradient level using Lyapunov-like stability approaches. Our methods are based on methods that originated from control problems for enforcing safety or stability properties in learning-based controllers, and we show that fairness properties can be handled in a similar framework with the specific design of regularization that we proposed in the paper. The proposed methods are novel, and as demonstrated in the case studies, can obtain policies that outperform previously known methods in the sense of achieving both higher utility and less fairness violation. 

As described in detail in the paper, one approach that we focus on comparing against is the framework of Constrained Markov Decision Processes (CMDPs). CMDP formulation requires learning algorithms to lower the expectation of constraint violation asymptotically and the training time can be arbitrarily long before any fairness properties can be established. In addition, CMDP-based methods require multiple rounds of numerical optimization (e.g. linear programming) at every step to satisfy the constraints, which can be expensive. Our formulation only requires the additional computation of the regularization term, which is constant time and is thus far more computationally efficient than the former. Empirically, we find that Constrained Policy Optimization (CPO), a CMDP-based policy optimization method, takes 2-3x more time than our A-PPO formulation to train a policy that achieves high overall utility and low violations of the fairness requirement. We can provide additional supporting evidence in our revision.


 The paper proposes a modification to PPO algorithm and presents an algorithm to constrain advantage disparity in the dynamic setup. The empirical results on multiple practical scenarios are also provided. The strength of the paper comes from a detailed presentation of the conducted experiments. The results are reported in three case studies (following D'Armor et al., 2020) and the comparisons involve various algorithms and fairness criteria.

Weakness of the paper comes from the way materials are presented. In particular, it would be very helpful if authors can clarify several questions (detailed in Section `Questions`):

Q1: what does the term "human-designed policies" mean in the summary of experimental results?

Q2: what is the contribution of the paper, is it in terms of technical modification of the PPO algorithm, or there are additional takeaway messages?

Q3: how does the proposed modification of PPO algorithm overcome the drawbacks/difficulties of current RL approaches (to algorithmic fairness) outlined in the introduction? ### Q1: what does the term "human-designed policies" mean in the summary of experimental results?

In presented experimental results, it is mentioned several times that the proposed A-PPO outperforms human-designed policies. I am a little bit confused of the used term here. Is the term "human-designed" referring to reward engineering (line 46)? If so, it seems that the proposed algorithm also involves an explicitly defined trade-off between utility and fairness violation, e.g., Equation 3. A further clarification would be very helpful.

### Q2: what is the contribution of the paper, is it in terms of technical modification of the PPO algorithm, or there are additional takeaway messages?

I think readers can benefit more from a clearer presentation of the contribution of the paper.

### Q3: how does the proposed modification of PPO algorithm overcome the drawbacks/difficulties of current RL approaches (to algorithmic fairness) outlined in the introduction?

In introduction (e.g., paragraph 2 of Section 1), the paper listed several drawbacks/difficulties of current RL approaches towards fairness in dynamic settings. I am wondering how the proposed technical modification of the PPO algorithm overcomes some or all of the listed drawbacks. In light of the questions detailed in Section `Questions`, while the experimental results are relatively well documented, readers can benefit more from clearer presentations of the takeaway messages of the paper.

 The paper studies the long-term fairness problem in MDPs by proposing advantage regularization policy optimization methods to achieve high overall utility and mitigate violation of the fairness requirements.  *Novelty* The paper studies the long-term fairness problem in MDPs. The problem is well-motivated, and this work is a novel combination of PPO and Lyapunov stability. Overall, the problem setting and methods are novel, but the authors fail to cite some recent works of fairness in MDPs [1-2].

*Quality*: The paper is technically sound. In particular, I find the authors have a reasonable and interesting RL environment setup. 

*Clarity*: The paper is well written, but the fairness definition is unclear at the beginning before the case study. 

*Significance*:  The long-term fairness problem is important. The paper seems to be a useful contribution to the literature in this area by proposing practical algorithms to achieve long-term fairness.   

[1] Towards Long-term Fairness in Recommendation, in WSDM 2021

[2] Towards Return Parity in Markov Decision Processes, in AISTATS 2022 I suggest the authors discuss the fairness notion in the problem setup. The fairness requirement is a function of the state, and depending on the context, it has different implications. I did not know the fairness notion until I read the experiment. The authors have adequately addressed the limitations and potential negative societal impact of their work.

 This paper suggests a method for optimizing a reinforcement learning model which has some auxiliary objective beyond reward (e.g. a fairness objective). They run some experiments and analyze the difference in performance between their method and several others, focusing on application to fairness-type questions, showing the advantages of their method. Strengths:
- this method seems very general and useful
- exposition of the method and motivation is very clear
- experiments support the claims of the method on a variety of simple RL problems

Weaknesses:
- the novelty of the work is not clear to me as I think I am not quite familiar enough with the relevant RL literature. I'm not sure how many similar methods exist for optimizing auxiliary objective functions like this
- I'm not quite sure how specific the proposed method is to fairness - if very specific, it should be explained why. If very general, I think the exposition of the paper should be a little bit different to communicate that it's a general method they are specifically exploring with respect to fairness

Minor points:
line 142: Q value is never defined
line 168: there are a few hyperparameters to balance this function (the \beta's). I'm not sure how this relates to the critique of existing methods on line 45, where the authors say that the hyperparameters weighting fairness and accuracy constraints may result in reward hacking.
line 168: I may just not be familiar enough to the relevant literature but I don't understand why the reference to Lyapunov stability is helpful here
Experiments: can you clarify more how to interpret the difference between A-PPO and R-PPO?
Line 303: what is the "min-max normalization" and what is it for? - Would be useful to clarify (to me) what the novelty is in this paper: is it the method for optimizing and auxiliary objective in the RL setting, the exploration of fairness, or something else?
- How general is the proposed method? If can be used beyond fairness but you're just choosing to explore fairness here for some reasons, would be nice to walk through that
- some clarifications on how to interpret the difference between A and R-PPO none

 The authors propose an approach to extend policy optimization algorithms to consider fairness requirement. To achieve this, they extend the MDPs to include two fairness related terms: (i) fairness metric regularization which penalizes when the fairness metric is above a threshold, and (ii) improvement over time term, which penalizes if the fairness metric increases in the next time step. This makes the model fairness aware. The extension proposed is simple, intuitive and easy-to-implement. Further, the experiments and results are neatly presented with several case studies. 

The primary weakness is the magnitude of novelty. The authors propose two simple regularization terms without any mathematical analysis. They have also not presented other possible alternatives they considered or any detailed justification about their final choice.
 * Did you try other regularization terms or variations of the value-thresholding term and decrease-in-violation term in Eq. 3? Can you justify mathematically why the terms used would be optimal? Yes, authors have adequately addressed the limitations and potential negative impact of their work.

Meta-review: I recommend acceptance due to the positive opinions of the reviewers. This paper proposes a method relevant to fairness in RL, for enforcing constraints during policy optimization. The reviews judge the method (inspired by Lyapunov stability ideas) and its empirical evaluation as technically sound. Many initial points of confusion or uncertainty were addressed and resolve during the author discussion. 

Decision: Accept