Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a modification to PPO algorithm and presents an algorithm to constrain advantage disparity in the dynamic setup. The empirical results on multiple practical scenarios are also provided.
strengths_and_weaknesses: The strength of the paper comes from a detailed presentation of the conducted experiments. The results are reported in three case studies (following D'Armor et al., 2020) and the comparisons involve various algorithms and fairness criteria.

Weakness of the paper comes from the way materials are presented. In particular, it would be very helpful if authors can clarify several questions (detailed in Section `Questions`):

Q1: what does the term "human-designed policies" mean in the summary of experimental results?

Q2: what is the contribution of the paper, is it in terms of technical modification of the PPO algorithm, or there are additional takeaway messages?

Q3: how does the proposed modification of PPO algorithm overcome the drawbacks/difficulties of current RL approaches (to algorithmic fairness) outlined in the introduction?
questions: ### Q1: what does the term "human-designed policies" mean in the summary of experimental results?

In presented experimental results, it is mentioned several times that the proposed A-PPO outperforms human-designed policies. I am a little bit confused of the used term here. Is the term "human-designed" referring to reward engineering (line 46)? If so, it seems that the proposed algorithm also involves an explicitly defined trade-off between utility and fairness violation, e.g., Equation 3. A further clarification would be very helpful.

### Q2: what is the contribution of the paper, is it in terms of technical modification of the PPO algorithm, or there are additional takeaway messages?

I think readers can benefit more from a clearer presentation of the contribution of the paper.

### Q3: how does the proposed modification of PPO algorithm overcome the drawbacks/difficulties of current RL approaches (to algorithmic fairness) outlined in the introduction?

In introduction (e.g., paragraph 2 of Section 1), the paper listed several drawbacks/difficulties of current RL approaches towards fairness in dynamic settings. I am wondering how the proposed technical modification of the PPO algorithm overcomes some or all of the listed drawbacks.
limitations: In light of the questions detailed in Section `Questions`, while the experimental results are relatively well documented, readers can benefit more from clearer presentations of the takeaway messages of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the long-term fairness problem in MDPs by proposing advantage regularization policy optimization methods to achieve high overall utility and mitigate violation of the fairness requirements. 
strengths_and_weaknesses: *Novelty* The paper studies the long-term fairness problem in MDPs. The problem is well-motivated, and this work is a novel combination of PPO and Lyapunov stability. Overall, the problem setting and methods are novel, but the authors fail to cite some recent works of fairness in MDPs [1-2].

*Quality*: The paper is technically sound. In particular, I find the authors have a reasonable and interesting RL environment setup. 

*Clarity*: The paper is well written, but the fairness definition is unclear at the beginning before the case study. 

*Significance*:  The long-term fairness problem is important. The paper seems to be a useful contribution to the literature in this area by proposing practical algorithms to achieve long-term fairness.   

[1] Towards Long-term Fairness in Recommendation, in WSDM 2021

[2] Towards Return Parity in Markov Decision Processes, in AISTATS 2022
questions: I suggest the authors discuss the fairness notion in the problem setup. The fairness requirement is a function of the state, and depending on the context, it has different implications. I did not know the fairness notion until I read the experiment.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper suggests a method for optimizing a reinforcement learning model which has some auxiliary objective beyond reward (e.g. a fairness objective). They run some experiments and analyze the difference in performance between their method and several others, focusing on application to fairness-type questions, showing the advantages of their method.
strengths_and_weaknesses: Strengths:
- this method seems very general and useful
- exposition of the method and motivation is very clear
- experiments support the claims of the method on a variety of simple RL problems

Weaknesses:
- the novelty of the work is not clear to me as I think I am not quite familiar enough with the relevant RL literature. I'm not sure how many similar methods exist for optimizing auxiliary objective functions like this
- I'm not quite sure how specific the proposed method is to fairness - if very specific, it should be explained why. If very general, I think the exposition of the paper should be a little bit different to communicate that it's a general method they are specifically exploring with respect to fairness

Minor points:
line 142: Q value is never defined
line 168: there are a few hyperparameters to balance this function (the \beta's). I'm not sure how this relates to the critique of existing methods on line 45, where the authors say that the hyperparameters weighting fairness and accuracy constraints may result in reward hacking.
line 168: I may just not be familiar enough to the relevant literature but I don't understand why the reference to Lyapunov stability is helpful here
Experiments: can you clarify more how to interpret the difference between A-PPO and R-PPO?
Line 303: what is the "min-max normalization" and what is it for?
questions: - Would be useful to clarify (to me) what the novelty is in this paper: is it the method for optimizing and auxiliary objective in the RL setting, the exploration of fairness, or something else?
- How general is the proposed method? If can be used beyond fairness but you're just choosing to explore fairness here for some reasons, would be nice to walk through that
- some clarifications on how to interpret the difference between A and R-PPO
limitations: none
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose an approach to extend policy optimization algorithms to consider fairness requirement. To achieve this, they extend the MDPs to include two fairness related terms: (i) fairness metric regularization which penalizes when the fairness metric is above a threshold, and (ii) improvement over time term, which penalizes if the fairness metric increases in the next time step. This makes the model fairness aware.
strengths_and_weaknesses: The extension proposed is simple, intuitive and easy-to-implement. Further, the experiments and results are neatly presented with several case studies. 

The primary weakness is the magnitude of novelty. The authors propose two simple regularization terms without any mathematical analysis. They have also not presented other possible alternatives they considered or any detailed justification about their final choice.

questions: * Did you try other regularization terms or variations of the value-thresholding term and decrease-in-violation term in Eq. 3? Can you justify mathematically why the terms used would be optimal?
limitations: Yes, authors have adequately addressed the limitations and potential negative impact of their work.
review_title: null

Meta-review: metareview: I recommend acceptance due to the positive opinions of the reviewers. This paper proposes a method relevant to fairness in RL, for enforcing constraints during policy optimization. The reviews judge the method (inspired by Lyapunov stability ideas) and its empirical evaluation as technically sound. Many initial points of confusion or uncertainty were addressed and resolve during the author discussion. 
confidence: Certain
Decision: Accept