Remembering to Be Fair Again: Reproducing Non-Markovian Fairness in Sequential Decision Making
Authors: Domonkos Nagy
Lohithsai Yadala Chanchu
Krystof Bobek
Xin Zhou
Jacobus Smit
Venue: Accepted by TMLR ()

Abstract: Ensuring long-term fairness in sequential decision-making is a key challenge in machine learning. Alamdari et al. (2024) introduced FairQCM, a reinforcement learning algorithm that enforces fairness in non-Markovian settings via memory augmentations and counterfactual reasoning. We reproduce and extend their findings by validating their claims and introducing novel enhancements. We confirm that FairQCM outperforms standard baselines in fairness enforcement and sample efficiency across different environments. However, alternative fairness metrics (Egalitarian, Gini) yield mixed results, and counterfactual memories show limited impact on fairness improvement. Further, we introduce a realistic COVID-19 vaccine allocation environment based on SEIR, a popular compartmental model of epidemiology. To accommodate continuous action spaces, we develop FairSCM, which integrates counterfactual memories into a Soft Actor-Critic framework. Our results reinforce that counterfactual memories provide little fairness benefit and, in fact, hurt performance, especially in complex, dynamic settings. The original code, modified to be 70% more efficient, and our extensions are available on GitHub: https://github.com/bozo22/remembering-to-be-fair-again.

Published in Transactions on Machine Learning Research (05/2025)
Remembering to Be Fair Again: Reproducing Markovian Fairness in Sequential Decision Making
Non-
Domonkos Nagy* Informatics Institute University of Amsterdam
domonkos.nanyy2@sstuden.
Lohithsai Yadala Chanchu" * Informatics Institule
whhithsai.addala.coo www.aaaa..caacccnoocsaaa........
University of Amsterdam
Kryštof Bobek
brgstof.bobckk@ssuudde...
Informatics Institute University of Amsterdam
Xin Zhou"
xin, St..sho@@student. uvo.
Informatics Institute
Universily of Amsteriam
Martin Smit Informatics Institute
J.m.m. mw..msmmit... Curva.nl
University of Amsterdam
Reviewed OpenReview:
net/ forum? id= H6DtMc2f5s
Abstract
Ensuring leng-term fairness in sequential decision-making is a key challenge in machine learn- ing. Alamdari et al. (2024) introduced FairQCM, a reinforcement learning algorithm that enforces fairness in non-Markovian scttings via memory augmentations and counterfactual reasoning. We reproduce and exlend their findings by validating their claims and introduc- ing novel enhancements. We confirm that FairQCM outperforms standard baselines fair- ness enforcement and sample efficiency across different environments, However, alternative fairness metrics (Egalitarian, Gini) yield mixed results, and counterfactual memories show limited impact on fairness improvement. Further, we introduce realistic COVID-19 vaccine allocation environment based on SEIR, a popular compartmental model of epidemiology. To accommodate continuous action spaces, we develop PairSCM, which integrates counterfac- tual memories into a Soft Actor-Critic framework. Our results reinforce that counterfactual memories provide little fairness benefit and, in fact, hurt performance, especially in complex, dynamic settings. The original code, modified be 70% more efficient, and our extensions are available GitHub: https: /ttp//ggttttb/t www/bozo22 htt.://ooo2eeermmmmeernn-tt-ttoo fa-
1 Introduction
As machine learning is increasingly involved high-stakes decision-making lasks such as healthcare triaging Salmerró al. 2022) and loan approvals (Sheikh et al.) 2020), becoming ever more critical understand and address the risks of deploying such system. One associated risk that directly affects end- users is that of fairness, in which decision-maker may, intentionally otherwise, benefit certain users over others. However, as shown in numerous studies (Liu et al. 2018 Hashimoto et al.) 2018 Hu and Chen 2018 D'Amour et al.) 2020), actions that promote fairness in the short term can result in unexpected negative outcomes in the long term, which motivates the need to consider fairness in a non-Markovian light.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (05/2025)
Consider, for example, two countries that each receive equal share of COVID-19 vaccines to promote fairness, allocated on month by month basis. Even if they end up with the same amount of vaccines (which we would intuitively consider a fair allocation), it might be the case that one country received all its share at once, while the other only received it in small portions over time. Although the equal allocation appears fair in the long term, over time the outcomes diverge sigrificantly, highlighting the need to consider fairness over the entire trajectory rather than in isolated, immediate decisions.
Sequential macisinn-making is the process of making decisions where each action influences future states and subsequent choices. Unlike static decision- making, where choices are made in isolation, sequential decision- making considers the long-term consequences of actions. Naive sequential decision-making systems face the challenge of decisions that can get stuck in feedback loops. These feedback loops can reinforce existing disparities, potentially creating a vicious cycle (Chohlas- Wood et al., 2023). An example of where such self-reinforcing loops can be found is predictive policing systems (Ensign et al. 2018). These systems analyze crime data to forecast where future crimes are likely occur, guiding law enforcement resource allocation. However, if the data used is biased rellecinn over-policing in certain communities the system may perpetuate and worsen this bias. For instance, increased police presence in a neighborhood leads to more recorded offenses, which the system interprets as higher crime rates, prompting even more policing in that area. This creates a self-reinforcing cycle that disproportionately targets specific communities, exacerbating existing disparities.
These loops not only harm those directly affected in the short term, but also compound disadvantages over time. In recognition of these challenges, there is a growing ambition to leverage reinforcement learning promote long-term fairness in algorithmic decision-making (Nashed et al. 2023 given its ability to balance immediate gains with desirable outcomes across a broader time horizon.
Past work that considers long-term fairness includes Jabbari et al. (0117) who examined approaches for adapting fairness principles within RL systems. They emphasized a dynamic and future-aware perspective on fairness, ensuring that fairness is preserved over the course of many decisions rather than focusing solely on individual actions in isolation. Sequential decision-making in Markovian settings has also been studied in Hu et al. (2024), which examines fairness in dynamic systems. These dynamic systems are modelled by Markov decision process (MDP), where decisions made sequentially, acknowledging that each decision can alter the underlying distribution of features or user behavior. However, Alamdari et al. (2024) clain that fairness assessments often depend on past actions taken during the decision satsiinnmaiing process, rather than just the current state, rendering them inherently non-Markovian. To address this, they introduce the concept of non- Non-Markovian fairness, exploring how memory and historical context can support the construction of fair policies sequential decision-making settings. They propose FairQCM, a reinforcement learning approach designed to enhance fairness by leveraging counterfactual experiences, which allow the agent to learn how its actions would perform in hypothetical situations. While the results of the paper promising, a number of the claims made paper lack experimental verification. Namely, while they claim that the method works in a wide variety of scenarios, motivated primarily on the COVID-19 vaccine roll-out problem mentioned above, their experiments do not include heterogeneous or dynamic stakeholder behavior, two key aspects of more realistic scenarios.
In this report, we first present and attempt reproduce the main claims of (Alamdari et al. 2024 before conducting additional experiments that show how counterfactual experiences may negatively allect learning in environments with dynamic stakeholder behavior. In particular this report shows that:
The ratio of counterfactual to real memories is an important hyperparameter affecting the perfor- mance of FairQCM.
Counterfactual memories may actively hinder the learning of an effective policy in the case that the dynamics of the underlying system are more complicated. This is exemplified in our vaccine allocation scenario where the spread of an infection is modelled with an SEIR model (see section 3.2.4p.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2 Scope of reproducibility
One of the main contributions of Alamdari et al. (2024). is their proposal of the Non-Markovian Fair Decision Process (NMFDP) framework. This allows standard algorithms for Markovian problems, such as Deep Q- Networks (DQNs) (Mnih et al.) 2015), to be used for the non-Markovian problem of sequential fairness by extending the state with a "memory" that stores relevant information from preceding time steps.
They also introduce FairQCM, which uses DQN as a baseline but extends with counterfactual memory augmentation. It uses the NMFDP framework to help generate long-term fair policies that consider past actions. Their work also introduces methods of expressing several different forms of fairness over time and explores the effect of memory augmentations fairness performance.
We aim to reproduce the following claims made by Alamdari et al. (2024), which are tested using the Resource Allocation (Donut) (Katoh and Ibaraki 1998) and Simulated Lending (Liu et al. 2018) environments utilizing their reproducibility package
Claim 1: Counterfactual reasoning improves fairness enforcement and sample efficiency.
Claim 2: Memory augmentations improve model performance significantly.
To investigate the first claim, we aim to reproduce the Full and FairQCM trials from the paper for both euvironments. We investigale if FairQCM, which uses counterfactual memories, performs better in both enviromments than the Full baseline (which consists of regular DQN without counterfactual memories).
To further investigate the impact that counterfactual memories have on fairness performance, we test with varying balances of counterfactual memories to real memories (with a fixed memory busfer) to see what the optimal balance is.
To investigate the second claim, we replicate the trials for FairQCM and the other augmented memory baselines explored in the original paper to compare their results for both environments. We add No Memory baseline to determine to what extent memory is important for the investigated tasks.
The original paper proposes vaccine allocation as an idcal real-world use case for FairQCM and its non- Markovian framework, although did not evaluate it on environments that embodied the important traits of a realistic vaccine allocation environment. Such an environment is complex in the sense that it has changing dynamics sensitive to time and past actions. Additionally, FairQCM must demonstrate robustness across various fairness scoring functions, each of which offers distinct advantages and limitations that align with specific and potentially different fairness objectives users might have. The original paper tested FairQCM on environments with discrete action space. A realistic COVID-19 vaccine allocation environment (and many other sequential decision-making tasks) requires a continuous action space, which would need a different off policy learning algorithm. This leads naturally to the following extensions:
Varying the aggregation reward function We test the fairness performance of FairQCM under dif- ferent fairness scoring functions and compare it with various memory-angmented baselines. While many scoring functions were suggested in the original paper, only a limited number of them were actually lested. As such, it is not clear which fairness metrics can be effectively used by FairQCM.
Introducing more complex stakeholder behavior: We introduce dynamic stakeholder behavior and dynamic behavior that depends on the agent's actions. This allows us to evaluate the original claims in scenarios with greater complexity and real-world accuracy.
We design COVID-19 vaccine allocation environ- ment according to the SEIR model (iillett (1922) with realistic hyperparameters for the COVID-19 virus, which allows modeling different populations, unique start states for each group, variable infection spread rales, and customized vaccine production schedules. We also propose a modified version of FairQCM called FairSCM, that works with continuous action spaces using the Soft Actor-Critt model.
Assessing FairQCM in realistic environment:
http:://gttuub.com/praa////eeeee
3

---

Published in Transactions on Machine Learning Research (05/2025)
Different types of memories stored by different baselines and the proposed FairQCM. U(T.)) is a vector consisting of the utility values cach of the stakcholders, and denotes the number of stakcholders.
Table 1:
Method
Way in Which Memory Is Represented
Full
U(T)
Min
min
min Urr);
Instead of storing memories directly, it uses recurrent Q-network to remember relevant information from past actions.
RNN
Reset
Stores U(Te), and whenever the utility values are all equal, resets them to 0.
UTTT):
in resource allocation environment
FairQCM
U(T)) - min mimu (Tr), in simulated lending environment
3 Methodology
3.1
Methodology reproducing original paper
The replication package of the original paper was used to reproduce its experiments, in which the proposed method, FairQCM and four basclines were tested: Full, Min, Reset and RNN. Besides the above five methods, we also added a No-Memory baseline.
3.1.1 Introduction to the proposed method and baselines
The proposed method, Fair Q-Learning with Counterfactual Memories (FairQCM) is based Deep Q Networks (DQN) Mnih et al.] (2015). Its main contribution is that instead of feeding states of the envi- ronment into the Q-network, FainQCM uses memory-augmenter states. This augmentation converts a non-Markovian fairness problem into a Markovian one by keeping track of information related to past actions. For example, if the agent's goal to distribute goods between different stakeholders, could store the amount goods it has given to each stakeholder up the current time step in the memory.
At time step f in trajectory t, the stakcholder status U(T)) is vector consisting of the utility values of each of the stakeholders. The differences between the proposed baselines are the types of memory they store, the details of which are found in Table 1 Notably the RNN considers actions and memories in an end-to-end fashion, as memories are not stored explicitly, but are implicitly represented in the Q-network itself.
Although the original paper explored how different memory representations affect performance, they did not compare the performance of the various memory-augmented baselines to a classic memory-less baseline, which does not extend state with a memory. This is a crucial part of the evidence suggest that memory is needed to perform well at the examined tasks of resource allocation and lending, and gives more insight into how memory augmentations improve performance. This is why we added the No- No-Memory baseline, where the DQN receives the state and no memory.
We also investigated how performance depended on the composition of the replay buffer in terms of regular memories, those experienced by the agent during training, and counterfactual memories, those constructed algorithmically aid learning (sce Scction 3.2.1p. By analyzing different compositions of the replay buffer, we can assess whether an optimal ratio exists that maximizes learning efficiency while maintaining fairness and stability. This investigation provides insights into the role of counterfactual experiences in shaping policy updates and whether their inclusion leads to more robust generalization.
3.1.2 Resource allocation environment
The donut allocation experiment has the setup of n 5 customers, with customer i being in front of the counter with constant probability p, 0.8 at each time stop t. The domut shop bakes one domut at cach time step, and the server chooses one customer to give out the donut to. If the server chooses customer that is not at the counter, the donut is wasted. The fairness scoring function W serves the reward function. Tn
4

---

Published in Transactions on Machine Learning Research (05/2025)
the original paper, the reward is based on the Nash welfare score as shown in Equation
We ((U(T:)) (1o(()s((TT)) + 1)) Σ (og(U(T)):+
(1)
where ( is the number of donuts allocated customer so far at time step t. The agent's ' goal then is allocate donuts in fair manner across customers, as measured by the Nash welfare score.
We averaged the results over 10 experiments, each consisting of 500 episodes of 100 time steps. We used the same training hyperparaneters as the original paper, which are detailed in Appendix A.2
3.1.3 Simulated lending environment
In this experiment Alamdari et al. (2024) adopt the lending environment proposed by Liu et al (0118) as version of the established fair machine learning testbed, consumer lending Dwork et al. (2012 Hardt et al. (2016). The original experiment examines four loan applicants divided into two protected groups (A and B), with each applicant characterized by a credit score C that represents the probability of loan repayment. The credit score distribution varies between these two groups. At each step, a subset of applicants applies for a loan, and the agent (bank) determines which applicant to approve. Each applicant i applies with a fixed probability of p; 0.9. Successful loan repayment increases the bank's utility by and the applicant's credit score by e, while defaulting decreases both. States include the subset of applicants applying for loans, their credit scores, and the cumulative profit margin.
Fairness in Alamdari et al. (0244) is assessed per timestep using the Relozed Demographic Parity (DP) Score:
W(U(r)) = - EU(o)
where UTT);; represents the cumulative number of loans granted to applicant i up to time step t. The agent's goal is to maximize the discounted sum of W a U, subject to two constraints. A significant negative reward is imposed if at least 10% profit margin is not achieved by the end of an episode, and a minor penalty is applied for granting loans to non-applicants.
The following baselines are used for the evaluation of the experiment: Full, Min, and RNN. The Min memory is also utilized by FairQCM.
3.2 Methodology beyond original paper
The original paper proposes vaccine allocation as real-world use case for FairQCM but did not test it in environments that reflect the complexities of sequential decision-making, where stakeholder behavior changes over time. Additionally, FairQCM should be robust across various fairness scoring functions, cach suited to different fairness objectives. We aim lo explore the performance of FairQCM on different fairness scoring functions and dynamic stakeholder behavior. Furthermore, FairQCM is limited by the fact that it can only work with discrete action spaces, whereas many realistic sequential decision making tasks (such as vaccine allocation) are best modeled using a continuous action space, and consequently, a different off off-policy learning approach, such Soft Actor-Critic (Haarnoja et al., 2018
3.2.1 Evaluating the impact of counterfactual memories on fairness
To further investigate the impact that counterfactual memories have on fairness performance, we test with varying balances of counterfactual memories to real memories (with a fixed memory buffer) to what the optimal balance is. In the Domut environment with 5 stakeholders, we analyzed how FaiQQCM's performance varies with the number of counterfactual memories stored in a fixed replay buffer of size 6,400. During training, FairQCM samples from this buffer, which we populate with different amounts of counterfactuals per time step. We define m as the number of maximum additional donuts each stakeholder could receive, ranging from yielding 35 243 counterfactual states. We tested m [0, 1, 4), corresponding to 0, 1, 32, 243, and 1,024 uaunterfactuals per time step.
For instance, with m 2, cach stakcholder could have 0, 1, or 2 additional donuts,
5

---

Published in Transactions on Machine Learning Research (05/2025)
3.2.2 Exploring different fairmess scoring functions
In the original paper, only the Nash welfare score was applied as a reward function. order to verify the performance of FairQCM using different definitions of fairness, we use the Egalitarian and Givi welfare scores as reward functions train separate models from scratch for the Donut environment. Compared to Nash, these welfare scores emphasize demographic parity with decreased focus achieving high overall utilities.
The Egalitarian reward function (as shown in Equation 3 aims to minimize the differences from the mean, that is, equal the number of donuts allocated to each customer.
W₁ (U.(T))
Σ U(TT):
(3)
The Gini reward function at time step t is defined in Equation
Wt = 1 1 where G= Ei=1 Ez= (
(4)
(T)
G is a measure of inequity, defined as the mean of absolute differences between all pairs of individuals for some measure, which is ( E f1,2...., n in our case. The minimum value of is zero when all the & E (1,2, ..., n are equal. The thcoretical maximum is one for an infinitely large set of individuals where all i E f1. n) is zero except one individual (ultimate incquality).
Besides, also introduce the Rawlsian reward function as below, which is the reward function of COVID-19 allocation simulation environment. It aims to maximize the minimum number of the utility:
wr ((U()))) minn)(()))
(5)
We applied the Egalitarion and Gini fairness scores to train and evaluate FairQCM along with four baselines: Full, Min, Reset and RNN. We averaged the results over 10 experiments, each consisting of 1000 episodes of 100 time steps. We used the same training hyperparameters as the original paper which can be found in the Appendix A.2
3.2.3 Adding dynamic stakeholder behavior
The original paper considers customer appearance probabilities that are both constant over time and uniform across customers. To better simulate real-world variability, we introduce two new variants in the donut allocation example.
the first experiment, we assign distinct, static appearance probabilities to each customer: p = [0.6, 0.8, 0.9, 1.0). These values are chosen arbilrarily and do not change throughout the episode.
In the second experiment, we introduce an action-dependent probability for customer appearance. Initially, each customer has probability of p 0.6 of appearing. However, when customer reccives donut, their probability of coming in the subsequent round increases by 0.1. Conversely, if customer does not receive a donut for two consecutive iterations, their appearance probability resets to 0.6. This mechanism allows customer appearance dynamics to be inffuenced by past allocations, introducing simple agent-driven nonstationarily.
3.2.4 COVID-19 allocation simulation
To investigate the applicability and effectiveness of counterfactual memory augmentations in addressing complex, real-world problems, we created a COVID vaccine allocation gym environment. In this scenario, supply constraints necessitate that allocation decisions be made over time, with each decision impacting future outcomes. For example, prioritizing specific demographic in one period can influence overall pop- ulation immunity or fairness in subsequent periods. As the vaccine allocation strategy greatly affects all infected regions, this problem serves perfect testbed to compare different fair learning methods.

---

Published in Transactions on Machine Learning Research (05/2025)
Furthermore, vaccine allocation problems are often modeled as multi-stakcholder systems, where regions, hospitals, or demographics act as agents competing for limited resources (Li and Huang, 2022 Yarahmadi ett al. 2023; Dalgıç et al. 2017). The ability to integrate fairness into such multi-agent systems provides a robust framework for tackling these challenges. By combining previous extensions of our experiments, such as the behavior under different aggregation or reward functions and the adaptability to dynamic conditions, we found it to be naturally suited to study the task of vaccine allocation.
We developed our vaccine allocation environment to reflect both the complexity of discase spread and the challenges of distributing limited resources fairly, At its core is the well-known SEIR model (Taghizadeh and Mohammad-Djafari 2022), an opidemiological framework that categorizes individuals according to their disease status: Susceptible (S) for those who can contract the disease, Exposed (E) for those infected but not yet infectious, Infected (I) for those actively transmitting the disease, and Recovered (R) for those
who have either gained immunity or passed away. The model captures how individuals transition among these compartments over time as governed by Equation G
ds
d = --B-S
AE AE d
3 -
di d E O.
7.
dR d
-1
(6)
N
where N is the total population size; f is the transmission rate; a is the rate of progression from exposed to infectious (1/a is the incubation period); is the recovery rate (1/7 is the infectious period).
Our environment models multiple regions with distinct populations, infection dynamics, and initial states (e.g.: exposod or infected counts). It reflects real-world disparitics, where infection rates and susceptibility vary by demographics and infrastructure. Additionally, vaccine production fluctuates over time, adding com- plexity to allocation decisions. To address this, we implement customizable vaccine production schedule, requiring the allocation agent to balance infection control and fairness under dynamic supply conditions.
We define the stakeholder utility for region i at time step L as:
U(T)): (.04(V(T)): - P(T.))) - E(Te);
where V(T:); is the percentage of allocated vaccines up to time step that were given to region i, P(Tc)s is the percentage of the overall population that lives in region i, and E(Tr); is the percentage of the overall population in region i that got exposed to the virus at time step t.
Intuitively, we can assume that each region has two main objectives: they want to minimize the number of inhabitants that become exposed to the virus (hence -E(r))), while also making sure that the percentage of vaccines they receive is not disproportionately smaller than the percentage of the overall population they account for (hence V(rz): - P(rr).). We can further assume that. the main focus of each region is to reduce the number of exposed inhabitants (as it has a direct effect on the spread of discase), while maintaining fair proportion of received vaccines can be thought of as a regularizer, hence the scaling factor 0.04. The reward at each time step is the Rawlsian welfare score. The utility and aggregation functions serve as feasible fairness baselines, and are by no means attempts at establishing an objective measure of a fair vaccine allocation.
As our environment has continuous actions, using DQN would only be possible with severe limitations, such as quantizing the action space. To avoid such limitations, we opted to use Soft Actor- Crrtic (SAC) (Haarnoja) et al., 2018) instead, which is an off- off-policy learning method with continuous actions. Similar to FairQUM, we refer to the momory-augmente version of SAC as FairSCM (Fair SAC with Counterfactual Mcmories).
3.3 Computational results
The reproducibility of the results was based on the original code repository. We ran experiments using an AMD Ryzen 2600 CPU and a Nvidia RTX 3060 GPU. This original code took around 24 hours to run all the experiments. We significantly improved the efficiency and reduced the time to run all experiments to 7 hours, which is a speedup of around 70%. Running all experiments (reproducing original claims as well as running our extensions) took less than 12 hours in total. This code is available on GitHut The speedup was partly achieved through modifying the code to take advantage of GPU.
hhttp:///gthuu.ccm//ooo///////// httpa////.w..eeee////errrrrrrgea to- btto-b--aade -o-bt--------aaaee fair-

---

Published in Transactions on Machine Learning Research (05/2025)
4 Results
We were able to successfully reproduce the original experiments, running FairQCM and the other baselines on both the resource allocation and the simulated lending environments. In addition replicating the original findings, we extended the experiments to examine how FairQCM performs under alternative fairness metrics and within dynamic environments.
We evaluated different fairness scoring functions beyond those initially tested. We also explored FairQCM's adaptability environments where conditions change over time, specifically focusing on scenarios where stakeholder probabilities evolve dynamically.
4.1 Results reproducing original paper
4.1.1
Resource allocation
1000
97.5
900
Accumulated NSW
800
Allocated
92.5
700
FairQCM
87.5
Full
Full
85.0
NoMemory
82.5
NoMemory
Reset
80.0
Reset
300
Number of Episodes
Number Episodes
Figure 1: Replication of the resource allocation experiment from Alamdari et al. (2024) with the additional No-Memory baseline. Results correspond with the original paper.
The replication results shown in Figure | align with the results in the original paper. They support the claims that FairQCM outperforms other baselines and learns to allocate donuts effectively. Similarly to the original paper, Min has the least Nash welfare score, and RNN adapts faster than others to learn to allocate the donuts effectively.
Claim 1: Our results confirm that FairQCM outperforms the baselines, particularly in the resource al- location environment, where it achieves higher fairness and faster learning. Compared to Full, FairQCM demonstrates greater sample efficiency, requiring fewer steps for improved performance.
Our second focus was on memory augmentations and their significant impact on model perfor- mance. The results in both environments validated that different memory structures, such as Full, Reset, and RNN, influence fairness outcomes in distinet ways. We the same ordering in the performances of the different memory augmentations as in the original paper.
Claim 2:
4.1.2 Simulated lending
Our replication of the experiment verifies key claims made by the original paper. All baseline methods achieve the desired profit margin, supporting the validity of the experimental setup (Figure 2).
Claim 1: This claim holds in our replication. FairQCM, which enhances the Min baseline with counter- factual experiences, achieves higher fairness score than Min alone. Tt. also demonstrates higher sample efficiency, improving more rapidly than the other tested baselines.
Claim 2: Our findings support this claim. All memory-augmeenne baselines outperform both the RNN and No-Memory baselines by substantial margin.
8

---

Published in Transactions on Machine Learning Research (05/2025)
FairQCM
-200
Min
NoMemory
-300
PNNN
Accumulated Pelexed
-400
-500
-600
-700
-800
Number of Episodes
Figure Replication of the simulated lending experiment from Alamdari et al. (2024). incorporating an additional No-Memory baseline. The results align with the original findings, reinforcing their conclusions.
However, we observed that the Full and Min baselines performed better in our replication than reported in the original study. This discrepancy occurred despite using the provided replication package, the same seed, and the exact command sequence described by the authors.
4.2 Results beyond original paper
4.2.1 Effect of counterfactual memories on fairness
Although results from the original paper suggest that counterfactual memories improve fairness performance, further analysis of counterfactual balance experiments (Figure 3 suggests that the specific proportion of counterfactual and real memories in the replay buffer influences fairness performance significantly as well: too many counterfactual memories perform worse than no counterfactual memory augmentation present. This might be due to the increasing proportion of unrealistic counterfactual states that increase with n. With higher values for n, more states that would normally never appear in trajectory would be produced as counterfactual states, that the DQN learns from. These additional "impossible" counterfactual states add noise to the learning process. For low values of n, the additional benefit of increased (realistic) counterfactual memories seems outweigh the performance degradation caused by the added noise (of the unrcalistic counterfactual memories).
4.2.2 Results of different fairness scoring functions
In Figure f. FairQCM and four other baselines Full, RNN, Reset, and Min are trained using the Egali- tarian welfare score as the reward function and evaluated with the same score as the metric. In Figure 4 all five models are trained and evaluated using the Gini welfare score. As shown in Figure 4b, all basclines except FairQCM tend to give out fewer donuts in total as the training progresses. This likely due to the fact that the Egalitarion welfare score considers only the fairness of the allocation but not the overall amount of stakeholder utilities, wasting donuts does not decrease the reward. RNN achieves highest accumulated Egalitarian scores but this is because it learns to give out least amount of donuts. In contrast, FairQCM lends not to waste more donuts as it learns fair allocation policy and thus performs better than other baselines. However, Figure 4p shows that FairQCM underperforms Full, Min and Reset both in terms of welfare score and number allocated donuts under the evaluations of Gini welfare score. Based on these results, FairQCM is less "fair" across stakeholders and also produces less utility since it tends to waste more donuts compared to other baselines.

---

Published in Transactions on Machine Learning Research (05/2025)
1000
Accimulated NSW Scores
700
Thme
400
200
500
Episodes
Figure 3: FairQCM's performance the Donut environment (5 stakeholders) as a function of counterfactual memorics in fixed 6400-memory replay buffer. As E {0, 2, 4} increases, the number of counterfactual states grows exponentially. Too few counterfactuals limit learning, while too many overwhelm the bulfer, indicating optimal range.
Acommdated Eqaiitarian Scores
FACCCM FAWN Pout
Number
Number Episodes
Number
(a) Results of Egalitarian fairness
Number Albeated Danats
Accumllzed Gini Scaren
M/
Number of Episodes
Number of Episodes
(b) Results of Gini fairness
Figure 4: FairQCM underperforms RNN (top left) but learns to allocate donuts effectively (top right) under the guidance and assessment of Egalitarian welfare score; FairQCM underperforms Full, Min and Reset (bottom left) and wastes more donuts under the guidance and assessment of Gini welfare score.
Overall, FairQCM performs worse under the evaluations of the Gini welfare score compared to the Egalitarian and Nosh welfare scores. It is an interesting question for future work how well FairQCM can generalize multiple scoring functions simultaneously.
10

---

Published in Transactions on Machine Learning Research (05/2025)
1100
1100
1000
1000
Accumulated NSW Scores
wmm
900
Accumulated NSW
800
700
600
FairQCM
600
FairOCM
Full
Full
500
Min
500
Min
RNN
RNN
400
Reset
Reset
200
400
1000
200
400
600
1000
Number of Episodes
Number Episades
(a) Results of the resource allocation experiment with distinct constant customer probabilities p [0.6, 0.8, 0.9, 1.0j. A significant performance drop is evident for the RNN, whereas FaiQQCM per- forms comparably to other baselines and achieves the fastest convergence.
(b) Resource allocation experiment with dynamically changing customer probabilities, illustrating the derperformance of RNV N. The initial probability of customers coming is set to p 0.6, increasing by 0.1 for the customer receiving donut, resetting to the original value after 2 steps without donut allocation.
Figure Results of extended resource allocation experiments
4.2.3 Dynamic stakeholder behavior
The results reveal a significant performance drop in the RNN and Min baselines, which struggle to adapt the incrcased complexity of the environment for both distinct constant (Figure 5a) and dynamically changing probabilities (Figure 5b). In contrast, the FairQCM approach confirms its fast convergence and sample efficiency across both experimental settings, even achieving superior results in the action-lependent environment.
4.2.4 Evaluation of FairSCM under the COVID-19 vaccine allocation simulation
Our results show that all baselines were able successfully optimize for our dual reward function. As Figure G demonstrates, both the overall number of infections and the inequality in vaccine allocations decrease as the models learn, while the cumulative Rowlsian welfare scores increase. Thus, the models learn policy that distributes vaccines fairly between regions while also controlling the spread of the virus effectively.
The results also demonstrate that counterfactual memory augmentations for this complex environment have negative effect on performance. The policies that FairSCM learns underperform other baselines both terms of fairness (cumulative Rawlsian welfare) and effective discase control (number of infections across all regions). We hypothesize that the reason behind this decrease in performance is a result of filling the buffer with noisy counterfactual transitions. For simpler problems such donut allocation, the model can learn the environment dynamics casily, and the counterfactual transitions can provide additional robustness by showing the model transitions that might be unlikely under its policy. However, for more complex environments such as our COVID-19 simulation, the model already has a difficult time figuring out the dynamics of the environment, so an extensive amount improbable counterfactual transitions can effectively act as noise, hindering the model's ability to understand the true dynamics of the environment.
5 Discussion
Based on the results reproducing the original paper, we verified the two main claims which highlight the important roles of counterfactual reasoning and memory augmentations. However, we showed that these contributions do not generalize well to more complex environments and different fairness scoring functions.
11

---

Published in Transactions on Machine Learning Research (05/2025)
FairSCM (Full)
-0.15
Full
360
0.050
Number infected Regions, in Millions)
Allocation Imbalance
-0.20
Random
0.025
Cumulative Reward
255
0.000
FairSCM
-0.30
Full
-0.025
-0.35
350
Min
Random
-0.40
345
-0.075
-0.45
Region
-0.100
Region
-0.50
Region
340
-0.125
250
500
750
1000
1250
1500
0
250
500
750 1000 1250
1500
0
250
500
750
1000 1250
1500
Episodes
Episodes
Episodes
Figure 6: Results of the vaccine allocation experiment with the mean values plotted and the standard deviations shown shaded areas around the mean. The left plot shows the cumulative rewards, while the center plot shows the overall number of people who get infected during the simulation. The right plot shows the vaccine allocation imbalance (difference between percentage of allocated vaccines and ratio of overall population) for each of the three regions for the Full baseline.
The performance of FairQCM under the evaluation of the Gini welfare score suggests that counterfactual memories do not always improve fairness. Furthermore, our results in the COVID-19 environment show that counterfactual memories for complex environments can have a negative effect on the model's fairness performance.
The replication of the original results is not difficult to obtain the authors provided a sufficiently working codebase, however their implementation only supports training on the CPU, which is inefficient. In addition, the codebase contains many pieces of redundant code, the author copies classes and functions for different environments (resource allocation and simulated lending). We made additional efforts to aggregate all the source files and extend them to enable the training on GPU. Furthermore, we found some mathematical definitions, design choices and surprising results in the original paper somewhat difficult to understand. Wo sent our questions to the authors, but have not received response by the time of submission.
Environemntal Impact: Our optimizations of the original codebase yielded a 70% speedup, reducing the energy consumption to similar degree. While the reproducibility experiments took only 7 hours to run, running and finc-tuning our extensions (including re-rumning experiments for testing and hyperparamoter op- timization) look approximately hours in total. Experiments were conducted using private infrastructure, with an estimated carbon efficiency of approximately 0.22 kgCO2eq/kWi (average global energy efficiency from Ritchie et al. (2020)) Overall, 87 hours of computation was performed an NVIDIA RTX 3060 GPU (170W TDP). Total emissions are estimated to be 3.2538 kgCOzeq. To put this into perspective, single cow produces an estimated 8.2-11 kgCOzeq per day (Garcial (024)).
References
Parand A. Alamdari, Toryn Q Klassen, Elliot Creager, and Sheila A. Mcllraith. Remembering to be fair: 04772.
mon-markovian fairness in sequential decision making, 2024. URL
Alex Chohlas- Madison Coots, Sharad Goel, and Julian Nyarko. Designing equitable algorithms, 2023.
URL https://arxiv. org/ 2ttp//232000515
Özden O. Dalgıç, Osman Y. Özaltın, William A. Ciccotelli, and Fatih S. Erenay. Deriving effective vaccine al- location strategies for pandemic influenza: Comparison of an agent-based simulation and a compartmental model. PLOS ONE, 1212)211 February 2017. ISSN 1932-6203. doi: / 012261,
URL http: doi 1tw..//////..1////rrr/a....a pone 0172261
12

---

Published in Transactions on Machine Learning Research (05/2025)
Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D. Sculley, and Yoni Halpern. Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Trunsparency, FAT '20, page 525 534, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369367. doi: 10.1145/8351095. 3372878. URL hattps://dii org/10..145//35511 11w.1/151/3315.... 15145//885099333788888
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214 226, 2012.
Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasuubammanian. Runaway feedback loops in predictive policing. Tn Sorelle A. Friedler and Christo Wilson, editors, Pro- ceedings of the 1st Conference on Fairness, Accountabilityy and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 160 171. PMLR, 23 24 Feb 2018. URL press/v81/ensic
mlr.
html
Alvaro
Garcia.
Dairy
coWS
and
corn
farming:
Balancing
methane and
0o
scquestration,
Dec 2024.
URL https://w..aprroud com/ articles/
60618 wow.aair--ooossaddooooooo -anoo---aaamagglllllaaaannnnnane www.-aaaeaaaaadenooeee-ee----eee
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-poliey maximum
entropy deep reinforcement learning with stochastic actor. CoRR,, abs/1801.01290, 2018. URL http:
7/arxiv. www.ooggg/bbb/eeeeeeeeeee a10aaaas....11////////////11111a 1etps/199000009000
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Tatsunori B. Hashimoto, Megha Srivastava, Hongsook Namkoong, and Percy Liang. Fairness without demo- graphics in repeated loss minimization, 2018 URL https://arxiv. wttp://w.rgrabb///eaeeaaae 1800/8880..0 08010
Lily Hu and Yiling Chen. A short-term intervention for long-term fairness in the labor market. In Proceedings of the 2018 World Wide Web Conference on World Wide Web WWW '18, www '18, page 1389 1398, ACM Press, 2018. doi: 0.1145/3178876.318860
doi Uttp:tt::/// http::///xx.d...og////..... 11011114111188 3186044 www.:////11111//18//118111111111
Yaowei Hu, Jacob Lear, and Lu Zhang. Striking a balance in fairness for dynamic systems through rein- forcement learning, 2024. URL https://arxiv.or http:/awwwvv.r.ora/.a.e.a abs 0abbs//441.0660
Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in rein- forcement learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1617- 1626. PMLR, 06-11 11 Aug 2017. URL ltttps:/ mlr -pppes//v0//
Naoki Katoh and Toshihide Ibaraki. Resource allocation problems. 1998. URL https://api. semanticscholar. :ttp::aaa...o//oooss::::::11:1
Qingfeng Li and Yajing Huang. Optimizing global covid-19 vaccine allocation: An agent-based computational model of 148 countries. PLOS Computational Biology, 18(9):e1010463, September 2022. ISSN 1553-7335 doi: 10.1371 / peb. 1010463. URL http::ttt/./pddee http:///dx..oi..o/10/../ 1ttp://///...1//rrrraooo.m pcbi. 1010463
Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning, 2018. URL
04383
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattic, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015. URL hattps://app semanticscholar oww.aaaaaaaaabaaaooooooooaaaaaaa (ot.a/rrroo/opps00-2222222000000
13

---

Published in Transactions on Machine Learning Research (05/2025)
Samer B. Nashed, Justin Svegliato, and Su Lin Blodgett. Fairness and sequential decision making: Limits, lessons, and opportunities, 2023. URL
orgooog///oo ptrg/abs/23000 200.///23000553
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basclines:: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1-8, 2021. URL
Hannah Ritchie, Max Roser, and Pablo Rosado. Co and greenhouse gas emissions. Our World in Data, 2020.
https://ouuwottddddaaa.. /orott/aaooo/ooaaaaaaaaaaa coree-addrrreeooooeeeeee gaas-mmissions.
Rocío Sánchez-Salmerrón, José L. Gómez- Urquiza, Luis Albendin-Garee María Correa-Rodriguer, María Be- goña Martos- Cabrera, Almudena Velando-Soriano, and Nora Suleiman-Martos Machine learning methods applied to triage in emergency services: A systematic review. International Emeryency Nursing, 60: 100109, doi: 10.1016/j.ien jenj. 101109 URL htttps://pbbmes ncbi nlm. nih ntih.gov/349522822
Jamuary 2022.
H. Sharma, M. Mathur, S. D. Purohit, K. M. Owolabi, and K. S. Nisar. Parameter estimation and early dynamics of covid-19 disease. In Mukesh Saraswat, Sarbani Roy, Chandreyee Chowdhury, and Amir H. Gandomi, editors, Proceedings of International Conference on Data Science and Applications, pages 783- 795, Singapore, 2022. Springer Singapore. ISBN 978-981-16-5348-3,
Mohammad Ahmad Sheikh, Amit Kumar Goel, and Tapas Kumar. An approach for prediction of loan ap- proval using machine learning algorithm. In 2020 International Conference Electronics and Sustainable Communication Systems (ICESC), pages 490 0004,,4449 2020. doi: 10.100//CCSC44995520000.1111
Elham Taghizadeh and Ali Soohammad-Djafartt Seir modeling, simulation, parameter estimation, and their application for covid-19 epidemic prediction. Physical Sciences Forum, 5(1), 2022. ISSN 2673-9984. doi:
10.3390/psf22222050118 URL https:/ htttpp://www... http:/wwwwpppp...mmm wwtm/227739984/55/11/
HilaryE. Tillett. Infectious diseases of humans: Dynamics and control r. m. anderson, r. m. may, pp.
757. oxford university press; 1991 (#50.000). Epidemiology and Infection, 108:211-211, 1992. URL https:
//pppi.emanticceteeecca.a wew.seiescccchoorrr.....o.. htw.:/cccchoaaaooorrrrrrrspssaaa 70971300
Hossein Yarahmadi, Mohammad Ebrahim Shiri, Hamidreza Navidi, Arash Sharifi, Hasan Piriaei, and Mo- harram Challenger. Vaccine distribution modelling in pandemics through multi-agent systems: Covid-19 case. In 2023 13th International Conference on Computer and Knowledge Engineering (ICCKE), pages 277 285, 2023. doi: 10.1000/CCCKKE6633.22222 200
A Appendix
A.1
Implementation details for the COVID-19 simulation experiments
State Representation The (unnormalized) state space of our environment consists of the nunber of susceptible, exposed, infected and recovered individuals for each region, well the number of vaccines available for distribution at the current time step. These values are scaled with the overall population.
Memory Representation The (unnormalized) memory for the Full bascline at cach time step consists of the number of vaccines distributed far each of the regions. The Min baseline uses a similar repre- sentation, but subtracts the minimum value. These values scaled with the overall number of vaccines distributed so far. FairSCM uses the same memory as the Full baseline.
The action space made up of contimuuus values between -1 and for cach of the regions. The softmax function is applied the actions to determine the percentage of available vaccines to distribute each of the regions.
Actions
14

---

Published in Transactions on Machine Learning Research (05/2025)
Environment Dynamics For experiments, used three regions with overall population of one billion. The SFIR parameters for the regions were determined by choosing the parameters estimates of three random countries from Sharma et al. (2022). The region populations, as well as the initial ratios of susceptible, exposed and infected individuals were determined arbitrarily with a focus on diversity between regions.
Counterfactual Memories We constructed counterfactual memory augmentations by randomly shifting the values in the unnormalized memory (number of vaccines given to each of the regions). We sampled the perturbations uniformly between -10,000,000 and 10,000,000. For cach real transition, we added three counterfactual transitions to the buffer.
Neural Network Architectures Our SAC agent uses identical actor and critic architectures, consisting of three fully-connected layers of 64, 32 and 16 units with ReLU activation functions. We used the imple- mentation provided by 3 (Raffin et al. 2021] ), and extended it with our own modifications.
Hyperparameters See Table
Table 2: COVID-19 Simulation Hyperparameters
Hyperparameter
Full and Min approaches
FairSCM
Episode Length
24
24
Learning Rate
0.00003
0.00003
Discount Factor (7)
0.99
0.99
Replay Buffer Size
5000
15000
Batch Size
128
512
Soft Update Coefficient (T)
0.000001
0.000001
Entropy regularization coefficient
0
A.2 Implementation details for resource allocation experiments
Hyperparameters
See Table 3
Table 3: Resource Allocation Hyperparamcters
Hyperparameter
Full, Min, and Reset approaches
RNN approach
FairQCM
Episode Length
100
100
100
Learning Rate
0.0001
0.002
0.0001
Discount Factor (n)
0.95
0.95
0.95
Min Exploration Rate (e)
0.2
0.2
0.2
Replay Buffer Size
400
1000
6400
Batch Size
64
256
2048
A.3 Additional results of exploring different fairness scoring functions
Bosides the Egalitarian and Gini welfare scores, also conducted experiments on the Utilitarian and Rawlssan welfare scores.
The Utilitarian reward function simply takes the sum of the numbers of donuts allocated to each customer so far, and aims to maximize the overall number of domuts given out:
w: = W((((T)) = Σ U(T.)..
(8)
im1
15

---

Published in Transactions on Machine Learning Research (05/2025)
Results based the Utilitarian welfare score: As shown in Figure 7 after 200 episodes the Utilitarian score of FairQCM starts to decrease and the number of donuts given out also starts to decrease. It is surprising as the Utilitarion welfare score aims to maximize the total number of donuts given out.
200U
BarQQM
Purkia Are
Accumuletedd unittarien Scores
INN
Reset
4500
NinI
Number sllecated
90-
4000
2500
FaROCM
Raset
2000
M8n
200
400
000
000
1000
200
400
600
800
1000
Numbee Episodes
Number Episedes
Figure The Ublitariin score of FairQCM surprisingly starts decrease after 200 episodes.
Results based on the Rawlsian welfare score: As shown in Figure it can be found that Min has zero Rawlsian welfare score and hardly allocates any donut to customers. Besides, FairQCM underperforms Full and Reset baselines.
500
IiirgCM
Accumnlated) exclsinn Sccres
RNN
Reset
Min
Number of allecated
60
400
20
TairOC
WM
Roset
-20
200
100
600
800
1000
200
100
600
800
1000
Number Episodes
Number
Figure 8: Min performs poorly under the asscssment of Rawlsian welfare score.
A.4 Additional results of dynamic stakcholder probabilities
Heterogeneous probabilities (action-independdent): In this experiment, we introduce time-dependent probabilities, allowing customer appearance likelihoods to vary dynamically at each time step. This behavior is modeled using a sigmoid-lin function:
f(1) 1 e (to
(9)
where & the current time step, and steepness and biid eustomer-specíffc hyperparameters. Here, liid denotes the time step at which the probability reaches 0.5, while steepness controls the transition rate, shaping the curve's gradient.
The results indicated an unexpected increase in the variance of Fair after approximately 500 episodes, which remains uncxplained (Figure g) Additionally, the RNN's subpar performance observed in this exper- iment was consistent with our findings from previous experiments.
16

---

Published in Transactions on Machine Learning Research (05/2025)
2000
100
80
NOW
600
60
400
RarOCM
Full
Full
RMN
RMN
Reset
20:
Roset
200
600
800
1000
200
400
600
800
1000
Episodes
Figure 9: Resource allocation experiment with dynamically changing customer probabilities, illustrating the subpar performance of RNN and an unexpected increase in variance for FairQCM. The probabil- ity modeling hyperparameters used in this experiment were: 4mid = [50, 50, 50, 75, 25] and steepness [0.9, 0.99, 0.1, 0.1,0.6,6600.. 0.5).
17


Reviews:
[{"review_id": "4DURHwC2Gj", "reviewer": "Reviewer_4kBh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Extending the experiments to more complex and realistic environments inspired by real-life applications highlights the impacts of various choices in FairQCM in realistic scenarios. \n2. Demonstrating that different baselines are better for other fairness metrics such as gini score shows how the choice of method should rely on the fairness metric of interest. \n3. More fine-grained analysis of the impact of real and counterfactual memories. \n4. Reproducible code for the experiments.\n\nWeaknesses\n1. Lack of novelty in methods or experiments\n2. Lack of analysis and discussion of the observed results", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper experimentally analyzes an algorithm FairQCM proposed by previous work to deal with non-Markovian fairness effects i.e., when fairness outcomes of an action depend not only on the current state, but also on previous states. FairQCM is a reinforcement learning algorithm with states including memory beyond the current state and counterfactual memories. The goal of the experiments is to analyze the extent of improvement for fairness and utility when using memory and counterfactual memories. \n\nThe paper replicates the results of experiments on the algorithm from previous work and extends the experiments in the following ways: 1) varying the ratio of counterfactual memories to real memories, 2) analyzing two more fairness metrics (egalitarian score and Gini score), 3) considering dynamic parameters in the environment, and 4) testing on a new epidemiology simulation environment.\n\nThe paper replicates the original paper's findings. Through their extended experiments, they arrive at the following findings:\n- In the environments of past work, both real and counterfactual memories are beneficial. However, counterfactual memories are only beneficial up to a certain extent. Once the counterfactual memories exceed a certain ratio of the real memories, the outcome is worse than having no counterfactual memories at all.\n- FairQCM also performs well relative to the egalitarian fairness metric. However, it has lower fairness and utility when the fairness metric is Gini welfare score. \n- FairQCM offers more clear benefits when the parameters of the environment are dynamically changing.\n- In the epidemiology simulation environment, counterfactual memories have a negative impact. They hypothesize that this is due to noise in the approximation of counterfactual memories in the more complex environment of the epidemiology simulation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper needs more discussion of the observed results to secure my acceptance recommendation. Especially why some of the findings from previous work differ from those in the new environment and other fairness scores. \n\n1. What is it about the Gini score that makes FairQCM perform worse than other baselines? \n2. Why do you think having too many counterfactual memories is worse than having no counterfactual memories at all? \n2. To support your hypothesis that the noise in approximating counterfactual memories is the reason why using counterfactual memories has a negative effect in the epidemiology simulation, can you show how varying the accuracy (through varying the approximating model's parameters) changes the outcomes?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9USzOKwxrT", "reviewer": "Reviewer_pp2W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well-written and easy to comprehend. \n\nThe overall usage of space especially regarding figures leaves room for improvement. \n\nThe evidence for the impact of counterfactual memories could be clarified more. \n\nThere are multiple weakpoints I tried to address within the Requested Changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper reproduces the results of Alamdari et al. (2024) where the method FairQCM is introduced and evaluated on resource allocation and a simulated lending problem. \n- The paper extends the results by evaluating the resource allocation problem on varying amount of counterfactual memories in the replay buffer.\n- The paper extends the results by comparing different welfare scores for the resource allocation problem.\n- The paper extends the results by also using dynamic stakeholder behavior. \n- The paper adjust the method FairQCM to FairSCM, the SAC equivalent of the original method.\n- The paper evaluates FairSCM on a complex environment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It is not clear at the end of Section 2 what the reference is for. \n\n- Can you elaborate why in Table 1, method RNN, it translates the RNN to a recurrent **Q**-network? What makes the RNN a Q-network here? \n\n- Please add a short sentence in the resource allocation environment (3.1.2) to specify the desired goal. \n\n- I would strongly encourage cross-linking throughout the paper. The SEIR model is explained in 3.2.4, but already referenced several times before in the paper. In my opinion, it strengthens the comprehensibility if a link / hint is given where the explanation can be found since it is explained in a later part in the paper.\n\n- If an equation is referenced explicitly like \"(as shown in equation 3)\", *E*quation should be uppercased. \n\n- The used line for mean should span across the whole term of which the man is meant. (Equation 3 and ff)\n\n- If indices of part of a set, then the set should be noted as a set, e.g., $i \\in \\lbrace1, 2, ..., n\\rbrace$ with brackets (in latex they need \\ to be displayed).\n\n- \"n = [0,1,2,3,4]\" is not an array. Please fix this.\n\n- Subsection 3.3 has a typo - the efficiency improvement leads *to* 7 hours computation time in my understanding.\n\n- Can you elaborate why the COVID-19 allocation simulation needs continuous actions? To me it looks it has a big discrete action space, which could be efficient to handle continuously.\n\n- Please streamline lower-upper-casing throughout all plot labels.\n\n- Streamline how to reference figures throughout the paper. \n\n- Please label Figure 5 appropriately. There seems to be something wrong. \n\n- The ax ticks for the middle plot of Figure 6 should be abbreviated. \n\n- Be more specific what is exactly plotted in the plots. Explicit explanation of shaded region etc. \n\n- Please use the decimal separator, e.g., 6,400. \n\n- What is the current state of \"Furthermore, we found some mathematical definitions, design choices and surprising results in the original paper somewhat difficult to understand. We sent our questions to the authors, but have not received a response by the time of submission.\"? As this should be changed before acceptance. \n\n- What I'm still confused about: Is it even possible to do well on different fairness scoring functions at the same time?\n\n- Can you design an experiment that clarifies the impact of counterfactual memories in the COVID-19 environment? And / or elaborate more precisely what impact these have?", "broader_impact_concerns": "I have no critique with respect to broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ft0OzHfFWx", "reviewer": "Reviewer_NiT1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-structured, with clearly stated motivation and contributions. It does not oversell and also covers appropriately related and previous works on the topic of fairness in sequential decision making. The additional experiments are interesting and reasonable. The most important contribution in my opinion is the newly proposed COVID environment. I believe that it could aid research on RL and causal RL which lacks simulation testbeds on healthcare, especially in the continuous action space. Given the success that similar environments, e.g., Sepsis, had over the past years I am optimistic that this testbed can have an impact.\n\n**Weaknesses**\n\nArguably, this paper has very limited novelty as it simply extends the experimental section of an existing paper. However, in my opinion this is fine, and does not diminish the importance of the contribution. I have several suggestions on how to improve the paper, please see my comments on the **Requested Changes** section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides additional empirical evaluation for the FairQCM method introduced by  Alamdari et al. (2024). FairQCM is an DQN algorithm that sees environment states augmented with past actions (non-Markovian) and its goal is to enforce fairness constraints in the context of sequential decision making. The new experiments extend the existing evaluation along several axis including: additional baselines; more realistic settings with continuous action space; robustness over different fairness definitions; the impact of counterfactual memories on fairness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please find below my suggestions on how to improve the paper. I am already positive about this paper and none of the following recommendations are \"mandatory\" from my side. That said, I would be greatful to the authors if they take my comments into consideration and implement whatever they might find it could help improve the quality of the paper.\n\n- Even thought the paper covers well the existing work it is based on, I would appreciate a more clean and concentrated explanation on why additional experiments are needed and what past evaluation lacked. This should be added in the Introduction section, in order to highlight the gap that the current paper tries to fill.\n\n- I know that FairQCM is derived from prior work, but it is not motivated in the current one. In other words, the authors should mention why they believe that this is an appropriate approach to enforcing fairness in sequential decision making.\n\n- A more clear explanation of the distinction between regular and counterfactual memories on 3.1.1 is needed.\n\n- The exploration of additional RL algorithms for both the discrete and continuous space cases would significantly strengthen empirical results.\n\n- The results on the robustness over different fairness functions are disheartening. I appreciate that the authors included these results and did not hide them. However, I believe that this motivates a more extended investigation, i.e., experiments with additional fairness functions. Furthermore, the authors should include additional discussion about these results.\n\n- Regarding related work section, papers that study notions related to fairness in sequential decision making could be also covered for generality. For example, works that explore notions such as interpretability, accountability, harm etc.", "broader_impact_concerns": "I do not have any such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: One reviewer raised a question about the methodology and contributions around the different scoring functions.  The authors responded that:

When comparing the performance on different scoring functions, we trained our models specifically for each scoring function separately. We didn’t train a single model and evaluate it on different scoring functions at the same time. It is an interesting question whether it is possible for a model to perform well on different scoring functions at the same time, but it is out of the scope of this paper.

This makes sense, but it would be good to incorporate this into the paper itself.  In the latest version it mentions training separate models from scratch for each in 3.2.2, but the results in 4.2.2 just talk about FairQCM rather than being more precise about which version in each instance and the fact that this question isn't explored should be explicitly noted.

With these small changes to 4.2.2 I believe all concerns will have been addressed.
Decision: Accept (poster)