FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning
Authors: Woosung Kim
Jinho Lee
Jongmin Lee
Byung-Jun Lee
Venue: NeurIPS 2025 poster ()
Keywords: Offline Reinforcement Learning
Multi-objective Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: Fair offline mult-objective Reinforcement Learning

Abstract: Multi-objective reinforcement learning (MORL) aims to optimize policies in the presence of conflicting objectives, where linear scalarization is commonly used to reduce vector-valued returns into scalar signals. While effective for certain preferences, this approach cannot capture fairness-oriented goals such as Nash social welfare or max-min fairness, which require nonlinear and non-additive trade-offs. Although several online algorithms have been proposed for specific fairness objectives, a unified approach for optimizing nonlinear welfare criteria in the offline setting—where learning must proceed from a fixed dataset—remains unexplored. In this work, we present FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare objective. FairDICE leverages distribution correction estimation to jointly account for welfare maximization and distributional regularization, enabling stable and sample-efficient learning without requiring explicit preference weights or exhaustive weight search. Across multiple offline benchmarks, FairDICE demonstrates strong fairness-aware performance compared to existing baselines.

Fai:DICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning
Woosung Kim!"
Jinho Lee!"
Jongmin Lee
Byung-Jun Lee
Korea University Yonsei University
Gauss Labs Inc.
{wsk208, jinho0997, syuuggiunlee@koora... ac.ak.c... ac kr
'Equal contribution
*Corresponding authors
Abstract
Multi-objectivv reinforcement learning (MORL) aims to optimize policies in the presence of conflicting objectives, where linear scalarization is commonly used reduce vector-valued returns into scalar signals. While effective for certain preferences, this approach cannot capture fairness-oriented goals such as Nash social welfare max-min faimess, which require nonlinear and non-additive trade-offs. Although several online algorithms have been proposed for specific fairness objectives, a unitied approach for optimizing nonlinear welfare criteria in the offline setting where learning must proceed from a fixed dataset-remains unexplored. In this work, we present FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare objective. FairDICE leverages distribution correction estimation to jointly account for welfare maximization and distributional regularization, enabling stable and cample-efficieee learning without requiring explicit preference weights or exhaustive weight search. Across multiple offline benchmarks, FairDICE demonstrates strong fairness-aware performance compared to existing baselines.
Introduction
Sequential decision-making in real-world domains often requires balancing multiple conflicting objectives, seen in applications like autonomous driving (1,2), robotic manipulation [3 and wireless network resource allocation [5]. Multi-objective reinforcement learning (MORL) addresses this challenge by providing principled framework for learning policies that maximize aggregated returns over conflicting objectives. While standard MORL with linear scalarization focuses on maximizing a weighted sum of objective returns, MORL with nonlinear scalarization, or fair MORL, promotes fair outcomes through concave scalarization objectives, such as Nash social welfare 66.
The nonlinearity of fair MORL presents major optimization challenge due to its nonlinear scalariza- tion over objective returns and has been widely studied in online settings where agents learn through interaction. Some approaches maximize a lower bound-the expected scalarized return 7 g wwile others focus solely on max-mil mix fairness or use policy-gradien methods directly optimize the original objective [ [00] However, fair MORL in the offline setting remains unexplored where the agent needs to learn fair policy from fixed dataset and avoids risky or costly interaction with the environment.
Recent studies have explored offline MORL but primarily focus linear scalarization, learning policies conditioned on fixed preference weights [II] to perform well along the Pareto front [22 B3 However, these methods are unsuitable for fair MORL, which aims maximize its fairness objectives (welfare) without explicitly specifying preferences. To this end, we formulate offline fair MORL
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

problem to directly optimize the trade-off between welfare and distribution shift regularization required for offline RL. To the best of our knowledge, this is the first work to optimize welfare- maximizing policy from fixed dataset.
In this paper, also show that while MORL with linear scalarization and fair MORL funda- mentally distinct problems with different solution spaces, they can be theoretically connected under offline regularization, sharing the same optimal solution. Building on this insight, we extend the DICE-RL framework, which optimizes the stationary distribution for offline policy learning, to handle the nonlinearity of fair MORL and develop FairDICE, our sample-bace based offline MORL algorithm. FairDICE effectively finds welfare-optimal policies in both discrete and continuous domains with minimal additional parameters, outperforming cefference-conditione baselines even with exhaustive weight search. In summary, our contributions are threefold:
A regularized offline MORL formulation that optimizes nonlinear welfare objectives while mitigating distributional shift.
A theoretical connection between our formulation and linear scalarization under regu- larization, showing that FairDICE implicitly optimizes preference weights for welfare maximization.
Fair a practical, sample-based algorithm for offline welfare optimization.
2 Related Work
Multi-Objective RL and Welfare Objectives Linear scalarization is a common approach within Nevertheless, fails to capture complex trade-offss like fairness or risk sensitivity [15] 16 18 This limitation has led to growing interest
MORL that optimizes weighted sum of returns [4.
in nonlinear scalarization, which enables more expressive preference modeling. Recent theoretical work has shown that it is tractable to optimize nonlinear scalarizations under smooth, concave utility functions [19 [O]. Such scalarization functions, including Nash social welfare and Gini indices, optimized via online interactions 7 20). Max-min objectives, another form of fairness-aware scalarization, have been addressed in model-free online settings with entropy regularization 9
Offline RL and Offline MORL Offline reinforcement learning aims to learn policies from fixed datasets without further environment interaction, avoiding costly or risky exploration. major challenge in this setting is distribution shift: deviations from the behavior policy can lead to inaccurate value estimates and suboptimal performance. To address this, various strategies have been proposed, including conservative value estimation [21], diverrenne-rrggularieedd optimization [22 23), and
return-conditioned sequence modeling [24]. In the multi-objective setting, most offline approaches assume linear scalarization and require explicit preference conditioning during training evaluation 25 (6). However, direct optimization nonlinear objectives in offline MORL remains largely underexplored.
3 Preliminaries
3.1 Markoy Decision Process (MDP) and Multi-objective RL
A Markov Decision Process (MDP) is defined by the tuple
T,,,oo 7). where S and A are the state and action spaces, T' is the transition probability, r is the reward function, po is the initial state distribution, and E [0, 1) is the discount factor. For policy æ, the stationary distribution d, (s, a) represents the discounted visitation frequency of state-action pairs and satisfies the Bellman flow constraint:
d=( T(ass) (1-7))0(0) + Σ add,s) As, a
The expected return of policy म, defined as the discounted sum of rewards, can be represented as ((x) : Σr, (s, a)r(s, a), and standard RL aims to find π that maximizes J(T).
In Multi-Ohjectice Reinforcement Learning (MORL., the return is represented as a vector [Ji())....,,J (T)), where each (TT is defined under corresponding reward function rs(s.a).

---

MORL methods typically scalarize the return vector to optimize a scalar objective of the form E, (T)) Linear scalarization applies fixed weights w;, reducing the problem single-objective RI. with the aggregated reward r(s,a) = E: (1, (8,a). This approach focuses on recovering optimal policies across different weight configurations.
However linear scalarization does not consider fairesss across objectives. To address this, we use strictly concave scalarization functions, referred to Fair MORL, which maximize the welfare L; Uj (T)( as in Nash Social Welfare with u; (z) = log(æ)) However, this nonlinearity breaks the Bellman recursion, making Temporal Difference (TD) targets difficult define and thus unsuitable for standard value- based RL methods.
3.2 Offline Reinforcement Learning and DICE-RL framework
In offline reinforcement learning (RL), an agent learns a policy from a fixed dataset
{ss az, Ti, s(h=r. without further environment interaction, making it well-suited for scenarios where exploration costly or unsafe. core challenge in offline RL is distribution shift, where the learned policy deviates from the behavior policy to improve performance. Excessive deviation leads to significant off- oolicy estimation errors and degrades real-world performance.
To address this, the DICE-RL. framework 22
regularizes policy optimization using an f-divergence between the stationary distribution of the learned policy d and the empirical distribution dn, solving:
d(s,a)
max d20
Σ
d(s,a))(ss,)) -
>>a dd(s a)f
dp(s,a))
8,4
8,a
s.t. Σ d(s,a) = =1--7)p0( + T(sss)
Vs
3,0
where the marginalized Bellman flow constraint ensures the optimized stationary distribution is valid. The hyperparameter 3 > 0 controls the trade-off between return maximization and distribution shift regularization. In finite domains, the optimal policy x² is recovered from d' (a.a)))
via x² (als) E. (s.a) Vs, a, with policy extraction methods used in continuous domains. Full derivations appear in Appendix A
4 Fair MORL as Convex Optimization
We present convex optimization formulation for Fair MORL. Before extending it to practical offline algorithm, we analyze its structure and reveal connection the Eisenberg- Gaaberrg--lleeeeeeeee Gale program 27 We reformulate the objective to facilitate future sample-based methods and highlight key differences from linear scalarization.
4.1 Convex Formulation via stationary distribution
We introduce convex formulation of Fair MORL, where the objective returns J; (T)) are expressed in terms of the stationary distribution. This formulation extends the dual of V-LP (28), the linear programming formulation of RL, by replacing its linear objective with welfare function, similar Specifically, we consider the following problem, where each uz(x) is
approach explored in 9 10
strictly concave:
(PI):
max
Σ 41 Σ ((ssa)))((s,a))
d20
3,4
8.1. Fa(s) (8) = 0, Vss
where Fa(s) = 7)p + a.a T's a) a)d(s,a) En d(s, a) denotes the violation degree of the Bellman flow constraint
Uniqueness of the solution to (P1) follows from the strict concavity of the objective and the convex, non-empty feasible set of valid stationary distributions. When u, (xr = log(x) Vi, the optimization takes form similar to the Hisenberg-Gale convex program, classical model for computing fair and

---

Pareto-efficient market equilibria:
max 20
Σ
log Σ UijJis
s.t. Σ
xT S 1Vj
where ziy denotes the allocation of good j to buyer i, and Mij is the utility of buyer i. In our setting, this corresponds allocating actions in single-state MDP maximize Nash social welfare, with representing actions and representing objectives. With additional constraint ( (PI) can be viewed sequential version of the Eisenberg- program, extending the allocation problem sequential decision-making.
4.2 Reformulation for sample-based optimization
Problem (P1) involves an expectation inside concave function, which complicates the future derivation of our sample-based optimization method. To address this, we reformulate (PI) as (P2) by introducing slack variables, moving the expectation outside the concave function.
(P2):
max d20.
Σ us (K:)
s.t.
Σ [d(s,a)ri(s.a) = kg Vi, Fa(s) = 0, bs
where k; is slack variable representing the expected return for objective i. The Lagrangian dual of (P2) introduces Lagrange multipliers μ4 for the return constraints and v(s) for the Bellman flow constraints. The dual formulation is given by:
max min d20,k 2.
Σ uss (ks)s) +
Σ 14g
d((s,a)) kz) -u(()F((
(3)
5,4
In the dual function 33. the Lagrange multiplier 14 modulates each objective's 's return, resembling the role of preference weights in Linear MORI. At optimality, the following relationship holds:
1 = (k())) = v Σ d' (s) 5,0
Since (x) is decreasing due to the strict concavity of u; #4 acts as implicit preference weight that penalizes large returns. For example, when ux(x) = log(x)), this yields μ that corresponds to the reciprocal of ith return, assigning higher weight to objectives with lower returns and promoting more balanced, fair allocation-com consisstnt with the goal of Nash social welfare and Fair MORL.
However, although plays a role similar to preference weights in Linear MORL, fixing μº in Linear MORL (P3) leads to a fundamentally different optimization problem from (P2) learning μı as part of the nonlinear welfare objective. (P3) is defined below, and a simple counterexample are provided in Appendix B
(P3): max d20
d(s, s.t. Fa(s) O, ws 8,
Unlike (P1), which has unique solution by strict concavity, (P3) may have multiple optimal solutions with different welfare outcomes. Thus, the welfare-maximinn policy of Fair MORL cannot generally be found by simply sweeping over weight vectors in Linear MORL.
FairDICE: Welfare Optimization for Offline Fair MORL
In this section, we introduce the Regularized Welfare Optimization framework and corresponding sample-based algorithm for effective welfare optimization the offline setting. Building on DICE-RL framework applied to (P2), our method, FairDICE, directly optimizes implicit preference weights to maximize welfare. We further provide theoretical support by recovering an equivalent Regularized Linear MORL formulation.

---

5.1 Regularized Welfare Optimization framework
We formulate our framework by incorporating an f-divergence between the optimized stationary distribution d and the empirical data distribution dp into (P2). The trade-off between the welfare and distributional shift is controlled by hyperrparameter 0. and is assumed be strictly convex with f(1) = 0. The resulting convex optimization is:
Σ u (ki) - 3 Σ Edo(s,a)f d(s.
(P2-reg):
max d200,
(ki)
a))
5,a
s.t. Σ d(s, ()),a)r,(s, ky Vi, Fa(s) = 0, Vs 2,0
Following the DICE-RL framework, we derive a sample-based optimization method from the La- grangian dual of (P2-reg). We also highlight the challenges of extending (PI) directly to sample-base optimization and show how (P2) cireumvents the issues. The Lagrangian is expressed as:
d(s,a)
max min L(, U. = Σ d20,k vi
Σ 3,
d(s, a)s(s,)) a) Kss -
-BΣ Edn(s,a)) dd(s,a) 3,0
E (k:) Em(s)Fals
We reparameterize the stationary distribution as d(s, a) w(s a a),p(s a) and express the dual in terms of the importance weights w, using the identity E. v(s) E.,o a))3,,
E., d(s,a) a) E a)s()s, This yields the following optimization problem:
max min Exppo ())(s)] + Edo [w(s, as,, a) -35 (w(s,a))] w20.
Σ (ncks + u;(ks))
where ( (,, a) + E. Tss
v(s) Vs,a.
Applying the Lagrangian dual directly to the regularized (P1) retains expected returns inside the concave functions un (.). preventing direct use importance sampling. Moreover, naive estimator such as Es.o (s. a) > uu(w(s, a)r(s, a)) introduces bias, violating the validity of importance- weighted estimation.
We further simplify the optimization by reducing parameters. Using strong duality of (P2-reg), switch the optimization order to min,,a max and derive closed-form solutions for w and h; from first-order conditions:
Wi,, (s, a) max (0, (Smal vs,
ki,, = (14) Vi
Substituting the closed-form solutions into the Lagrangian dual yields the final optimization, which defines the loss function of our offline algorithm, FairDICE (Fair MORL via Stationary Distribution Correction):
min Exmpo (1--)u((s) + Eys 85ō 2,
(Cam
(Cap a)
+ EE (-()))
(4)
B
where (*(y) = max+20 xy fxx and uz (v) == max, ay + uz(x))
convex conjugate functions. - (s, (s,a)dn a). In finite Solving gives the optimal stationary distribution d' (s,a) a) w domains, the optimal policy is directly recovered via n"" (als) = d (s. Σ. d= (x, a) Vs,a.
Compared to its single-objective counterpart, OptiDICE, FairDICE introduces only one additional scalar parameter per objective, incurring minimal overhead and scaling efficiently with the number of objectives. Moreover, FairDICE extends naturally large and continuous domains by approximating v, μ and æ with function approximators. continuous domains, we adopt weighted behavior cloning to extract the policy from the optimal stationary distribution using the following loss:
max T
[saa)dd [w- .a- (s, a) log (als))
Further details in experimental settings and algorithmic descriptions for continuous domains are provided in Appendix F
5

---

(a) Data collection policy
(h) Utilitarian welfare
(c) Nash social welfare
(d) NSW (8 (bjeccives))
Figure 1: Visualization of FairDICE policies in M00-Fou-Roomms (a) Uniformly random policy for
data collection, (b) FairDICE policy maximizing Utilitarian welfare (sum returns), (c) FairDICE policy maximizing Nash social welfare (NSW). (d) FairDICE maximizing NSW in domain with eight objectives. Red arrows indicate the policy, and blue heatmap shows state visitation.
5.2 Equivalence to Regularized I Linear MORT
Previously, we showed that unregularized linear and fair MORL converge to different solutions. We now demonstrate that their regularized forms converge the same unique solution. This equivalence supports that FairDICE implicitly optimizes preference weights corresponding to those linear MORL to directly maximize target welfare.
To formalize this, we introduce (P3-reg), an extension of DICE-RL framework to Linear MORL. While any preference weight can be used, we set it to the optimal dual variable μ² from 4 for analysis:
d(s,a) dd(s,a)
(P3-reg): max d20
Σ d(s, Σ Huirs(s,a) Edd(s,a)f
8,0
s,a
s.t. Fa(s) Ys
Proposition (Equivalence between the regularized problems). Let μa be the optimal multipliers obtained from ( Then, the optimal solutions of FairDICE and (P3-reg) with Pa yield the same unique optimal policy. (Proof Appendix
This equivalence suggests that FairDICE reduces to offline Linear MORL algorithm when its is fixed to the preference weights. We refer to this special case as FairDICE- and leverage it to optimize utilitarian welfare- the sum of objective returns by setting all μs equal.
The equivalence also implies that sweeping over preference weights in regularized Linear MORL can recover the optimal policy found by FairDICE. However, this requires training policies across wide range of weights and selecting the one that achieves the highest welfare, which becomes impractical as the number of objectives increases. In contrast, FairDICE effectively and efficiently optimizes implicit preference weights, directly producing an offline MORL policy that maximizes welfare.
6 Empirical Behaviors of FairDICE
In this section, we empirically validate our theoretical insights using multi-objective adaptation of the classic Four-Room environment 22 and Random MDP 22 30 as a toy example. The
visualization shows how FairDICE effectively optimizes the trade-off between welfare and distribution shift (Section 5.1)) and aligns with Regularized Linear MORL while optimizing its implicit preference weight for offline welfare optimization (Section 5.2).
In the experiments, we use a-fairesss to aggregate objectives, generalized social welfare function that balances total return and fairness. The trade-off is controlled by the parameter a: as a - 0, it approximates Utilitarian welfare, the sum of returns; + oc, it approaches the max-min fairnsss. The scalarization function is defined as:
1, it recovers Nash social welfare; and as
[11 - (1-a))-aa* ---l--aa
(a + 1)
u, (x)
Vi
(log(x)
(a

---

-154
0.
0.73
-13.
13.
3.
30
0.600
0.75
0.70
0.75
0.40
-146
0.40
14.
0.55
-150
0.50
10-
22
10.
10.4
227
30"
100
13:
35-
10:1
1.2
10"
10:1
30-
10
20-4
20"
10
Figure 2: Policy performance on Random MOMDP domain across different a and values, evaluated on Nash social welfare, Utilitarian welfare, and Jain's fairness index. Results are averaged over 1000 seeds, and reported with 95 % confidence intervals.
We evaluate the resulting policies using three metries: Utilitarian welfare Σi=1 R., Jain's Fairness Index (E R₂)² (n Ei=1 R,)) and Nash social welfare (NSW) Eiz1 log( log(R.). Utilitarian wel- fare measures the total return, Jain's 's Fairness Index evaluates fairness across objectives, and Nash
social welfare captures trade-off between efficiency and fairness. Details of the environments and experiments introduced in this section are provided Appendix
6.1 MO-Four-Room Experiment
We extend the Four-Room domain to a MORL setting, referred to as MO-Four-Room, by introducing three distinct goals, each associated with a separate objective. As shown in Figure the agent starts from the initial state (orange) and moves toward the goal states (green). Upon reaching goal, receives a one-hot reward: [1, 0, 0], [0, 0]. or [o, 0, 1]. To simulate offline RL, we construct a dataset of 300 trajectories collected from a uniformly random behavior policy.
Even when data is collected under fair, suboptimal policy, FairDICE success- fully learns offline MORL policies that maximize target welfare. Figure shows how varying o-fairness objectives shape FairDICE's behavior. The utilitarian objec- tive favors the nearest goal as it results in the best return, achieving the highest sum of returns despite being unfair. However, FairDICE maximizing Nash social welfare (FairDICE-NSW), encourages balanced goal visitation, even when the environment is expanded to include eight objectives, shown in Figure
Table MO-Four- Room Performance Table
Behavior
Utilitarian
FairDICE-NSW
NSW
-16.10
-17.33
-10.77
Util
0.024
0.086
0.082
Jain
0.710
0.500
0.996
6.2 Effective Optimization over Two Distinct Trade-offs
The Regularized Welfare Optimization framework balances two trade-offs controlled by a and 3: one between objective returns under a-fairness and the other between welfare and distributional shift. To empirically demonstrate how these trade-offs evolve with varying parameters and generalize across different MOMDP environments, extend the Random MDP to MORL setting, called Random MOMDP, following a similar approach used for the Four-Room domain.
Figure 2 illustrates how the three performance metrics vary under these trade-offs. Increasing shifts the objective toward max-mi fairness, improving Jain's fairness index, while decreasing a prioritizes total return, enhancing utilitarian performance. Higher 8 values constrain the learned policies to remain closer to the data collection policy across all metrics. In contrast, lower values reduce regularization, allowing each objective more effectively pursue its own a-fairesss However, if 8 excessively low, the resulting distribution shift can degrade practical performance.
A surprising finding is that while FairDICE achieves the highest utilitarian performance when maxi- mizing utilitarian welfare (a = 0), higher a values also sustain strong utilitarian performance across wide range of B. This stems from the concavity of the a-fairness objective, where returns contribute less to overall welfare as they grow. Consequently, the incentive for pure return maximization tempered, implicitly regularizing against distributional shift.

---

6.3 Welfare Optimization via Implicit preference weight
In Section 5.2 we established the equivalence between regularized linear and fair MORL. We validate this by perturbing the optimal p obtained by FairDICE-NSW in the Random MOMDP experiment and apply it to FairDICE-fixed, as reported in Figure 3 noise with standard deviation is generated and applied each dimension of μx by scaling it as (1+noisse). FairDICE- the highest NSW without perturbation, and the NSW decreases as the preference weights deviate from the optimal value. This indicates that our regularized welfare optimization framework shares the same optimal solution with regualrized MORL with linear scalarization and optimizes its implicit preference that maximize NSW.
Gaussian
achieves
531 o
10
Figure 3: FairDICE-fixed with perturbed μ
MD-Hopper
Swimmer
Expert 15
11.5
Bug Nesh Social Welfare
31.0
135
130
22.5
MO-Hoppeer
MO-Walker22d
MO-Ani
MO-ttal/Cheelah
MO-Swimmer
22.0
125
326
13%
130
31.0
20
10
38
20
23
28
20
22
Weight - [0.0.
BCIP)
MODT(P)
MORVSIP)
RaiDDICE
Figure 4: Nash social welfare scores for five two-objective tasks, evaluated across 30 linearly spaced preference weights. Each curve shows the average NSW over 5 seeds and 10 evaluation episodes per seed. Red line indicates the average NSW performance of FairDICE.
7 Welfare Maximization in Continuous Domains
Environments We evaluate our method on the D4MORL benchmark 31 a standard MORL benchmark in continuous control domains. D4MORL builds upon the D4RL benchmark [32 by decomposing the original MuJoCo rewards into multiple objectives, such as speed, height and energy efficiency. The dataset for each domain consists of two types of data, collected using either expert or stochastically perturbed (amateur) behavioral policies, and is annotated with preference vectors. Our main experiments include five two-objective tasks (e.g., MO-Hopper, MO-Walker244 MO-HalfCheetah, MO-Ant, MO-Swimmer) and one three-objective task (MO-Hopper-3obj). Further details on the environments and experiments reported in Appendix
Baselines While FairDICE seeks a single offline RL policy that maximizes welfare, no existing algorithm directly optimizes this objective in offline MORL. Therefore, we compare against three offline MORL approaches with linear scalarization, additionally searching preference weights that maximize NSW by uniformly discretizing the simplex and evaluating NSW at each point. Specifically, we adopt three basclines that learn prefcrence-conditione policies [31].
BC(P) performs behavioral cloning by conditioning on the state and a preference vector to imitate observed actions.
MODT(P) extends Decision Transformer by modeling trajectories as sequences of (state, action, return-to-go) tokens concatenated with preference vector, using transformer to predict actions.
MORvS(P) simplifies this setup using feedforward model that takes the current state and prefgrence-weiighted return-to-go as input, enabling more efficient training.
While the baseline includes approaches that do not concatenate the state and linear preference ratio, do not evaluate them as they generally underperform compared to their prefereece--ooniitioned

---

Hopper
MO Walkerdd
Art
HalfChhesth
MD SwlmSee
2500
2003
2500
Expert
2003
2003
2000
1003
1003
15
1060
50
500
1008
2000
250
Amateur
4000
2000
5000
1500
1303
200e
1000
1003
1000
1000
ScB
1003
2006
3703
1008
2000
3200
2701
300
1000
3200
2000 2000
100
200
Speed vs. Height
Speed vs. Energy
Horizontal vs. Vertical Speed
Speed vs. Energy
Speed vs. Energy
BC(P)
MODT(P)
MORvSIP)
FairDICE
Figure 5: Raw return evaluations on five two-objective MuJoCo tasks from D4MORL. Each point represents policy performance under a specific preference weight; Pareto frontiers and dominated regions are shown.
MO-Hopper-3ob/
Expert
Amateur
Expert
Amateur
[:SS1LL, 1572.0 2134.1)
(2308.1 1. 2616.2. 2001.7)
16.31
00035 MSN
200D M5W
106
1000
Speed
Preference Weight Simplex
Preference Weight Simplex
BCIP)
MODT(P)
MORWS(P)
FairDICE
Figure 6: Raw returns and Nash social welfare evaluations N0-Hooper-3obj with three objectives: speed, height, and energy. 50 preference weights are sampled uniformly from the 3D simplex. Red plane indicates average NSW performance of FairDICE.
counterparts. In contrast, method does assume behavior policy preference weights but directly optimizes them to maximize welfare, making it applicable even to datasets without this information.
Offline Fair MORL Performance Figure 4 shows that FairDICE achieves competitive or superior NSW performance across all two-objective D4MORL tasks, compared to the best results obtained by extensively searching over preference weights in existing methods. Since NSW summarizes multiple objectives into a single scalar, it does not fully capture how well each objective is optimized. To better illustrate FairDICE's effectiveness across all objectives, we also position its returns relative to the Pareto frontier formed by preference-conditione, baselines. Figure 5 shows that the FairDICE solution lies the Pareto frontier, highlighting the strong practical performance of approach.
In the MO-Hopper-3obj task with three objectives, the preference weight space expands substan- tially, making it increasingly difficult find weights that maximize NSW using MORL with linear scalarization. However, Figure iillustraaess that FairDICE identifies a welfare- -axifiiinm policy without requiring explicit preference conditioning. Notably, while optimizing for NSW, the resulting policy also achieves raw returns that lie close to or even surpass the Pareto frontier formed by cenereicc-conniioonee baselines, indicating that FairDICE achieves strong efficiency in addition to fairness. Moreover, a single additional scalar parameter is sufficient to handle the increased number of objectives, highlighting the scalability and practical efficiency of FairDICE, particularly in high-dimensional preference spaces.
8 Conclusion
In this paper, we introduce a novel regularized welfare optimization framework for maximizing welfare in offline MORL, enabling fair outcomes across objectives using fixed datasets-a setting

---

not addressed by prior work. We establish a theoretical connection between regularized MORL with linear scalarization, showing that our framework implicitly learns preference weights that maximize welfare. Building on this, we extend the DICE RI. framework to derive our sample based algorithm, FairDICE, that overcomes the optimization challenge caused by the nonlinearity of the objective. Empirically, FairDICE achieves strong fairness aware performance across both discrete and continuous domains with a fixed dataset, effectively balancing trade-off' between objectives and between welfare and distributional shift.
Limitation While our method effectively handles MORL with strictly concave scalarization func- tions, it does not cover all forms of nonlinear scalarization. Our formulation assumes convexity, and thus the DICE-based approach relying on the Lagrangian dual does not apply to non-convex scalarization in MORI.. Additionally, although concave scalarization helps mitigate sensitivity to distribution shift, as an offline RL. method, the final performance still depends on the choice of hyperparameters controlling distribution shift and the quality of the dataset.
Acknowledgements
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-1122031 Develop- ment of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects, No. RS-2024-00457882. AI Research Hub Project, No. RS-20 19911999,,1,00 Artificial Intelligence Graduate School Program (Korea University), and No. RS-2025-254108% Beyond the Turing Test: Human-Level Game-Playing Agents with Generalization and Adaptation), the IITP (Institute of Information & Communications Technology Planning Evaluation)-ITK (Information Technology Research Center) grant funded by the Korea government (Ministry of Science and ICT) Ministry of Science and ICT, Korea, BK21 Four project of the National Research Foundation of Korea, the National Research Foundation of Korea (NRF) grant funded by the Korea government the ITTP under the Artificial Intelligence Star grant funded
2---22255--2222 --224-00436857) the NRF (RS-2024-00451162) funded by the
(MSIT) 50S-2022500550335
SSS-002-22400330
Fellowship support program to nurture the best talents 2EP---22--SS2----0000 by the Korea government (MSIT), and KOREA HYDRO & NUCLEAR POWER CO., LTD (No. 2024-Tech-09). This work was also supported by the Institute of Information & Communications Technology Planning & Evaluation (ITP) grant FRS-2020-IBP 1220-11201661 161,, Artificial Intelligence Graduate School Program (Yonsei University) and the Yonsei University Research Fund of 2025--2--
References
[1] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE transactions intelligent transportation systems, 23(6):4909 492499999999,,, 2021.
[2] Xin- Min-Qiang Cai. Pushi Zhang, Li Zhao, Jiang Bian, Masashi Sugiyama, and Ashley Llorens. Distributional pareto-optimal multi-objective reinforcement learning. Advances in Neural Information Processing Systems, 36: 1. 15593-150000 15613, 2023.
[3] Sandy H Huang, Martina Zambelli, Jackie Kay, Murilo F Martins, Yuval Tassa, Patrick M Pilarski, and Raia Hadsell. Learning gentle object manipulation with curiosity deep reinforcement learning. arXiv preprint arXiv:1903.08544 2019.
[4] Jikun Kang, Miao Liu, Abhinav Gupta, Christopher Pal, Xue Liu, and Jie Fu. Learning multi- objective curricula for robotic policy learning. In Conference on Robot Learning, pages 847-858. PMLR, 2023.
[5] Chouaib Messikh and Nacereddine Zarour. Towards multi-objective reinforcement learning based routing protocol for cognitive radio networks. In 2018 International Conference on Smart Communications in Network Technologies (SaCoNeT), pages 84-89. IEEE, 2018
[6] Hervé Moulin. Fair division and collective welfare. MIT press, 2004.
[7] Zimeng Fan, Nianli Peng, Muhang Tian, and Brandon Fain. Welfare and fairness in multi- objective reinforcement learning. arXiv preprint 2121v2221..013822222 2022.
10

---

[8] Conor F Hayes, Timothy Verstraeten, Diederik M Roijers, Enda Howley, and Patrick Mannion. Expected scalarised returns dominance: new solution concept for multi- nulti--bjeective decision making. Neural Computing and Applications, pages 1-21, 2022.
[9] Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, and Youngchul Sung. The max-min formulation of multi-ohjective reinforcement learning: From theory to model-free algorithm. arXiv preprint (tte::240600722266 2024.
[10] Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. Multi-objective reinforcement learning with non-linear scalarization. In Proceedings of the 2 International Conference on Autonomous Agents and Multiagent Systems, pages 9-17, 2022.
[11] Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning, pages 11-20. PMLR, 2019.
[12] Toygun Basaklar, Suat Gumussoy, and Umit Ogras. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. In The Eleventh International Conference on Learning Representations, 2023.
[13] Runzhe Yang. Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. Advances in neural information pro- cessing systems, 32, 2019.
[14] Daniel Lizotte, Michael H Bowling, and Susan A Murphy. Efficient reinforcement learning with multiple reward functions for randomized controlled trial analysis. In ICML, volume 10, pages 695-702, 2010.
[15] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48: 67-113, 2013.
[16] Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective rein- forcement learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pages 191-199. IEEE, 2013.
[17] Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Leurning Research, 15(1):3433,352, 2014.
[18] Joar Skalse and Alessandro Abate. On the limitations of markovian rewards to express multi- objective, risk-sensitive, and modal tasks. In Uncertainty in Artificial Intelligence, pages 1974-1984. PMLR, 2023.
[19] Nianli Peng, Muhang Tian, and Brandon Fain. Multi-objective reinforcement learning with nonlinear preferences: Provable approximation for maximizing expected scalarized return. arXiv preprint urXiv:2311.02244, 2023.
[20] Umer Siddique, Abhinav Sinha, and Yongcan Cao. Faimess in preference-based reinforcement learning. arXiv preprint arXiv:2306.0990 09995, 2023.
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33: 1179-1191, 2020.
[22] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 613006130. PMLR, 2021.
[23] Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regu- larized behavior value estimation. arXiv preprint 2003i::103300575 2021.
11

---

[24] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34: 34:15084-15097 2021.
[25] Runzhe Wu, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Offline constrained multi- objective reinforcement learning via pessimistic dual value iteration. Advances in Neural Information Processing Systems, 34:25439-25451 2021.
[26] Qian Lin, Chao Yu, Zongkai Liu, and Zifan Wu. Policy-regularized offline multi-objective reinforcement learning. arXiv preprint arXiv:2401.02244, 2024.
[27] Nikhil R Devanur, Christos H Papadimitriou, Amin Saberi, and Vijay V Vazirani. Market cquilibrium via a primal-dual algorithm for a convex program. Journal of the ACM (JACM), 55 (5):1-18, 2008.
[28] Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafede duality, arXiv preprint OrXiv:2001.0/860 2020.
[29] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[30] Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference on machine learning, pages 3652-3661. PMLR, 2019.
[31] Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via of- fline multi-objective rl. In The Eleventh International Conference on Learning Representations, 2023.
[32] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.00 07219, 2020.
[33] Jie Xu. Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction guidedd multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pages 10607-10616. PMLR, 2020.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
12

---

Justification:
Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
Appendix
D
Appendix
Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
13

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Appendix D Appendix Appendix
F
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guudeeee http://oeeuuiiiiiioooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
14

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Appendix D Appendix
E
Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section Appendix
H
Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix G
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
15

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpge///aarrps.c.cobb/ll//lclle
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code for Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not directly engage with societal elements or have immediate applications that could identified as having positive or negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
16

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our L.M policy httppp//eeaarrrreeec/////feeeeee for what should or should be described.
18

---

A DICE-RL framework
In this section, introduce the DICE-RL framework along with its corresponding offline single- objective reinforcement learning algorithm, OptiDICE, as proposed in [22]. Our algorithm can be viewed as a multi-objective extension of OptiDICE. The DICE-RL framework is offline RL framework where return maximization is regularized with an f-divergence between the stationary distribution of the learned policy d and the empirical distribution dp, solving:
d(s,o)
Σ
d(ssa))(ss - 3 Σ dp(s, a)f dp(s,a)
max 420
8,0
3,0
s.t. Σ
= (1- + Σ 8,0
T As
where the Bellman flow constraints ensure that the optimal d' d'(s, a) constitutes valid stationary distribution. Lagrangian dual of the convex optimization problem is given by.
3 Edp(s,a)s d(s,a) dp(s,a)
min Σ (((s))) + Eu(s)F((s)
d20
where Fa(s) (1-
+ Es,o T(s)s,a)d(3, -. d(s We reparameterize the a) and express the dual in terms of the importance
stationary distribution as d(ss, a) = weights w, using the identity Σ, ( Es, d(s,a) = Σ., d(s, a) Es T(slss This yields the following optimization problem:
max min L(w,2 v) = Ex-poll [11 (s){s}] + E (sa (w(s, (), a) 38 (wo(s, a))l
w20 "
a) where (8, a) a( +
u(s) Vs,a.
Strong duality holds by Slater's condition, since the problem is convex and valid stationary distribution exists in the strictly feasible set. Therefore, the order of optimization can be swapped to min, maX Optimal w" (s, a) is computed from the first-order condition given by:
aL(w, v) @w(s,a) E(s,))dd (e) 33
where a)) max (F)- (e((w))) )) By plugging as, in L(w,v) results in the following v loss OptiDICE.
(v(8,a) )
min Ba [11 - [))()) + do gfō
B
where (z(v) max;20 (xx
After minimizing over v. the optimal stationary distribution is given by d' (s, a) = wy (s, a)).sa)) This distribution can then be used to recover the optimal policy that induces d' (s, a) via n- (als)
d' (s.a) E. (s,0) for all 8, a, or through policy extraction method such as weighted behavior cloning.
B Counterexample
In this section, we present counterexample demonstrating that applying the optimal implicit weights 4 from Fair MORL (P2) Linear MORL (P3) does not recover the optimal policy of Fair MORL (P2). Consider an MDP with single state and terminal state reached immediately after taking an action. There are two available actions, a E (a), a2), with corresponding stationary distributions d(s, a1) and d(s, 02). Each action yields reward vector consisting of rewards for objectives A and B: r(s, a1) = (r,(s, (1), TB a1)) = [1,4] and r(s, a2) = [r a2), TB(S, az)] = =[3,1] 1].
Using this setup, we construct the corresponding (PI) optimization problem, while assuming Nash social welfare (u; log(x) Vi):
(P1): max log (d(s, a1) + 3d(s, az)) + log (4d(s,a1) + d(s.
d20
s.t. d(s + a2) =
19

---

where log(x:) is applied to the returns of each objective. An equivalent optimization (P2) is given by:
(P2): max log (k1) + log (k2)
d20,6
s.t. d(s,, 01) - d(s, 02) = 1
d(s, - 3d(s, 02) = 11
4d(8, 01) + d(s, 02) 1-2
Its Lagrangian dual is given as,
mAx min log (ki) + log (k2)) ((((s, 01) + a2) -1) d20,6 H.M.
+ Mi a1) + 3d(ss a2) kn) + p2
d(s, 02) K2)
By solving the first-order conditions, the optimal Lagrange multipliers are μr R 0.5455 and j s 0.3636, resulting in the optimal stationary distribution [d (s, a1 d (s, a2)] = [0.5834, 0.4166). The optimal policy of Fair MORL (P2) is a stochastic policy that prefers action a₁ while still assigning probability to d2. resulting in objective returns k* = [1.8332, 2. 2.7500]. While μ is applied to the objective returns in manner analogous Linear MORL, we demonstrate that the two formulations are fundamentally distinct by applying the implicit preference weight μ to (P3):
(P3): max (1 μ + 4- 15)d(s, a1) + (3 + +1 (11.55)) 12)dss
d20
s.t. d(s, 01 + d(s, =
In this case. 1- + 4 μ1z = 3 μu + μ² 2.0, indicating that all policies are the all optimal policies of (P3). As result, any policy is optimal under Linear MORL with fixed weights μ.. This demonstrates that (P2) and (P3) are distinct optimization problems.
C Proof of Proposition 1
In this section, we provide proof of Proposition [ showing that regularized Linear MORL (P3-reg), when using preference weights equal to the optimal dual variable μr from regularized Fair MORL (P2-reg), converges the same solution as (P2-reg). To facilitate the explanation, we begin by rewriting the (P3-reg) formulation:
Ed(s, - dd(s,))f d(s,a)
(P3-reg): max
(d((sa))
s.t. (2)
d20
The Lagrangian duals of (P2-reg) and (P3-reg) are given by:
d(s,a) dd(ssa))
max min Lpg (V, u1, k) = Σ Σ d(s; -k:) )1)) -B Σ Edo(s,a)f 8,a
d20,k u.
8,0
+ Σ Us (k₂) + Σ v(s)Fa(s)
Σ d(s,a) a) Σ ()ri(s,a) Σ do((,a)f ( d(s,a)
max maxmin Lp3-rog (V, d) == d20
dp(s,a)
8,a
8,0
Σ (s)FFass
Assuming the optimal 4" from (P2-reg) is given, we compute the gradient of each Lagrangian with respect to v(s) and d(s, and show that both share the same gradient at their optimal solutions.
ÖLpz-cg aL (a)
Ov(s)
Ou(s)
ÖLp2-re
ÖLpuee
- - (=(s,a) d
Σ [TT(d)saa)010)--
ad(
Od(ssa) a)
5.a
While (P3) is linear program, (P3-reg) becomes convex optimization problem due to regularization with a strictly convex function f, making the KKT conditions applicable. From the stationarity conditions, (P3-reg) with μ² converges the same optimal solution as (P2-reg).
20

---

C.1 Empirical evidence
We adapt the counterexample from Appendix to demonstrate that, under offline regularization. Linear MORL and Fair MORL can share the same optimal solution. In the offline setting, we additionally assume a fixed data distribution over actions given by [dD(s, a1). The (P2-reg) formulation of the counterexample is given by:
(nn).dp(s.og (2. [0.7, 0.3].
Σ do(s,a)f d(s.a)
(P2-reg): max log (k1) + log (k2) d20,k
dp(s,a)
a
s.t. d(. + d(s. = d(s,a1)+ 02) = kı
4d(s, (1ls,a1)+ d(s a2) k2
where we adopt x²-divergence f(x) -- 1)2. The optimal Lagrange multipliers are p R 0.5959 and 222 R 0.3352, resulting in the optimal stationary distribution [d (s, di),d" a2) = [0.6609, 0.3390J. This indicates that as 0, is more common in the data distribution than a2, offline RL policy of (P2-reg) favors a₁ compared to the unregularized case. This aligns with the goal of offline reinforcement learning where the optimized stationary distribution should not deviate excessively from the dataset distribution. We apply μ² to (P3-reg).
-dd Edn(s,a)s d(s,a) d))
(P3-reg): max (1 1i + /4 ( ai) + (3 p1 + -2)d(s, - d20
s.t. d( a1) d(s) 02) 11
where 1 μ + 4 μ1 1.936 and 3- μ + μ12 2.123.
As (P3-reg) a convex optimization problem with strictly concave objective, the uniqueness of the optimal solution is guaranteed. The optimal stationary distribution of (P3-reg) is equivalent to that of (P2-reg) as [d" (s, a1). d² (s, a2)] = [0.6609, 0.3390). This establishes the connection between regularized Linear MORL and Fair MORL theoretically and empirically.
D Finite Domain Experiment Setting
In this section, provide detailed explanation of the experimental setup used in Section 6 We begin by describing how the classic Four-Room environment [22] 29 and Random MDP [2 30 are
adapted for the offline multi-objective reinforcement learning (MORL) setting. We present additional visualizations of the MO-Four-Roon results to further support the findings in Section 6
D.1 Environment detail
MO-Four-Room In MO-Four-Rowe domain, three distinct goals, each associated with separate objective. The agent starts from the initial state (orange) and navigates toward one of the goal states (green). Upon reaching goal, it receives a one-hot reward vector: [1, 0, 0) for the lower-left room, [0, 1, for the upper-right room, and [0, 0, 1] for the lower-right room. The agent selects one of four actions, {left, right, up, down): however, the environment is stochastic, and with probability 0.1, 1, the agent transitions in different direction than the one intended. To simulate offline RL, dataset of 300 trajectories are conducted with a E (0.0, 1.0), where a 0.0 corresponds to a policy that maximizes Utilitarian welfare, and = 1.0 corresponds to one that maximizes Nash social welfare. The regularization coefficient is fixed at B = 0.01 and f(x) = 0z - 1)2.
collected from uniformly random behavior policy. The experiments
In the Random MOMDP domain, a multi-objective Markov decision pro- cess is generated with |S| 50, [A| 4, and discount factor 7 0.95. For each state-action pair, the next-state transitions are defined over four possible next states, with transition proba- bilities sampled from Dirichlet distribution, Dir(1, I, Among the 49 states excluding the fixed initial state, three are randomly selected as goal states, and each is assigned distinct one- hot reward vector the same manner as in the MO-Four-Room environment. To simulate the offline RI. setting, dataset of 100 trajectories is collected using behavior policy with an opti- mality level of 0.5. Here, optimality defined as the normalized performance relative to a uni- formly random policy Tunif (optimality 0.0) and optimal policy x" (optimality = 1.0). This
Random MOMDP
21

---

Nash
Utiliartaa
Jain's Falmess
-20
0.073
-0
6.0%
4.0%
6.0%
22
0.550
3.0
2.0%
24
%%
-2.0%
2.0%
2.0%
0.575
0.5
6.0%
5.0%
0.850
6.0%
0.4
10.0%
-22
10.0%
0.52
10.0%
Objective
Objective 2
Objective 3
10.0%
0.008
10.0%
0.000
10.0%
-00077
0.007
3.006
0.006
0.006
3.005
2.0%
0.005
2.0%
0.005
2.0%
D22
20%
2.0%
2.0%
3.033
0.003
-0002
-0.002
3.002
-0000
00.01
3.001
10.0%
10.0%
10.0%
10.9%
2.0%
2.05
1049h
4.%
24%
2055
8.0%
10
-000
4.%
2299
2,0%
10
4Bg
43)
A:3)
Figure 7: Visualization of FairDICE-fixe performance with different μ in MO-Four-Room The center point is the optimal μ² obtained by FairDICE. The x-axis and y-axis represent the degrees of perturbation applied to #2 and μa from their optimal values.
implies that the behavior policy achieves performance halfway between that of the optimal and random policies. The experiments are repeated over 1000 seeds, with a E {0.0, 0.5, 1.0, 1.25) and / E {0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0), to provide comprehensive analysis of the trade-off between welfare maximization and distributional shift.
D.2 Additional visulaization: MO-Four-Room
Figure I illustrates that FairDICE maximizes the welfare objective, resulting in behavior that is distinct from conventional MORL approaches. In this subsection, we extend the experiment from Section 6.2 to further visualize two key insights proposed in our paper: (1) μ corresponds to the preference weights used in regularized Linear MORL, and (2) these weights are implicitly optimized to maximize the welfare objective-ang any deviation from the optimal μ leads to a reduction overall welfare. In Section 6.2] all preference weights were perturbed within Random MOMDP setting. While this effectively demonstrated that FairDICE selects the welfare-maximizaee weights, not easy to visualize the consequence of deviation.
Given the optimal preference weight of FairDICE within So-Foou--Romn μz' [A1. U2, U3], we perturb #a and μą while keeping μ1 fixed. The perturbed weights are applied to Fair to obtain the corresponding optimal policy, whose performance then evaluated as shown above.
The center point corresponds to the optimal solution of FairDICE, achieving the highest Nash social welfare. Increasing μl2 boosts return objective 2, while increasing A3 improves return objective 3. Return on objective! increases when both μ22 and AAg decrease. The point that maximizes Utilitarian welfare shifts toward prioritizing objective 1, but this comes the cost of reduced Nash social welfare and lower Jain's 's fairness index. These results highlight that FICE implicitly optimizes preference weights maximize welfare, enabling fair behavior the offline MORL setting.
E D4MORL Benchmark
To evaluate the efficacy of offline multi-objective reinforcement learning (MORL) algorithms, we utilize the D4MORL benchmark (with MIT License) introduced by (31). D4MORL is the large- scale benchmark designed for offline MORL and includes high-dimensional continuous control environments derived from MuJoCo. We adopted settings and configurations of D4MORL without additional modifications.
22

---

Te forward speed E.1 Environments and Objectives
D4MORL comprises six environments: MO-Ant, MO-HaliCheetah, MO-Hopper, -wimmer, and MO- H0-Walker2d, each with two conflicting objectives (e.g., speed vs. energy efficiency), and (O-Hopper-3oby) which includes three objectives-maken it more challenging benchmark. Each environment is defined by multiple, often confficting, objectíves such as forward velocity. jumping stability, or energy consumption. These objectives induce trade-offs, thereby enabling the study of Pareto-optimal policy learning in continuous control settings.
We summarize the objectives for each environment in Table including their physical interpretations and reward formulations. The rewards are computed based on physical quantities such as displace- ment, height, and control cost. Most environments include survival bonus term r, and penalize excessive actions via an action cost term Na = Σx oz. The time delta At determines the resolution of velocity and height- based rewards and is aaviromeen--speeie
Reward Terms and Environment--Seecin Constants. The reward functions in Table 2 include shared terms whose values vary across environments:
Action penalty (ra): Typically defined as the squared sum of action magnitudes, i... ng
Σn a. However, different environments apply distinct scaling factors: Ta = 0.5 Σ. 2 in MO-Ant; Ta = 2 x 10- >x of in MO-Hooper.
Survival bonus (ra): A constant reward to encourage survival, set to T3 1.0 in all enviromments except MO-Swimmer, where it is omitted.
Time delta (At): Represents the duration between timesteps used in computing velocities or other dynamic terms. Its value is Al = 0.05 in MO-Ant, MO-HalfCheetah, and MO- Swimmer; At 0.01 in MO-Hopper and MO-Hopper-3obj: and At 0.008 in MO- Walker2d.
Initial height (hmit): In MO-Hopper and MO-Hopper-3obj, vertical jump rewards are defined relative to a fixed starting height of hiit 1.25.
These constants follow the environment configurations detailed in Appendix A of [1].
Table 2: Objectives in D4MORL. Environments
Environment
Objective Name
Reward Description
MO-Ant
Tezi velocity in x direction
Fuz
Br At
+ rs Ta
Tey: velocity in y direction
Toy
Al
+ rs - Va
MO-HalfCheetah
To: forward speed
r. min(4.0,
zt -t-Ie-1
+ Ts.
At
Te: energy efficiency
Te = 4.0 - -To + T'
MO-Hopper
r forward running
r, = 1.5 01-31-1
+ T2 Ya
At
Ty: vertical jumping
T3 = 12
h Ninit At
+ +1 To
MO-Hopper-3obj
ry forward running
r, 1.5
81 81-Bi-1 Ts
At
ry: vertical jumping
T 12.
h, Wimil
At
re: energy efficiency
Te = 4.0
Ta + T
MO-Swimmer
Tel forward speed
r. = #1 01-P1-1
At
re: energy efficiency
Te = 0.3 0.15 To
= 31 21-31-1 + T, At
re: energy efficiency
Te = 4.0 ra + T's
E.2 Behavioral Policy Quality
Each dataset in D4MORL is collected using:
Expert policy: Selected from a large PGMORL -rained ensemble to match the target preference closely.
23

---

Amateur policy: Amateur policy: Perturbed version of the expert policy, where actions have certain probability being stochastic. For most environments, stochastic actions are generated by scaling the expert action by random factor sampled from Uni5( 1.65) (65%), while the expert action is retained otherwise (35%). In the MO-Swimmer environ- ment, stochastic actions (35%) are instead uniformly sampled from the action space to better approximate amateur-level performance.
The target preferences used during data collection are uniformly sampled from the full (n - 1)- dimensional simplex. This promotes diverse trade-offs across objectives and ensures that the dataset covers a wide range of possible preferences. All preference vectors w E R² are normalized to satisfy wi and Σ; wi
E.3 Reward and State Normalization
We introduce how rewards and states are normalized in the D4MORL benchmark (31].
Reward normalization. The reward values are normalized for each objective to the [0, 1] range using min-max normalization:
.orm 7 7'ax Tmin
Tmin
where m'inn 7max are computed empirically from the offline dataset for each objective dimension.
State normalization. All state vectors are standardized using environment-specifee statistics provided by the D4MORI. benchmark. Specifically, the raw state is normalized as:
norm
where and denote the per-dimension mean and standard deviation of the state distribution, computed from the offline dataset.
F
Implementation Details
We use the Soft-x² divergence as the regularization function ff in FairDICE. This function is defined piecewise as:
if < 12 1)2 otherwise
x log
f(x)
This divergence combines the smooth behavior of KI. divergence near x 0 with the quadratic growth of the standard x² divergence for larger æ.
Although the definition of the convex conjugate f"y may suggest a bi-level optimization, once f=) specified, both (z) and (f')) can obtained in closed form. For the chosen Soft- x²² divergence, we have:
( 1, y <<
< 0,
(v)
(x)- 11+2, otherwise.
5y2² +y, otherwise,
Therefore, the final FairDICE objective can be computed directly without solving any inner maxi- mization loop.
The FairDICE algorithm, summarized Algorithm alternates between optimizing the dual variables (v, (4) and updating the policy æ via weighted behavior cloning. Both the policy and critic # networks are implemented as multilayer perceptrons, parameterized by & and 6 respectively. The scalar parameters μ are updated to maximize the desired social welfare function, and fix 1 to correspond to the Nash social welfare objective. The initial state distribution po is estimated from the offline dataset.
Table provides a summary our default hyperparameters. The policy and value networks are constructed with three hidden layers, each containing 768 units. Optimization is performed using the Adam optimizer with learning rate of 3 x 10-4 and discount factor of 7 = 0.99. To study the effect of the regularizatio arizaaiio coefficient 3-which governs the trade-off between distributional robustness and optimization stability- we conduct a hyperparameter sweep over 3 E {1.0, 0, 0.01, 0.001, 0.0001). Our code is available at: https: htt///gitttbb.t

---

Algorithm FairDICE
Input: Offline dataset D. initial state distribution Po, policy ne. dual parameters Ves p4, divergence regulrarization parameter 3, concave scalarization function u, policy T
Output: Welfare-maximizze
Initialize all parameters
2: while not converged do
Update Vgl l to minimize:
Lu Essopo [(1 ()) + (s,a)) 35% (cosul(ssa)
)
Σ us (-
+
B
4: Compute optimal weights:
"y (3.0) = max (0.15)-1 (crssulo
3
5: Update policy mo via weighted behavior cloning:
Lo -REas ( (8, a) log To s)
6: end while
Table 3: Implementation Details for MO Environments
Hyperparameter
Value
B
(1.0, 0.1, 0.01, 0.001, 0.0001)
Hidden dim of uy and me
768 (512 for MO-Ant)
layer of Vi and me
3 (4 for Mo-Hopper-3ob))
Learning rate
x 10
7 (discount factor)
0.99
Optimizer
Adam
Values in parentheses indicate environment-specifcc overrides.
G Experiments Compute Resources
All experiments were conducted on single machine equipped with an Intel Xeon Gold 6330 CPU (256GB RAM) and an NVIDIA RTX 3090 GPU. Training single FairDICE policy on each D4MORL task required approximately 10 to 20 minutes on average. During training, GPU memory usage remained below 20GB.
H Robustness of FairDICE to Limited Data Quality and Coverage in Offline Multi-OOjecctve RL
As offline RL methods are often sensitive to the quality and coverage of the dataset, we provide empirical evidence that FairDICE exhibits degree of robustness suboptimal data quality and limited coverage. Regarding data quality, we evaluated FairDICE on both the expert and amateur datasets from the D4MORL benchmark, and consistently achieved high Nash Social Welfare across both settings. To further assess robustness to limited data coverage, we conducted additional experiments where we filtered out trajectories whose preference weights lie near the center of simplex. Specifically, we removed trajectories in which all preference weights fall between 0.4 and 0.6. This filtering removes data points likely to represent balanced fair trade-offs, resulting in more challenging offline dataset.
25

---

Table 4: Comparison of Nash Social Welfare before and after trajectory filtering across different
environments and dataset qualities.
Environment
Dataset Quality
Traj. Removed
NSW (Full)
NSW (Filtered)
VO-Swiimer-v2
Expert
24.0%
11.597 0.091
11.489 0.192
HO-Swimmer-ve
Amateur
24.0%
11.820 0.005
11.819 = 0.001
MO-Walker2d-M
Expert
34.9%
11.534 + 0,039
9.562 = 1.161
MO-Walkerd-vv2
Amateur
35.0%
11.396 + 0.291
11.339 + 0.016
MO-Ant-v2
Expert
43.1%
11.535 0.018
11.320 + 0.332
MO-Ant-v2
Amateur
43.2%
11,509 + 0.049
11.384 + 0.027
HIO-HalCCheetab--2
Expert
43.6%
11.828 0.017
11.714 - 0.035
MO-
HD-HalfCheetth-v22
Amateur
43,9%
11.994 + 0.146
11.709 - 0.074
MO-Hopper-v2
Expert
67.3%
11.058 1 0.395
11.157 1 0.014
MO-Hopper-v2
Amateur
67.7%
11.570 0.003
11.548 0.003
MO-Hopper
MO-Swimmer
11.580
NSW Score
11.576
11.800
11.785
11,572
11.770
11.568
0.0001
0.001
0.0001
0.001
MO-Walker2d
MO-Ant
11.5
NSW Score
10.5
11.2
11.0
9.5
Beta
Beta
Figure 8: Performance of FairDICE with varying 3 values on amateur datasets in D4MORL. Results are averaged over 10 seeds and 10 episodes, with error bars denoting +1 standard errors.
The results are shown without any filtering, while Nash Social Welfare (Filtered) reports performance after removing the trajectories. Each result shows the average Nash Social Welfare (NSW) over seeds. FairDICE continued to perform reliably, demonstrating its ability to optimize fairness-driven objectives even under biased or sparse data conditions.
Nash Social Welfare (Full) refers to performance on the original dataset
I Impact of f-divergence on FairDICE
In this section, we investigate how varying 8, which controls the strength of regularization, af- fects FairDICE's performance. Figure g reports Nash social welfare (NSW) performance for (10.0, 1.0, 0, 0.01, 0.001, 0.00001). We show that the trade-off between NSW and distributional shift observed in FairDICE in the finite domain (Figure 2 generally extends to the continuous domain. NSW performance typically improves as 3 decreases, reflecting a stronger emphasis on maximizing NSW and reduced reliance on the dataset distribution. While FairDICE
26

---

displays strong NSW across wide range of 3, excessive distributional shift at very small 8 can degrade its practical performance.
An exception is observed in the MO-Swimmer environment, where NSW consistently decreases as & decreases. This is likely because the MO-Swimmer dataset already contains trajectories with high NSW, making further deviation harmful. This is supported by Figure which shows that although trajectories were generated using different preference weights in existing preference-based baselines, they consistently achieve high NSW.
27


Reviews:
[{"review_id": "ZtEK069DQT", "reviewer": "Reviewer_Rez3", "summary": "This paper studies the problem of multi-objective offline reinforcement learning with fairness-oriented goals. Fairness requires a nonlinear and non-additive combination of objectives which makes the standard linear scalarization method ineffective. This work extends DICE-RL framework to the multi-objective setting which adopts the regularized linear programming formulation of RL. Based on this framework, they derived a practical sample-based objective FairDICE for fairness-oriented goals. They also build the relation between FairDICE and the standard linear formulation by showing FairDICE implicitly optimizes the linear weights. Finally, they conducted experiments to show the effectiveness of their method.", "questions": "1. How is the FairDICE objective (4) optimized practically?\n2. If the setting is online, can we design a practical algorithm based on the dual formulation (3)?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This work studies the setting of fairness-oriented multi-objective offline reinforcement learning which is relatively under-studied.\n2. The experiments are comprehensive including both discrete and continuous domains.\n3. It is an interesting result that optimizing the regularized fairness objective is implicitly optimizing the preference weights of all objectives.\n\nWeaknesses:\n1. The main contribution of this work is the FairDICE objective for multi-objective offline RL problem. However, formulating fair MORL as a convex optimization (P1) and converting it to the dual formulation (P2) is a standard derivation. It is not clear what specific challenges are and how they are solved. And with (P2), the derivation of the final FairDICE objective is to incorporate the existing DICE-RL objective to take the distribution shift into account. Based on these facts, I think the innovation of this work is a little bit limited.\n2. Different from the objective of DICE-RL, the FairDICE objective contains convex conjugate functions which makes it a bi-level  optimization problem, especially the term $f^*(e_{\\nu,\\mu}(s,a))$. In the paper, there is no explanation on how the objective is optimized practically.", "was_revised": true, "final_justification": "I am recommending an accept. It is a good and clear work on fair offline multi-agent RL. I didn't give a score of 5 considering the technique of introducing slack variables is a standard trick in optimization area and the regularized offline RL framework also builds on DICE-RL which makes me feel the work has a moderate not high impact.", "rebuttal": "Dear Reviewer Rez3,\n\nThank you for your thoughtful and constructive feedback. We appreciate your careful review and hope that the following response addresses your questions and alleviates your concerns.\n\n>**[W1: Clarification on Novelty Beyond Standard Convex Reformulation and DICE-RL]**\n\nTo the best of our knowledge, introducing slack variables to derive an equivalent convex optimization problem is novel in the context of reinforcement learning. In typical formulations of Fair MORL, the objective takes the form of a concave utility function applied to an expected return, as in problem (P1). A direct approach to solving (P1) is challenging because the expectation is nested inside the non-linear (concave) function, which leads to **biased** sample-based estimation.\n\nTo address this challenge, we introduce slack variables to move the term $\\sum_{s,a} d(s,a) r_i(s,a)$ outside the concave utility function $u_i$ (to linear constraints)​, resulting in a reformulated problem (P2). This crucial step enables us to develop a sample-based algorithm, FairDICE, which **circumvents the bias issue** incurred by taking expectation inside the concave function.\n\nAlthough the derivation borrows elements from DICE-RL to account for distribution shift, simply applying DICE to a multi-objective setting is non-trivial—especially when fairness and optimality must be preserved across conflicting objectives. Our contribution lies in identifying how to retain these properties within a convex dual structure, which, to our knowledge, had not been done before in this context.\n\n>**[W2,Q1: Lack of Explanation on Practical Optimization of FairDICE’s Bi-Level Objective with Convex Conjugate Term]**\n\nThe use of a maximization in the definition of the convex conjugate may give the impression of a bi-level optimization. However, **once the function $f(x)$ is specified, its conjugate $f^{*}(y)$ can be derived in closed form**. For example, in our implementation of FairDICE (Appendix F), we use the soft-$\\chi^2$ divergence as the regularization:\n\n$\nf(x) = \n\\\\begin{cases} \nx \\\\log x - x + 1, & x < 1 \\\\\\\\\n0.5(x - 1)^2, & x \\\\geq 1\n\\\\end{cases}\n\\\\quad \\\\text{and} \\\\quad u(x) = \\\\log x.\n$\n\nThe corresponding conjugate functions and $(f')^{-1}(x)$ are given analytically by:\n\n$\nf^{\\\\ast}(y) = \n\\\\begin{cases} \ne^y - 1, & y < 0 \\\\\\\\\n0.5y^2 + y, & \\\\text{otherwise}\n\\\\end{cases}\n\\\\quad \\\\text{and} \\\\quad u^*(y) = 1 + \\\\log y.\n$\n\n$\n(f')^{-1}(x) = \n\\\\begin{cases}\ne^x, & x < 0 \\\\\\\\\n1+x, & \\\\text{otherwise}\n\\\\end{cases}\n$\n\nTherefore, the **final objective can be computed directly and does not require solving an inner optimization loop**.\n\nThe practical optimization procedure is provided in Algorithm 1 in Appendix F (p. 25). To improve clarity and accessibility, we will revise the main text to include a concrete example of the regularization function $f$, the utility function $u$, and their convex conjugates, and clearly reference the relevant section of the Appendix where the full algorithm is described.\n\n>**[Q2: If the setting is online, can we design a practical algorithm based on the dual formulation (3)?]**\n\nEquation (3) by itself is not directly suitable for constructing a practical online algorithm, for two main reasons. First, the variable $d(s,a)$ does not generally correspond to the stationary distribution of any policy unless it satisfies the Bellman flow constraints. Second, even if we were to compute expectations with respect to $d(s,a)$, the $F_d(s)$ term depends on transition probabilities for arbitrary $(s,a)$, which are typically unknown in standard model-free settings (both online and offline). These challenges limit the direct applicability of (3) and motivate the subsequent derivations in our paper that transform it into a practical, sample-based objective."}, {"review_id": "gAi6HRrjba", "reviewer": "Reviewer_YvDZ", "summary": "This paper presents FairDICE, a method to learn fairness-aware policies in offline multi-objective RL. Improving on the limitations of existing MORL techniques that rely on linear combinations of objectives, this approach allows for learning concave welfare functions which require non-linear scalarization. The authors cast fair MORL as a convex optimization problem, and extend ideas from DICE-RL, using regularized optimization to avoid distribution shift from the offline dataset while still learning welfare-oriented policies. The authors derive the optimization using a primal-dual formulation. They also show equivalence of the proposed algorithm to the regularized linear MORL.\nFinally, they perform experiments on discrete and continuous domains, and show FairDICE is able to learn fair or utilitarian policies, based on the alpha-fairness measure with different weights. They also discuss the evolution of these methods when random noise is added and when they are trained on one metric but measured on another. Finally, the authors show the policies generated by FairDICE lie on the pareto front when compared to other baselines with exhaustive search.", "questions": "Q1. Why is the non-linearity an issue? The bellman equation can still hold as long as the non-linearity is within a single time-step. Is there a time-dependence for the non-linearity?\n\nQ2. Why is the equivalence between fair MORL and regularized  linear MORL important?\n\nQ3. Why is FairMORL a single point in Figure 5? Is it possible to elicit different tradeoffs? If yes, where do they lie in this figure, and why is this not compared?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces fair MORL in the offline setting, providing a new and performant method for learning policies with non-linear scalarizations as required by many welfare functions. This is important for settings where balancing between objectives is key.\n2. The authors also extensively evaluate the method and show its strength compared to reasonable baselines and analyze the behavior in different regimes.\n3. The theoretical formulation for the method gives it credence.\n\nWeaknesses:\n1. The paper assumes that u is a concave function (line 113), without defining in what sense this concavity is applied. Is it concave over all objectives? Is it concave over time? While readers can figure this out, clearly stating this and defining $u$ concretely will help.\n2. It is not clear why the equivalence between the regularized linear and fair MORL methods is important. The authors should discuss the significance, as significant space is spent outlining this in the paper.\n3. In Figure 5, FairDICE is a single point, and often lies close to MORvSP. It seems MORvSP is similarly performant and able to capture a wider range of pareto-optimal solutions. If it is easy to train FairDICE with specific welfare functions, why are there not multiple points for FairDICE in this figure? It signals a lack of control rather than a strength (e.g. why is the tradeoff selected by FairDICE any good?)", "was_revised": true, "final_justification": "The authors sufficiently clarified my doubts about the paper. I will maintain my score.", "rebuttal": "Dear Reviewer YvDZ,\n\nWe sincerely thank you for your detailed comments and insightful suggestions. We hope that our responses below provide the necessary clarification and help resolve the points you raised.\n\n>**[W1: Clarification Needed on Domain and Interpretation of Concavity Assumption for Utility Function $u$]**\n\nTo clarify, the concavity we refer to applies individually to each objective's return component, not over time steps. That is, the total objective is of the form $\\sum_{i}u_{i}(R_{i})$, where each $u_{i}$ is a concave function applied to the return of the $i$-th objective.\n\nWe use concave functions to promote fairness across objectives. This approach is motivated by the principle of diminishing marginal utility, which implies that an identical gain in return provides more utility to underperforming objectives than to well-performing ones. As a result, the optimization naturally prioritizes weaker objectives, learning to more balanced and equitable outcomes.\n\nThis design choice is also supported by prior literature on fairness in multi-objective optimization, where per-objective concave functions or fairness-aware metrics (e.g., Nash Social Welfare) have been commonly used to balance outcomes [1, 2, 3]. We will revise the paper to clearly explain both the interpretation and the motivation behind this formulation.\n\n>**[W2,Q2: Clarification Needed on the Importance of the Equivalence Between Regularized Linear MORL and Fair MORL]**\n\nThe equivalence we establish serves a crucial **theoretical and practical purpose**: it demonstrates precisely why FairDICE offers a more principled and efficient approach compared to naive approaches.\n\n**Theoretically**, the equivalence proves that for a given fairness objective (e.g. Nash Social welfare), there exists a **single, specific** set of preference weights that a regularized linear MORL agent would need to find the truly optimal fair policy.\n\n**Practically**, however, finding these \"magic\" weights with a linear MORL method poses a major challenge. Since these optimal weights are unknown beforehand, one must perform brute-force grid search, a process that becomes computationally infeasible due to the curse of dimensionality as the number of objectives increases.\n\nThis is where the significance of FairDICE becomes clear. Instead of requiring this inefficient, extensive search, FairDICE treats the preference weights as optimization variables, directly learning the optimal weights as part of its end-to-end training process. In essence, it provides a principled way to find the optimal fair policy in a single run, without any weight sweeping.\n\nAlthough this motivation is briefly mentioned in the final paragraph of Section 5, we agree that a clearer and more explicit explanation would help readers better understand this advantage. We will revise the text accordingly in the final version.\n\n>**[W3,Q3: Why is FairMORL a single point in Figure 5? Is it possible to elicit different tradeoffs? If yes, where do they lie in this figure, and why is this not compared?]**\n\nWe would like to clarify that the primary goal of FairDICE differs fundamentally from that of linear MORL methods like MORvSP. Linear MORL approaches aim to find  a wide range of Pareto-optimal solutions, leaving the final decision of which trade-off is \"best\" to the user. In contrast, FairDICE, along with other non-linear fair MORL methods [1, 2, 3], is designed to  directly find a single, principled solution that is optimal under a pre-defined, meaningful fairness criterion (e.g., Nash Social Welfare). Therefore, the single point in Figure 5 is a deliberate feature of our design, not a limitation.\n\nIn Figure 5, FairDICE appears as a single point because it optimizes a fixed nonlinear scalarization function: Nash Social Welfare (NSW), defined by $u_i(x) = \\log x$. However, FairDICE allows for eliciting different trade-offs by changing the underlying utility function. For example, the widely used α-fairness formulation allows interpolation across a spectrum of fairness criteria:\n\n$\nu_i(x) =\n\\\\begin{cases}\n\\\\frac{x^{1-\\\\alpha}}{1 - \\\\alpha} & \\\\text{if } \\\\alpha \\\\ne 1 \\\\\\\\\n\\\\log(x) & \\\\text{if } \\\\alpha = 1\n\\\\end{cases}\n$\n\nSmaller values of $\\alpha$ (e.g., $\\alpha \\to 0$) emphasize efficiency and resemble utilitarian welfare, whereas larger values (e.g., $\\alpha \\to \\infty$) prioritize equality, approaching max-min fairness. In addition, a weighted Nash objective, such as $u_i(x) = w_i \\log x$, can be used to reflect asymmetric priority across objectives. Each of these alternative formulations would yield a different point on or near the Pareto front.\n\nWhile we did not compare multiple trade-offs in this figure, we agree that such an analysis could provide additional insight. We will clarify this design choice in the paper and discuss the potential to explore alternative fairness trade-offs as part of future work.\n\n**References:**\n\n\\[1] Zimeng Fan, Nianli Peng, Muhang Tian, and Brandon Fain. *Welfare and fairness in multi-objective reinforcement learning*. arXiv preprint arXiv:2212.01382, 2022.\n\n\\[2] Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, and Youngchul Sung. *The max-min formulation of multi-objective reinforcement learning: From theory to a model-free algorithm*. arXiv preprint arXiv:2406.07826, 2024.\n\n\\[3] Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. *Multi-objective reinforcement learning with non-linear scalarization*. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 9–17, 2022."}, {"review_id": "lrEg0y8iWo", "reviewer": "Reviewer_s6Th", "summary": "This paper investigates an important problem of fairness in multi-objective reinforcement learning (MORL) in the offline RL setting. Instead of using the standard linear scalarization function to optimize the expected sum of rewards, the authors propose to directly optimize nonlinear social welfare functions, which they claim better capture fairness goals such as Nash Social Welfare and $\\alpha$-fairness. To this end, they develop FairDICE, an offline MORL method that optimizes a nonlinear welfare objective while leveraging distribution correction estimation to handle distributional shift. The proposed method is evaluated across several domains and demonstrates promising results compared to existing baselines.", "questions": "See above and also below for more questions:\n\n1. How would FairDICE perform under different welfare functions, such as the generalized Gini or WOWA?\n\n2. Could the proposed fairness mechanism be extended to other offline RL methods, such as CQL, IQL, or BC?\n\n3. Offline RL methods are often sensitive to the quality and coverage of the dataset. How robust is FairDICE when data coverage is poor or biased, especially in real-world problems where multiple and possibly conflicting objectives are present?\n\n[1] Siddique, Umer, Paul Weng, and Matthieu Zimmer. \"Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Yu, Guanbao, Umer Siddique, and Paul Weng. \"Fair deep reinforcement learning with preferential treatment.\" ECAI 2023. IOS Press, 2023. 2922-2929.\n\n[3] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in neural information processing systems 33 (2020): 1179-1191.\n\n[4] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).", "limitations": "The limitations are adequately addressed by the authors.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and generally easy to follow.\n\n* The use of a nonlinear welfare function to address fairness in the offline MORL setting is an interesting and important contribution. To my knowledge, this is indeed the first work to formalize and solve the fair optimization problem in offline MORL.\n\n* I particularly like how the authors frame the fair MORL problem as a convex optimization problem and use a concave welfare function for policy learning. This is not only theoretically sound but also practically vert important and relevant.\n* The experimental results are also strong, comprehensive and considered baselines are strong and very relevant. \n\nOverall, I like this work. It addresses an underexplored but practically important problem. However, I have some concerns and questions, which I list below.\n\nWeaknesses:\n* The current work focuses on $\\alpha$-fairness, which has some well-known limitations. For example, if some utilities are negative, the Nash Social Welfare (which is equivalent to the geometric mean) becomes infeasible because it requires all utilities to be positive. This same issue can affect the general $\\alpha$-fairness for other values of $\\alpha$. This restricts practical applicability in real-world scenarios where some objectives may yield negative returns. I recommend that the authors also explore and if possibly include other general-purpose welfare functions, such as the Generalized Gini welfare function or Weighted Ordered Weighted Average (WOWA) as it has been considered in online RL settings [1-2].\n\n* Although the paper claims to be the first method to tackle fairness in offline RL, the proposed method relies heavily on the DICE-RL framework. It would be useful to explain why DICE-RL was chosen instead of other SOTA offline RL baselines such as CQL [3], IQL [4], or BC. For example, can the proposed method be integrated with these methods? This point is especially relevant given that similar extensions have been demonstrated in the online RL setting [1].\n\n* Another potential drawback of FairDICE is its dependence on the hyperparameter $\\beta$. As shown in the results, the performance appears quite sensitive to the choice of $\\beta$ which introduces additional tuning complexity on top of standard RL hyperparameters. More discussion on choosing this particular hyperparameter would be beneficial for the reader.", "was_revised": true, "final_justification": "The authors have addressed all of my questions and provided clear clarifications to my concerns. Specifically, my questions regarding the DICE-RL framework and the choice of the welfare function were thoroughly addressed. Based on these clarifications, I am increasing my scores accordingly.", "rebuttal": "Dear Reviewer s6Th,\n\nThank you for taking the time to review our work. We greatly value your feedback and aim to address your concerns thoroughly in the following response.\n\n>**[W1,Q1:Limitations of Alpha-Fairness with Negative Returns and Recommendation to Consider Alternative Welfare Functions like Gini or WOWA]**\n\nWe agree with the reviewer’s observation that the welfare functions used in our paper can become infeasible when negative returns are present, particularly functions like $\\log(x)$, which are undefined for non-positive values. A practical workaround for handling negative returns is to define the utility function piecewise. For the non-positive domain, we can use a concave function that handles negative inputs, and attach it to the original $\\log(x)$ utility function. For example, we can define the utility for $x<1$ as a quadratic segment, say $-0.5(x−2)^2 + 0.5$, and attach it smoothly to $\\log(x)$ for $x \\geq 1$. By doing so, we can ensure the function remains continuous, differentiable, and concave while being well-defined for all real-valued returns. Although this modifies the original Nash Social Welfare (NSW) objective slightly, it preserves its core fairness-promoting properties and can provide meaningful and robust solutions in practice.\n\nAlternatively, one can consider general-purpose welfare functions such as the Generalized Gini Function (GGF) and the Weighted Ordered Weighted Averaging (WOWA), which naturally handle negative returns. However, there are two structural differences that make optimizing rank-based objectives such as GGF or WOWA directly within the FairDICE framework nontrivial.\n\nFirst, the primary challenge with rank-based objectives like GGF or WOWA is the instability and optimization difficulties this introduces. These methods define the welfare function based on the **sorted order** of the objective returns. Since any off-policy estimation (OPE) of returns from a fixed dataset is inherently noisy, even small estimation errors can cause the predicted ranks of two closely-valued objectives to flip.  Furthermore, this rank-switching can make the overall objective function **non-smooth or even discontinuous** with respect to the policy being optimized. This can lead to unstable learning signals, making it challenging to optimize particularly in the **offline** setting. It is likely for this very reason that the works for GGF and WOWA mentioned by the reviewer, do so exclusively in the **online** setting.\n\nFurthermore, this practical challenge points to a difference in how these frameworks encode fairness. GGF and WOWA rely on explicit, discrete sorting to determine an objective's importance. In contrast, FairDICE uses a continuous and implicit mechanism: preference is not determined by rank, but is encoded directly into the smooth curvature of the concave utility function. This yields naturally smooth and stable prioritization based on an objective's current return level, which can lead to more stable optimization.\n\nWe appreciate the reviewer’s suggestion and agree that exploring alternative welfare formulations, particularly those that accommodate negative returns, is a promising direction for extending FairDICE to broader real-world scenarios. Such formulations could enable more stable and robust optimization of welfare in environments with high variance or unbounded return scales.\n\n>**[W2, Q2: Justification for Using DICE-RL Over Other Offline RL Methods Like CQL or IQL, and Potential for Integration]**\n\nOur choice of DICE-RL framework was driven by the unique structural requirements of Fair MORL objective, which are not easily addressed by value-based methods like CQL or IQL.\n\nThe key advantage of DICE-RL is its formulation as a convex optimization problem over the space of **stationary distributions** (d). This structure is well-suited for our objective, $\\sum_{i} u_i( J_i )$ where the expected returns $J_i$ are linear functions of the distribution $d$: $J_i = E_d [r_i]$. This allows us to use convex optimization techniques (i.e., introducing slack variables) to handle the concave utility function $u_i$ in a principled way, circumventing the difficult nested expectation problem.\n\nIn contrast, adapting value-based methods to non-linear fair objectives is fundamentally challenging. A naive approach might be to learn separate Q-functions $Q_i$ for each objective, and then derive a greedy policy. However, unlike standard linear scalarization, the standard policy improvement guarantee breaks down for non-linear objectives due to the fact that $\\sum_{i} u_i( J_i( \\pi ) )] \\neq E_{\\pi}[ \\sum_{i} u_i( J_i(s,a) )]$. Because these two quantities are not equivalent, there is no guarantee that optimizing the latter will improve the former (our true goal). Therefore, naively applying value-based methods would fail to learn truly optimal fair policy. It remains unclear how to optimize a fairness-driven policy when the scalarization is non-linear.\n\nIn light of such a challenge, investigating how to adapt SOTA offline RL algorithms like CQL and IQL to the fairness-aware MORL setting is a highly promising direction for future work. It has strong potential to improve the practical performance and scalability of fair MORL methods in complex environments.\n\n>**[W3: Concern About Sensitivity to Hyperparameter $\\beta$ and Need for Practical Guidance on Its Selection]**\n\nWe agree that the hyperparameter $\\beta$ plays a critical role in balancing the trade-off between welfare maximization and distributional shift, and as such, it can influence the final performance of FairDICE.\n\nHowever, our empirical results across both discrete and continuous domains indicate that FairDICE maintains robust performance for sufficiently small values of $\\beta$. To support this claim, we refer to the results in both domains. In the discrete setting (Random MOMDP, Figure 2), FairDICE consistently achieves high Nash Social Welfare across broad range of relatively small $\\beta$ values (e.g., from 0.001 to 1). This suggests that finding a suitable $\\beta$ is not a matter of delicate tuning. Similarly, in the continuous domain, as shown in Appendix H, FairDICE exhibits stable performance across a broad range of $\\beta$ values.\n\nWhile it is true that offline RL algorithms can be sensitive to hyperparameter choices, our findings suggest that FairDICE is relatively robust in this regard. We will revise the paper to clarify this point and provide additional guidance on selecting $\\beta$ in practice.\n\n>**[Q3: Robustness of FairDICE to Limited Data Quality and Coverage in Offline Multi-Objective RL]**\n\nAs offline RL methods are often sensitive to the quality and coverage of the dataset, we provide empirical evidence that FairDICE exhibits a degree of robustness to suboptimal data quality and limited coverage.\n\nRegarding data quality, as shown in Section 7, we evaluated FairDICE on both the expert and amateur datasets from the D4MORL benchmark [1], and it consistently achieved high Nash Social Welfare across both settings.\n\nTo further assess robustness to limited data coverage, we conducted additional experiments where we filtered out trajectories whose preference weights lie near the center of the simplex. Specifically, we removed trajectories in which all preference weights fall between 0.4 and 0.6. This filtering removes data points likely to represent balanced or fair trade-offs, resulting in a more challenging offline dataset.\n\n| Environment   \t| Dataset Quality | % Traj. Removed | Nash Social Welfare (Full)| Nash Social Welfare (Filtered) |\n| ----------------- | --------------- | --------------: | -----------------------: | ---------------------------: |\n| MO-Swimmer-v2 \t| Expert      \t|       \t24.0% |       \t11.597 ± 0.091 |           \t11.489 ± 0.192 |\n| MO-Swimmer-v2 \t| Amateur     \t|       \t24.0% |       \t11.820 ± 0.005 |           \t11.819 ± 0.001 |\n| MO-Walker2d-v2\t| Expert      \t|       \t34.9% |       \t11.534 ± 0.039 |            \t9.562 ± 1.161 |\n| MO-Walker2d-v2\t| Amateur     \t|       \t35.0% |       \t11.396 ± 0.291 |           \t11.339 ± 0.016 |\n| MO-Ant-v2 \t    | Expert      \t|       \t43.1% |       \t11.535 ± 0.018 |           \t11.320 ± 0.332 |\n| MO-Ant-v2     \t| Amateur     \t|       \t43.2% |       \t11.509 ± 0.049 |           \t11.384 ± 0.027 |\n| MO-HalfCheetah-v2 | Expert      \t|           43.6% |       \t11.828 ± 0.017 |           \t11.714 ± 0.035 |\n| MO-HalfCheetah-v2 | Amateur     \t|       \t43.9% |       \t11.994 ± 0.146 |           \t11.709 ± 0.074 |\n| MO-Hopper-v2  \t| Expert      \t|       \t67.3% |   \t    11.058 ± 0.395 |           \t11.157 ± 0.014 |\n| MO-Hopper-v2  \t| Amateur     \t|       \t67.7% |       \t11.570 ± 0.003 |           \t11.548 ± 0.003 |\n\nThe results are shown in the table above. Nash Social Welfare (Full) refers to performance on the original dataset without any filtering, while Nash Social Welfare (Filtered) reports performance after removing the trajectories. Each result shows the average Nash Social Welfare (NSW) over 5 seeds. FairDICE continued to perform reliably, demonstrating its ability to optimize fairness-driven objectives even under biased or sparse data conditions.\n\nWe will incorporate these additional robustness results into the updated version of the manuscript to better highlight FairDICE’s robustness.\n\n**References:**\n\n\\[1] Baiting Zhu, Meihua Dang, and Aditya Grover. *Scaling pareto-efficient decision making via offline multi-objective rl*. In The Eleventh International Conference on Learning Representations, 2023."}, {"review_id": "84E30mt1Uy", "reviewer": "Reviewer_SCn5", "summary": "This paper makes the following contributions:\n\n1. It proposes FairDICE, a novel method for addressing the offline multi-objective reinforcement learning (MORL) problem.\n\n2. It provides an analytical perspective on FairDICE, showing its equivalence to linear MORL under specific weighting schemes.\n\n3. It demonstrates the effectiveness of the proposed method through extensive experiments, supports the theoretical analysis, and includes ablation studies on hyperparameters.", "questions": "1. I am curious about the behavior of $\\mu$ during training. Does it continue to evolve throughout the entire training process, or does it typically converge quickly in the early stages and remain relatively stable afterward?  \nIf the latter is true, one could potentially view the algorithm as an optimizer for $\\mu$, which could then be combined with other algorithms that perform well in linear MORL settings—instead of relying on DICE as the policy solver. Would this be a viable direction?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. The writing is clear and fluent, with a well-balanced level of detail in both the main text and the appendix. The theoretical derivations are easy to follow.\n\n2. The proposed idea is natural yet novel: building on the DICE framework, the authors incorporate Lagrangian methods to adaptively handle nonlinear multi-objective reinforcement learning (MORL) settings.\n\n3. The paper includes detailed ablation studies and analyses regarding the hyperparameters involved in the method.\n\n4. The experimental setup is comprehensive, covering both continuous and discrete action space environments.\n\n---\n\n### Weaknesses\n\n1. To the best of my knowledge, there are no major flaws. The following points are minor and do not significantly affect the overall quality of the paper.\n\n2. A small suggestion regarding notation: the paper refers to the Bellman flow constraint using “(2)”, which sometimes appears at the end of equations. This can be misleading, as it may be mistaken for a subscript or label—particularly in the definitions of P2, P3, and P2-reg. While careful readers will not be confused, using a clearer reference format (e.g., avoiding placing “(2)” at the end of expressions) could improve the reading experience.\n\n3. Typos:\n   - Line 158: The position of $\\sum_{\\bar{s}, \\bar{a}}$ appears to be incorrect.\n   - In the loss expression below line 159, a summation symbol seems to be missing before the final term $u_i(k_i)$.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed responses and am satisfied with the clarifications provided. I have also read the rebuttals to other reviewers and found no points that significantly changed my overall evaluation. This is a solid and worthwhile submission.\n\nThat said, I believe the work is not particularly strong in terms of novelty, and there is still room for improvement in the design and completeness of the experiments, including ablations, as also noted by other reviewers. Therefore, I do not consider an increase in score to be warranted and will retain my current rating.", "rebuttal": "Dear Reviewer SCn5,\n\nWe sincerely thank you for your close reading and constructive suggestions. We hope the responses below address your comments.\n\n>**[Q1: Behavior and Role of $\\mu$ During Training and Its Potential Use with Other MORL Algorithms]**\n\nIn our experiments, we observe that the dual variable $\\mu$, which implicitly represents the preference weights across objectives, tends to converge relatively early in the training process (e.g., stabilizing around 20k steps out of total 100k training steps). This suggests that FairDICE quickly identifies the correct trade-off priorities by the fairness objective.\n\nHowever, while your suggestion of two-stage approach (using FairDICE to find an optimal $\\mu^{\\\\ast}$ and then plugging it into another high-performing linear MORL algorithm) sounds appealing, this direction is non-trivial due to the tight coupling between the policy optimization and the preference weight optimization in our framework.\n\nThe key point is that the equivalence we established in Section 5.2 holds **specifically for the regularized setting**. The optimal weights $\\mu^{\\\\ast}$ are optimal only for the regularized problem. Without regularization, as we show in Appendix B, the optimal policy for Fair MORL (which can be stochastic) can be fundamentally different from the one found by linear MORL (often deterministic), even if they use the same weights. Therefore, one cannot simply take the $\\mu^{\\\\ast}$ from our **regularized** FairDICE and plug it into a different family of offline RL algorithms (like CQL or IQL) and expect it to remain optimal. Those algorithms employ entirely different regularization principles to handle distribution shift. The optimality of $\\mu^{\\\\ast}$ is tied to the specific DICE-style optimization landscape.\n\n>**[W2, W3: Notation Clarity for Bellman Flow Constraint and Minor Typographical Corrections]**\n\nWe sincerely appreciate your suggestion regarding the use of “(2)” to refer to the Bellman flow constraint. To avoid potential confusion with subscripts or equation components—especially in the definitions of P2, P3, and P2-reg—we will adopt a clearer notation. Specifically, we will use $BF(d) = 0$ to denote the Bellman flow constraint after it is formally defined in the paper.\n\nWe will also fix the following typos in the updated manuscript:\n\n  * Line 158: Fixing the summation expression by rewriting as $\\sum_{s}\\nu(s)\\sum_{\\bar{s},\\bar{a}}T(s|\\bar{s},\\bar{a})d(\\bar{s},\\bar{a})$.\n\n  * Line 159: Revising $\\\\sum_{i}\\\\mu_{i}k_{i}+u_{i}(k_{i})$ to $\\\\sum_{i}(\\\\mu_{i}k_{i}+u_{i}(k_{i}))$ to ensure correctness."}]

Author rebuttal:

Meta-review: This paper considers fairness-aware policy learning in off-line multi-objective reinforcement learning, and proposes a nice solution to this problem based on the DICE framework. Theoretical analysis and numerical validations are sound.  AC recommends the acceptance of the paper. Please include the additional results obtained during the rebuttal period in the final version.
Decision: Accept (poster)