A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Authors: Ying-Tu Chen
Wei Hung
Bing-Shu Wu
Zhang-Wei Hong
Ping-Chun Hsieh
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-objective reinforcement learning
reward-free reinforcement learning
Primary area: reinforcement learning

Abstract: Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.

Published as a conference paper at ICLR 2026
A REWARD-FREE VIEWPOINT ON MULTI-OBIECTIVE REINFORCEMENT LEARNING
Ying-Tu Chen !-f Wei Hung Bing-Shu Wu'-f Zhang-We Wei Hong Ping-Chun Hsieh
NNational Yang Ming Chiao Tung University, Hsinchu, Taiwan
Massachussett Institute of Technology
(shu0 924. cs13, pinghsiehhenncce edu.tw. tw, zwhongemit.edu
ABSTRACT
Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt different user preferences. In multi- objective reinforcement learning (MORL), one widely studied approach addresses this by training a single policy network conditioned on preference-weighted re- wards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has histor- ically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL'S challenge of han- dling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge shar- ing beyond the multi-objective reward function given training time. To this end, we adapt a state-of-the-ar RFRL algorithm to the MORL setting and intro- duce a ciserrnnce-uuide' exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior perfor- mance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as scalable and empirically effec- tive solution to oujei-obeecctvve policy learning."
INTRODUCTION
Many sequential decision-making tasks require optimizing multiple, often conficciigg objectives. For example, in robot control, there a trade-off between minimizing energy consumption and maximizing speed. One common approach to find Pareto optimal policy is to maximize weighted sum of the objectives, where the weights represent user preferences. User preferences depend on context- for instance, prioritizing speed in emergencies and energy efficiency in routine operations. Since user preferences are unknown in advance, solving multi-objective decision decision-making requires learning set of policies for different preferences before testing.
Reinforcement learning (RL) (Sutton & Barto, 2018) has achieved strong performance in sequential decision- making, making multi-objeccive RL (MORL) widely studied approach for learning poli- cies for different user preferences (Hayes et al., 2022). A naive but inefficient solution is to train separate policy for each preference. Another more scalable approach is to train single policy network (Yang et al., 2019; Basaklar et al., 2023; Hung et al., 2023) conditioned on preferences, enabling parameter sharing and generalization across preferences. During training, the policy is optimized over a range of sampled preferences, each defining a reward function weighted by the preference. test time, users can specify a preference to obtain the corresponding policy.
Another approach to handling unknown user preferences at test time is reward-free reinforcement learning (RFRL) (Jin et al., 2020; Touati et al., 2023), which has historically been developed inde- pendently of MORL despite addressing a similar problem. In RFRL, the agent explores the envi- ronment without receiving reward signals during training and instead learns set of optimal policies
Equal contribution. 'https:/ bandd t.s 1ab. web.githu.... wtw.uuu...///.....o... O/MORIIFFA/

---

Published as a conference paper at ICLR 2026
for any possible reward function in the environment. MORL can be seen special case of RFRL (Alegre et al., 2022), as RFRL does not restrict the reward function to be a weighted sum of pre- defined reward functions. However, despite their similarities, no prior work has explicitly adapted RFRL methods to solve MORI. problems.
In this paper, we ask: Can RFRL inform MORL? We hypothesize that the objective of RFRL to learn optimal policies for any reward function could serve as a useful auxiliary task for MORL (Jaderberg al., 2016; Rafiee al., 2022; Veeriah et al., 2019). Although MORL under linear scalarization only needs optimal policies for linear combinations of known objectives, learning beyond these combinations could accelerate MORL via effective knowledge sharing. To investigate this question, we adapt a state-of-the-ar RFRL algorithm (Touati et al., 2023) to the MORL setting, treating the preference-wette weighted reward as the test-time reward function given to RFRI.. However, this naive approach performs poorly compared to existing MORL methods, likely because purely reward- free exploration does not prioritize states that are important for optimizing the preference-weighte reward in the given MORL task. As result, the policies learned by RFRL for these reward functions could be suboptimal. To address this, we propose guiding exploration using sampled preferences and mini-batch sampling, directing the agent visit states that maximize the corresponding preference- weighted reward function. This ensures that learning is focused on policies most relevant to MORL.
We highlight the main technical novelty of this paper: (1) A new perspective of solving MORL: We identify the close connection between MORL. and RFRL, which have evolved independently despite tackling similar challenges of unknown user preferences at test time. This insight motivates new MORL algorithms by rethinking policy learning with multiple objectives from the perspective of RFRL. (2) Algorithmic enhancements for adapting RFRL to MORL: Even though RFRL and MORL are closely related, vanilla RFRL can perform poorly in the MORL setting (see Section exploration: We propose to use the preference vector to sample latent vectors aligned with the target rewards to facilitate exploration in the latent space; (ii) Training on latent vectors computed by mini-batch sampling from replay buffer auxiliary tasks: Our approach trains the policy network on latent vectors computed from mini-batch transitions sampled from the replay buffer. This design learns broader range of policies than required for MORL and can be beneficial by providing auxiliary tasks; (iii) Auxiliary Q loss: To better adapt RFRL to MORL, we further facilitate the learning of representations from the observed reward vectors (instead of pseudo rewards as in RFRL) via an auxiliary Q loss as an additional learning signal.
4). To address this, we introduce three key enhancements: (i) Preference-guiee
Our experimental results demonstrate that approach is both simple and effective. First, our method significantly outperforms the state-of-the-art MORL algorithms across various tasks in the MO-Gymnasium benchmark suite (Felten et al., 2023), including discrete and continuous control. Second, when trained with a limited number of preference samples, our method achieves substan- tially higher performance than other MORL approaches. This highlights that decoupling environ- ment knowledge from reward information enhances generalization, particularly in scenarios with limited preference samples. To the best of our knowledge, this is the first work adapt RFRL for MORL and present a practical algorithm that performs well across diverse deep RL tasks.
PRELIMINARIES
This section provides a brief review of MORL, along with the notation used throughout the paper. We use boldface symbols for vectors and matrices. For any n E N, we use [n]] as a shorthand for {1.-.., n). For set Z, we let A(Z) denote the set of all probability distributions over Z.
We formulate the MORL problem as an Multi-Objective Markov Decision Process (MOMDP) de- fined by the tuple (S.A, P. R, y u). where S and A are the state and action spaces, P : S x A - A(S) is the transition function, R. : S x A - IRd is a vector-valued reward function of d objectives, y [0, 1) is the discount factor, and μ is the initial state distribution. Let II denote the set of all stationary randomized policies. Let St. at, r, be the state, action, and reward received at time l. For policy म E II, define V Ex,,so~p E 7 R((s,, az)] as the expected total discounted return vector achieved by т. Let V denote the i-th entry of V For pair of policies π and x', we say that π Pareto-dominates n² (denoted by > x if V V for all E [d] and there exists some j E [d] such that V Vj
2

---

Published as a conference paper at ICLR 2026
The general goal of MORL is discover the Pareto front, which is defined as the set of non- dominated policies. That is, for each policy т in the Pareto Front, there exists no other policy x' E II such that n' > л. To search for the Pareto front, one common approach is to leverage scalarization utility function SA I + IR under a user preference vector À E A, where denotes the preference set. In this paper, as commonly adopted in the MORL literature (Abels et al., 2019; Yang et al., 2019; Basaklar et al., 2023; Hung et al., 2023; Lu et al., 2023). Without loss of generality, we presume that A the d- dimensional unit simplex. Notably, it has recently been shown by Lu et al. (2023) that any point on the Pareto front can be achieved by training a policy using linear scalarization due to the convexity of the policy-induced value function's range. Since the preference Xlest at test time is unknown during training, our goal to learn a preference-conditioee, policy т : :S X + A(A) that can maximize the total discounted scalarized reward E [E== 7'> re]. for any À A.
focus the linear scalarization setting where fx A'r,
3 REWARD-FREE RL FOR MULTI- RL
This section explains why MORL can be seen as a special case of RFRL. We then discuss how this perspective enhances MORL by improving generalization and sample efficiency.
MORL as special case of RFRL, The goal of RFRL is to compute an optimal policy for any scalar reward function R S x A + R provided test time, without observing any reward signal during training (iee, "reward-free") nor requiring additional environment interaction at test time. Formally, RFRL solves the following optimization problem test time: arg max, Er,,s--y so 1 where T3 is the reward realization that corresponds to the test-time reward function.
MORL under linear scalarization addresses a similar problem, but presumes the vector-valued ward signal from R(s, a) can be observed during training, and assigns R(s,a) = A R(ss, a). where À (a user-specified preference vector) defines a linear combination of multiple reward components in R. Both RFRL and MORL aim to retrieve an optimal policy for a given reward function at test time, but their approaches differ. While MORL typically focuses on finding the Pareto front by learning a set of optimal policies for various preferences X. RFRL learns policies for all possi- ble reward functions, potentially including optimal policies for scalarized MORL rewards. RFRL achieves this by training conditional policy network (Touati & Ollivier, 2021) or leveraging pre- collected dataset through planning or batch RL (Jin et al., 2020), providing a broader policy set than traditional MORL.
Key idea: RFRL as a source of auxiliary tasks. RFRL learns policies for a broader class of reward functions than required for MORL, but this can be beneficial by providing auxiliarr tasks. Prior research has shown that incorporating auxiliary tasks improves sample efficiency and generalization in RL (Jaderberg al., 2016; Veeriah et al., 2019; Rafiee et al., 2022). Since RFRL naturally trains policies across a spectrum of reward functions, it provides a structured way to design these auxiliary tasks. However, directly applying RFRL MORL can be data-inefficient since reward- free exploration may not prioritize states that are crucial for learning the Pareto front in MORL. The key challenge is: How to utilize the auxiliary tasks offered by RFRL effectively to accelerate the learning of optimal policies in MORL? In the sequel, we describe how RFRL can be adapted to effectively improve training in MORL.
3.1 FORWARD-BACKWARD MORL (MORL-FB)
In this section, we formally present MORL-FB by describing the implementation of RFRL and the key components that adapt RFRL to MORL by enhancing its learning efficiency.
RFRL Implementation. We implement RFRL for MORL using the state-of-the-a Forward- Backward (FB) RI. algorithm (Touati & Ollivier, 2021). The FB method learns a set of policies optimized for different reward functions by decomposing the Q-value of an optimal policy for a scalar reward function R into two neural networks: Fa (forward representation) and B.,, (backward representation), where 0 and w denote their parameters. This decomposition allows the Q-function for given reward function R to be expressed as:
Q(s, a, 2R) = Fo a, 2.) xR) 2R-
(1)
3

---

Published as a conference paper at ICLR 2026
where zR Rd is d: d--dimensional latent vector, and both Fø and B are neural networks producing d, d--dmenssionll outputs. Intuitively, *R is meant to encode the optimal policy that corre- sponds to the current reward function. Once reward function R is revealed, nn is defined as:
zn (( (B a) R(s,
where represents an arbitrary state-action distribution. In our implementation, we use as the distribution induced by the replay buffer collected by the agent during training. Using this formula- tion, the greedy policy for given reward function defined as:
π (5, ZR) arg max Fø(s, a, ZR) ZR.
(3)
Now we are ready to present the design of MORL-FB at both training time and test time.
(a) Desp Treesure
Reware Dimensson (Tima Panaloy)
Contour Deep Treasure
Algorithm MORL-FB
UTT
1: Input: dimension ds, sample num-
1.5
ber Ti
1.0
2: Initialize replay buffer M <- Ø
[b) HV on Deep Sea Treasure
000
-20
3: for each iteration i do
HVT
4: Sample preference À uniformly
from A
125
5:
* + PIG-EXPLORE(M
60538
75
18.0
12.5
32.0
17.
20.2
Batch Size
Reward Dimension (Treasure Value!
6:
Generate rollouts using
FS
256
7: Sample n, transitions D M
4096
8: Update FB networks Fo, B. and
Figure 1: A motivating experiment on Deep Sea Treasure. (a)(b) Training performance (UT and HV defined in the se-
policy
9: end for
quel) of MORL-FB under different batch sizes for xx. (c)
10: function PG-EXPLORE(A)
KDE contour of return vector distributions of π(. x) in-
11:
M Sample n, transitions
duced by 2Λ (with various batch sizes b) and (0, Td=).
12:
= =+ >+D(sanne Bu(saje
This shows that Σ corresponds to learning for more diverse
13:
Normalize 2 to Vd Tz]la
and relevant behavior in MORL than yy and the z sam- 14:
return z
pling strategy of the original FB, The detailed configuration is provided in Appendix C.
15:
end function
Test Time: At test time, can easily adapt Equation (2) by replacing R(s, a) with user-specified scalarized multi-objective reward based on a preference vector À as R(s,a) = > ARR(s, ((ss,a) a). Next, given the learned Fe and B.,, compute the corresponding zæ and use it in the policy defined by Equation (3). This effectively retrieves an optimal policy for the given preference X.
Training with Preference-Guided Exploration: During training, Fs, B and must be trained by sampling z and conditioning the networks on these sampled values. Since the test-time user preference Agess unknown at this stage, we cannot directly compute z using Equation (2). At high level, training on diverse set of samples is equivalent to training the agent on variety of reward functions, since z is inherently linked to rewards through Equation (2).
In principle, can be sampled from any distribution without restriction. In (Touati & Ollivier, 2021), is drawn from a standard normal distribution (0, Id.) in a dy-dimensional space. However, we found that this approach leads to poor sample efficiency when testing the agent on MORL tasks. We hypothesize that sampling z from a normal distribution produces representations that differ sig- nificantly from the actual æR obtained from preference-weipeee weighted multi-objective reward function (see Figure for a visualization empirical distributions).
To address this issue, we propose Preference- Guided Exploration (PG-Explore), which constructs a more relevant z distribution via sampling guided by preference-weighted rewards. The design of PG-Explore builds on the following insights:
Using {*A Jeea only leads to limited exploration of 2: Recall that in MORL, we observe multi-objective rewards R(s, a) (or its noisy version) during training. One direct approach is to compute z
4

---

Published as a conference paper at ICLR 2026
2 = a (0) a a)
(b) EE BB (s) ()
(4)
where (a) holds due to the fact that A ATR(s,a) a) is a scalar and can be swapped in the matrix multiplication with (s,s, a) and (b) follows from that À can be moved out of the expectation. Equation (4) suggests that z is in the span of d preference-agnosicc column vectors of the d2 x d matrix H, for any preference X. In practice, since the number objectives d is usually much smaller than dg. the coverage of AEA in Rd can be extremely small. This leads to limited exploration of z during training such that the agent can commit to a set of improper 2, especially in the early training stage when Fø and B are not well trained.
Constructing Σ by mini-batch sampling for exploration: To encourage exploration of z rel- evant to MORL, we propose conceptually simple and yet effective technique that leverages mini-batch sampling construct x Specifically, we sample batch of n, data samples (denoted by D) from the replay buffer and compute y = = (s,...,,') (s, a)r X/n.. Figure 1 shows a comparison training with 2x. å under various batch sizes, and drawn from (0, I as in the original FB method, in Deep Sea Treasure (DST), which is goal-oriented navigation task with two-dimensional rewards as (treasure value, step cost). As the true y is not available, we use & with large ny surrogate for Zx. We can that: (i) ž indeed corresponds to learning more diverse behavior than just learning for zx. (ii) y 's are more relevant to the reward functions in MORL encounteed at test time than the sampled from (0, Td+), improving sample efficiency.
Learning induced by 2 serves auxiliary tasks: Recall that in PG-Explore, construct a (s.o.r,a")) B_ (8, a)r' A/n, by mini-batch sampling. This means that for any given preference X, the agent can learn beyond z and, moreover, from multiple values z from different batches of transitions, providing richer learning signals. This approach is closely related to the auxiliary tasks in deep RI., where training objectives that are not directly or totally aligned with the target objective have been shown to accelerate learning (Jaderberg al., 2016; Veeriah al., 2019; Rafice et al., 2022).
Training Objective Functions: (i) Measure loss: To train the Fg and B networks in MORL-FB, the standard measure loss (Fs. Bui zx), which minimizes the Bellman residual on the successor measure (Touati et al., 2023). Fğ and Bo are target networks. This loss is defined as:
(Fa, Bwi xx)=
(e [Fo(se, ae. 2x) (us'''a') (s,a')
FFF (8 T (S xx). 2x) Bs )²)
2B 2B Fo(se, ar 2x) B (S1+1,
(5)
(ii) Auxiliary Q loss: In the context of MORL, we propose to employ auxiliary Q loss to facilitate the learning of FB representations from the observed reward vectors, instead of the pseudo rewards in the original FB (also see the ablation study in Section 4.1). Specifically, the Q-loss is constructed as the squared temporal difference error represented in Fg and B and the transitions are sampled from the replay buffer to compute yY via our preference-gui guidee function:
(F 2>) = (.s..ar,)"'' [(Fo T z> (AT (ATr+F (1), =x)T =x))
(6)
We summarize the implementation Algorithm 1. The detailed pscudo code (Algorithm 2) and further details about loss functions and implementation provided in Section B. Note that the FB framework can use either state sete-dependent or vute-actionndeeendde both perform well in practice (see Appendix D.4). As the original FB (Touati al., 2023) presumes a state-dependent design, we focus mainly on state-dependent ones in the subsequent experiments.
backward representation, and
4 EXPERIMENTS
Evaluation Domains. We leverage the MO-Gymnasium benchmark suite (Felten et al., 2023) and consider various discrete and continuous control tasks. For the continuous control tasks, we consider
5

---

Published as a conference paper at ICLR 2026
robot locomotion tasks in Multi-objective MuJoCo with up 5 objectives, including Walker2d, Halfcheetah2d, Ant3d, Hopper3d. Humanoid2d, and Humanoid5d. Each environment presents a unique set of objectives, e.g.. the goal of Ant3d is to optimize both x-axis and y-axis speeds while minimizing energy consumption. The experiments of the discrete control tasks are provided in Appendix D.3. The detailed configurations for all tasks provided in Appendix C.
Benchmark Methods. To evaluate the effectiveness of our proposed approach, we compare MORL- FB against various benchmark methods, including: (i) Single snnferencceconddtioonedd policy meth- ods: PD-MORL (Basaklar et al., 2023), Q-Pensieve (Hung et al.. 2023), CAPQL (Lu et al., 2023), Envelope Q-Learning (EQL) (Yang et al., 2019), and PCN (Reymond et al., 2022); (ii) Multi-policy MORL: PG-MORL (Xu et al., 2020). SFOLS (Alegre et al., 2022), MORL/D (Felten et al., 2024), GPI-LS, and GPI-PD (Alegre al., 2023): (iii) Rewand-free RI:: We take the original FB approach (Touati et al., 2023) as a baseline.
Regarding PD-MORL, Q-Pensieve, and FB, we leverage their official implementations from (Basak- lar et al., 2023; Hung et al., 2023; Touati et al., 2023). To ensure fair comparison among all the benchmark methods, we adopt the standard PD-MORL without the auxiliary pre-trained preference interpolator, which essentially requires substantial amount of additional data for pre-training and could bias the comparison. Regarding CAPQL, EQL. PCN, PG-MORL, MORL/D, GPI-LS, and GPI-PD, we leverage the implementation of MORL-Baselines (Felten et al., 2023) for better repro- ducibilityy As the PG-MORL. MORL-Baselines can only support two-objective tasks, we extend this code base accommodate those tasks beyond two objectives. On the other hand, the orig- inal SFOLS only focuses on discrete control tasks by default. For a more thorough comparison, utilize its official implementation for discrete problems and extend SFOLS with TD3 back- bone for evaluation on continuous control. Moreover, we apply hyperparameter optimization to the benchmark algorithms and MORL-FB. Appendix C.2 details the selection range of hyperparameters and the final selected values. For all the tasks, we run each algorithm for 3 million environment most of the existing MORL studies. Below we report the average
steps, which is comparable performance and the empirical standard deviation over 5 random seeds for each task. More detailed configurations of the experiments and benchmark methods are provided in Appendix C.
Halfcheetah2d
Walker2d
Hopper3d
Ant3d
Humanoid2d
Humanoid5d
8930
2000
2000
2008
UT1
1000
(000
3000
-250
HVT
0.75
0.30
0.50
0.25
0.20
0.5
0.30
0.00
0.0
ED
0.2
60
ap
AO-MORL
CAPOL
MORUD
SPOLS GAILS
GPARD
MORL-B (Ours)
Q.Perdieve
PGMORL
PCN
Figure 2: Evaluation of MORL-FB and several MORL benchmark algorithms on diverse continuous control tasks within the MO-Gymnasium suite, assessing performance using key metrics. These results demonstrate the clear advantage of MORL-FB across all tested benchmarks.
Evaluation Metrics. We evaluate the performance of each algorithm using three metrics that are widely used in the MORL literature (Van Moffaert & Nowé, 2014; Yang al., 2019; Kyriakis & Deshmukh, 2022; Basaklar et al., 2023; Hung et al., 2023; Lu et al., 2023):
Utility (UT): To evaluate the scalarized total reward across different preferences at aggregate level, we employ the utility metric defined as Ex|Σ, X & rr.) where the expectation is taken with respect the uniform distribution over the preference set A (iee, d-dimensional unit simplex).

---

Published as a conference paper at ICLR 2026
Hypervolume (HV): As a standard metric in the literature of general multi-objective optimization, hypervolume naturally captures the inherent trade-off among different objective functions using one aggregate scalar value (Zitzler & Thicle. 1999). Specifically, given reference point U R and any collection for return vectors U S RR. the hypervolume of M can be formally defined as ž
where pa()) denotes the d-dimensional Lebesgue HV(:; uher) := ( y 4er))
wEl
measure. In practice, Ure is selected based on the range of possible total return and is task- dependent. The configuration of Uhe for each MORL task is provided in Appendix C.
Episodic Dominance (ED): As a metric complementary to HV and UT, ED meant to capture the relative strength of pair of algorithms under different preferences. Specifically, given any two algorithms ALGI, ALG2, we define ED(AG), ALG2) == [I{A ((TaL,)) 2 A g Taray HH. where 9(-) denotes the trajectory-wiss cumulative return vector, Tang, and TAtG; are the trajecto- ries generated under the policies of ALG, and 2 and À is drawn uniformly from A. Note that we use 500 uniformly sampled preference vectors and evaluate across 5 distinct random seeds for each preference vector for statistical robustness.
To ensure rigorous evaluation, we further follow the guidelines of (Agarwal et al., 2021) by tak- ing the normalized UT scores and reporting the aggregated performance across tasks in median, mean, and interquartile mean (IQM). Regarding the normalized scores, we follow the procedure in (Fu et al., 2020), which (i) employs a random policy- where actions are selected uniformly at random- as the baseline with normalized score of 0 and (ii) expert policy trained by single- objective SAC-as as the topline with normalized score of 100. The above normalization is done on a per-preference basis.
Does the reward-free viewpoint of MORL-FB improve sample efficiency over the benchmark methods? Figure shows the performance of all the methods in UT. HV, and ED for continuous control tasks. Regarding ED, for each baseline algorithm ALG, we report ED [LG,, MORL-FB) to show the pairwise comparison. We can make the following observations: (i) MORL-FB achieves either the best or close to the best UT and HV among all methods on all the tasks, regardless of the number of objectives. This showcases that MORL-FB is indeed sample-efficient in the sense that it can discover a diverse collection of tiih--erffrming policies across various preferences using only as few as million samples as used by the expert policy. (ii) Given that ED(ALG, MORL-FB) are consistently smaller than 0.5 for all baselines, we see that MORL-FB outperforms all bench- mark methods (including the state-of-the-ar methods like PD-MORL and Q-Pensieve), under most preferences. (iii) PD-MORL and Q-Pensieve perform well on two-objective tasks (e.g., Halfchee- tah2d and Walker2d) but underperform when the number of objectives is larger (e.g., Ant3d and Humanoid5d). Additional results discrete control environments can be found in Section D.3.
Moreover, regarding the aggregated results, Figure shows that MORL-FB reliably outperforms the benchmark methods both in conventional statistics (e.g., mean and median) and robust metrics like IQM. MORL-FB achieves the best IQM scores by a large margin vis-â-vís other methods, confirming the significant improvements over the state-of ofte-offthee-rt MORL.
Recall that MORL-FB leverages PG-Explore to address the fundamental exploration issue of
Modian
IQM
Moan
NORLLY (Ougg
vanilla FB, which suffers from sample ineffi- ciency in MORL. Remarkably, the per-task re-
SFIPD PCN
sults in Figure 2 and aggregated results in Fig- ure 3 show that MORL-FIB enjoys significantly better UT and HV across tasks. Accord- ingly, the ED scores ED(FB, MORL-FB) re- main nearly zero in all tasks.
POMORI
-55
0.0
0.5
1.0
0.25
0.50
0.75
-0.5
0.0
05
1.
Normalized Weighted Reward
Figure 3: Evaluation of MORL-FB and several MORL benchmark algorithms using aggregate metrics, including median, mean, and interquar- tile mean (IQM). These results show the superior performance of MORL-FB across all metrics.
Does MORL-FB achieve effective general- ization across preferences? To better assess the generalization capabilities of MORL-FB across preferences, we further evaluate the al- gorithms in a stylized setting where they are trained only on a small set of preference vectors Aquin (rather than the whole A) and aim for gen- cralization over A at test time. Specifically, for a d-objective task, we let Again include only the standard basis preferences, ie., d-dimensional one-hot vectors, and the uniform preference vector
7

---

Published as a conference paper at ICLR 2026
Hopper3d
Ant3d
Huideo
2000
1000
2 Distribution During Training on Humanold2e
UT1
2002
MURL-FB
Original
1000
10
Feature
HVT
1:0
0.5
0.0
ED1
40
20
10 0 20
30
PD MORL
O-PenGivee
MORL-FB (Ours)
Figure 5: Empirical x distribution under t-SNE for Humanoid2d with MORL-FB (preference- guided sampling. blue) and original FB (stan- dard normal, red): The multi-modal distribution observed with MORL-FB suggests a more di- verse set of latent representations compared unimodal nature of original FB.
Figure 4: Evaluation of MORL-FB and bench- mark methods (PD-MORL and Q-Pensieve) un- der a reduced preference set Amin during train- ing: These demonstrate the generalization capa- bility of MORL-FB across preferences.
[1/d,... ... 1//d) The testing setup is exactly the same as that for Figure 2. Here we focus on compar- ing MORL-FB PD-MORL and Q-Pensieve, which the top two benchmark methods in Figure and utilize conditioned networks of structures different from MORL-FB.
As shown in Figure 4, both PD-MORL and Q-Pensieve exhibit notable decline in performance across all three metrics compared to those in Figure 2. In contrast, MORL-FB maintains consistent performance across the evaluated tasks, with only minimal degradation in UT and HV values com- pared to Figure 2. These findings showcase that MORL-FB can generalize more effectively over the entire preference set, even when trained on limited of preference vectors. More detailed results, such as the numerical values and the aggregated performance (e.g., IQM) are in Appendix D.
Ant3d
Ant3d
to
4000
10
UT 3000
UTf
2000
HV1
10
MORL-F15 (w/o PG-Explore! MORLFF (wle measure loss)
MORL-FU (wil Q loes)
MORLFFB (pseuda Q loss)
FB
F65
MORL-FB (Ours)
Figure 6: Evaluation of MORL-FB and its ab- lated versions the Ant3d task. The results highlight the importance of PG-Explore and auxiliary losses, removing these components leads to performance degradation.
Figure Zero-shot cross-objective transfer from Hopper2d to Hopperšd and Hopper4d us- ing vanilla FB and MORL-FB: Results demon- strate effective transfer by MORL-FB, support- ing the efficacy of its proposed enhancements.
4.1 ABLATION STUDY
Exploration (PG-Explore). To investigate the benefits of sampling = from reffrrecce-uuiee distribution, we perform an ablation study on comparing the proposed MORL- FB and variant of MORL-FB that samples from (0, TT,,, iee, the distribution adopted by the vanilla FB. Figure (specifically the bars gray and orange) shows that MORL-FB indeed benefits significantly from preference-guiee guidedd distribution across the tested task, highlighting its importance in enabling directed exploration and samplc-efficient policy learning.
Preference-Guider
Moreover, visualize the empirical distributions of the sampled of MORL-FB and the original FB. Specifically, we record the z vectors used throughout training and apply t-SNE (Van der Maaten & Hinton, 2008) for visualization in two-dimensional space. The results Humanoid2d in Fig- ure show that sampling x from a normal distribution results in a unimodal empirical distribution (contours in red). By contrast, MORL-FB with the preference-guided sampling exhibits multi-
8

---

Published as a conference paper at ICLR 2026
modal distribution (contours in blue), indicating richer and more diverse set of distributions. This multi modality allows MORL-FB to better capture the underlying reward structure, achieving improved generalization and adaptation to various objectives. More visualization of distributions for other tasks can be found in Appendix D.
Auxiliary Q loss. To corroborate the efficacy of the auxiliary Q loss, we further conduct abla- tion study on this term. From Figure 6 (specifically the bars in black and orange), the Q loss can facilitate the learning of forward and backward representations MORL-FB and thereby boost the performance in both UT and HV. More ablation results across environments are in Appendix D.2.
4.2 ZERO-SHOP -BOSS-OBEECTIVE TRANSFER
Recall that one salient feature of MORL-FB is use to encode the X-dependent scalarized reward function. Accordingly, MORL-FB is endowed with the ability to achieve zero-shot transfer even across tasks of different number of objectives. Such zero-shot cross-objective transferability allows us to add new factors to the reward function without the need for retraining and hence is a very useful feature in practice. To validate this, we use MORL-FB learn the F and B networks on Hopper2d and directly evaluate them on Hopper3d and Hopper4d, which involve additional reward terms like "jump height" and "z-axis speed", at test time. Details of the environment configurations are in Appendox C. We conducted the same evaluation for vanilla FB (Touati et al., 2023) as a base- line. From Figure 7, vanilla FB cannot achieve effective cross-objective transfer given that vanilla FB already suffers in the standard MORL. setting (cf. Figure 2). By contrast, MORL-FB achieves effective transfer across objectives in a zero-shot manner, corroborating the proposed enhancements.
5 RELATED WORK
5.1 MULTI-OBIECTIVE RL
Single preference-conditioneed policy methods. Single preference-conndiiooee policy methods learn one policy network that adapts to different objective trade-offs by conditioning on prefer- ence. Many of these methods employ scalarization techniques (Van Moffaert et al., 2013; Yang et al., 2019), transforming multi-objective problems into weighted single-objective problems. They allow policies dynamically adjust at inference time. However, relying solely on linear reward ag- gregation without proper representation learning can lead to suboptimal solutions. To address this, CAPQL (Lu et al., 2023) introduced the concave reward terms for better optimization landscapes, while CN-DER (Abels et al., 2019) proposed a cerfffrrnnc-conditioonnnn ence replay mechanism to handle dynamic weights and mitigate non-stationarity. Q-Pensieve (Hung et al., 2023) improved sample efficiency by reusing past policy snapshots. Without using scalar- ization, Abdolmaleki et al. (2020) proposed to learn action distributions per objective and fitted parametric policy via supervised learning. To improve adaptability to diverse and unseen preference vectors, methods like PCN (Reymond et al., 2022) formulated MORL as a classification problem, and MOAC (Zhou et al., 2024) finds pareto-stationary points by adapting multi gradient descent to MORL without scalarization. PD-MORL (Basaklar et al., 2023) also trains a single preference- conditioned network but directly incorporates preference vectors, for example, through cosine sim- ilarity measures within its value-function update rule to efficiently learn a comprehensive set of policies across the continuous preference space.
Q-network with an experi-
Multi-policy methods. Multi-policy methods explicitly learn multiple policies to cover the Pareto front, capturing diverse trade-off's in the training process. A key challenge in multi-policy MORL is efficiently constructing a coverage set that represents the full Pareto front while maintaining scal- ability. To refine policy selection and handle dominated actions, Lizotte al. (2012) introduced a structured approach using linear value function approximation. Subsequent methods (Kyriakis & Deshmukh, 2022; Van Moffaert & Nowé, 2014; Xu al., 2020) focused on improving explo- ration efficiency across the preference space but lacked structured learning mechanisms to gener- alize across diverse preferences. Building on the idea of incorporating structure into policy learn- ing, Felten et al. (2024); Mossalam et al. (2016) extended structured learning for multi-objective RL, employing decomposition and sequential single-objective optimization to enhance efficiency. However, scalability and adaptability remained challenges. DG-MORL (Lu et al., 2024) leveraged demonstrations and a self-evolving mechanism to improve scalability. As for improving adaptabil- ity, Mossalam et al. (2016) extended Optimistic Linear Support (OLS) to deep RI., constructing

---

Published as a conference paper at ICLR 2026
convex coverage set through sequence of single-objective, providing a structured way represent diverse trade-offs, but lacked effective transferability.
5.2 SUCCESSOR FEATURES FOR TRANSFER ACROSS REWARD FUNCTIONS
One related transfer setting in RL is to learn policies for all reward functions that are linear combi- nations of a finite set of known features. Barreto et al. (2017) proposed the successor feature (SF), which reflects the state-action occupancy of policy and can be viewed as extension of the classic successor representation (Dayan, 1993). By design, SFs achieves transfer across reward functions in two ways: (i) For fixed policy π, SFs can enable fast policy evaluation of ι across different reward functions; (ii) Given a set of policies, SFs can be combined with the generalized policy im- provement (GPI) update to generate a new policy that is no worse than the given set of policies and has a sub-optimality gap characterized by the task differences, for any reward function (Barreto al.. 2017; 2018). Accordingly, SFs has been combined with deep neural networks and applied to various tasks like subgoal extraction (Kulkarni et al., 2016) and robot navigation (Zhang et al., 2017). Subsequently, Borsa et al. (2019) extended the SFs by decoupling the policy and task de- scription for better representational flexibility. Moreover, Alegre et al. (2022) applied SFs and GPI to solve MORL by enabling transfer across different scalarized reward functions. Chua et al. (2024) proposed practical method for learning the SI's directly from pixels for image-based control. More recently, Zhang et al. (2024) established the convergence analysis with generalization guarantees for SFs under neural function approximation.
Despite the transfer capability, the limitations of SFs and its variants are mainly two-fold: (i) SFs require a set of pre-defined reward features, which could miss important aspects of the environment and limit adaptability. (ii) While SFs can achieve fast policy evaluation across reward functions, SFs cannot directly induce optimal policy when given arbitrary new reward function at test time.
5.3 REWARD- FREE RL
Among the studies in the RFRL literature, (Touati & Ollivier, 2021: Touati et al., 2023) are the most relevant to our work. To address the aforementioned issues of SFs, Touati & Ollivier (2021) adopted the reward-free MDP formulation and proposed a low-rank model termed the forward- backward (FB) representations, which capture the state-action occupancy of the optimal policies for all the reward functions by learning the required features directly from data. The FB framework has been implemented and validated for both reward-fir discrete control (Touati & Ollivier, 2021) and continuous control (Touati et al., 2023). Subsequently, the framework of FB representations has been extended to various settings, such as offline RL. with low-quality data (Jeen et al., 2024), online unsupervised RL (Sun et al., 2025), imitation learning (Pirotta et al., 2024; Tirinzoni et al., 2025), and partially-observable MDPs (Jeen et al., 2025).
Another line of RERL research focuses on achieving provably efficient exploration without using any reward information, in both tabular settings (Jin al., 2020; Kaufmann et al., 2021; Ménard et al., 2021; Wu et al., 2022) and under function approximation (Qiu et al., 2021; Wagenmaker et al., 2022: Wang et al., 2020; Zanette et al., 2020: Zhang et al., 2021).
Inspired by the RFRL literature, we propose to rethink MORL via RFRL and adapt the FB method to boost the sample efficiency and generalization in MORL.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
We propose MORL-FB, which offers a new perspective by rethinking MORL through the lens of RFRL. By using RFRL as auxiliary tasks. MORL-FB provides the first systematic adaptation of RFRL to MORL and enhances its sample efficiency with critical algorithmic enhancements, includ- ing preference-guidee
exploration with mini-batch sampling and an auxiliary Q loss based on the observed reward vectors. MORL-FB achieves strong performance in a variety of discrete and contin- uous control tasks, offering superior efficiency, better generalization, and zero-shot cross-objective transfer. A key limitation, inherited from FB-based RFRL, is the need for more advanced explo- ration, especially in complex or sewrse-reward environments, where dedicated strategies are crucial. Future work includes exploring RFRL methods beyond FB, such representation learning of suc- cessor measures (Agarwal et al., 2025; Farebrother et al., 2023) and learning ristance-preserviin state representations (Park al., 2024), to further reveal RFRL's advantages in MORL.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research is partially supported by the National Science and Technology Council (NSTC) of Taiwan under Grant Numbers 1 114-628-E-AA49000 and 14-26644--AA99-022MMBB We also thank the National Center for High-performance Computing (NCHC) for providing computational and storage resources.
ETHICS STATEMENT
Our work develops and evaluates reinforcement learning methods purely in simulated environments, without involving human subjects or sensitive data. This submission follows the code of ethics.
REPRODUCIBILITY STATEMENT
We release our code in the supplementary material and describe the commands needed to execute the code in a Readme file attached in the supplementary material. Additionally, we attach the list of package dependencies which can be used to build the environment.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large language models (LLMs) were employed exclusively for language editing and polishing of the manuscript. They were not used for designing methods, conducting experiments, analyzing results.
BIBLIOGRAPHY
Abbas Abdolmaleki, Sandy H. Huang, Leonard Hasenclever, Michael Neunert, H. Francis Song, Martina Zambelli, Murilo F. Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A dis- tributional view multi-objective policy optimization. International Conference on Machine Learning. 2020.
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International Conference on Machine Learning, 2019.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Advances in Neural Infor- mation Processing Systems, 2021.
Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor measure: Repre- senting the behavior space of RL agent. In International Conference on Machine Learning, pp. 566-586, 2025.
Lucas Nunes Alegre, Ana Bazzan, and Bruno C Da Silva. Optimistic linear support and successor features basis for optimal policy transfer. In International Conference Machine Learning, 2022.
Lucas Nunes Alegre, Ana LC Bazzan, Diederik M Roijers, Ann Nowé, and Bruno da Silva. Sample-efficient multi-objective learning via generalized policy improvement prioritization. In International Conference on Autonomous Agents and Multiagent Systems, 2023.
André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501-510, 2018.
11

---

Published as a conference paper at ICLR 2026
Toygun Basaklar, Suat Gumussoy, and Umit Ogras. PD-MORL: eiferrnceedrrvvrd multi- Objective reinforcement learning algorithm. In International Conference on Learning Representations, 2023.
Diana Borsa, Andre Barreto, John Quan, Daniel Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In International Conference on Learning Representations, 2019.
Raymond Chua, Amn Ghosh, Christos Kaplanis, Blake A. Richards, and Doina Precup. Learning successor features the simple way. In Advances in Neural Information Processing Systems, 2024.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural computation, 1993.
Theresa Fimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how tune them. In International Conference on Machine Learning, 2023.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G. Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. International Conference on Learning Representations, 2023.
Florian Felten, Lucas Nunes Alegre, Ann Nowe, Ana Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C da Silva. A toolkit for reliable benchmarking and research in multi-objective rein- forcement learning. In Advances in Neural Information Processing Systems, 2023.
Florian Felten, El-Ghazali Talbi, and Grégoire Danoy. Multi-objective reinforcement learning based on decomposition: A taxonomy and framework. Journal of Artificial Intelligence Research, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto. Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018
Conor F. Hayes, Roxana Rádulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. A practical guide multi-objective reinforce- ment learning and planning. In International Conference on Autonomous Agents and Multiagent Systems, 2022.
Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, haiyin piao, Yi Chang, and Lichao Sun. Dis- tributional reward estimation for effective multi-agent deep reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Wei Hung, Bo-Kai Huang, Ping-Chun Hsieh, and Xi Liu. Q-Pensieve: Boosting sample efficiency of multi-objective RI. through memory sharing of Q-snapshots. In International Conference on Learning Representations, 2023.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: /. 1.05977, 2016.
Scott Jeen, Tom Bewley, and Jonathan Cullen. Zero-shot reinforcement learning from low quality data. Advances in Neural Information Processing Systems, 111116894-16444 2024.
Scott Jeen, Tom Bewley, and Jonathan Cullen. Zero-shot reinforcement learning under partial ob- servability. In Reinforcement Learning Conference, 2025.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward exploration for reinforcement learning. In International Conference on Machine Learning, 2020.
Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent,
and Michal Valko. Adaptive reward-free exploration. In Algorithmic Learning Theory, 2021.
12

---

representation. In Published as a conference paper at ICLR 2026
Tejas D Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.0223: 2016.
Panagiotis Kyriakis and Jyotirmoy Deshmukh. Pareto policy adaptation. In International Confer- ence on Learning Representations, 2022.
Xi Lin, Xiaoyuan Zhang. Zhiyuan Yang. Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalarization for multi-objective optimization. In International Conference on Ma- chine Learning, 2024.
Daniel Lizotte, Michael Bowling, and Susan A Murphy. Linear fitted-Q iteration with multiple reward functions. Journal of Machine Learning Research, 2012.
Haoye Lu, Daniel Herman, and Yaoliang Yu. Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality. International Conference on Learning Representations, 2023.
Junlin Lu, Patrick Mannion, and Karl Mason. Demonstration guided multi-objective reinforcement learning. arXiv preprint MxXiv:2404003997 2024.
Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In Inter- national Conference on Machine Learning, 2021.
Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint arXiv:1610.02771 2016.
Scohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with Hilbert representa- tions. In International Conference on Machine Learning, pp. 39737-39761, 2024.
Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imita- tion via behavior foundation models. In International Conference on Learning Representations, 2024.
Shuang Qiu, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. On reward-free RL with kernel and neu- ral function approximations: Single-agent MDP and Markov game. In International Conference on Machine Learning, 2021.
Shuang Qiu, Dake Zhang, Rui Yang. Boxiang Lyu, and Tong Zhang. Traversing Pareto optimal poli- cies: Provably efficient multi-objective reinforcement learning. arXiv preprint arX1v:240..17466 2024.
Banafsheh Rafiee, Jun Jin, Jun Luo, and Adam White. What makes useful auxiliary tasks in rein- forcement learning: investigating the effect of the target policy. arXiv preprint arXiv:2204.00 00565, 2022.
Mathieu Reymond, Eugenio Bargiacchi, and Ann Nowé. Pareto conditioned networks. Interna- lional Conference on Autonomous Agents and Multiagent Systems, 2022.
Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, and Joelle Pineau, Re- ward estimation for variance reduction in deep reinforcement learning. In Conference on Robot Learning, 2018.
Jingbo Sun, Songjun Tu, Haoran Li, Xin Liu, Yaran Chen, Ke Chen, Dongbin Zhao, et al. Un- supervised zero-shot reinforcement learning via dual-value International Conference Learning Representations, 2025.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Bradford Book, 2018
Aviy Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling, In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
13

---

Published as a conference paper at ICLR 2026
Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behav- ioral foundation models. In International Conference Learning Representations, 2025.
Ahmed Touati and Yann Ollivier. Learning one representation optimize all rewards. In Advances in Neural Information Processing Systems, 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations, 2023.
Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning, 84 (1):51-80, 2011.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 2008
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of Pareto dominating policies. Journal of Machine Learning Research, 2014.
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforce- ment learning: Novel design techniques. In Adaptive Dynamic Programming and Reinforcement Learning, 2013.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. Advances in Neural Information Processing Systems, 32, 2019.
Andrew J Wagenmaker, Yifang Chen, Max Simehowitz, Simon Du, and Kevin Jamieson. Reward- free RL no harder than reward-aware RL in linear markov decision processes. In International Conference on Machine Learning, 2022.
Ruosong Wang. Simon S Du, Lin Yang, and Russ R Salakhutdinov. On reward- free reinforcement learning with linear function approximation. Advances in Neural Information Processing Sys- tems, 2020.
Jingfeng Wu, Vladimir Braverman, and Lin Yang. Gap-dependent unsupervised exploration for reinforcement learning. In International Conference on Artificial Intelligence and Statistics, 2022.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In In- ternational Conference on Machine Learning, 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan, A generalizzed algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, 2019.
Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Provably effi- cient reward-agnostic navigation with linear value iteration. In Advances in Neural Information Processing Systems, 2020.
Jingwei Zhang, Jost Tobias Springenberg. Joschka Boedecker, and Wolfram Burgard. Deep re- inforcement learning with successor features for navigation across similar environments. In IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371-2378, 2017.
Shuai Zhang, Heshan Devaka Fernando, Miao Liu, Keerthiram Murugesan, Songtao Lu, Pin-Yu Chen, Tianyi Chen, and Meng Wang. SF-DQN: Provable knowledge transfer using successor feature for deep reinforcement learning. International Conference on Machine Learning, pp- 58897-58934, 2024.
Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-frre free model-based reinforcement leamn ing with linear function approximation. Advances in Neural Information Processing Systems, 2021.
14

---

Published as a conference paper at ICLR 2026
Tianchen Zhou, FNU Hairi, Haibo Yang, Jia Liu, Tian Tong, Fan Yang, Michinari Momma, and Yan Gao. Finite-time convergence and sample complexity of actor-critic oulti-objective reinforcement learning. In International Conference Machine Learning, 2024.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Compulation, 1999.
15

---

Published as a conference paper at ICLR 2026
APPENDICES
A Additional Background: Successor Measure and Forward-Backwant Representations
16
A.I Successor Measure
16
A.2 Forward-Backward (FB) Representations
17
B Detailed Pseudo Code of MORL-FB
18
Detailed Configurations of Experiments
19
C.1 Evaluation Environments
19
C.2 Experimental Sctup
20
C.3 Compute Resources
24
D Additional Experimental Results
25
D.1 Visualization of x Distribution in Different Environments
25
D.2 Ablation Study
26
D.3 Performance Comparison of MORL-FB Under Different Q-loss Coefficients
29
D.4 Performance Comparison of MORL-FB Under State-Action-Baut Rewards and State-Based Rewards
30
D.5 Performance Comparison of MORL-FB Under Stochastic Rewards
30
D.6 Performance Comparison of MORL-FB With Nonlinear Scalarization
D.7 Sample Efficiency of MORL-FB
31
D.8 Cross-Objective Transfer Capability MORL-FB
D.9 RFRL source of auxiliary tasks
31
D.10 Comparison of Pareto Fronts
32
D.11 Robustness under Worst-Case Preferences
34
D.12 Computational Cost Analysi........
34
D.13 Detailed Experimental Results of Section
34
A ADDITIONAL BACKGROUND: SUCCESSOR MEASURE AND FORWARD-BACKWARD REPRESENTATIONS
A. SUCCESSOR MEASURE
Recall that in standard single-objective RL, the Q-function under a policy æ with respect reward function R: S x A + IR is defined as
X Σ
(= E
R(R( (:) 80 = =,
which captures the long-term expected discounted reward under the policy म. One way to interpret the Q function is through the lens of successor measure M S A + ASS x A), which reflects the discounted, expected future occupancy of the state-action pairs X when starting from (s, a) and following policy Formally, the successor measure A is defined as follows: For any subset
16

---

Published as a conference paper at ICLR 2026
X S x A, define
20
(s, x) = E
(st E
x3 80 8, ao = a
where } is the indicator function. The key property is that this measure M is agnostic to the reward function R as only depends on the environment dynamics and the policy я. Note that viewing MM
deal with both the discrete and continuous cases. For ease of exposi- tion, the sequel, we focus on the case of discrete state and action spaces. With that said, we also let AM (s, a, s', a') denote the successor measure of the state-action pair (s', a') when starting from (s, a) and following policy т. We can connect the Q function in Equation (7) with the successor measure in Equation (8) follows:
measure
Q"
E (" R(s'
(s',a))ESXA
Crucially, we can make two observations: (i) If we can directly learn the successor measure of policy v, then we can derive the corresponding function in zero-shot manner given any reward function. (ii) For any reward function Rt if we can learn the successor measure of optimal policy for R (denoted by RR). then we can leamn the optimal Q function Q* and thereafter derive an optimal policy for R. Therefore, we can directly encode the information about = the successor measure, then we can learn the optimal Q function for any possible reward function.
A.2 FORWARD-BACKWARI (FB) REPRESENTATIONS
Our MORL-FB method is built the Forward- Backwarrd (FB) representation approach (Touati & Ollivier, 2021; Touati et al., 2023), which provides a compact, low-rank factorization of the successor measure. The FB framework decomposes the successor measure into the product of two vector-valued functions, namely a Forward function F S x A x Rd + Rds and Backward function : A + Rd., Let RT be a policy that depends on some vector Rdd Specifically, under the FB framework, the successor measure M is represented by:
MM a') = F(s, a, B(s'.a"),
(10)
where acts some latent vector that captures the required information about the policy π.. Using this factorization, the Q-function for any policy T. can be expressed as
(**(s,0)
Σ
F(s,a, 2) B(s' ( B(d.dd))(ss'
(11)
= 2s(
B(s B(s',a'R(sssd)) (AAA))SSA
Σ
(12)
Note that Equation (12) holds for any (and hence any policy Rz). Therefore, have the freedom to set R to be the latent vector that results in an optimal policy for the reward function Rt i.e., Tzg = The To achieve this, shall simply enforce the following Bellman optimality equation (Touati & Ollivier, 2021), iee.
(s)) arg F(s, a,) TRR- a
(13)
The final optimal policy for the given preference is then directly constructed by maximizing the ap- proximated Q-function: Therefore, F and are well learned, then one can directly retrieve an op- timal policy for any reward function R given at test time by computing zg and apply Equation (13). This factorization provides the foundation of our MORL-FB algorithm to efficiently explore and generalize across preferences during training.
17

---

Published as a conference paper at ICLR 2026
Algorithm 2 MORL-FB
Input: Network parameters 0, 0 8, @2, 7), f, preference sampling distribution Px, preference set
A, actor learning rates Mm, FB presentation learning rate pra. dimension dz. sample number ns. update frequency Nuy warm up steps th. and target smoothing coefficient
2: Initialize networks Fe, w "y and target networks Fg, Ba: "n.
3: Initialize replay buffer M 4- Ø
4: for each iteration do
5: Sample a preference vector ~ Px
6: if S. Tow then
Warm-up stage
7: Sample 2 from multivariate normal distribution / (0,, Id.)
8:
Normalize & such that # - vdg ||
9: else
= 1-PG-Explloe(A))
11:
end if
for each environment step do
13:
Ge T, (-18;;;
81+1 ((-1s,,.)))
15:
M + MU Mss, ar ru. d, 8c-1)}
16:
end for
17:
for each gradient step do
18:
Sample a batch transitions {(s, a, X, 8'}} from the replay buffer M
19:
2j PG-Explore(A
20:
- μFB V Lo (0: A) + w;; A))
21:
pm (n (w: X) L0((0,;; (0)
22:
end for
i % nu 0 then
24:
- ! - Mr V v, (1; A)
25:
+ =0 + (1 +)
Tw (1 -
27:
- 7n + (1 T)
end if
29:
end for function PG-EXPLORE(2)
31: Sample batch D of non-terminal transitions a, 8 from M
(..... (s,a)))r
32:
Normalize z such that z dd x
return
35:
end function
B DETAILED PSEUDO CODE OF MORL-FB
Algorithm details the proposed MORL-FB method. Initially, during the warm-up phase (lines 6-8), the latent vector z is sampled from standard multivariate normal distribution. After the warm-up. determined using the preference-guided sampling scheme (line 10). This z is then used to generate trajectories within the environment, which are stored in the replay buffer M (lines 12-16). Model updates are performed by sampling transitions from M (lines 17-22). A delayed actor update mechanism is employed for the actor model (lines 23-24), and target networks are updated via a soft update scheme (lines 25-27). The Preference-Guidee Exploration function (lines 30-35) normalizes the sampled latent vector z (line 33) as
This normalization step, motivated by the prior work (Touati et al., 2023), has been observed to improve performance.
/d |
As shown in Algorithm the training of MORL-FB involves the following loss functions:
Measure Loss. The Measure loss, (M w: xx), is central to learning a task-agnostic representation of environment dynamics, Fo 0, #x), encoding command-come conditioond successor measures. It enforces Bellman consistency for these measures when projected onto a learned basis (s' a'), shown in Equation (14). This mechanism, drawn from (Touati al., 2023), aims to separate the
18

---

Published as a conference paper at ICLR 2026
environment structure from specific rewards. This disentanglement is erucial for enabling zero-shot generalization, allowing the agent to understand "what happens next" irrespective of the immediate goal, forming reusable foundation for various tasks.
(Fg. Bwi = os (aap://((FFFFFFFF (Fo(st, Bw (s',a')
(8' a'D''
TB ,.
- B (x) [ (s at =>) (5(541,,0411)
(14)
where p denotes the underlying distributions of the dataset.
Auxiliary Q Loss. To ensure the learned representation Fg is relevant for decision-making, the Auxiliary Q Loss, (o(0;zx), connects it to task-specific values. When explicit reward signals and corresponding preferences À are available, Equation (15) minimizes a standard temporal difference error. This vital for MORL contexts, effectively teaching Fg to support optimizing diverse rewards.
(0; 2x) B(spa)rss)))) (Fo(s), 2x) (ATre ( zA) =x))²
(15)
Orthonormality Loss. The Orthonormality Regularization Loss, La(w), acts crucial regular- izer for the learned basis functions (s, Its purpose, as reflected Equation (16), is to promote well-conditioned and non-degenerate basis. By encouraging properties such as orthogonality be- tween basis vectors and unit norm, this loss helps prevent representational collapse and redundancy within B This, in turn, ensures that the successor measures are projected onto a stable and diverse set of features, enhancing the robustness and quality of the learned representations Fg.
(w(w) = -.s..))D,(,v
(B
| (s, a) [B (ss('s''')) a')12 (16)
Policy Loss. The agent's behavior refined through the Policy Optimization Loss, Lx (x zx), which trains the policy T, within actor-critic paradigm. The actor's objective is to maximize the Q-values estimated by the critic, where these Q-values are derived from the Iearned representation Q(s, a; zx)= Fo(s, a, 2x) z (Equation (17)). This loss drives the policy select actions that are optimal for the task specified by the current command zx- It thus enables the agent to translate its universal understanding of the environment into effective, task-adaptive behavior.
Cx(11 ES-D Q(s n. (s); 2x)1, where Q(s, az 2x) F(s, a, 2x) 2x.
(17)
C DETAILED CONFIGURATIONS OF EXPERIMENTS
In this section, describe the experimental setup used to evaluate the performance of our approach. We detail the hyperparameters used in our experiments, as well as the reference points chosen for HV evaluation across different environments.
C.I EVALUATION ENVIRONMENTS
We evaluate the performance of our proposed method, MORL-FB, across a diverse set of multi- objective reinforcement learning environments. These environments, detailed below, encompass both established MuJoCo-based locomotion tasks and discrete problems, allowing us to assess its adaptability across distinct settings with varying state spaces, action spaces, and objective numbers.
MuJJCo-Based Continuous Control:
The state space and action space defined as S C R and A C R respectively. The two objectives for this environment are maximizing moving speed along the x-axis and minimizing energy cost.
Halfcheetah2d:
19

---

Published as a conference paper at ICLR 2026
The state space and action space defined 8 S R and A S R respec- tively. The two objectives for this environment are maximizing moving speed along the x-axis and minimizing energy cost.
Walker2d:
The state space and action space are defined as S S R and C R The three objectives include maximizing moving speed along the x-axis, maximizing jumping height along the z-axis, and minimizing energy cost.
Hopper3d:
The state space and action space are defined as S C R2? and A C R The three objectives are maximizing moving speed along the x-axis, maximizing moving speed along the y-axis, and minimizing energy cost.
Ant3d:
Humanoid2d: The state space and action space are defined as S S R and A C IR The two objectives maximizing moving speed along the x-axis and minimizing energy cost. Additionally, we set the healthy reward parameter to 1.0 to encourage exploration and stability.
The state space and action space are defined as S S _3766 and A C R This environment has five objectives: maximizing moving speed along the x-axis, max- imizing moving speed along the y-axis, maximizing angular velocity maximizing angular velocity on the right elbow, and minimizing energy cost. Similar to Humanoid2d, the healthy reward parameter is 1.0 ensure meaningful evaluation.
Humanoid5d:
the left elbow,
Classic Control:
Deep Sea Treasure (DST): The Deep Sea Treasure environment is a classic MORL prob- lem in which the agent controls submarine in 2D grid world (Vamplew et al., 201 I). The two conflicting objectives are typically maximizing collected treasure value and mini- mizing the time cost of collection.
Fruit Tree Navigation (FTN): This environment is structured as full binary tree of depth d = 5, 6, or 7 (Yang et al., 2019). The agent navigates from the root to a Ieaf node. Every leaf contains fruit with values for six objectives: Protein, Carbs, Fats, Vitamins, Minerals, and Water.
C.2 EXPERIMENTAL SETUP
To begin with, we describe the hyperparameters the benchmark MORL methods and the proposed MORL-FB for better reproducibility.
Hyperparameters for Experiments. To ensure fair comparison, for those benchmark methods that already provide tuned task-speciff hyperparameters original papers for the hyperparameter configurations, including PGMORL (Xu et al., 2020) and CAPQL (Lu et al., 2023). Table and Table 3 list the detailed hyperparameters used in our periments. For PGMORL, the hyperparameters reflect its evolutionary population-bass -based design. The parameter defines the number of parallel reinforcement learning tasks in each generation. Each task includes mw warm-up iterations and evolutionary iterations. Paum and Pazee define the number and size of the performance buffers. The PPO parameters used across all environments are summarized in Table 2. For CAPQL, the hyperparameter a controls the strength of a concave regularization term added the reward. The general hyperparameters shared by CAPQL are listed in Table 4.
MuJoCo, primarily refer to their
Table Hyperparameters of PGMORL.
Environments
m+
P
num
K
P
size
HalfCheetah2d
80
20
100
1
Walker2d
6
80
20
100
2
7
-1
Hopper3d
200
40
210
- -106
Ant3d
15
200
40
210
-106
Humanoid2d
200
40
100
2
-1
Humanoid5d
35
200
40
550
-106
20

---

Published as a conference paper at ICLR 2026
Table 2: PPO hyperparameters used in PGMORL.
Parameter
Value
Timesteps per actor batch
2,048
Processes number
4
Learning rate
x 10-4
Discount factor (?)
0.995
GAE lambda
0.95
Batch size
32
PPO epochs
10
Entropy coefficient
Value loss coefficient
0.5
Table Augmentation strength of CAPQL.
Environments
HalfCheetah2d
0.1
Walker2d
0.05
Hopper3d
0.2
Antöd
0.2
Humanoid2d
0.005
Humanoid5d
0.005
Table 4: Hyperparameters of CAPQL.
Parameter
Value
Optimizer
Adam
Learning rate
x 10
Discount factor (n)
0.99
Number of hidden units per layer
256
Replay buffer size
10
Batch size
256
Nonlinearity
ReLU
Target smoothing coefficient (r)
0.005
For algorithms without officially tuned or specified hyperparameters, we perform hyperparameter optimization (HPO) by following (Eimer al., 2023). Specifically, we employ Bayesian optimiza- tion by using the Weights & Biases Sweeps. Each episode refers to simulation run for 1M envi- ronment steps, and the optimization performed over 10 episodes. Tables 5 to 11 summarize the search ranges and the final selected hyperparameters.
MORL-FB: For MORL-FB, which is built on the implementation of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm (Fujimoto et al., 2018), we prioritized the tuning of parameters inherent to TD3. Specifically, this included critical elements such learning rates for actor and critic networks, the target network update rate, and policy delay, which are known to significantly influence the stability and performance of TD3- based agents.
Q-Pensieve: Our tuning efforts centered on parameters associated with its core Q-Snapshot mechanism (Hung et al., 2023). Given that Q-Pensieve's 's efficacy in improving sample efficiency stems from the storage and utilization of these Q-function snapshots, parameters governing the snapshot buffer, the frequency of snapshot capture, and their influence on the policy update were key areas focus during hyperparameter optimization.
PD-MORL: For PD-MORL, which can be viewed as the objeet- extensdion of TD3 (Fujimoto al., 2018), prioritized the tuning of parameters inherent to TD3. This
21

---

Published as a conference paper at ICLR 2026
included critical elements such learning rates for actor and critic networks, the target net- work update rate, and policy delay, which are known to significantly influence the stability and performance of TD3-based agents.
MORL/D: For MORL/D, which is built upon SAC, we tuned the standard hyperparame- ters of SAC. In addition, we focused on four components that are central to the MORL/D //D framework: population size, neighborhood size, scalarization method, and weight adap- tation method. These elements directly affect how the algorithm decomposes the multi- objective space and maintains policy diversity.
PCN: For PCN, since the hyperparameter settings for continuous control tasks such as MuJoCo environments were not specified in (Reymond al., 2022), we tuned three critical parameters: learning rate, batch size, and hidden dimension, which affect training stability, sample efficiency, and the model's ability to generalize across diverse preference vectors.
SFOLS: For SFOLS, since utilize its official implementation for discrete problems and extend with a TD3 backbone for evaluation on continuous control tasks, we tuned the parameters inherent TD3, including learning rates for the actor and critic networks, pol- icy noise, and target policy smoothing noise. These components are known to significantly influence the stability and performance of TD3-based agents.
GPI-PD: For GPI-PD, since GPI-PD relies learned dynamics model for sample gen- eration, we tuned four key parameters: dynamics rollout length, rollout frequency, model training frequency, and real data ratio. These factors critically affect model accuracy, sta- bility of planning updates, and the overall effectiveness of Dyna-style training.
Table 5: Hyperparameter tuning for MORL-FB.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0001
Policy update delay
{1,2, 5, 10)
10
Steps per update
{1,2, 5, 10)
Latent dimension (z dimension)
(50, 150, 300)
300
Exploration noise std.
00.1,
0.1
Target policy smoothing noise std.
0011, 1.0]
0.2
Table 6: Hyperparameter tuning for Q-Pensieve.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0001
Q replay buffer size
{1, 21,2,49 4}
4
Preference set size
{1, 2, 4)
4
Table 7: Hyperparameter tuning for PD-MORL.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0003
Exploration noise std.
[0.1, 1.0]
0.15
Target policy smoothing noise std.
[0.1, 1.0]
0.25
Policy update delay
{1, 2, 5, 10)
10
Hyperparameters of MORL-FB, The hyperparameters for the MORL-FB experiments were cho- to ensure fair evaluation and stable learning, guided by prior research and the HPO results. The
22

---

Published as a conference paper at ICLR 2026
Table Hyperparameter tuning for PCN.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0023
Batch size
{64, 128, 256]
64
Number of hidden units per layer
2256,512, 1024)
1, 0242
Table 9: Hyperparameter tuning for MORL/D.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0013
Batch size
{64, 128, 256}
256
Number of hidden units per layer
/256,512, 1024]
1, 1,024
Pop size
{4, 6,8}
Neighhorhood size
{0, 1,2]
Scalarization method
{none, ws}
ws
Weight adaptation method
{none, PSA}
PSA
Table 10: Hyperparameter tuning for SFOLS.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0006
Batch size
(64, 128, 2563
256
Number of hidden units per layer (256, 512, 1024]
1, 0202
Target smoothing coefficient (r)
[0.001, 0.02]
0.0061
Exploration noise std.
[0.1, 0.3]
0.1736
Target policy smoothing noise std.
[0.1, 0.5]
0.4232
Table II: Hyperparameter tuning for GPI-PD.
Parameter
Value
Optimal Value
Learning rate
[0.0001, 0.01]
0.0003
Batch size
{64, 128, 256}
256
Number of hidden units per layer
{256, 512, 1024]
1, 020
Dynamics rollout length
[5, [5,20]
8
Dynamics rollout fregency
[50, 500]
312
Dynamics train freqency
[100, 500]
139
Dynamics real ratio
[0.05, 0.2]
0.0845
hyperparameters tuned via HPO presented in Table 5, while the remaining values primarily drawing from the default settings of original FB and TD3 algorithms. The complete configuration is detailed in Table 12.
Reference Points for HV Evaluation. We compute the HV indicator using predefined reference points (Ref. Point) for each environment. These reference points serve baselines to measure the
23

---

Published as a conference paper at ICLR 2026
Table 12: Hyperparameter configuration for MORL-FB experiments.
Parameter
Value
Total number of environment steps
x 10
Replay buffer size
106
Latent dimension (z dimension)
300
Interface batch size
5,120
Number of hidden units layer
1,024
Preprocessing feature dimension
512
Batch size
1,024
Target smoothing coefficient (r)
0.01
Discount factor (7)
0.99
Learning rate
x 10
Policy update delay
10
Steps per update
Exploration noise std.
0.1
Target policy smoothing noise std.
0.2
Clipping parameter
0.5
Value loss coefficient
coverage of the Pareto front obtained during training. Table provides the specific reference points used in different environments.
Table 13: Reference points used for HV calculation different environments.
Environment
Ref. Point
Deep Sea Treasure
(0, -50)
Fruit Tree Navigation
(0, 0, 0, 0, 0, 0)
Halfchectah2d
0000))00
Walker2d
(0,00000)
Hopper3d
(0, 0, -8000)
Hopper4d
(0, 0, -8000,
Ant3d
(0, -8000)
Humanoid2d
(0, -8000)
Humanoid5d
(0, 0, 0, 0, -8000)
C.3 COMPUTE RESOURCES
All models were trained workstation featuring single NVIDIA RTX 4090 GPU, Intel Core 17-13700K CPU, and 64 GB of system memory.
24

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 VISUALIZATION OF & DISTRIBUTION IN DIFFERENT ENVIRONMENTS
We analyze the learned latent representations of MORL-FB by visualizing the distribution of sam- pled across various environments using t-SNE (Van der Maaten & Hinton, 2008). These visualiza- tions offer insight into the method's s ability to capture the underlying structure of the multi-objective tasks. Consistent with the observations on Humanoid2d (Figure 5), the visualizations for Walker2d, Hopper3d, and Ant3d (Figure 8, Figure 9, and Figure 10, respectively) demonstrate that preference- guided sampling yields distinct distributions compared to sampling from a standard normal dis- tribution. These results further support the hypothesis that preference-guided sampling promotes the exploration more diverse set of latent representations, which may contribute to improved generalization and adaptation various objectives.
Z Distribution During Training Walker2d
MORL-FB Original
20
Feature 2
10:
-20
-30
-20
-10
0
10
20
30
Feature
Figure 8: Empirical distribution under MORL-FB with preference-guide sampling versus Origi- nal FB with simple normal distributions on Walker2d.
Distribution During Training on Hopper3d
20
MORL-FB
Original FB
10
Feature
10
-20
-30
Feature
Figure 9: Empirical distribution under MORL-FB with preference-guided sampling versus Origi- nal FB with simple normal distributions on Hopper3d.
To shows the multi-modality of MORL-FB, we visualized the positions of latent variables = inferred from different preferences the t-SNE plot in Figure 11. This demonstrates that MORL-FB ef- fectively encodes different preferences into separate regions of the latent space, leading to more diverse policies. To further illustrate this, provide demo of the policies learned MORL-FB and vamilla FB in this link: https://intt /ttm////...//////////////////../ com/ a/ ehxlv7g, where the z s are selected from different positions the t-SNE plot.
25

---

Published as a conference paper at ICLR 2026
Z Distribution During Training on Ant3d
20
MORL-FB
Original FB
10
Feature 2
-10
-40
-20
Feature
Figure 10: Empirical z distribution under MORL-FB with preference-guide sampling versus Orig- inal FB with simple normal distributions on Ant3d.
Z Distribution During Training on Humanoid2d
60
40
Feature 2
-20
-00
-4
-20
60
Feature 1
Preferences
Original-FB: [0.9,
MCRL-FB:
Original-Fa: 10.7. 0.31
MORL-FB: [0.7. 0.31
Original-FB:
Original-FB: [0.3, 0.7]
MORLFBB: [0.3, 0.7]
Original-FB: [o.1, 0.9]
MORL/FB: [0.1, 0.9]
Figure 11: Empirical z distribution under MORL-FB with preference-guide sampling (blue) versus Original FB with simple normal distributions (red) on Humanoid2d. This figure is slightly different from Figure due to the additional preference points and the inherent randomness of t-SNE.
D.2 ABLATION STUDY
D.2.1 EXPERIMENT ON CARFERENCE-GUDDER EXPLORATION
To assess the data efficiency of our proposed MORL-FB, we compared its performance against PD- MORL (Basaklar et al., 2023). PD-MORL necessitates three to five times training samples to train its preference interpolator. Can MORL-FB maintain superior performance while reducing data requirements?
Table 14 reveals that MORL-FB consistently outperforms PD-MORL across various environments. Specifically, MORL-FB achieves higher UT values in five out of six tasks, demonstrating its effec-
26

---

Published as a conference paper at ICLR 2026
tiveness in most scenarios, In more complex settings, such MO-Humanoie how does MORL-FB compare in HV results? The results indicate that MORL-FB remains competitive, underscoring its significant data efficiency gains. By eliminating the need for additional data to pretrain an interpo- lator, MORL-FFB achieves competitive or superior performance while requiring significantly fewer training samples in multi-objective environments.
Table 14:
Performance comparison between MORL-FB and PD-MORL (with interpolator) across key metrics (UT, HV, and ED) on various multi-objective tasks.
Environments
Metries
PD-MORL (w/i interpolator)
MORL-FB
UT(x 10")
5.62 + 0.05
7.69 = 0.08
Halfcheetah2d
HV(x 10%)
1.08 0.00
1.24 = 0.00
ED
0.07 0.01
UT(x 10²)
2,18 0.02
2.23 # 0.03
Walker2d
HV(x 10")
5.45 + 0.01
4.32 0.02
ED
0.56 + 0.00
UT( 10%)
2.26 0.01
0.01
Hopper3d
HV(x 10
1,88 0.00
1.15 0.00
ED
0.20 0.01
UT(x 10")
3.59 + 0.06
3,43 + 0,22 4.18 +..0 0.04
Ant3d
HV(x 10
4,20 = 0.04
ED
0.60 + 0.00
UT(x 102)
2.93
8.13 0.01
Humanoid2d
HV(x 107)
1.06 4 0.01
1.75 + 0.02
ED
0.33 0.01
UT(110
0.04
1.11 0.00
Humanoid5d
HV
6.64 0.09
6.99 0.06
ED
0.43 0.01
D.2.2 EXPERIMENTS ON DIMENSION
We investigate the impact of the dimension in the Hopper3d environment. As shown in Table 15, the performance metric increases with the z dimension. However, when the z dimension reaches 300, performance declines, likely due to insufficient training steps for the larger network.
Table 15: Empirical study on z dimension on Hopper3d.
dimension
Metries
MORL-FB
UT(x 10a)
2.21 + 0.01
50
HV(x 10")
1.03 + 0.05
UT(x 10a)
2.25 0.00
100
IV(x 10t4)
1.13 1 0.011
UT(x 10a)
2.36 4: 0.01
150
HV(X 10th)
1.15 4 0.00
UT(X 10a)
2.01 4 0.01
300
HV(x 10
0.97 + 0.02
D.2.3 EXPERIMENTS ON AUXILIARY LOss
While Touati et al. (2023) also includes similar auxiliary loss, there is one salient difference between theirs and our Q loss Equation (6): As the original FB is designed for RFRL, does not have the reward signal available training and hence needs to construct pseudo reward as
27

---

Published as a conference paper at ICLR 2026
B BTEIBB"111 (ef. Equation (9) in (Touati et al., 2023)). On the other hand, MORL-FB ad- dresses MORL and can observe vector reward signals at training, we propose to use the actual scalarized reward A r in the loss. While this algorithmic difference appears seemingly subtle, the performance. Below we show an ablation study that Q loss (denoted "Original") and MORL-FB with the auxiliary loss using pseudo reward in FB (denoted as "Pseudo Q Loss"). The results are summarized in Ta- ble 16 and Fig. 12. This shows that the auxiliary loss of FB cannot be directly applied and needs to be adapted properly in the context of MORL. Note that we use the term "auxiliary" since the original FB directly built on the measure loss and hence the loss auxiliary for learning FB representation, rather than being unimportant for MORL.
this design makes a huge difference compares MORL-FB with
Table 16: Ablation study of MORL-FB on preference-guidde exploration.
Environments
Metrics
MORL-FB
MORL-FB
MORL-FB (Ours)
(w/o PG-Explore)
(Pseudo Q Loss)
UT( 10")
1.45 + 0.02
0.02
1.27 0.16
3.93 + 0.04
Ant3d
HV(x 10
1,25
1.91 0.02
3.85 + 0.01
Hopper3d
Ant3d
Humanoid2d
750
2000
UTf
1500
2000
1000
250
500
0
1e11
1e11
1e7
1.00
1.5
HV t
0.75
i
1.0
0.50
0.5
0.25
0.00
0.0
Preference Serttrence-Guidd-EExppeee Guided-Explloation)
MORL-FB (Durs)
MORL-FB (w/o loss)
Figure 12: Evaluation of MORL-FB and its ablated versions across different enviromments. The results highlight the importance of preference-guie -ggide sampling and the auxiliary Q-loss for MORL performance.
D.2.4 EXPERIMENTS ON MEASURE LOSS
Q-loss is surely important in our method as it guides our representation to learn the reward function used in the environment. On other hand, both the preference-guided exploration and the measure loss also essential to the success of MORL-FB based on Figure 12 and additional ablation study on the measure loss shown below.
Table 17: Ablation study of MORL-FB on measure loss and Q loss.
Environments
Metrics
MORL-FB
MORL-FB (w/o Q loss)
MORL-FB (Ours)
(w/o measure loss)
UT( 10º)
1.37 0.28
-1.53 + 0.00
3.93 : 0.04
Ant3d
HV(x 10
2.74 0.02
0.00 + 0.00
3.85 + 0.01
28

---

Published as a conference paper at ICLR 2026
D.2.5 EXPERIMENTS ON Q-LOSS COEFFICIENT
In our implementation, the measure loss LM and the Q loss Lg are directly added together without an additional weighting term. We chose this default configuration because it already yields strong and stable performance across all evaluated tasks, and did not observe a need for further hyperpa- rameter tuning during development. Moreover, we have conducted an additional sensitivity analysis better understand how the balance between these two losses affects performance. Specifically, we varied the Q-loss coefficient aq (with aq = 1 as the default setting) and reported the corresponding results in Ant3d. Table 18 shows that MORL-FFB's performance largely insensitive to the choice of aq, indicating that directly adding LM and Lo is reasonable and robust design choice.
Table 18: Performance of MORL-FB with different Q-loss coefficients.
MORL-FB
Environments
Metries
QQ 0.25
QQ 0.5
Qq (Default)
aq =
UT(x10*)
3.76 = 0.41
3.79 + 0.32
3.77 = 0.11
3.95 + 0.13
Ant3d
HV(x10"!)
4.03 + 0.22
4.13 + 0.19
3,95 + 0.13
4.23 0.04
D.3 PERFORMANCE COMPARISON OF MORL-FB UNDER DIFFERENT Q-LOSS COEFFICIENTS
We evaluate MORL-FB and several baselines on two classic discrete control tasks: Deep Sea Trea- sure (DST) and Fruit Tree Navigation (FTN). DST features the fundamental trade-off between two conflicting objectives. FTN, on the other hand, involves a tree search process with 6-dimensional terminal rewards, as previously detailed in the evaluation environments section. The detailed con- figurations for these experiments are provided in Section C.
Figure 13 shows the performance of all the methods in UT, HV, and ED for discrete control tasks. Regarding ED, for each baseline algorithm ALG, we report ED(ALG, MORL- (BRRL-BB) to show pair- wise comparison. We observate that MORL-FB consistently achieves competitive superior performance across all three metrics on the discrete control tasks Deep Sea Treasure and Fruit Tree Navigation.
Deep Sea Treasure
Fruit Tree Navigation
UTf
4
2
1000
10000
HV1 T
500
5000
H
0.8
0.8
ED f 0.6 84
83
0.2
0.0
SFOLS
Optimal
EQL
MORL-FB (Ours)
Figure 13: Competitive Results of MORL-FB on Discrete Control Tasks. We evaluate MORL- FB and several benchmark MORL algorithms on classic discrete control tasks in MO MO-Gymnasium. Performance is measured using UT. HV and ED. MORL-FB demonstrates competitive results against specialized discrete MORL algorithms.
29

---

Published as a conference paper at ICLR 2026
D.4 PERFORMANCE COMPARISON OF MORL-FB UNDER SOATE-ACTION-BLL REWARDS AND STATE- BASED REWARDS
In this paper, we primarily focus on state-based rewards. However, as the original FB supports both state-based (Touati et al., 2023) and state-action-bass -bssed rewards (Touati & Ollivier, 2021), MORL-FB can also be extended to state-action- cestoon-based rewards by replacing (ss) with [(s, a). Since some MO MuloCo rewards depend on both states and actions, compare MORL-FB and the extended one. As shown the Figure 14, the state-action- based variant yields slight performance improvements on several tasks.
Walker2d
Hopper3d
Ant3d
2000
3000
UT1
1500
2000
1000
1000
500
1000
1e7
1e11
1611
HVT
0.75
0.50
0.25
0.00
0
state-action based
state based
Figure 14: Evaluation of MORL-FB with different reward function representations. This figure presents the performance of MORL-FB when reward functions depend on states only (i.e., R(s)) versus state-action pairs (i.e., R)s,a)).
D.5 PERFORMANCE COMPARISON OF MORL-FB UNDER STOCHASTIC REWARDS
As vanilla FB naturally handles stochastic rewards, MORL-FB inherits this capability. To further demonstrate this, we evaluated MORL-FB under stochastic rewards by adding zero-mean Gaussian noise (0, )²), similar to prior work (Romoff al., 2018; Hu al., 2022). The result is shown in Figure 15.
Ant3d
3000
UT
1000
HV
1
MORL-FB MORL-FB (a=0.1)
MORL-FB (a=0.3)
Figure 15: Evaluation of MORL-FB under stochastic reward. This figure assesses the perfor-
mance of MORL-FB in environments featuring stochastic reward functions.
30

---

Published as a conference paper at ICLR 2026
D.6 PERFORMANCE COMPARISON OF MORL-FB WITH NONLINEAR SCALARIZATION
While we focus linear scalarization in this paper, MORL-FB be readily extended nonlinear scalarization schemes by replacing r À with fx(r) when sampling for weeference-guide explo- ration and when computing Q loss, where (r) is the general scalarization function. This feasible since the original FB is designed to handle any scalar reward function, and MORL-FBB inherits this property from FB and can also handle nonlinear scalarization. To demonstrate this generalizability. we further evaluate MORL-FB on Halfcheetah2d by training under smooth Tchebycheff scalariza-
log (E exp (x(r-Tret)) )) where X; = is the i-th entry of preference vector, the smoothing parameter and set to 0.1, and Fref is set to [2.0, 0.0] across training (Lin al., 2024: Qiu et al., 2024). We see that MORL-FB still achieves comparably strong performance in HV and UT under nonlinear scalarization.
tion as (x(r
Table 19: Performance of MORL-FB with Smooth Tchebycheff scalarization.
Environments
Metrics
MORL-FB (linear scalarization)
MORL-FB (Smooth Tchebycheff scalarization)
UT(x 10²)
7.69 + 0.08
6.33 + 0.02 1.00 4 0.01
HalfCheetah2d
HV(x 108)
1.24 + 0.00
D.7 SAMPLE EFFICIENCY MORL-FB
We demonstrate the sample efficiency of MORL-FB by evaluating its performance at intermedi- ate stage of 1.5M training steps. Notably, as shown in Figure 16, MORL-FFB achieves superior HV and UT scores across most tasks compared to bascline methods trained for full 3M steps. This indicates that MORL-FB can attain high performance with significantly fewer environment interac- tions. Further evidence, presented in Figure 17 and Figure 18, corroborates that MORL-FB reaches proficient performance levels with reduced training data.
D.8 CROSS-OBJECTIVE TRANSFER CAPABILITY OF MORL-FB
To investigate how well MORL-FB handles transfer across different numbers of objectives, we con- ducted an empirical study on Hopper across different objective dimensions. We analyze the follow- ing cases:
Hopper2d: Moving forward speed the x-axis, control cost of the action
Hopper3d: Moving forward speed on the x-axis, jumping height on the z-axis, control cost of the action
Hopper4d:
Moving forward speed the x-axis, jumping height on the z-axis, jumping up speed on the z-axis, control cost the action
Table 20 summarizes the quantitative results presented visually in Figure 7. MORL-FB consistently outperforms FB across all configurations in terms of utility and hypervolume.
D.9 RFRL AS A SOURCE AUXILIARY TASKS
During training, the z vectors computed for each preference are diverse, covering both CCS and non-CCS policies. Learning from non-CCS policies serves as auxiliary tasks. From Figure 19, we find that the return vectors achieved by those z-induced polices at the 1.5 million training step span both non-CCS and CCS regions.
31

---

Published as a conference paper at ICLR 2026
3opper3d
Ant3d
Humanoid2d
Humanoid5d
800
1200
3000
600
800
UTf
1500
1000
200
400
1000
1.2
1.5
1.0
HV
0.6
0.4
0.5
0.2
PD-MORL
CAPQL PGMORL
MORL/D
SFOLS
MORL-FB (Ours) MORL-FB (1.5M)
Q-Pensieve
PCN
FB
Figure 16: Performance of MORL-FB on continuous control tasks. We evaluate MORL-FB (1.5M training steps) against several benchmark MORL algorithms (3M training steps) diverse continuous control tasks from MO-Gymnasium. Utilizing key metrics, these results demonstrate that MORL-FB outperforms baselines, achieving superior HV and UT across most tasks despite significantly fewer training steps.
Hypervoluma on Ant3d
Jtility on Antäd
O Persieve
Pensieve
1041
SPOLS
PCN
PGMORL
10t
PCN
GPILS
GP
SFOLS
MORUD
MORLD
30
Clegs
125
Figure Learning curves for MORL-FB and benchmark algorithms on Ant3d. This figure presents the learning curves terms Hypervolume (HV) for MORL-FB and sev- eral benchmark MORL algorithms evaluated on Ant3d.
Figure 18: Learning curves for MORL-FB
and benchmark algorithms on Ant3d. This figure presents the learning curves in terms of Utility (UT) for MORL-FB and several bench- mark MORL algorithms evaluated on Ant3d.
10 COMPARISON OF PARETO FRONTS
To evaluate the sample efficiency of MORL-FB, we conduct experiments on 2-objective MuJoCo tasks with a whole range of 21 preference vectors ((0.0, 1.0], [0.05, 0.95), [0.1, 0.9], ..., [1.0, 0.0]). consider SORL which trains separate policy for each individual
As comparison baseline, preference. Each single-object SAC (SOSAC) model requires 3M steps, resulting a total training budget of 63M steps for all 21 preferences. By contrast, MORL-FB only uses 3M steps in total to learn policies that generalize across the entire preference set. Figure 20 shows the return vectors attained by MORL-FB and the collection of 21 SOSAC models the Walker2d task. MORL-FB
32

---

Published as a conference paper at ICLR 2026
Table 20: Zero-shot cross-objective transfer from Hopper2d Hopper3d and Hopper4d using vanilla FB and MORL-FB: This figure presents the results demonstrating effective transfer by MORL-FB, supporting the efficacy of proposed enhancements.
Environments
Metries
FB
MORL.FBB
UT(x 10")
0.02 + 0.00
1.77 0.00
2D to 3D
HV(x 10"))
0.40 = 0.00
7,81 + 0.06
ED
0.07 0.03
UT(x 10")
0.06 + 0.00
1.59 0.00
2D to 4D
HV(x 10
0.05 = 0.00
7.78 + 0.10
ED
0.05 1 0.01
Walker2d preference (0.7, 0.3]
Rewerd Dimersion 2 (Energy Cost)
-400
steps 1.5M steps
steps
1000
2000
2002
Reward Dimension [Moving Speed)
Figure 19: Return vectors (Moving Speed vs. Energy Cost) achieved at the initial, intermediate, and final training stages in Walker2d with preference [0.7, 0.3]: The scatter plot, particularly at 1.5M steps, highlights the diverse policies beyond the CCS policy, supporting that RFRL serves as auxiliary tasks beneficial for MORL.
achieves comparable even superior return vectors with only 1/21 of the samples, demonstrating its strong sample efficiency and generalization ability across diverse preference vectors.
Humanoid2d
Walker2d
5054C
50SAC
Reward Dimersion 2 (Energy Cost)
Reward Dimansion 2 (Energy Cosz)
-900
-00
-3900
-130
-3000
100
128
294
2013
2990
12.
1000
ohe
Kee
Reward Dimension
(Maving Speed)
Reward Dimension 1 (Moving Speed)
(a) Humanoid24
(b) Walker2d
Figure 20: Return vectors (Moving Speed vs. Energy Cost) achieved under 21 different preference vectors J0, 1], [0.05, 0.95 ... [0.95, 0.05), [1, 0] across different methods. Each scatter cloud corre- sponds to the learned policies under a specific preference, illustrating how MORL methods adapt to diverse trade-offs between objectives.
........
33

---

Published as a conference paper at ICLR 2026
D.11 ROBUSTNESS UNDER WORST-CASS PREFERENCES
Beyond average performance metrics, it is crucial to assess the robustness of multi-objective rein- forcement learning (MORL) algorithms under unfavorable preference settings. In practice, poli- cies deployed in dynamic environments may encounter user preferences that significantly diverge from those seen during training. To capture this aspect, we evaluate each method under worst-case preferences using the Conditional Value-at-Risk (CVaR). Following the risk- learning formulation of Tamar al. (2015), we compute C'VaR @0.1 the mean of the lowest 10% scalarized returns. Specifically, for each algorithm we uniformly sample 500 preference vectors, apply linear scalarization to obtain utility values, sort the results, and average the worst 10%. This quantifies the expected return conditioned on being in the lowest a-quantile:
reinforcement
( [ R S (o),
(18)
Table 21 reports @V. across HalfCheetah2d and Walker2d. MORL-FB achieves the highest CVaR both environments, significantly outperforning prior baselines. These results indicate that MORL-FB not only excels
metries but also demonstrates superior robustness against adverse preferences, avoiding catastrophic failures more effectively than existing methods.
Table 21: CVaR@0.1 performance across HalfCheetah2d and Walker2d. Higher is better.
Algorithm
HalfCheetah2d
Walker2d
PD-MORL
2425.91
284.89
Q-Pensieve
6275,89
1516.20
CAPQL
4874.64
1297.40
PGMORL
2630.02
1113.49
MORL/D
407.66
40.47
PCN
-0.11
98.28
SFOLS
1390.73
402.19
GPI-LS
3884.67
1546.96
GPI-PD
3999.36
276.74
MORL-FB (Ours)
7123.87
2304.02
D.12 COMPUTATIONAL COST ANALYSIS
To provide fair and comprehensive comparison of computational efficiency, we evaluated all meth- ods under the same hardware environment. Each algorithm was trained for 100K environment steps on workstation equipped with a single NVIDIA RTX 4090 GPU, an Intel Core 17-13700K CPU, and 64 GB of system memory. Table 22 reports the total wall-clock time required by each method, ensuring fair comparison of time and resource usage across methods. Compared to strong base- lines such as PD-MORL, Q-Pensieve, and GPI-LS, FB-MORL maintains reasonable training time. While some methods require higher computational costs, approach strikes good balance be- tween performance and efficiency.
D. DETAILED EXPERIMENTAL RESULTS OF SECTION 4
In this part, we will provide the table we use on plotting the bar chart on Section 4.
The results in Tables 23 26 correspond to the visualizations shown in Figure 2, Figure 4, Figure 12 and Figure 13, respectively.
34

---

Published as a conference paper at ICLR 2026
Table 22: Wall-clock time comparison (100K steps).
Algorithm
Wall-Clock Time (seconds)
PD-MORL
1166
CAPQL
3369
MORL/D
793
SFOLS
550
Q-Pensieve
12960
PGMORL
4237
PCN
4445
GPI-LS
3611
GPL-PD
5237
MORL-FB (Ours)
1874
Table 23: Performance of MORL-FB and benchmark algorithms in discrete MORL environments.
Enviromments
Metries
PD-MORL (w/i interpolator)
Envelope
PCN
MORL/D
SFOLS
MORL-FB
Optimal
(Ours)
UT
0.52 + 0.00
6,53 + 0.00
6.12 + 0.95
3.88 + 0.26
5.65 + 0.95
6.43 0.10
6.89
Deep Sea Treasure
HV(x10²)
9.33 + 0.00
9,91 = 0.00
8.71 + 1.49
5.63 = 0.41
8.46 + 0.01
9.92 : 0.00
9.92
ED
0.69 1: 0.00
0.77 1 0.00
0.81 0.01
0.15 1 0.10
0,25 0.00
UT
5.03 + 0.00
5.07 = 0.00
5.00 0.02
4.19 = 0.03
4.77 0.01
5.01 + 0.01
5.08
Fruit Tree Navigation
1.25 1: 0.00
1.16 1 0.00
1.01 0.06
0.18 0.05
0.87 1 0.01
1.16 116.10.00
1.25
0.76 0.01
0.95 0.00
0.72 0.03
0.01
0.08 + 0.01
Table 24: Performance of MORL-FB, PD-MORL, and Q-Pensieve under constrained preference training.
Environments
Metrics
PD-MORL
Q-Pensieve
MORL-FB
UT(x 103)
1.05 0.02
1.72 0.01
2.26 + 0.01
Hopper3d
HV(x 10ah)
0.61
0.88
1.16 0.01
ED
0.01
0.00
UT( 10a)
1.69 + 0.04
0.49 + 0.02
3.11 + 0.24
Ant3d
HV(x 10")
2.18 0.03
0.51 0.00
3.17 + 0.05
ED
0.22 + 0.02
0.26 + 0.04
UT(x 102)
-0.04 0.00
4.51 + 0.38
8.19 0.03
Humanoid2d
HV(x 107)
0.06 0.00
1.51 + 0.04
1.83 - 0.01
ED
0.00 - 0.00
0.40 + 0.05
Table 25: Performance comparison of MORL-FB and its ablated versions across environments. This table evaluates MORL-FB against variants lacking seference-guided exploration or the Q-loss component, showing their performance across different environments.
Environments
Metrics
MORL-FB (w/o preference-guided exploration)
MORL-FB
MORL-FB
(w/o q loss)
UT(x 103) HV(x 1)
2.03 0.42 0.91 + 0.02
0.33 0.07
2,36 0.00
Hopper3d
0.44 + 0.02
1.15 - 0.01
UT(x 103) HV(x 11
1.23 0.03
-1.01 0.01
3.43 0.22
Ant3d
1.47 + 0.03
0.06 + 0.00
4.18 - 0.04
UT(x 102)
2.38 0.30
-1.43 0.14
8.13 0.01
Humanoid2d
HV(x 107)
1.78 0.00
0.96 + 0.00
1.75 + 0.02
35

---

Published as a conference paper at ICLR 2026
7.69 0.088
1.87 0.05
4.56 0
2.36 0.01
1.15 0.00
3.43 0.22
4.18 0.04
8.13 0.01
1.75 0.02
1.11 0.00
6,99 + 0.I6
MORL-FI
(Ours)
1.24
1.26 0.00
0.26 + 0.00
+0002
0.29 1=001
1.97 +
1:000
1:000
+5000
1=0.00
1:001
0.08 1:000
0.03 1:002
1:000
1.19 1:055
0.06 1.0.00
0.55 1.0.3
4.69 #=0.0
1.0.0
Table 26: Comparative performance MORL- RRL-FBBRR and various benchmark algorithms across continuous control tasks in MuloCo.
FB
0.04
0.00
0.01
0.00
0.00
0.01
2.44
0.03
GPI-PD
4.06 +. 0.03
0.68 0.00
0.01
0.54 0.00
1.13 0.00
0,15
0.02
1.32
0.51 0.00
0.05
0.02 32 0.10 0,111
1. 0.00
0.00
0335
0.08 1 0.00
0.00
0424 0.03
0.14..0000
0.30
0,20
0.03 0.03 0.00
0.06
0,66
1.35
GPI-LS
4,15 0.84
0.67 0.12
0.05 0.00
1,92 = 0.18
3.35 = 0035
0 0.23
0.29 0,00
0.10
1 0010
0.73 015
0.17 1 000
144 0.12
0.77 0.77
0.15 115200 000
172:0000 0.02
3.88 = 0.03
0.09 = 0.00
0.32
0.57
0.72
SFOLS
1.39 + 0.03
0.59 +
0.01 +.0.1
0.41 0.41 0.01
1.22 +.000
0.15 0.15
#.00
0.03 1:001
0.03 +0000
0,00 1:000
0.28 1.0.01
0.12 4:2.00000
0.06
1:00::04
0.45 4.0.8
0.75
0.04:0000
0.54 1.000
2.15
0.05 4.000
0.00 4: 4.000
0.00 0.00
0.01
0.10 0.00
0.17 # 0.00
0.03 + 0.01
0.0
0.00 + 0.00
0.00 + 1.00
1.0
0.07 1 0.00
0,00 1 0.00
3,79 3.79 0.72
0.25 0:25 0.1
0.00 1 0.00
-0.10 0:0
+ 0.00
1 0.00
PCN
0.05
0,01
0,05
-0.10
0.00
0.00
MORL/D
0.41 + 0.01
0.00
0.01
0.05 4: 0.00
0.35 4 0.00
0.11 1: 0.00
0.01 = 0.00
0,04 11 0.00
0.01 = 0.00
0,05 0, 0.00
0.62
0.00
0.18 0.01
0.58
0.31
0.00
0.0:1:0000
0.00
0.00
2.71
0.08
0.05
0.03
0.62
0.31
0.08
PGMORL
2.72 = 0.03
0.03 0.01
1.15 0.02
352 002=000 0.00
0.78 0.00
0.00
0.80 = 0.00
0.11 -00
0.08 = 0.0
1.5 = 004
1.1 0.0
0.54 144=000 0.00
0.59 0.00
0.10 = 0.00
0.73
0.18
0.01
0.03
0.07
5.39 = 0.01
0.06 0.01
1.49 = 0.01
0.03 0.01
0.00 = 0.00
1.25 1 0.01
0.57 0.01
0.00 = 0.00
1.01 + 0.00
0,45 0.00
0.08 = 0.05
-0.4 + 0.00
0.01 = 0.01
0.08 = 0.00
0.08 0.00
0.00 = 0.00
0.04 - 0.00
CAPQL
0,39 =
Q-Pensieve
6.85 0.01
0.08 0.01
1.66 + 0.08
0.36 0.01
1.82 1: 0.02
1.25 + 0.01
0.35 1 0.01
1.81 : 0.08
2.61 + 0.02
0.14 0.04
4,51 1 0.03
1.51 + 0.04
0.12::0000
1.80 0.02
0.00 0.00
4.54 +
1.13
0.40
PD-MORL interpolator)
3.17 0.04
0.95 0.00
0.06 0.01
1.70 0.01
4.02 0.01
0.27 0.03
1.29 0.02
0.92 0.00
0.05 0.00
1.29 0.1.1
2.42 0.05
0.11 0.05
-0.05 0:00
0.06 0.00
0.38 1: 0.01
0.03 0.00
0.07
3.91
Metries
UT(x109)
HV(x10*)
UT(x
HV(x10')
UT(x109)
HV(x10')
UT(x109)
HV(x10")
UT( ED
HV(x10*)
UT(x109)
HV(x10'5
ED
ED
ED
ED
ED
Enviromments
Halfcheetah2d
Humamoid2d
Humanoid5d
Walker2d
Hopper3d
Ant3d
36

---

Published as a conference paper at ICLR 2026
We also evaluated the probability of improvement (POI) between MORL-FB and the benchmark algorithms suggested by Agarwal et al. (2021) in Figure 21. POI quantifies the likelihood that one algorithm will outperform another. The results demonstrate that MORL-FB consistently achieves superior performance compared to baselines.
Additionally, assessed the performance of training with constraint preferences using the method from Agarwal et al. (2021), presented in Figures 22 and 23.
Algorithm
Algorithm
GPI-PD
MORL-FB (Ours)
GPI-LS
MORL-FB (Ours)
FB
MORL-FB (Ours)
SFOLS
MORL-FB (Ours)
PCN
MORL-FB (Ours)
MORL/D
1
MORL-FB (Ours)
PGMORL
MORL-FB (Ours)
CAPQL
MORL-FB (Ours)
Q-Pensieve PD-MORL
MORL-FB (Ours)
MORL-FB (Ours)
0.6
0.7
0.8
0.9
1.0
P(X Y)
Figure 21: Probability of Improvement (POI) of MORL-FB against benchmark algorithms.
This figure illustrates the POI of MORL-FB relative to various benchmark MORL algorithms.
Median
IQM
Mean
MORL-FB (Ours)
Q-Pensieve PD-MORL
0.4
0.6
0.8
1.0
0.4
0.6 0.8
0.4
0.6
0.8
1.0
Normalized Weighted Reward
Figure 22: Medium, IQM, and Mean performance of MORL-FB and benchmarks trained with small preference sets. This figure displays the Median, Interquartile Mean (IQM), and Mean per- formance for MORL-FB and other benchmark algorithms, trained with only small set of preference vectors.
Algorithm
Algorithm Y
Q-Pensieve
MORL-FB (Ours)
PD-MORL
MORL-FB (Ours)
0.5 0.6
0.7
0.8 P(X < Y)
0.9 1.0
Figure 23: Probability of Improvement (POI) of MORL-FB under constrained preference training. This figure illustrates the POI of MORL-FB relative to other benchmark algorithms, all trained with specific constraint sets of preference vectors.
37


Reviews:
[{"review_id": "Yn0sL1KUN9", "reviewer": "Reviewer_cCvf", "summary": "This paper considers the problem of multi-objective reinforcement learning. Current MORL algorithms typically assign preference weights to individual objectives during the training process, modifying the underlying utility function to discover one or multiple policies to satisfy different preferences. \nThis work proposes to instead use techniques adapted from reward-free reinforcement learning (RFRL), where no such preference weighting is included in the utility function. RFRL aims to learn a policy that is optimal for arbitrary reward functions. The authors note the conceptual connection to MORL problems, and suggest an extension of a state-of-the-art RFRL algorithm to the multi-objective setting. The proposed algorithm contains three important modifications: preference-guided exploration, training on latent vectors sampled from the replay buffer as auxiliary tasks, and auxiliary Q loss.\nThe algorithm is validated experimentally, showing clear improvement over other MORL algorithms in several standard multi-objective metrics.", "questions": "1. Perhaps this has been missed, but will the implementation of this methods be made available, ideally for integration with the existing benchmarking suite?\n\n2. How exactly were hyperparameters chosen for the proposed method (MORL-FB)? The appendix only mentions a decision “guided by prior research and the HPO results”, which seems not quite comparable to the HPO-tuning of the remaining algorithms.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper draws a natural connection between RFRL and MORL, with a generally insightful discussion of the similarities and differences between the two. According to the authors, this is the first work to do so explicitly. This is a valuable contribution, and an interesting starting point for further work. (This reviewer is not intimately familiar with the current state of the art, and so must take the assertions about the novelty of the paper at face value)\n\n- The paper is well-written and largely well-organised. Experiments are well-constructed, with a solid statistical approach, hyperparameter search, reproducible parameters, and meaningful ablation studies.\n\n- The experimental validation of the algorithm shows good results compared to various state-of-the-art approaches and baselines.", "weaknesses": "- The paper occasionally seems to conflate the field of MORL as a whole with the single MORL technique of linearly combining objectives, e.g. in the introduction and Section 3. As mentioned in the related work section, other MORL approaches exist. This should be clarified.\n\n- At times, this paper relies quite heavily on the appendix, in particular when relegating the discussion of related work from RFRL to the appendix. With the extension of the page limit during the revision phase, this should be moved to the main section to allow the paper to stand alone.\n\n- The conclusion in its current form is also quite short, and could benefit from the addition of slightly more detail about the method and context. Similarly, some figures are quite cramped (see additional comments).\n\n\nAdditional (minor) comments\n\n- Some of the figures, e.g. 1 and 3, are somewhat difficult to read. These could be made more readable despite the space constraints, e.g. by increasing font size and line thickness. Similarly, it is good that variance/error is represented in the plots, but in the current format this is barely readable.\n\n- Some minor language and formatting issues:\n1. line 050: “no prior work has explicitly adapt RFRL...”\n2. “across various tasks in … benchmark” (line 080/081)\n3. line 210: “where (a) holds by that ...”\n4. Eq. 5 extends into the page margin.\n\n- Occasionally, a citation is used as the subject of a sentence, e.g. in line 465 “(Felten et al., 2024; Mossalam et al., 2016) extended …”. In such cases, ‘\\citet{}’ can be used in LaTeX to generate a textual citation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fsFI4vfueE", "reviewer": "Reviewer_hbj3", "summary": "This paper views MORL problems from a reward-free perspective and then solves them using the RFRL method. Based on RFRL, it introduces a preference-guided exploration strategy that incorporates reward functions and preferences into the original RFRL. Specifically, rather than directly sampling latent vectors from a normal distribution, it samples preferences uniformly and uses them to generate latent vectors. Finally, the proposed method is evaluated on a series of tasks in MO-Gymnasium and demonstrates superior performance and data efficiency.", "questions": "- For completeness, I recommend briefly describing the future visited probabilities $M=FB$ in the preliminary section or appendix, as it is an important concept that helps readers understand why FB can be applied to arbitrary rewards.\n- In Figure 1, the lines are too dense and intertwined, making it difficult to interpret. It may be better to eliminate the curves corresponding to batch sizes 256 and 4096, or adjusting the scaling to improve clarity.\n- I recommend adopting more RFRL methods beyond FB to solve MORL in the future, which could lead to more insightful conclusions on RFRL's advantages in MORL.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated. It solves MORL problems via RFRL, which is intuitive and novel and could serve as a good bridge between the two sub-fields.\n- The empirical evaluation and ablation studies are comprehensive and insightful, carefully demonstrating the effectiveness of the proposed approach from multiple aspects.\n- The paper is well-written and well-organized, with implementation details that are easy to understand.", "weaknesses": "I didn’t find major issues in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pmm1GXD7rj", "reviewer": "Reviewer_d51d", "summary": "This paper proposes a reward-free viewpoint for MORL and an approach, MORL-FB, that integrates reward-free RL methods into MORL. Extensive experiments are conducted to show the effectiveness and generalization of the proposed method.", "questions": "1. How is the transfer implemented through the framework of RFRL? A brief explanation based on the motivating experiment in Figure 1 would be appreciated.\n2. How are $L_Q$ and $L_M$ balanced in line 20 of Algorithm 2? Is there any evidence that directly adding them up is plausible?\n3. Why is the Q loss regarded as “auxiliary”? Does this mean the Q loss is less important compared with the measure loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work investigates the integration of a reward-free framework into multi-objective RL methods for better performance and transferability, which is novel yet natural.\n2. A motivating example is provided to show the design consideration of the exploration of *z*, revealing the core insight of this work.\n3. Extensive experiments are conducted on both discrete and continuous tasks, comparing with a wide range of baselines.\n4. The visualizations are intuitive and show that the method is capable of handling multi-modal distributions, implying its improved generalization.", "weaknesses": "1. The layout on page 4 may need optimization. Algorithm 1 and Figure 1 can be presented on one line.\n2. This work focuses on model design and experimental analysis, while the “reward-free viewpoint” in the title seems more theoretical. Further theoretical analysis may be needed, or the title and abstract should focus more on the model’s effect.\n3. The title of Figure 20(a) seems mis-specified; it may be “Humanoid2d.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7MOtAredo", "reviewer": "Reviewer_3sX6", "summary": "This paper presents a reward-free viewpoint for multi-objective reinforcement learning (MORL). It adapts the Forward–Backward (FB) representation from reward-free RL to MORL, aiming to learn a task-agnostic policy/value representation that can generalize to unseen preference vectors without retraining. Experiments on MO-Gymnasium tasks show improved performance and generalization compared with existing MORL baselines.", "questions": "1. The proposed method and the goal of fast adaptation to unseen preferences appear conceptually close to SF. What concrete limitation of SF does the FB formulation address? Could the authors provide an ablation comparing FB + PG-Explore with an SF-based approach?\n2. In Section 3 \"Training objective function\", the paper introduces both a measure loss and an auxiliary Q-loss. How are these two losses combined, particularly for the forward network? If they are combined, how sensitive is performance to their relative weighting?\n3. In Figure 6, except for MORL-FB (w/o Q-loss), the hypervolume of other baselines is close to the proposed method, while UT performs significantly worse. Could the authors explain the reason for this discrepancy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a new perspective on MORL, suggesting that learning beyond specific scalarized reward combinations can accelerate MORL through more effective knowledge sharing across preferences.\n2.The empirical evaluation is comparatively thorough for MORL: it includes multiple tasks (both discrete and continuous control), standard metrics such as hypervolume, analyses of generalization, and ablations (e.g., with and without PG-Explore and auxiliary losses).", "weaknesses": "1. The comparison with related work based on Successor Features (SF) and its variants is insufficient.\n2. Some methodological and experimental descriptions are unclear. Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)