MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Authors: Yifu Yuan
Zhenrui Zheng
Zibin Dong
Jianye HAO
Venue: ICML 2025 poster (2025)
Keywords: Generalization
Diffusion
Multi-Objective Reinforcement Learning
Slider Guidance
Primary area: reinforcement_learning->batchoffline
TL;DR: We propose the multi-objective diffusion planner with sliding guidance that has better generalization in OOD preferences.

Abstract: Multi-objective Reinforcement Learning (MORL) seeks to develop policies that simultaneously optimize multiple conflicting objectives, but it requires extensive online interactions. Offline MORL provides a promising solution by training on pre-collected datasets to generalize to any preference upon deployment. However, real-world offline datasets are often conservatively and narrowly distributed, failing to comprehensively cover preferences, leading to the emergence of out-of-distribution (OOD) preference areas. Existing offline MORL algorithms exhibit poor generalization to OOD preferences, resulting in policies that do not align with preferences. Leveraging the excellent expressive and generalization capabilities of diffusion models, we propose MODULI (Multi-objective Diffusion Planner with Sliding Guidance), which employs a preference-conditioned diffusion model as a planner to generate trajectories that align with various preferences and derive action for decision-making. To achieve accurate generation, MODULI introduces two return normalization methods under diverse preferences for refining guidance. To further enhance generalization to OOD preferences, MODULI proposes a novel sliding guidance mechanism, which involves training an additional slider adapter to capture the direction of preference changes. Incorporating the slider, it transitions from in-distribution (ID) preferences to generating OOD preferences, patching, and extending the incomplete Pareto front. Extensive experiments on the D4MORL benchmark demonstrate that our algorithm outperforms state-of-the-art Offline MORL baselines, exhibiting excellent generalization to OOD preferences.

MODULI:
Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Yifu Yuan Zhenrui Zheng"
Zibin Dong'
Jianye Hao
Abstract
tion to OOD preferences. Codes are available this repository.
Multi-objective Reinforcement Learning (MORL) seeks to develop policies that simultaneously optimize multiple conflicting objectives, but it requires extensive online interactions. Offline MORL provides promising solution by train- ing on pre-collected datasets to generalize to any preference upon deployment. However, real- world offline datasets are often conservatively and narrowly distributed, failing comprehensively cover preferences, leading to the emergence of out-of-distribution (OOD) preference areas. Exist- ing offline MORL algorithms exhibit poor gener- alization to OOD preferences, resulting in policies that do not align with preferences. Leveraging the excellent expressive and generalization capa- bilities of diffusion models, we propose MOD- ULI (Multi Objective DiffUsion planner with sLIing guidance), which employs preference- conditioned diffusion model as planner to gen- erate trajectories that align with various prefer- ences and derive action for decision-making To achieve accurate generation, MODULI introduces two return normalization methods under diverse preferences for refining guidance. To further en- hance generalization to OOD preferences, MOD- ULI proposes a novel sliding guidance mecha- nism, which involves training an additional slider adapter to capture the direction of preference changes. Incorporating the slider, it transitions from in-distribution (ID) preferences to generat- ing OOD preferences. patching, and extending the incomplete Pareto front. Extensive experiments the D4MORL benchmark demonstrate that our algorithm outperforms the state-of-tee oate--of-the-ar Offline MORI. baselines, exhibiting excellent generaliza-
1. Introduction
Real-world decision-making tasks, such as robotic con- trol (Liu et al., 2014: Cui et al., 2025; Yuan al., 2025), autonomous driving (Li & Czarnecki, 2018), and industrial control (Bae et al., 2023), often require the optimization of multiple competing objectives simultancously. It neces- sitates the trade-offs among different objectives to meet diverse preferences (Roijers et al., 2017; Yuan et al., 2024; Liu et al., 2024; Zhou et al., 2025). For instance, in robotic locomotion tasks, users typically focus on the robot's move- ment speed and energy consumption (Xu al., 2020). If the has high preference for speed, the agent will move quickly regardless of energy consumption; if the user aims to save energy, the agent should adjust to lower speed. One effective approach Multi-objective Reinforcement Learning (MORL), which enables agents to interact with vector-rewarded environments and learn policies that sat- isfy multiple preferences (Hayes et al., 2022: Roijers et al., 2013; 3; Yang al., 2019). These methods either construct set of policies to approximate the Pareto front of optimal solutions (Alegre et al., 2023; Felten et al., 2024) or learn preference-conditions policy to adapt to any situation (Lu et al., 2023b; Hung et al., 2022). However, exploring and optimizing policies for each preference while alleviating conflicts among objectives, requires extensive online inter- actions (Alegre, 2023), thereby posing practical challenges due high costs and potential safety risks.
Offline MORL (Zhu et al., 2023a) proposes promising solution that can learn preference-conditie pre-collected datasets with various preferences, improving data efficiency and minimizing interactions when deploying in high-risk environments. Most Offline MORL approaches focus extending conditiond methods (Thomas et al., 2021; Zhu et al., 2023a) or encouraging consistent preferences through policy regularization (Lin et al., 2024), However, real-world offline datasets often come from users with different behavior policies, making it difficult to cover the full range of preferences and leading to missing prefer-
policies from
Equua contribution 'College of Intelligence and Computing, Tianjin University Harbin Engineering University. This work was done when ZZ was visiting Tianjin University. Correspondence to: Jianye Hao siianychaooe Saannyc.hao@ju.cuu...o edu.edd.cn>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Complete Dataset
Shattered Dataset
Narrow Dataset
Approximate Pareto
NODULI MORvS
Soo
Preference Region
Cojective 2
Cijective
Oojective
Oojective
de
Objective
Objective
Objective 1
Objective
Figure 1. (Left) Trajectory returns for Hopper- Aepper-Amateur datasets in D4MORL. We visualize Complete, Shattered, and Narrow datasets for comparison. OOD preference regions are marked by the circles. (Right) The approximated Pareto fronts learned by MORvS and MODULI on the Shattered dataset.
ence regions, i.e. OOD preferences, resulting in a shattered or narrow data distribution, As shown in Figure 1, we visu- alize the trajectory returns for Hopper poapeer-Amateee Amateur dataset. The Complete version of the dataset can comprehensively cover a wide range of preferences. In many cases, subopti- mal policies can lead to Shattere or Narrow datasets, resulting in OOD preference regions as marked by the red circles. We then visualize the approximate Pareto front for MORvS baseline (Zhu et al., 2023a) and our MODULI in such shattered dataset. We find that the current Offline MORL baselines exhibit poor generalization performance when learning policies from incomplete datasets. They fail to adapt to OOD preference policies. The approximate Pareto front of MORYS shows significant gaps in the OOD preference areas, indicating that the derived policies and preferences are misaligned. Therefore, we aim to derive a general multi- nulti-objecctve policy that can model wide range of preference conditions with sufficient expressiveness and strong generalization OOD preferences. Motivated by the remarkable capabilities of diffusion models in expres- siveness and generalization, we propose MODULI (Multi Objective DiffUsion planner with sLIding guidance), trans- forming the problem of Offline MORL into conditional generative planning. In Figure (right), MODULI demon- strated better approximation of the Pareto front, achieving returns beyond the dataset. MODULI also patched the miss- ing preference regions and exhibited excellent generaliza- tion for the OOD preferences.
methods adapted to multi objett- tasks, which can better measure the quality of trajectories under different prefer- ences and achieve a more refined guidance process. Besides, intro-
to enhance generalization to OOD preferences, duce sliding guidance mechanism that involves training an additional slider adapter with the same network structure to independently learn the latent directions of preference changes. During generation, we adjust the slider to actively modify the target distribution, gradually transitioning from ID specified OOD preferences. This better captures the fine-grained structure of the data distribution and allows for finer control, thereby achieving better OOD generalization. In summary, our contributions include (i) conditional dif- fusion planning framework named MODULI, which models general multi-objective policies as conditional generative models, (ii) two preference-reture normalization methods suitable for multi-objective preferences to refine guidance, (fii) novel sliding guidance mechanism in the form of an adapter and, (iv) superior performance in D4MORL, partic- ularly impressive generalization to OOD preferences.
2. Related Work
Offline MORL Most MORL research primarily focuses on online settings, aiming to train a general preference- conditioned policy (Hung al., 2022; Lu et al., 2023a) set of single-objective policies (Natarajan & Tadepalli, 2005; Xu et al., 2020) that can generalize to arbitrary pref- erences. This paper considers an offline MORL setting that learns policy from static datasets without online interac- tions. PEDI (Wu et al., 2021) combines the dual gradient ascent with pessimism to multi-objective tabular MDPs. MO-SPIBB (Thomas al., 2021) achieves safe policy im- provement within a predefined set of policies under con- straints. These methods often require priori knowledge about target preferences. PEDA (Zhu et al., 2023a) first develops D4MORL benchmarks for scalable Offline MORL with a general preference for high- high-dimensional MDPs with continuous states and actions. It also attempts to solve the
MODULI constructs a multi-objective conditional diffusion planning framework that generates long-term trajectories aligned with preference and then executes as planner. Fur- thermore, appropriate guiding conditions play a crucial role in the precise generation, which requires the assessment of trajectory quality under different preferences for normaliza- tion. We found that simply extending the single-objective normalization method to multi-objective causes issues, as it fails to accurately align preferences with achievable high returns. Therefore, we propose two return normalization
2

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Offline MORL via supervised learning, expanding return- conditioned sequential modeling to the MORL,, such as MORvS (Emmons et al., 2021), and MODT (Chen et al., 2021). Lin et al. (2024) integrates the MORI. methods to alleviate the sreference-inconsistee demonstration problem. This paper investigates the general- ization problem of OOD preference, aiming to achieve better generalization through more expressive diffusion planners and the guidance of transition directions by slider adapter.
of the true one. While not knowing the true front, we can define set of metrics for relative comparisons among algo- rithms, e.g., hypervolume, spprrss and return deviation, to which we give a further explanation in Section 5.
into
Denoising Diffusion Implicit Models (DDIM) Assume the random variable x E R follows an unknown distribu- tion (x DMs define a forward process (x* hejo, by the noise schedule {as, reEeoo.TT s.t., Vt E [0, T):
dx' = f(t)x'd dt + g(t)dw', x² ~ go(x"),
(1)
Diffusion Models for Decision Making Diffusion mod- els (DMs) (Ho al., 2020; Song al., 2021b) have be- come mainstream class of generative models. Their re- markable capabilities in complex distribution modeling demonstrate outstanding performance across various do- mains (Kingma et al., 2021; Ruiz et al., 2023), inspiring research to apply DMs to decision-making tasks (Chi et al., 2023; Yang et al.. 2024; Xian al., 2023). Applying DMs for decision-making tasks can mainly be divided into three categories (Zhu et al., 2023b): generating long horizon trajectories like planner (Janner et al., 2022; Dong et al., 2024a; Ajay et al., 2023), serving as more expressive mul- timodal polices (Wang et al., 2023; Gansen-Estruch et al., 2023; Yuan et al., 2023) and performing as the synthesizers for data augmentation (Lu et al., 2024). Diffusion models exhibit excellent generalization and distribution matching abilities (Kadkhodaie et al., 2024; Li et al., 2024b), and this property has also been applied to various types of decision- making tasks such as multi-task RL (He et al., 2024), meta RL (Ni et al., 2023) and aligning human feedback (Dong et al., 2023). In this paper, we further utilize sliding guid- ance to better stimulate the generalization capability DMs, rather than merely fitting the distribution by data memoriza- tion (Gu et al., 2023).
where w' E RR. is the standard Wiener process, and f(t) alaga gt) do?
di 20? The SDE for- ward process in Equation (1) has an probubility flow ODE (PF-ODE) reverse process from time T to 0 (Song et al., 2021b):
dx'
di - 1()x( 1 (1V log xt or (x("));
(2)
where the score function V log (x') is the only unknown term. Once we have estimated it, we can sample from qu (x²") by solving Equation (2). In practice, we train a neu- ral network cs(x", () to estimate the scaled score function of V x log (x²)) by minimizing the score matching loss:
(0) (O,T) .) go(== _o00)). [Ilea(x', t) E|
Since c, t) can be considered as a predicted Gaussian noise added to x'' it is called the noise prediction model. With a well-train noise prediction model, DDIM (Song et al., 2021a) discretizes the PF-ODE to the first order for solving, i.c., for each sampling step:
xx - 0150(2 ,8) : + 0,60(=",s)
x² = at
(4)
3. Preliminaries
holds for any 0 S t < < T. To enable the diffusion model to generate trajectories based on specified properties, Classifier Guidance (CG) (Dhariwal & Nichol, 2021) and Classifier-free Guidance (CFG) (Ho & Salimans, 2022) are the two main techniques. We use CFG due to its stable and easy-to- to-ttiin properties. CFG directly uses a conditional noise predictor to guide the solver:
Multi-objective RL The MORL problem can be for- mulated as a multi-objeccive Markov decision process (MOMDP) (Chatterjee et al., 2006). MOMDP with n objectives can be represented by a tuple (S, R. 22,,) where S and A demote the state and action spaces, P is the transition function, R S x A + R²" is the reward function that outputs n--dim vector rewards r = R(s, a), Ω is the preference space containing vector preferences w, and y is a discount factor. The policy æ is evaluated for m distinct preferences { resulting policy set be represented as { where Tp Taa 8, wp),, and Fh = Ea,~,, EE RRs,,, a:)] is the corresponding un- weighted expected return. We say the solution TD is domi- nated by 4 when G.' < G for Vi E [1, 2, .... mn.. The Pareto front P of the policy π contains all its solutions that are not dominated. In MORL, we aim to define a policy such that its empirical Pareto front is a good approximation
Eg (a",, f, c) == w Eg (2. c, + (1 w)-
ca(x",,) (.w))aaaaa
(5)
where w is used control the guidance strength.
Guided Sampling Methods To enable the diffusion model to generate trajectories based on specified proper- ties, Classifier Guidance (CG) (Dhariwal & Nichol, 2021)
'To ensure clarity, we establish the convention that the su- perseript denotes the diffusion timestep, while the subscript t represents the decision-making timestep.
3

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
and Classifier-free Guidance (CFG) (Ho & Salimans, 2022) the two main techniques. CG requires training addi- tional classifier log logpo(clas Po(cla' t) to predict the log probability of exhibiting property c. During inference, the gradient of the classifier is used to guide the solver:
(()) = E -ttp:/aaaa-...//))-------- [Ileo(a", (. g) <112]
MODULI uses the CFG guidance for conditional genera- tion. During training, MODULI learns both conditioned noise predictor cø(x', t, w. g) and an unconditioned one ca (ax (,,2) 2). We adopt a masking mechanism for training, with a probability 0 < p < to zero out the condition of trajectory, equivalent to an unconditional case. We employ DiTid (Dong et al., 2024b) network architecture instead of UNet (Ronneberger et al., 2015), which has been proven to have more precise generation capabilities. MODULI also employs a loss-weight trick, setting a higher weight for the next state 81 of the trajectory x² a²(rr encourage more fo- cus on it, closely related to action execution. For details on the loss-weight trick, please refer to Appendix F.
Ea(x", c) = Es(2", t)
or V VaCa( t),
(6)
on the contrary, CFG directly uses a conditional noise pre- dictor to guide the solver:
Co(2", t, c)
co(a", 1, e) (1 (1-ww) w) co(a", f); (7)
where w is used to control the guidance strength.
4. Methodology
To address the challenges of expressiveness and generaliza- tion in Offline MORI., we propose novel Offline MORI. framework called MODULI. This framework utilizes con- ditional diffusion models for preference-oriented trajectory generation and establishes a general policy suitable for a wide range of preferences while also generalizing to OOD preferences. MODULI first defines a conditional genera- tion and planning framework and corresponding preference- return normalization methods. Moreover, to enhance gen- eralization capability, we design sliding guidance mecha- nism, where a slider adapter actively adjusts the data distri- bution from known ID to specific OOD preference, thereby avoiding simple memorization.
4.2. Refining Guidance via Preference-Return Normalization Methods
Global Normalization To perform generation under dif- ferent scales and physical quantities of objectives, it is es- sential to establish appropriate guiding conditions, making return normalization a critical component. We first con- sider directly extending the single-objective normalization scheme the multi-objective case, named Global Normal- ization. We normalize each objective's ' RTG using min-max style, scaling them to a unified range. Specifically, we cal- culate the global maximum RTG g gmaa = [gg for each objective in the offline dataset. The same applies to the minimum RTG gmin, The conditioned RTG g are first normalized to (g - g min) /g max - mii) E [0. 1]", then con- catenated with the preference w as inputs. During deploy- ment, aim to generate xTT) to align the given preference warget and maximized scalarized return using the highest RTG condition g = 1". Note that we compute the global extremum values based on the offline dataset with differ- ent preferences, which are not necessarily the reachable min/max value for each objective. Also, some objectives are inherently conflicting and can't 't maximized at the same time. As result, direct guidance using the target conditions of y = [Wrrrget, 17] is unachievable for some preferences, leading to performance drop. To address these issues, we match preferences with vector returns and then propose two novel normalization methods that adapt to multi-objective scenarios to achieve preference-related return normalization. providing more refined guidance: Preference Predicted Nor- malization and Neighborhood Preference Normalization:
4.1. Offline MORL as Conditional Generative Planning
We derive policies on the Offline MORL dataset D with trajectories 7 = (w, 80, ag; To, 8T-1, 4T-1, TT-1) to respond to arbitrary target preferences w E Ω during the deployment phase. Previous Offline MORL. (Zhu et al., 2023a) algorithms generally face poor expressiveness, un- able to adapt to a wide range of preferences through gen- eral
policy, resulting in sub-optimal approximate Pareto front. Inspired by the excellent ex- pressiveness of diffusion models in decision making (Ajay et al., 2023; Dong et al., 2023), MODULI introduces diffu- sion models model the policy. We define the sequential decision decision-making problem as a conditional generative ob- jective max, ErrD [log pa (r) y(])]], where 0 is the trainable parameter. The diffusion planning in MORL aims to generate sequential states
(r) = [sp.... ===1] with horizon H to satisfy the denoising conditions y = [w> g]. where w a specific preference. Similar to Zhu et al. (2023a), 9: is defined as the vector-valued Returns-to- Go (RTG) for single step, i.e. g: = Σ. R(st,a)) The trajectory RTG is the average of the RTGs at each step.
Preference Predicted Normalization To avoid the issue in Global Normalization, we hope to obtain correspond- ing RTG conditioned on the given preference. The dataset contains a discrete preference space and it is not possi-
Training Next, we can sample batches of (7) and y from the dataset and update pa as the modified score match- ing objective in Equation (3):

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
ble to directly obtain the maximum RTG for each prefer- ence. Therefore, we propose Preference Predicted Normal- ization training a generalized return predictor additionally gm (w) = (w) to capture the maximum expected re- turn achievable for a given preference w. Specifically, we first identify all undominated solution trajectories Tp E Dp. where Dp C D and P denotes Pureto front. Then the gen- eralized return predictor is trained by minimizing the return prediction error conditioned on the preference w:
extract the action d, to be executed from generated æ. It is worth noting that we fixed the initial state as x* [0] x² [0]
to enhance the consistency of the generated results.
4.3. Enhancing OOD Preference Generalization with Sliding Guidance
After using diffusion models with appropriate normalization, MODULI can exhibit remarkable expressive capabilities, fully covering the ID distribution in the dataset. However, when faced with OOD preferences, tends to conserva- tively generate the trajectory closest to the ID preferences. This may indicate that the diffusion model has overfitted the dataset rather than learning the underlying distribution. To enhance generalization to OOD preferences, inspired by Gandikota et al. (2024) and Du et al. (2020), we pro- pose novel sliding guidance mechanism. After training the diffusion models po. We additionally trained a plug- and-play slider adapter with the same network structure of diffusion models, which learns the latent direction of pref- erence changes. This approach allows for greater control during generation by actively adjusting the target distribu- tion, thereby preventing simple memorization. We define the adapter as pa-. When conditioned on this method boosts the possibility of attribute c+ while reduces the pos- sibility of attribute c_ according to original model pg:
((v) = () [w)) - 91ا2
(9)
Then we calculate normalized RTG for each trajectory& т with preference w as 9- (mmx(()) E [0. 1]". With RTG prediction, we can follow closer the training distribution, guiding the generation by specifying different maximum vector returns for each preference.
Neighborhood Preference Normalization We train (w) using only non-dominated solutions to predict the expected maximum RTG. If the quality of the behavior policy in the dataset is relatively low, the non-dominated solutions in the dataset are sparse, which introduces predic- tion errors that result in inaccurate normalization. Therefore we propose a simpler, learning-free method named Neigh- borhood Preference Normalization. In a linear continuous preference space, similar preferences typically lead to sim- ilar behaviors. Therefore, we can use neighborhood set of trajectories to obtain extremum values, avoiding inac- curate prediction. For a trajectory t and its corresponding preference w,, we define c-neighborhood:
" Pg (c+ a (T))
pe (z) + pa (x (T)
Ph (c_.. (T))
(13)
First, train diffusion model p to generate trajectories corresponding to ID) preferences. Then, we attempt to learn the pattern of change (for example, when the preference shifts from energy efficiency to high speed, the amplitude of the Swimmer's 's movements gradually increases). There- fore, our goal is to learn an adapter p", which applies this pattern OOD preferences achieve better generaliza- tion. In Equation (13), the exponential term represents increasing the likelihood of preference c+ and decreasing the likelihood of preference c-. 17 represents the guidance strength weight in classifier-free guidance for the diffusion model. Based on Equation (13), we can use an adapter-style method to adjust the original noise prediction model (p). Now, the noise prediction combines the noise from the orig- inal model and the adapter. Then, we can derive direct fine-tuning scheme that is equivalent to modifying the noise prediction model. The derivation process is provided in the Appendix D:
(T) = (T' ||0,,- 5
(10)
Therefore, we can use the maximum expected return within the neighborhood trajectories to approximate the maximum expected return, normalized as follows:
9r min (T)
9- max (g).()) min (g))
E {0,, 1], (11)
(B.(T) = [gr | E B.))).
(12)
Overall, we empirically find that the NPN method achieves consistently good performance across datasets of varying quality, making it the preferred method. On the other hand, PPN, which requires training NN estimate the return, performs well only in high-quality datasets. We provide depth analysis of the three normalization methods in the Section 5.3 and Appendix C
ea. (a' c,t) + Eg (x', + (Ea (e',c,,,)) - Ep (af.c_, .c,,t))) t))
(14)
Action Extraction Now, MODULI can accurately gener- ate desired trajectories with arbitrary preference and RTG in the CFG manner according to Equation (7). Then, we train an additional inverse dynamics model a; = h(st, 8141) to
The score function proposed in Equation (14) justifies the distribution of the condition c by converging towards the positive direction e+ while diverging from the negative di-
5

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
rection c_.. Therefore, we can let the 0* model capture the direction of preference changes, that is, we hope that the denoising process of <ā(x", c,t) t) at each step meets the unit length shift, i.e. eg. (x*,e,t) c,t) = [cala' (o
Narrow: Similar to the Shattered dataset, but it mainly evaluates the extrapolation OOD generalization ability. The Narrow dataset is generated by removing portion of the trajectories from both ends of the Complete dataset, resulting in a narrower preference distribution.
In MORL tasks, the condition is w. and ct and c are the changes in preference Aw in the positive and negative direc- tions, respectively. We minimize the loss [(1)) to train the adapter:
We compare MODULI with various strong offline MORL baselines: BC(P), MORvS(P) (Emmons et al., 2021), MODT(P) (Chen et al., 2021) and MOCQL(P) (Kumar et al., 2020). These algorithms are modified variants of their single-objective counterparts and well-studied in PEDA (Zhu et al., 2023a). MODT(P) and MORvS(P) view MORL, as a preference-conditiones sequence modeling problem with different architectures. BC(P) performs imi- tation learning through supervised loss, while MOCQL(P) trains a reeference-conditioned Q-function based on tem- poral difference learning. Further details of datasets and baselines are described in Appendix A.
((0")= B Aw)DD [|eeo-aet w,t) Ieaiinn wutt-ii d)-rale' iiiirr (--Aw
2
(15)
During deployment, when we encounter OOD preferences WOOD. we can first transition from the nearest IDD prefer- ence WID. Then, we calculate Aw- WOOD WID, and at each denoising step, simultaneously apply the diffusion model and adapter E + Aweg., so that the trajectory distribution gradually shifts from ID preferences to OOD preferences to achieve better generalization. We provide the pseudo code for the entire training and deployment process in Appendix E.
Metrics We use two commonly used metrics HV and SP for evaluating the empirical Pareto front and an additional metric RD for evaluating the generalization of OOD pref-
5. Experiments
erence regions: Hypervolume (HV) (P) measures the space enclosed by solutions in the Pareto front P: H(PP)
We conduct experiments on various Offline MORL tasks to study the following research questions (RQs): Expressive- ness (RQ1) How does MODULI perform on Complete dataset compared to other baselines? Generalizability (RQ2) Does MODULI exhibit leading generalization perfor- mance in the OOD preference of Shattered or Narrow datasets? Normalization (RQ3) How do different normal- ization methods affect performance? All experimental de- tails of MODULI are in Appendix B.
Je (=(P)(z)dz, where H(P) = {z E RT Bi 1 Pl To 3 x 3 P(i)}. To is reference point determined by the environment, 3 is the dominance relation operator and 1 ((P)) equals if x E (P) and 0 otherwise. Higher HV is better, implying empirical Pareto front expansion. Spar- sity (SP) S(P) measures the density of the Pareto front P: S(P)= - E. - - P( + 1))², where P. represents the sorted list of values of the i-th target in P and Pk())) is the k-th value in P,. Lower SP, indicat- ing a denser approximation of the Pareto front, is better. Return Deviation (RD) R(O) measures the discrepancy between the solution set O in the erence region and the maximum predicted value: R(O)- To E [0(k) - R. (WO(R)) ll2, where RS is pretrained generalized return predictor mentioned in Section 4.2, (O)) is the corresponding preference of O Lower RD is bet- ter, as it indicates that the performance of the solution is closer to the predicted maximum under OOD preference, suggesting better generalization ability. Following the Zhu et al. (2023a), we uniformly select 501 preference points for 2obj environments and 325 points for 30bj environments. For each experiment, we use 3 random seeds and report the mean and standard deviation.
Datasets and Baselines We evaluate MODULI on DAMORL (Zhu et al., 2023a), an Offline MORL benchmark consisting of offline trajectories from multi-objective Mu- JoCo environments. D4MORL contains two types of quality datasets: Expert dataset collected by pre-trained expert behavioral policies, and Amateur dataset collected by per- turbed behavioral policies. We named the original dataset of D4MORL as the Complete dataset, which is used to verify expressiveness ability to handle multi-objeccivv tasks. Additionally, to evaluate the algorithm's 's generaliza- tion ability to -itt---iiitiibtiion (OOD) preferences, similar to Figure 1, we collected two types of preference-deficient datasets, namely Shatteres and Narrow. We refer to the Appendix A.2 for the dataset generation.
pref-
Simulates incomplete preference distribu- tion by removing part of the trajectories from the Com- plete dataset, creating regions with missing preferences. It primarily evaluates the interpolation OOD generalization ability, assessing how well the algorithm generalizes when encountering unseen preferences with some missing.
Shattered:
5.1. Evaluating Expressiveness on Complete Datasets
We first compare MODULI with various baselines on the Complete datasets. In this dataset version, the trajectories have comprehensive coverage preferences, and the policy needs sufficient expressive capability to cover complex dis-
6

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Table HV and SP performance on full Comp1 ete datasets. (B: behavioral policy).
The hest scores emphasized in hold.
Env
Metries
B
MODT(P)
MORvS(P)
BC(P)
CQL(P)
MODULI
HV (x109) f
5.61
5.924.04
6,07+.02
4.372.06
5,624.23
6,082.03
Ant
SP (x10*)) +
8.72+.77
5.24+.52
25.90+16.40
1.064.28
0.53+.05
HV (x
5.68
5.69+.01
5.77+.00
5.46+.02
5.54+.02
5.76+.00
HalfCheetah
SP (x100)
1.162.42
0.57#.09
2.222.91
0.45a.27
0.072.02
HV (x107) f SP (x
1.97
1.81..05
1.762.03
1.352.03
1.64+.01
2.01#.01
Hopper
Amateur
1.61+.29
3.50+1.54
2.42+1.08
3.30+5.25
0.10#.01
HV (x10*)
2.11
1.67+.22
2.79#.03
2.82+.04
1.694.93
3,20#.00
Swimmer
SP (x10))
2.87+1.32
1.03#.20
5.05+1.82
8.87+6.24
9.50+.59
HV (x10%)
4.99
3.10+.34
4.98+.01
3,42+.42
1.78
5.06t.00
Walker2d
SP (x11))
164.20#13.50
1.94+.06
53.10=34.60
7.33+5.89
0,25+.03
HV (
3.09
1.04..16
2.77+.24
2.424. 18
0.59+.42
3,33+.066
Hopper-3obj
SP (x100)
10.23+22778
1.03#.11
0.87+.29
2,00+1.72
0.10+.00
HV (x10%)
6.32
6.214.01
6.362.02
4.88
5.762.10
6,39a.02
Ant
SP (x10))
8.26+2.22
0.87+.19
46.20#1 16.40
0.58+.10 10
0.79
HV 1x
5.79
5.73+.00
5.78+.00
5.54+.05
5.63+.04
5.79:.00
HalfCheetah
SP (x10))
1.244.23
0.67#.05
1.78*.39
0.10+.00
0,07
HV (x107) t
2.09
2.00#.02
2.02+.02
1.234.10
0,334.39
2.09+.01
Hopper
Expert
SP (x110))
16.30+10.60
3.03+.36
52.50+4.88
2.8442.46
0.09+.01
HV (x10*) f SP (x10))
3.25
3.24+.00
3.21+.00
3.224.08
3.24+.00
Swimmer
15.00#7.49
4.39
4.50+.39
13.6045.31
4.43+.38
HV (x10%)
5.21
4.89+.05
5.142.01
3.74
3.21#.32
5.20+.00
Walker2d
SP 5x110)
0.99+.44
3,224.73
75.60+52.30
6,23
0.11#.01
HV (x1019)+ +
3.73
3.38+.05
3.424.10
2.29+.07
0.78+.24
3.57+.02
Hopper-3obj
SP (x10*)
1.40 44
2.72+1.93
0.72+.09
2.60+3.144
0.07+.00
HalfCheetah-amateur
Swimmer-amateur
competitive with behavioral policies due to the poor expres- siveness of the context policy. Overall, MODULI achieves the most advanced policy performance according to HV, and the most dense solution with the lowest SP. We visualize the empirical Pareto front in Figure 2, demonstrating that MODULI a good approximator under different data qual- ities. In the expert dataset, MODULI accurately replicates the behavioral policies. In the amateur dataset, due to the inclusion of many suboptimal solutions, MODULI signifi- cantly expands the Pareto front, demonstrating its ability to stitch and guide the generation high-quality trajectories.
2000
Objeecivee 2
20e0
-180
HalfCheetah-expert
Swimmer-expert
2980
Objjective
1000
.500
2000
2004
Objective
5.2. Evaluating Generalization on Shattered and Narrow Datasets
Figure 2. Approximated Pareto fronts Complete dataset MODULI. Undominated / Dominated solutions are colored in red / blue, and dataset trajectories are colored in grey.
Performance We evaluate the generalization performance on the Shattered and Narrow datasets. Table shows the comparison results on the expert datasets, where MOD- ULI still demonstrates the best performance across all base- lines. Specifically, MODULI achieves significant improve- ments on the new RD metric, indicating better generalization capability and better approximation of the Pareto front in OOD preference regions. We further visualize the powerful OOD preference generalization ability of MODUL.I
tributions under wide range of preferences. As shown in Table 1, most of the baselines (MOCQL, MOBC. MODT) exhibit sub-optimal performance, fail to achieve HV close to behavioral policies, and are prone to sub-preference space collapses that do not allow for dense solutions. MORvS can exhibit better HV and SP but fails to achieve performance

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Table HV, SP and RD Shattered/ Naw datasets. The best scores are emphasized in bold. See Appendix G for full results.
Shattered
Narrow
Env
Metries
(ODT(P)
MORvS(P)
MODULI
MODT(P)
MORvS(P)
MODULI
HV (x10") f
5.88t.02
6.38+.01
6.392.01
5.05#.08
6.06#.000
6.362.(0)
Ant
RD (x 1x10 4
0.71
0.16+.00
0.142.01
0.98#.01
0.37+.03
0.252.01
(x10%)
2.224.50
0.75+
0.53+.13
0.88#.41
0.80#.11
0.80=.25
HV (x10") f
5.69
5.73+.00
5.78+.00
5.04#.00
5.46#.01
5.762.00
HalfCheetah
RD (x10%) (x
0.184.00
0.10+.00
0.07+.00
0.43#.00
0.29+.00
0.17=.00
0.184.01
0.15a.02
0.11
0.04
0.05a.00
0.04
HV (x10") f
1.95..03
2.06+.01
2.07+.00
1.85#.02
1.98+.00
2.042.01
Hopper
RD (x10%)
1.42+.00
1.10#.04
0.22:.06
4.24#.02
1.37+.05
2.42+005
SP (x1000))
0.85
0.20#106
0.11#.01
0.34
0.16#.02
0.25+.05
HV (x10") f
3.20+.00
3.24+.00
3.24+.00
3.03#.00
3.10+.00
3.21+.00
Swimmer
RD (x 4
0.38
0.16+.00
0.062..0
0.38*.00
0.39+.01
0.10=.00)
(x
15.50#.3
7.36#
5.792.43
3.65#.22
4,38+.80
3.28
HV (x10") t
5.01+.01
5.14+.01
5.20+.00
4.75#.01
4.854.01
5.10=.01
Walker2d
RD (x10))
0.862.01
0.65+.02
0.15#.01
1.07=.01
1.45+002
0.28
SP (x
0.49
0.27+.04
0.12+.01
0.34
0.19a.02
0.13#.01
HV (x10"9) f
2.83+.06
3.28+.07
3.43+.02
3.18#.05
3.32+.00
3.37=.05
Hopper-3obj
RD (x10))
1.60+.01
1.60+.02
1.28t.03
2.48#.00
2.24#.02
2.38
SP 4
0.09
0.062.00
0.10+.01
0.1
0.22+.02
0.12:.01
MODUU
NORVSIPI
BC(P)
chart in Figure We found that in most tasks, MODULI with slider exhibits higher HV and lower RD, indicating leading performance in MO tasks. At the same time, the presence or absence of the slider has little impact on SP performance, indicating that sliding guidance distribution does not impair the ability to express the ID preferences. Besides, we visualized the empirical Pareto front in Fig- ure for mechanism significantly extends the Pareto of OOD preference regions, exhibiting better generalization. Due to space limits, we provide more comparative experiments on the different guidance mechanisms (CG, CG+CFG) in Appendix H.
dataset. The sliding bootstrap
126
Objective
200
Objective
Objective
Figure Solution (Blue points) under OOD preference for differ- ent algorithms in Walker2d-exper--S55 Shattered (Top) and Hopper- amateur-Narrow (Bottom). Dataset trajectories are colored in grey.
HV
RD
SP
Sumeera
slider
OOD Generalization Visualization In Figure 3, we visu- alize the actual solutions obtained under OOD preferences. Gray points represent the trajectories in the dataset, where the preference of the dataset is incomplete. We observe that BC(P) exhibits policy collapse when facing OOD prefer- ences, showing extremely low returns and mismatch with the preferences. RvS(P) demonstrates limited generalization ability, but most solutions still do not meet the preference requirements. Only MODULI can produce correct solutions even when facing OOD preferences. In the Shattered dataset, MODULI can patch partially OOD preference gions, while in the Narrow dataset, MODULI can slightly extend the Pareto front to both sides.
Peapese
No tr/pp
Figure 4. Comparison HV. SP, and RD performance of MOD- ULI with and without slider across various environments. All metrics are rescaled by dividing by their maximum values. The suffix "-e" denotes Expert and "-a" denotes Amateur.
5.3. Comparison of Normalization Methods
We compared the impact of different return regularization methods on MODULI's performance on the Complete dataset. As shown in Table 3, Neighborhood Preference Normalization (NP Norm) consistently has the highest HV performance. On the expert dataset, both NP Norm and
Ablation of Sliding Guidance We conducted an ablation study of the sliding guidance mechanism on 9 tasks within Na rrow datasets. The results are presented in the radar
8

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Hopper-amateur-Narrov
ducted in low-dimensional state space and continuous action space. We hope to extend it to multi-objective tasks with image inputs, further enhancing its generalization ability, We also believe that integrating evolutionary reinforcement learning (Li et al., 2024c;a; Hao et al., 2023) is a possi- ble direction for future extension. Additionally, nonlinear preference spaces may present new challenges.
w/o Slider
with Slider
Objective ?
4500
#300
3300
2050
2200 Objective 1
2000
2502
2000
Impact Statement
Objective
This paper investigates Offiie MORL algorithms, focusing on their potential to address real-world problems where si- multaneous optimization of multiple conflicting objectives is required and data collection is limited or expensive. By advancing Offline MORL methods, this work contributes to safer, more efficient, and practical deployment of rein- forcement learning in fields such healthcare, autonomous driving, and resource management, where learning from offiin data is crucial. The findings and methodologies pre- sented in this chapter are expected to inspire further research and facilitate the adoption of MORL in complex decision- making scenarios.
Figure 5. The Pareto front illustrations for MODULI with and without the Slider, generated on the Hopper-Amat eur ur-Narrow dataset, demonstrate that the inclusion the Slider significantly extends the boundary of the policies.
PP Norm exhibit very similar HV values, as they can ac- curately estimate the highest return corresponding to each preference, whether through prediction or neighborhood analysis. However, on the amateur dataset, PP Norm fails to predict accurately, resulting in lower performance. We also noted that the naive method of Global Norm fails to approx- imate the Pareto front well, but some datasets, it achieved denser solutions (small SP). We believe this is because the Global Norm produced inaccurate and homogenized guid- ing information, leading to different preferences for similar solutions, and resulting in dense but low-quality solutions. We refer to Appendix C for in-depth visual analysis.
A Acknowledge
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132), We would like to thank all the anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved the quality of this paper.
Table Comparison of normalization on Complete ete datasets. (NP: Neighborhood Preference, PP: Preference Predicted)
Euv
Metrics
NP Norm
Norm
Global
Norm
HV (x10')
1982.02
1.92.01
1.58..02
References
Amanzur
Hopper
(x
0.21+:06
0.24:.06
0.12+.03
Walker2d
HV (x10P)
5003.000
4.942.00
4,02+.01
Ajay, A., Du Y., Gupta, A., Tenenbaum, B., Jaakkola, T. S., and Agrawal, P. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Leurning Representations, ICLR, 2023.
SP (xx10)
0.20+001
0.31+.01
0.04x.01
(x10")
2.07
2.07=:01
1.48+.02
Exper
Hopper
SP (x10")
0,144.04
0.124.02
2.69&1.97
Walker2d
HV (x:10))
5.20
5.18+.00
2,70+.04
(x119)
0.11+:01
0.17#.01
1.1626883
Alegre, L. N. Towards sample-efficient multi-objective reinforcement learning, In AAMAS, pp. 2970-2972 2023.
6. Conclusion
In this paper, we propose MODULI, diffusion planning framework with two multi-objective normalization schemes, enabling preference-retum condition condition-guided trajectory gen- eration for decision- mariiii.nnmmiin........ To achieve better generaliza- tion in OOD preferences, MODULI includes a sliding guid- ance mechanism, training an additional slider adapter for active distribution adjustment. We conduct extensive ex- periments on Complete, Shattered, and Narrow datasets, demonstrating the superior expressive and generalization ca- pabilities of MODULI. The results show that MODULI is an excellent Pareto front approximator, capable of expanding the Pareto front and obtaining advanced, dense solutions. However, there are limitations, such as experiments con-
Alegre, L. N., Bazzan, A. L., Roijers, D. M., Nowé, A., and da Silva, B. C. Sample-efficient multi-objective learning via generalized policy improvement prioritization. AA- MAS, 2023.
Bae, J., Kim, J. M., and Lee, S. J. Deep reinforcement learning for a multi-objective operation in nuclear power plant. Nuclear Engineering and Technology, 55(7))3277~ 3290, 2023.
Chatterjee, K., Majumdar, R., and Henzinger, T. A. Markov decision processes with multiple objectives. In STACS 2006: 23rd Annual Symposium on Theoretical Aspects

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
of Computer Science, Marseille, France, February 23-25, 2006. Proceedings 23, pp. 325-336. Springer, pprinee, 2006.
Hansen-Esstuuch P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implieit q-learning actor- critic method with diffusion policies. arXiv preprint arXiv:2304.1033 2023.
Chen, L.,, Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15004-15555 15097, 2021.
Hao, J., Li, P., Tang, H., Zheng, Y., X. Fu, and Meng, Z. Erl-re?: Efficient evolutionary reinforcement learning with shared state representation and individual policy representation. ICLR, 2023.
Chi, C., Feng, S., Du, Y. Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems XIX, 2023.
Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond. M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., al. A practical guide multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
Cui, H., Yuan, Y., Zheng, Y., and Hao, J. Aharobot: A low- cost open-source bimanual mobile manipulator for em- bodied ai, 2025. URL https: /ete://arra /arxiv.viv..///.i//.a.aa/aaaavva org/ abs/ 2503. 10070.
He, H., Bai, C., Xu, K., Yang, Z., Zhang, W., Wang, D., Zhao, B., and Li, X. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, NIPS, 36, 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Dong, Z., Yuan, Y., HAO, J., Ni, F., Mu, Y., ZHENG, Y., Hu, Y., Lv, T., Fan, C., and Hu, Z. Aligndiff: Aligning diverse human preferences via behavior-customisable dif- fusion model. In The Twelfth International Conference on Learning Representations, ICLR, 2023.
Ho, and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Dong, Z., Hao, J., Yuan, Y., Ni, F., Wang, Y., Li, P., and Zheng, Y. Diffuserlite: Towards real-time diffusion plan- ning. NeurlPS, 2024a.
Hung. W. Huang, B.-K., Hsieh, P.-.,, and Liu, X. Q- pensieve: Boosting sample efficiency of multi-objective rl through memory sharing of q-snapshots. The Tenth International Conference on Learning Representations, 2022.
Dong, Z., Yuan, Y. Hao, Ni, F., Ma, Y., Li, P., and Zheng, Y. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. NeurlPS, 2024b.
Janner, M., DU, Y., Tenenbaum, J., and Levine, S. Planning with diffusion for flexible behavior synthesis. In Proceed- ings of the 39th International Conference on Machine Learning, ICML, 2022.
Du, Y., Li, S., and Mordatch. I. Compositional visual gen- eration with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
Kadkhodaie, Z., Guth, F., Simoncelli, E. P., and Mallat, Generalization in diffusion models arises from geometry- adaptive harmonic representation. The Twelve Interna- tional Conference on Learning Representations, ICLR, 2024.
Emmons, S., Eysenbach, B., Kostrikov, I,, and Levine, S.
Rvs: What is essential for offline rl via supervised learn- ing? ICLR, 2021.
Felten, F., Alegre, L. N., Nowe, A., Bazzan, A., Talbi, G., Danoy, G., and C da Silva, B. A toolkit for reliable bench- marking and research in multi-objective reinforcement learning. In Advances in Neural Information Processing Systems, 2024.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-2111 21707, 2021.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
Gandikota, R., Materzynska, J., Zhou, T., Torralba, A., and Bau, D. Concept sliders: Lora adaptors for precise control in diffusion models. ECCV, 2024.
Gu, X., Du C., Pang, T., Li, C., Lin, M., and Wang, Y. On memorization in diffusion models. arXiv preprint arXiv:23 10.02664, 2023.
Li, C. and Czarnecki, K. Urban driving with multi- objective deep reinforcement learning. arXiv preprint arXiv: /8/ /. 1.85586, 2018
10

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Li, P., Jianye, H., Tang, H., Zheng, Y., and Barez, Z. Value- evolutionary-basel reinforcement learning. In ICML, 2024a.
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley,
A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Li, P., Li, Z., Zhang, H., and Bian, J. On the generaliza- tion properties of diffusion models. Advances in Neurul Information Processing Systems, 36, 2024b.
Roijers, D. M., Whiteson, S.. Brachman, R., and Stone, P.
Multi-objeccivv decision making. Springer, 2017.
Li, P., Zheng, Y., Tang, H., Fu, X., and Hao. J. Evorainbow: Combining improvements in evolutionary reinforcement learning for policy search. In ICML, 2024c.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In MICCAI, 2015.
Lin, Q. Yu, C.. Liu, Z., and Wu, Z. offline multi-objective reinforcement learning. AAMAS, 2024.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 22500-22511 2023.
Liu, C., Xu, X., and Hu, D. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385- 398, 2014.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. International Conference on Learning Representations, 2021a.
Liu, J., Yuan, Y., Hao, J., Ni, F., Fu, I.., Chen, Y., and Zheng, Y. Enhancing robotic manipulation with ai feedback from multimodal large language models, 2024. URL https: //tpxvaaavvv.//aa//eaa wttesiiirrr//eeo org/abs//// 2402. 14245.
Song. Y., Sohl-Dickstein, J.. Kingma, D. Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 2017.
Thomas, P. S., Pineau, J., Laroche, R., et al. Multi-objective spibb: Seldonian offline policy improvement with safety constraints in finite mdps. Advances in Neural Informa- tion Processing Systems, 34:2004-2017, 2021.
Lu, C., Ball, P., Teh, Y. W., and Parker-Holder, J. Syn- thetic experience replay. Advances Neural Information Processing Systems, NIPS, 36, 2024.
Lu, H.. Herman, D., and Yu, Y. Multi-objective reinforce- ment learning: Convexity, stationarity and pareto optimal- ity. In ICLR, 2023a.
Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. The Eleventh International Conference on Learning Representations, ICLR, 2023.
Lu, H., Herman, D., and Yu, Y. Multi-objective reinforce- ment learning: Convexity, stationarity and pareto opti- mality. In The Eleventh International Conference Learning Representations, 2023b.
Wu, R., Zhang, Y., Yang. Z., and Wang. Z. Offline con- strained multi-objective reinforcement learning via pes- simistic dual value iteration. Advances in Neural Infor- mation Processing Systems, 34:25439-25444 2021.
Misra, D. Mish:
A self regularized non-monotonic neural activation function. arXiv preprint 2019.
Xian, Z., Gkanatsios, N., Gervet, Ke, T.-W., and Fragki- adaki, K. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
Natarajan, S. and Tadepalli, P. Dynamic preferences in multi-criteria reinforcement learning. In ICML, 2005.
Ni, F., Hao, J., Mu, Y., Yuan, Y., Zheng, Y., Wang, B., and
Xu, J., Tian, Y., Ma, P., Rus, D., Sueda, S., and Matusik, W. Prediction-guice guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, 2020.
Liang, Z. MetaDiffuser: Diffusion model as conditional planner for offline meta-RL, In Proceedings of the 40th International Conference Machine Learning, ICML. 2023.
Yang, R., Sun, X., and Narasimhan, K. A generalized al- gorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information pro- cessing systems, 32, 2019.
Nichol, A. Q and Dhariwal,
P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. PMLR, 2021.

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Yang, S., Du, Y., Ghasemipour, S. K. S., Tompson, J., Kael- bling, L. P., Schuurmans, D., and Abbeel, P. Learning interactive real-world simulators. In The Twelfth Inter- national Conference on Learning Representations, ICLR, 2024.
Yuan, Y., Hao, Ni, F., Mu, Y., Zheng, Y., Hu, Y., Liu, J., Chen, Y., and Fan, C. Euclid: Towards efficient unsuper- vised reinforcement learning with multi-choice dynam- ics model, 2023. URL https: /ttp/aaxivvv//..///vva///aa//aaa wrw. abs/ 2210. 00498.
Yuan, Y., Hao, J., Ma, Y., Dong, Z., Liang. H., Liu, J., Feng. Z., Zhao, K., and Zheng, Y. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback, 2024.
Yuan, Y., Cui, H., Chen, Y., Dong, Z., Ni, F., Kou, L., Liu, J., Li, P., Zheng, Y., and Hao, J. From seeing doing: Bridg- ing reasoning and decision for robotic manipulation, 2025. URL https::///// / /aaaiaivvv.../....//.aaa./aavavv org/ abs / 2505. 08548.
Zhou, X., Yuan, Y., Yang. S., and Hao, J. MENTOR: guiding hierarchical reinforcement learning with human feedback and dynamic distance constraint. IEEE Trans. Emerg. Top. Comput. Intell., 92)):192-1306,, 2025.
Zhu, B., Dang, M., and Grover, A. Scaling pareto-efficient decision making via offline multi-objective rl. The Eleventh International Conference on Learning Repre- sentations, 2023a.
Zhu, 7., Zhao, H., He, H., Zhong, Y., Zhang, S., Yu, Y., and Zhang, W. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223, 2023b.
12

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
A. Dataset and Baselines
A.1. D4MORL Benchmark
In this paper, we use the Datasets for Multi-Objective Reinforcement Learning (D4MORL) (Zhu et al., 2023a) for experiments, a large-scale benchmark for Offline MORL. This benchmark consists of offline trajectories from 6 multi- objective MuJoCo environments, including 5 environments with 2 objectives each, namely MO-Ant, MO-HalfCheetah, MO-Hopper, MO-Swimmer, MO-Walker2d, and one environment with three objectives: MO-Hopper-3ohj. The objectives in each environment are conflicting, such as high-speed running and energy saving. In D4MORL, Each environment uses PGMORL's (Xu et al., 2020) behavioral policies for data collection. PGMORL trains a set of policies using evolutionary algorithms to approximate the Pareto front, which can map to the closest preference in the preference set for any given preference. Therefore, in D4MORL, two different quality datasets are defined: the Expert Dataset, which collected entirely using the best preference policies PGMORL for decision making, and the Amateur Dataset, which has a probability P of using perturbed preference policies and a probability 1 - p of being consistent with the expert dataset, with p 0.65. We can simply consider the Amateur Dataset as a relatively lower quality dataset with random perturbations added to the Expert Dataset. Each trajectory in the D4MORL dataset is represented as: [w, 8u, au To. ,87, ar, &r,TT] where 7 is episode length.
A.2. Complete, Shattered and Narrow Datasets
In this section, we provide a detailed description of how three different levels of datasets are collected, and what properties of the algorithm they are used to evaluate respectively:
Complete datasets i.e. the original version of the D4MORL datasets, cover wide range of preference distributions with fewer OOD preference regions. These datasets primarily evaluate the expressive ability of algorithms and require deriving the correct solution based on diverse preferences.
Shattered datasets simulate the scenario where the preference distribution in the dataset is incomplete, resulting in areas with missing preferences. This kind of dataset primarily evaluates the interpolation OOD generalization ability of algorithms, aiming for strong generalization capability when encountering unseen preferences with some lacking preferences. Specifically, the Shattered dataset removes part of the trajectories from the Comples cemplete dataset, creating n preference-missine regions, which account for m% of the trajectories. If the total number of trajectories in the dataset is N, then total of Mack = N m% trajectories are removed. First, all trajectories in the dataset are sorted based the first dimension of their preferences w, then points are selected equal intervals according to the sorting results. Centered on these n points, an equal number of Mack /n trajectories around each point are uniformly deleted. In this paper, m, n are fixed to 30, and 3, respectively.
The role of Narrow datasets is similar to thred datasets but mainly evaluates the extrapolation OOD general- ization ability of algorithms. If we need to remove a total of m% of the trajectories, we can obtain the Narrow dataset with a narrow preference distribution by removing the same amount, m/2%, from both ends of the sorted Complete dataset. In this paper, m is fixed to 30.
A.3. Baselines
In this paper, we compare with various strong baselines: MODT(P) (Chen et al., 2021), MORvS(P) (Emmons et al., 2021), BC(P), and MOCQL(P) (Kumar et al., 2020), all of which are well-studied baselines under the PEDA (Zhu et al., 2023a), representing different paradigms of learning methods. All baselines were trained using the default hyperparameters and official code as PEDA. MODT(P) extends DT architectures to include preference tokens and vector-valued returns. Additionally, MODT(P) also concatenates w to state tokens [s. w) and action tokens [a, w| for training. MORvS(P) and MODT(P) are similar, extending the MIP structure to adapt to MORL tasks. MORvS(P) concatenates the preference with the states and the average RTGs by default as one single input. BC(P) serves as an imitation learning paradigm to train a mapping from states with preference to actions. BC(P) simply uses supervised loss and MLPs as models. MOCQL(P) Q-function. Then, MOCQL(P)
represents the temporal difference learning paradigm, which learns a uses underestimated Q functions for actor-critic learning, alleviating the OOD phenomenon in Offline RI..
13

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
A.4. Evaluation Metrics
In MORL, our goal is train a policy to approximate the true Pareto front. While we do not know the true Pareto front for many problems, can define metrics for evaluating the performance of different algorithms. Here We define several metrics for evaluation based the empirical Pareto front P. namely Hypervolume (HV), Sparsity (SP), and Return Deviation (RD), we provide formal definitions as below:
Definition A.1 (Hypervolume (HV)). Hypervolume H(P) measures the space enclosed by the solutions in the Pareto front P:
H(P) S.-. (2(p)
where = E 1 I I x N P(i)), To is reference point determined by the environment, 3 is the dominance relation operator and 1 (P)) equals if = E H(P) and O otherwise. Higher HV is better, implying empirical Pareto front expansion.
Definition A.2 (Sparsity (SP)). Sparsity S(P)
measures the density of the Pareto front P:
PP-1
S(P)
P Σ
Σ
P.ck-
where P. represents the sorted list of values of the i-th target in P and P.(k)) is the k-th value in P.. Lower SP, indicating denser approximation of the Pareto front, is better.
Definition A.3 (Return Deviation (RD)). The return deviation R(O) measures the discrepancy between the solution set O in the out-of-distriutiion preference region and the maximum predicted value:
101 Σ 10(k) (2((w)))) A=1
R(O) = 10|
where Rs is a generalized return predictor to predict the maximum expected return achievable for a given preference w. To accurately evaluate RD, we consistently train this network Ry using the Complete Exppette-Expert dataset, as shown in Equation (9). In this way, for any preference of missing dataset, Please note that Rg here is only used to evaluate algorithm performance and is unrelated to the Preference Predicted Normalization method in MODULI. Wo(k) is the corresponding preference of Okk). A lower RD is better, as it indicates that the performance of the solution is closer to the predicted maximum under OOD preference, suggesting better generalization ability.
predict and know the oracle's solution.
B. Implementation Details
In this section, we provide the implementation details of MODULI.
We utilize DiT structure similar to AlignDiff (Dong et al., 2023) as the backbone for all diffusion models, with an embedding dimension of 128, attention heads, and 4 DiT blocks. We use a 2-layer MLP for condition embedding with inputs (w.g). For Preference Prediction Normalization, RTG for given preferences. For Neighborhood Preference Normalization, The neighborhood hyperparameter is fixed at 1e-3 3 for 2-obj experiments and 3e 3 for 3-obj experiments. For the slider adapter, we use a DiT network that is the same as the diffusion backbone of the slider. We observe that using the Mish activation (Misra, 2019) is more stable in OOD preference regions compared to the ReLU activation, which tends to diverge quickly. Finally, the model size of MODULI around 11M.
use 3-layer MLP to predict the maximum expected
All models utilize the Adam W (Loshchilov & Hutter, 2017) optimizer with a learning rate of 2e - 4 and weight decay of le - 5. We perform a total of 200K gradient updates with a batch size of 64 for all experiments. We employ the exponential moving average (EMA) model with an ema rate of 0.995.
For all diffusion models, we use a continuous linear noise schedule (Nichol & Dhariwal, 2021) for a and a,, We employ DDIM (Song et al., 2021a) for denoising generation and use a sampling step of 10. For all tasks, we use a sampling temperature of 0.5.
14

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
The length of the planning horizon H of the diffusion model should be adequate capture the agent's behavioral preferences. Therefore, MODULI uses different planning horizons for different environments according to their properties. For Hopper and Walker2d tasks, we use planning horizon of 32; for other tasks, we use planning horizon of 4. For conditional sampling, we use guidance weight w as 1.5.
The inverse dynamic model is implemented as a 3-layer MLP. The first two layers consist of a Linear layer followed by Mish (Misra, 2019) activation and a LayerNorm. And, the final layer is followed by Tanh activation. This model utilizes the AdamW W optimizer and is trained for 500K gradient steps.
For all 2-obj tasks, we evaluate the model on 501 preferences uniformly distributed in the preference space. For all 3-obj tasks, we evaluate the model on 325 preferences uniformly distributed in the preference space. We perform three rounds of evaluation with different seeds and report the mean and standard deviation. Please note that in the Complete dataset, we default to not using the slider adapter for sliding guidance because there is no OOD preference.
We standardized the parameter size of all baseline algorithms to the same magnitude as much as possible. MORvS: 8.55M, MOBC:8 83M, MODT:9.75M.
Compute Resources We conducted our experiments on an Intel(R) Xeon(R) Platinum 817IM CPU @ 2.60GHz processor based system. The system consists of 2 processors, each with 26 cores running at 2.60GHz (52 cores in total) with 32KB of LI, 1024 KB of L2, 40MB of unified L3 cache, and 250 GB of memory. Besides, we use Nvidia RTX3090 GPUs to facilitate the training procedure. The operating system is Ubuntu 18.04.
C. In-depth Analysis of Different Normalization Methods
1.0
0.8-
Objective 2
0.8
PPN
0.2
NPN
Global Norm
Sample target
0.0
Objective
Figure 6. Visualization of trajectory returns for the -aaeerrddd-amate-----ppppeee dataset after applying different normalization methods. (PPN: Preference Predicted Normalization, NPN: Neighborhood Preference Normalization, Sample Target: In the deployment phase, we fix the condition (w>, 1 to guide the generation of high-quality trajectorics.)
As shown in Figure 6, we visualize the trajectory returns of the Walker2d-amateur amateur-Complete dataset after applying different normalization methods. order to achieve refined guided generation, we require consistency between the training objectives and the sampling objectives, meaning the sampling objectives should correspond to high-quality trajectories in the dataset. simple Global Normalization fails achieve this, as it cannot identify a definitive high-return guiding objective that matches the return distribution of dataset. Setting the return to 1 may be unachievable for Global Normalization. On the contrary, Preference Predicted Normalization (PPN) and Neighborhood Preference Normalization (NPN) successfully normalize the trajectories in the dataset into distinctly different trajectories. High-quality trajectories that conform to preferences are clustered around 12, and trajectories points farther from 1 represent lower vector returns. Note that at this time, the training objective and the sampling objective are fully aligned. Compared with NPN, the PPN method uses neural networks to predict the maximum expected RTG for each preference, facing prediction errors in low-quality datasets (e.g. amateur
15

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
dataset). After PPN process using 9- 9- ( a small number of trajectories even slightly exceed 1", indicating that (w) fails accurately predict the maximum expected return. This ultimately causes the guidance target 1 used in the deployment phase to mismatch the target corresponding to the high trajectory return in training, resulting in suboptimal policy. The NPN method can achieve accurate normalization for multi objettvbee targets, significantly distinguishing the quality of trajectories in the dataset, while ensuring that the sampling targets during the training phase and the deployment phase are consistent.
D. Proofs for Sliding Guidance
In this section, provide derivation process of the sliding guidance mechanism, which enables controlled continuous target distribution adjustment. Given trained diffusion model po and condition and define the adapter par, our goal is to obtain H that increase the likelihood of attribute while decrease the likelihood of "opposite" attribute c_ when conditioned on c:
Po (T) + po (2) po- (c+ 1: (T)) po- (c_ (T))
(1)
where pa (+) c) represents the distribution generated by the original diffusion model when conditioned on According the Bayesian formula p (c+ xz (T) r(="(r))c+ )p(e)) the gradient of the log probability
((x
Po- (2+)2 Po (e (+))
7
v log po (T) c))
can be expanded to:
pa- (x" c+ (4))((c)) po- (x) c- p-)p(c)))
V log (po (z)
(2)
which is proportional to:
v log Pe (z) ( VVogoe- pa- (z) c4) Vlog pg- (x)())
Based the reparametrization trick of (Ho et al., 2020), introduce time varying noising process and express each score (gradient of log probability) denoising prediction (æ', t). thus Equation (3) is equivalent to modifying the noise prediction model:
co (x',c,t) + eo c,t) + 17 (co- (x',c+,0) eg. (e',c_.t)
(4)
16

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
E. Pseudo Code
In this section, we explain the training and inference process of MODULI in pseudo code. MODULI first trains a standard diffusion model for conditional generation; then, we freeze the parameters of this model and use an additional network, the slider, to learn the latent direction of policy shift. Finally, we jointly use the original diffusion and the slider to achieve controllable preference generalization. The pseudo code given below.
Training Process of MODULI model and the training of the slider adapter, which are described in Algorithm and Algorithm 2, respectively. Given a normalized dataset D containing trajectories of states 8, normalized RTG g and preferences w. MODULI first trains a diffusion planner θ:
The training process of the MODULI consists of two parts: the training of the diffusion
Algorithm Diffusion Training
Require: normalized dataset Dg. diffusion model 0 while not done do
(x w.g) ~ Dg
Uniform (1. TT) (0, I)
((0) Exe (..0)-=1--//077/.-000.0. [||ea(") (, - -E112)
Update 0 to minimize [(0)
end while
After training the diffusion model freeze its parameters prevent any changes when training the slider. At each step, sample random preference shift from uniform distribution Aw Uniform({ -A max)) }), where Aw is preset maximum preference shift to avoid sampling ood preference. In implementation, Aw is set to le - 3. We then use the diffusion model to sample a pair of predicted noises (eø(x", it, w + Aw, g), <s(x", ft w Aw, g)) whose preferences are positively and negatively shifted, and divide them by the shift value of the preference 2Aw to represent the "noise shift caused by unit preference shift". The slider model is used to explicitly capture this shift. The specific training process and gradient target are detailed below:
Algorithm 2 Slider Training
Require: normalized dataset Dg, pretrained diffusion model 0, slider adapter 0*, preference shift Aw max while not done do
(a:", w.g) g) ~ D Aw ~ Uniform (w-Aw mux mix
Iniform((1,, ... TT)
E I)
|eo- (20" (t,,,, t, (eo( dow Aw, (w.))wa.))))lloo cc (aalaa"tttn s.w Aw.g))
C(0') (ww.
2Aw
Update 6" to minimize L(Be)
end while
Inference Process of MODULI During deployment, MODULI utilizes both the diffusion model 0 and the slider 0" to guide the denoising process. Given a desired preference w, first query the closest preference wo in the dataset. In the S-step denoising process, we expect the distribution of the generated trajectory æ to gradually shift from the in-distribution preference Wi to the desired preference w through the guidance of the slider. Formally, we expect that the trajectory x at the i-th denoising step falls within the distribution p (x* | w;, K;). After one step of diffusion denoising, falls within the distribution p ( ws, 14-1). Subsequently, after the slider shift, falls within the distribution p pa= W;-1, (4-11,6411) 16-1))
The pseudocode for the implementation provided in Algorithm
17

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Algorithm 3 MODULI planning
Require: dataset D, diffusion model 0, slider 0", target preference w, inverse dynamics hg. S-length sampling sequence Ks. classifier-free guidance scale
Wo +- arg min || wol:: Wo E D wo
Aw = = Wo while not done
Observe state St; Sample noise from prior distribution x= I) for S.... do
Wi f- wo + S
Eo + (1 + ww)co
w:, 1n) wro(x"s g Eg. (x K4 w, )
Aég- Aw
- f- Denoise( Eo + Aco-)
end for
Extract St, 8141 from xNo and derive a; (1((5,8841))
Execute
end while
F. Details of Less-weight Trick
The diffusion planner generates state sequence (7) = (so. -11- Since closed-loop control is used, only and 91 are most relevant to the current decision. When calculating the loss, the weight of $1 is increased from to 10, which is considered more important. For clarity, we provide pseudocode below for the loss calculation the diffusion model incorporating the loss-weight trick and explain in detail.
Create loss woss_weight and assign a higher loss weight to the next observation (o_1).
bext_obs_loss_weltt weigh = 10 loss_weight = torch. ones ( (batch_size, horizon, obs_dim))
loss Loss_weight[:: 1, :] = next_obs ctcit_oos__os_________ casos__ossswwiiiip
def diffusion_loss_peeeooooooso pscudocosseeeceeeeeeeee (x0, condition)
Pseudocode function to demonstrate the loss function in diffusion models
Variable descriptions:
x0: Original noise-free data btatch_size, horizon, obs [dm]
condition: Conditional information (batch_size, condition_dim) xt: Noisy data after noise addition same shape as Time points in the diffusion process step [batch_size] eps: Noise added to x0 same shape as x0 Loss_weights Loss weight coefficient - scalar or tensor matching x0's s shape
Step 1: Add noise to the original data xt, t, eps add_nois (x0) Generate noisy xt from
Step 2: Process conditional information Encode the original condition information condition condition_encodee (condition)
Step 3: Predict noise through the diffusion model
predicted_eps diffusion_model (xt, t, condition) # Model predicts the added noise
18

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objectiv Reinforcement Learning
Step 4:
Calculate mean
squared error
Mean squared error between predicted noise and actual noise
loss = (predicted_eps - eps) ** 2
Step 5: Apply loss weights
We assign higher weight the prediction of the s_ loss less loss_weight.
Step 6: Calculate average loss final_ loss Joss.mean () i Calculate the average loss
return final _oss
19

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Table HV and SP performance full Comp1 ete datasets. (B: behavioral policy).
The hest emphasized in hold.
Env
Metries
B
MODT(P)
MORvS(P)
BC(P)
CQL(P)
MODULI
HV (x109)
5.61
5,92#.04
6,07
4.374.06
5.622.23
6,082.03
Ant
SP (x10+) 4
8.72+.77
5.24+.52
25.90=16.40
1.06+.28
0.53+.05
HV (x109) 1
5.68
5.69+.01
5.77+.00
5.46+.02
5.54+.02
5.76+.00
HalfCheetah
SP SP(100
1.162.42
0.572.09
2.221.91
0.45a.27
0.07#.02
HV 1
1.97
1.812.05
1.762.03
1.352.03
1.62.01
2.01#.01
Hopper
Amateur
SP (P
1.61
1.50
2.42+1.08
3.30+5.25
0.102.01
HV (x10
2.11
1.67+.22
2.792.03
2.82+.04
1.69..33
3.20=.00
Swimmer
SP (x100)
2.87#1.32
1.03#.20
5,05#1.82
8.87+6.24
9.502.59
HV (x100)1
4.99
3.10#.34
4.98
3,42#.42
1.782.33
5.062.00
Walker2d
SP SP(100 +
164.20+ 13.50
1.94
53.10434.60
7.33#5.89
0.25+.03
HV (x10 +
3.09
1.04
2.77
2,42+.18
0.59
3.33#.06
Hopper-3obj
SP (x11)) 4
10.23+2.78
1.03#.
0.87#.29
2.00+1.72
0.10#.00
HV (x109) f
6.32
6.214.01
6.362.02
4.88
5.76
6.392.02
Ant
SP (x1000)
8.26+2.22
0.87
46.20#16.40
0.58+. 10
0.79#.12
HV (x (x109) 1
5.79
5.73*.00
5.78
5.54+.05
5.63+.04
5.79:.00
HalfCheetah
SP (x10)) J
1.244.23
0.67
1.782.39
0.102.00
0.07=.00
HV 1
2.09
2.00#.02
2.02a.02
1,234.10
0.332.39
2.09
Hopper
Expert
SP (x110))
10.60
3.03+.36
52.50+4.88
2.84+2.46
0.09+.01
(x10+) f SP (x10") J
3.25
3.15#.02
3.24+.00
3.21+.00
3.22+.08
3.24+.00
Swimmer
15.00=7.49
4.39
4.50+.39
13.60#5.31
4.43+.38
HV (x 10%)
5.21
4.89#.05
5.14#.01
3,74#.11
3.21
5.20#.00
Walker2d
SP SP (x
0.99+.44
3.22+.73
75.60+52.30
6,23+10.70
0.11#.01
HV (x1002) f
3.73
3.38+.05
3.42a.10
2.294.07
0.78+.24
3.57+.02
Hopper-3obj
SP (P110))
1.40#.44
2.72+1.93
0.72+.09
2.60#3.14 14
0.07#.00
G. Full Results
See Table 4, Table 5 and Table for full results of HV, RD, SP Complete Swwt/Shatttttteeeeeeeeeeeeeeeeee Narrow datasets.
H. Additional Experiment Results
H.1. Performance with Different Guidance Mechanism
As shown in Table 7, we also compare the slider with different guided sampling methods on the expert datasets, namely CG and CG+CFG. Classifier Guidance (CG) requires training an additional classifier log Pe (w, g j| t) predict the log probability of exhibiting property. During inference, the classifier's gradient is used guide the solver. CG+CFG combines the CG and CFG methods, using preference w for guidance in the CFG part and RTG classifier log p (g x', t) for guidance in the CG part. We found that the CFG+CG method performs better than the CG method in most environments, indicating that CG is more suitable for guiding trajectories with high returns. However, MODULI w/ slider still demonstrates the best performance in most environments, especially in the HV and RD metrics.
H.2. Approximate Pareto Front on all Environments
We present the approximate Pareto front visualization results of MODULI for all environments in Figure 7. MODULI is a good Pareto front approximator, capable of extending the front on the amateur dataset and performing interpolation or extrapolation generalization on the Shattered or Narrow datasets.
20

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-bjeectie Reinforcement Learning
Table HV, RD and SP performance on full Expert -Shtttered and Expert-Naal -Narrow datasets. The best scores are emphasized in bold.
Env
Metries
(ODT(P)
MORvS(P)
BC(P)
MODULI
HV (x10%) f
5.88+.02
6.38+.01
4.76#.05
6.392.01
Ant
RD (x10*) +
0.714.03
0.162.00
0.61+.01
0.14*.01
SP (x10*)
2.22+.50
0.75
5.16+1.18
0.534.13
HV (x109) 1
5.69+.00
5.73+.00
5.66#.00
5.78
HalfCheetah
RD (x109)
0.18+.00
O. 0.10+.00
0.28+.00
0.07+.00
SP (x10f)
0.18
0.15+.02
0.34+.03
0.11+.02
Expert-Shattered
HV (x107) 1
1.95+.03
2.06+.01
1.49+.01
2.07:.00
Hopper
RD (x100) +
1.42#.00
1.10t.04
3.92+. 11
0.22*t066
SP (x10%)
0.85+.27
0,20+.06
2.58+1.01
0.11#.01
HV (x10f)
3.20+.00
3.24
3.192.00
3.24+.00
Swimmer
RD (x (x10)) +
0.38+.00
0.16t.00
0.07+.00
0.06:.00
SP (x100)
15.50+.33
7.36+.81
9.05+.30
5.79+.43
HV (x109)
5.01+.01
5.14s.01
3.47#.12
5.20+.00
Walker2d
RD (x109) +
0.86+.01
0.65+.02
1.72+.01
0.15#.01
SP (x10*) J
0.49+.03
0.27+.04
16.61
0.12t.01
HV (x1019) +
2.83+.06
3.28+.07
1.95+.16
3.43+.02
Hopper-3obj
RD (x10*)
1.60+.01
1.60+.02
3.21#.01
1.28*t.03
SP (x109)
0.09+.01
0.06+.00
0.17#.06
0.10#.01
HV (x 10%)= 1
5,05+.08
6,06+.00
4.90+.02
6.36#.00
Ant
RD (x10*) +
0.98+.01
0.37+.03
0.78+.03
0.25+.01
SP (x10*)
0.88+.41
0.79 11
1.56+.28
0.80+.25
HV (x10%) 1
5.04+.00
5,46+.01
4.88+.02
5.76*.00
HalfCheetah
RD (x (x10*) +
0.43*.00
0,29+.00
0.44+.00
0.17*.00
SP (x10*)
0.04
0.05+.00
0.02+.00
0.04+.00
HV (x107))
1.85+.02
1.98+.00
1.29+.01
2.04+.01
Napert-Narrow
Hopper
RD (x109) 4
4.24+.02
1.37+.05
3.83+.05
2.42+.05
SP (x109)
0.34+.07
0.16t.02
0.59+.24
0.25+.05
HV (x10*)
3.03+.00
3.10+.00
3.06#.00
3.21#.00
Swimmer
RD (x (x109)
0.38+.00
0.39+.01
0.19+.00
0.10#.00
SP (x10") 4
3.65
4.38+.80
5.53+.20
3.28+.15
HV (x10%)
4.75+.01
4.85+.01
1.88+.95
5.10t.01
Walker2d
RD (x (x109)
1.07+.01
1.45+.02
1.88+.00
0.28+.02
SP (x109)
0.34+.09
0.19+.02
10.76+1 1.58
0.13t.01
HV (x1010)+ 1
3.18+.05
3.32+.00
2.32+.10
3.37+.05
Hopper-3ob)
RD (x10²)
2.48+.00
2.24+.02
2.49+.02
2.38+.01
SP (x10%)
0.11+.01
0.22+.02
0.14+.02
0.12+.01
21

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Ojeectie Reinforcement Learning
Table HV. RD and SP performance on full in bold.
and Mmattur-Narrrov datasets. The best scores are emphasized
Env
Metries
MODT(P)
MORvS(P)
BC(P)
MODULI
HV (x10%) f
5.742.04
6.092.01
4.20#.05
6.04#.01
Ant
RD (x10*) +
0.362.01
0.132.03
1.01+.07
0.09*.01
SP (x10*)
2.47*.61
0.79
2.29+1.38
0.63#.23
HV (x109) 1
5.61
5.74+.00
5.54+.00
5.75#.00
HalfCheetah
RD (x109) 4
0.62+.00
0.11+.00
0.11+.00
0.07+.00
SP 4
0.11#.01
0.09
0.09#.01
0. 0.13#.01
Amateur-Shate Shatteredd
HV (x10²) 1
1.46+.00
1.69+.01
1.63+.00
2.01#.01
Hopper
RD (x10*) +
2.51+.01
1.76t.02
0.99+.06
0.49
SP (x10%)
0.54
0.18
0.11
0.15#.04
HV (x10f) 1
0.75+.00
2.81+.00
2.81+.00
3.17#.00
Swimmer
RD (x (x109) +
0.94+.00
0.40+.00
0.35+.00
0.294.00
SP (x100)
6.75+.23
1.26+.06
1.89+.05
8.85+.77
HV (x10%)
3.86+.07
4.99+.00
3.42#.02
5.03:.00
Walker2d
RD (x109) +
1.44+.02
0.37+.02
1.18+.01
0.35#.02
SP (x10*)
11.1343.58
0.20#.03
4.56#.08
0.29+.01
HV (x1019) +
1.48+.02
2.844.04
1.34+.02
3.02+.03
Hopper-3obj
RD (x10*)
2.86+.01
1.29+.01
2.87+.00
1.01#.04
SP (x109)
0.56+.08
0.13+.01
0.59+.08
0.10#.01
HV (x 1x0011 1
5.174.03
5,80+.03
4.19t.10
5,882.03
Ant
RD (x109) +
0.774.01
0.27
1.93+.04
0.53+.00
SP (x10*)
0.43+.10
0.51+.08
4.42+1.44
0.62+.11
HV (x10%) 1
4.29+.00
5,55+.00
5.08+.01
5.71*.00
HalfCheetah
RD (x10%) +
0.98
0,36+.00
0.36#.00
0.35#.00
SP (x10*)
1.02+.44
0.02*.00
0.02+.03
0.06+.02
HV (x107))
1.73+.01
1.69+.01
0.94+.34
1.99+.00
Amateur-Narrow
Hopper
RD (x109)
1.38+.04
2.88+.08
5.22+.01
1.33+.11
SP (x10%) 4
0.27+.07
0.25+.03
5.81+8.21
0.22+.06
HV (x10*)
0.66+.03
2.95+.00
2.80+.00
3.15+.00
Swimmer
RD (x (x109) +
1.144.00
0.48+.00
0.49+.00
0.45*.00
SP (x10")
14.2543.73
1.91+.01
2.95+.08
5.79+.43
HV (x10%)
3.24
4.83+.00
2.83+.01
4.91#.01
Walker2d
RD (x (x109)
1.91+.00
1.77+.00
1.87+.00
0.97+.02
SP (x10*)
5.6443.72
0.22+.01
0.01#.00
0.24+.01
HV (x1010)+ 1
1.85+.04
2.274.05
2.26+.02
2.90+.05
Hopper-3ob)
RD (x10²)
2.44+.02
1.60+.02
1.86+.03
1.84+.04
SP (x109)
3.12+1.21
0.07+.01
0.09+.00
0. 0.122.02
22

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-bjeectie Reinforcement Learning
Table 7. HV, RD and SP performance with different guided sampling methods the expert datasets.
Env
Metrics
MODULI
CG
CFG+CG
HV (x109)
6.362.00
6.232.03
6.12+.03
Ant
RD (x10%) 4
0.25+.01
0.44
0.85+.03
SP (
0.80#.25
0.52+.16
0.81
HV (x109) f
5.76*.00
5.692.01
5.76+.00
HalfCheetah
RD (x10%)
0.17+.00
0.142.03
0.52*.03
SP (
0.04+.00
0.47+.03
0.04+.00
HV (x10²)
2.04#.01
1.95+.01
2.03+.01
Expert-Naree -Nrrow
Hopper
RD (x109) 4
2.42+.05
1.11
2.97+.05
SP 4
0,25+.05
0.09
0.09
HV (x10*) t
3.21#.00
3.19t.01
3.16#.00
Swimmer
RD (x103) L
0.102.00
0.16#.01
1.12+.00
SP (x109)
3.28+.15
23.2843.56
5.26+.62
HV (x109) t
5.10t.01
5.01
5.08+.00
Walker2d
RD (x10%) L
0.28+.02
0.40#.04
0.63+.00
SP (x10*) +
0.13*.01
1.27
0.19+.00
HV (x10'))
3.37*.05
2.30#.08
3.342.03
Hopper-3obj
RD (x109) 4
2.38t.01
2.58+.04
2.45#.03
SP (x10%) 4
0.12+.01
0.112.01
0.11+.01
HV (x10%)
6.394.01
6.23*.03
6.36+.03
Ant
RD (x103) 4
0.14
0.46+.02
0.14+.00
SP (x100)) 4
0.53*.13
0.54
0.
HV (x109) t
5.78+.00
5.69
5.78+.00
HalfCheetah
RD (
0.07+.00
0.37
0.07+.03
SP (P100)
0.11+.02
0.47+.03
0.12+.01
Expert-Shatie Shattered
HV (x10²) t
2.07#.01
1.95#.01
2.07#.01
Hopper
RD (x109) 4
0.22+.06
0.81
0.23+.03
SP (x10%)
0.11+.01
0.13
0.12+.01
HV (x104)
3.24+.00
3.19a.01
3.24+.00
Swimmer
RD (x103) L
0.06+.00
0.29#.02
0.04+.00
SP (x109)
5.79t.43
23.2843.56
6.78+1.26
HV (x109) t
5.20*.00
5.19+.00
5.01+.06
Walker2d
RD (x109) 4
0.15+.01
0.17+.01
0.73+.04
SP (x104)
0.12+.01
0.12+.01
1.274.32
HV (x10"))
3.43+.02
2.30+.08
3.39+.02
Hopper-3obj
RD (x10*) L
1.28+.03
2.45+.10
1.66t.01
SP (x10*) 4
0.10+.01
0.12+.01
0.10#.01
23

---

MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
Ant
HalfCheetah
Swimmer
Hopper
Walker2d
Amateur- Compu--C
2000
203
124
033
10C
3900
1500-
1900
66
-2200
Expert-Compie Compiet
7000
3900
2003
1500
500
539
1900
592
-000
2.3
2590
SmmateurSShateerer
146
2000
209
1990
3900
9 Pnngenes
1800
Expert- Shattered
5.30
2000
4300
Occum 0 dRotcom
66
590
3300
2.00
Amateur-Narrow
2900
1000
100
3000
142
Expert-Narrow
2000
X09
120
100
3900
150
1990
48
-3000
-1200
180
1200
2050
180
1826
1989
2003
1980
1091
2000
4003
92
1959
2050
Figure Approximated Pareto fronts on Complete Shattered Narrow dataset by MODULL Undominated/ Dominated solutions are colored in red blue, and dataset trajectories colored in grey.
24


Reviews:
[{"review_id": "Q5LQ468ayg", "reviewer": "Reviewer_DFCw", "summary": "The paper studies the problem of offline multi-objective RL with preferences over the objectives. The key contribution of the paper is to introduce a new preference-conditioned diffusion model to generate trajectories aligned with specific preferences and derive actions accordingly. The generation process is enhanced with return normalization (across multiple objectives with different value scales and slide-guidance to learn preference direction to handle OOD preferences. \n\nExperiments are conducted using the D4MORL benchmark dataset, showing the proposed method outperforms baselines such as BC, MORvS, MODT, and MOCQL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and useful feedback, please see the following for our response.\n\n### Q1: Novelty of MODULI\n\nWe would like to clarify that MODULI is not merely an application of diffusion models in offline RL. The novelty of MODULI lies in the following three aspects:\n\n- **A new research problem**: When only incomplete offline data (narrow/shattered) is available, the preference for generalization to OOD is crucial for multi-objective RL. For the first time, **We model Offline MORL from the perspective of generative models** and introduce diffusion models into offline MORL. We found that their expressive and generalization capabilities are highly effective for MORL problems.\n- **Improved return normalization methods**: Traditional return normalization guided by the only maximum reward has critical flaws in scenarios with conflicting multi-objectives. We designed two **return normalization methods (NPN/PPN) tailored for multi-objective problems**, which are crucial for diffusion models in MORL.\n- **Better OOD Preference Generalization:** Through the study of multi-objective task datasets, we found that as preferences change, policies in multi-objective tasks will show certain changing trends, suggesting a potential direction for addressing the challenge of policy generalization in multi-objective optimization tasks. Inspired by prior work, we introduced a sliding adapter to enhance generalization capability, achieving significant results while many implementation details differed from [1], including the fine-tuning method, the calculating unit strength, and the final integration of the outputs from two models. We focus on stronger generalization ability rather than precise conceptual control over new attributes.\n\n### Q2: The impact of the degree of conflict between multiple objectives on the MODULI performance\n\nWe conducted a comprehensive evaluation on datasets of **varying quality (expert, amateur)** and datasets with **different levels of OOD (complete, narrow, shattered)**, all of which have varying degrees of conflicting objectives, such as speed and energy. As shown in Fig. 3 (page 7), we were surprised to find that when the dataset is incomplete, baseline algorithms like BC fail completely under OOD preferences due to conflicts, and RvS tends to fit a single solution regardless of different preferences. In contrast, MODULI adapts to varying levels of conflict and dataset quality, demonstrating excellent performance.\n\n### Q3: Can you comment on how your method would perform when the number of objective increases?\n\n- Our experiments include the **hopper-3obj** dataset, which involves 3 conflicting objectives. It is worth noting that as the number of objectives increases, the policy for given preference becomes more complex. MODULI demonstrates outstanding performance in the three-objective task, achieving over a **20% improvement** compared to the strongest baseline MORvS **(Table 1)**.\n- From the perspective of method design, MODULI does not assume any specific number of objectives. The proposed new normalization methods and sliding adapter can handle scenarios with more objectives.\n\n[1] Gandikota et al. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models", "claims_and_evidence": "The advantages of the proposed method are well supported by a comprehensive experimental analysis.", "methods_and_evaluation_criteria": "--- The D4MORL dataset used in the paper is a benchmark dataset used in multi-objective RL, which contains different types of data quality including expert and amateur datasets.\n --- The evaluation focuses on various metrics including hyper volume, sparsity, and return deviation, which are commonly used metrics in MORL. \n---The idea of applying diffusion models with two enhanced techniques (return normalization and slide guidance) are well justified.", "theoretical_claims": "The proof for sliding guidance is provided in the appendix and looks correct.", "experimental_designs_or_analyses": "The experimental design of the paper is quite standard in MORL with commonly used evaluation criteria and benchmark datasets and state-of-the-art baselines.", "relation_to_broader_scientific_literature": "Findings of the paper are related to the RL research community.", "essential_references_not_discussed": "I'm not aware of any missing essential references.", "other_strengths_and_weaknesses": "Diffusion models have been used extensively in offline reinforcement learning, which is not something new. However, the strength of the paper lies on the performance enhancement obtained though return normalization and slide guidance which enable the generation process to align with preferences over multiple objectives.", "other_comments_or_suggestions": "I don't have other comments or suggestions.", "questions_for_authors": "--- How does the level of conflict between multiple objectives impact the performance of our proposed method?\n--- Most of the evaluations are for the two-objective RL tasks. Can you comment on how your method would perform when the number of objective increases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PjfQ7oMPZX", "reviewer": "Reviewer_MyRm", "summary": "This work proposes MODULI(Multi Objective DiffUsion planner with sLIding guidance), which employs a preference-conditioned diffusion model as a planner to generate trajectories that align with various preferences and derive action for decision making. MODULI also introduces two techniques 1) new return normalization method 2) slider adapter to achieve better generation for both ID and OOD preferences. Extensive experiments on the D4MORL benchmark demonstrate superiority of MODULI.\n\n## update after rebuttal\nThe authors' responses addressed most of my concerns so I've raised my score from 3 to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and useful feedback, please see the following for our response.\n\n### **Q1: Comparison of Parameters for Different Baseline Algorithms**\n\nThank you for your suggestion! We would like to clarify that in our experiments, we standardized the parameter size of all baseline algorithms to the same magnitude as much as possible. A complete list is provided in the table below:\n\n| **Algo.** | **Model Size** |\n| --- | --- |\n| MORvS | 8.55M |\n| MOBC | 8.83M |\n| MODT | 9.75M |\n| MODULI Diffusion Model | 11.14M |\n| MODULI Inverse Dynamic Model | 1.11M |\n\nWe believe your suggestion is very correct, and we will include this comparison in the revised version.\n\n### Q2: Novelty of MODULI\n\nWe would like to clarify that MODULI is not merely an application of diffusion models in offline RL. The novelty of MODULI lies in the following three aspects:\n\n- **A new research problem**: When only incomplete offline data (narrow/shattered) is available, the preference for generalization to OOD is crucial for multi-objective RL. For the first time, **We model Offline MORL from the perspective of generative models** and introduce diffusion models into offline MORL. We found that their expressive and generalization capabilities are highly effective for MORL problems. Relying solely on Decision Diffuser[1] cannot achieve the performance of MODULI, the following two techniques are important.\n- **Improved return normalization methods**: Traditional return normalization guided by the only maximum reward has critical flaws in scenarios with conflicting multi-objectives. We designed two **return normalization methods (NPN/PPN) tailored for multi-objective problems**, which are crucial for diffusion models in MORL.\n- **Better OOD Preference Generalization:** Through the study of multi-objective task datasets, we found that as preferences change, policies in multi-objective tasks will show certain changing trends, suggesting a potential direction for addressing the challenge of policy generalization in multi-objective optimization tasks. Inspired by prior work, we introduced a sliding adapter to enhance generalization capability, achieving significant results while many implementation details differed from [2], including the fine-tuning method, the calculating unit strength and the final integration of the outputs from two models. We focus on stronger generalization ability rather than precise conceptual control over new attributes.\n\n[1] Ajay A et al. Is conditional generative modeling all you need for decision-making. ICLR2023.\n\n[2] Gandikota et al. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. ECCV2024.", "claims_and_evidence": "Yes. The article clearly reflects the main arguments and the experiments can support the authors' claims.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. The experimental designs are reasonable and can support the authors' claims.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "#### Strengths\n\n1. The article is well-written, with clear expression and logic, effectively reflecting the main argument.\n2. The experiments are very comprehensive. The authors validate the effectiveness of their method across a series of tasks including 'complete' datasets for expressiveness evaluation and 'shattered and narrow' datasets for generalization evaluation.\n\n#### Weaknesses\n\n1. I think the author should clearly specify the parameter sizes of this algorithm and the other baseline algorithms for comparison, and try to keep the number of parameters consistent across different algorithms as much as possible. This is because diffusion-based methods may benefit from a larger number of parameters, and doing so will make the results more convincing.\n2. I think the innovation of this work may be somewhat insufficient because the main diffusion-based planner is basically equivalent to the Decision Diffuser, and the proposed sliding guidance also draws on the work of predecessors. However, considering that these components are quite practical and have effectively improved the performance of the algorithm, I think it still meets the threshold for acceptance.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbfe6uAb50", "reviewer": "Reviewer_Dt55", "summary": "This paper introduces MODULI (Multi-Objective Diffusion planner with sliding guidance), a novel algorithm for offline multi-objective reinforcement learning (MORL). MODULI employs a conditional diffusion model for representing the agent’s policy. Besides MODULI, the paper also introduces two techniques for normalizing multi-objective returns that are used to condition the policy, and a “slider adapter” neural network used to make the diffusion model generalize to unseen OOD preferences.\n\n---\n\n### Post Rebuttal\n\nI thank the authors once again for providing detailed answers to my questions.\n\nI increased my score since the authors addressed my raised concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1: Comparison with the performance of MORvS\n\nMODULI consistently outperforms the strong MORvS baseline, leading in 20/24 metrics across 12 Complete datasets(Tab. 4) and surpassing the baseline in 59/72 metrics on Shattered & Narrow datasets(Tab. 5 & 6).\n\n## Q2: Why use 3 seeds?\n\nWe collect 501 sample points for each seed to calculate the metrics, fully covering the preference space, which results in small variances. At the same time, we aligned with the setting of D4MORL benchmark, it also uses 3 seeds. We conducted a large number of performance and ablation experiments(2 levels * 3 types * 6 tasks), demonstrating the reliability of the MODULI.\n\n## Q3: Significantly extend the Pareto front\n\nDue to task constraints, the Pareto front in Fig.3 can only expand by a small margin. Compared to other methods, MODULI demonstrates stronger generalization and an \"expansion\" phenomenon. To avoid misunderstanding, we replace \"significantly\" → \"slightly\" in revision. Additionally, extensive experiments and visualization(fig.7) proves that MODULI can enhance generalization.\n\n## Q4: Confusing expressions, incomplete issue and typos\n\nWe apologise for the unclear description in the introduction. We will provide a brief introduction to the concept in the Introduction and explain them in detail in the Method. And we have carefully reviewed your suggestions part. Thank you for pointing them out! We will thoroughly review the paper, check for grammatical issues and verify the references in revision. \n\n## Q5: Intuition for Equation 11\n\nWe trained a diffusion model $p$ to generate trajectories corresponding to ID preferences. Then, we attempt to learn the pattern of change (for example, when the preference shifts from energy efficiency to high speed, the amplitude of the Swimmer's movements gradually increases). Therefore, our goal is to learn an adapter $p^*$, which applies this pattern to OOD preferences to achieve better generalization. In Equ. 11, the exponential term represents increasing the likelihood of preference $c+$ and decreasing the likelihood of preference $c−$. η Represents the guidance strength weight in classifier-free guidance for the diffusion model. Based on Equ. 11, we can use an adapter-style method to adjust the original noise prediction model ($p$). Now, the noise prediction combines the noise from the original model and the adapter. \n\nWe apologize again for the confusing expressions and undefined symbols. We will provide definitions and include thorough intuitive explanations.\n\n## Q6: Detailed limitations\n\nWe assume a linear preference space, such as $w_1 + w_2 = 1$, which is the standard practice in most MORL papers. When the linear assumption does not hold, our core method remains applicable, but additional representations are required. And very sorry for the confusion. We will change the “structured state space” to “low-dimensional state space” (Distinguished from image input). “Continuous action” should be “continuous action space”, distinguished from discrete action space. We believe that MODULI can support tasks with these new modalities with minimal additional adaptation (e.g., encoder). We left it for future work.\n\n## Q7:  Loss-weight trick\n\nSorry for the confusion, we will include detailed implementations in the appendix. The diffusion planner generates a state sequence $x^0(\\tau) = \\left[ s_0, \\cdots, s_{H-1} \\right]$. Since closed-loop control is used, only $s_0$ and $s_1$  are most relevant to the current decision. When calculating the loss, the weight of $s_1$ is increased from 1 to 10, which is considered more important.\n\n## Q8: Compute Equation 7\n\nIn D4MORL dataset, each trajectory contains a target preference $\\omega$, and $g$ is the return of the trajectory. Since $D_p$ consists of trajectories corresponding to Pareto front points, it represents the optimal level of policy in the dataset. Therefore, for each $(g, \\omega)$ sampled from $D_p$,  $g$ is the maximum return corresponding to $\\omega$. Then, a predictor $R(\\omega)$ for the maximum possible return can be obtained with Equ.7. It can be seen as a fit to the Pareto front in the dataset.\n\n## Q9: RD metric\n\n**Please note that we trained a predictor for each task (e.g., ant-expert) to evaluate RD, rather than one for each algorithm**. The detailed process: we fixed the Ant-**expert-complete** dataset to trained $R_\\psi$. Since expert and complete trajectories were used, we can estimate the optimal $R$ for each sampled preference $w$, these ground truth information is only used for evaluation. Then, for **Ant-expert-{complete/narrow/shattered}**, for the trajectories sampled from OOD preference points, RD is defined as the difference between the ground truth return from $R_\\psi$ and the actual obtained return. The results are averaged over all OOD preference points. A smaller RD indicates that the algorithm can better align with the target preference when the target preference is OOD, thus demonstrating better generalization ability.", "claims_and_evidence": "The claims are evaluated empirically by evaluating the method in offline datasets with missing examples for some regions of preferences, or with narrow distribution of solutions in the Pareto frontier. The empirical results seem to be in accordance with the claims. That is, 1) the proposed method achieves generally better metrics than the baselines, 2) the addition of the slider adapter network seems to indeed improve the generalization of the method, and 3) the normalization techniques introduced improve the method when compared to the global normalization baseline.\n\nHowever, in Figure 3 (bottom), the authors claim that “MODULI can significantly extend the Pareto front to both sides”. However, this is not true if we inspect the plot. MODULI can reproduce the solutions already in the dataset on the sides, but it does not generalize to more extreme points.\n\nAlthough the results seem to validate the claim, the performance improvements compared to the MORvS(P) baseline are not that significant.", "methods_and_evaluation_criteria": "The method is compared with other recent offline algorithms in the D4MORL benchmark, which is a recent Mujoco benchmark for offline MORL. They constitute appropriate baselines and benchmarks for evaluating the method.", "theoretical_claims": "The paper does not introduce any theorems or theoretical results.", "experimental_designs_or_analyses": "I checked the details of the experimental designs. One issue with the paper is that, for each experiment, the authors used only 3 random seeds to report the mean and standard deviation. The authors should justify why only 3 random seeds are enough.", "relation_to_broader_scientific_literature": "The paper tries to improve upon previous offline MORL approaches by employing different techniques that are related to diffusion models for sequential decision-making. Prior works have employed other supervised learning techniques to tackle this problem, and the paper tries to tackle the problem with a different set of techniques.", "essential_references_not_discussed": "The authors discussed all previous relevant papers in the field of offline MORL.", "other_strengths_and_weaknesses": "Below, I point out a few other weaknesses in the paper:\n\n* The Introduction has a few technical terms that are not explained, making it confusing to understand the contributions. For instance, what is a “slider adapter”, “latent directions of preference”, or “refine guidance”? Please explain such terms in more detail.\n* The paper has a few grammatical issues and sentences that are not flowing well. I suggest the authors review the text for such issues.\n* Section 4.3 is currently very difficult to follow. I strongly suggest the authors provide intuition for Equation 11 and the associated variables. For instance, in the sentence “we can derive a direct fine-tuning scheme that is equivalent to modifying the noise prediction model.”, do you mean modifying the noise prediction model to achieve what goal? Also, the variable $\\eta$ was not defined.\n* The paper does discuss its limitations in sufficient detail. For instance, in the conclusion the authors could better state the necessary assumptions and when they are not valid.", "other_comments_or_suggestions": "* A few references are citing the ArXiv version of papers that have been published in peer-reviewed venues, e.g., “Alegre et al. Sample-efficient multi-objective learning via generalized policy improvement prioritization. arXiv preprint arXiv:2301.07784, 2023” has been published at AAMAS 2023.\n\n* ”As a result, direct guidance using the target conditions of $y = [\\omega_{target}, 1^n]$ may be unachievable for some preferences, leading to a performance drop.”\nSince this is a multi-objective setting, the target $1^n$ will always be unachievable. The vector $1^n$ is only achievable if all objectives are aligned/not conflicting,\n\n* “This enables controlled continuous concept during generation”\nIt is not clear what a “controlled continuous concept” is.\n\n* Typo: ”and P demotes Pareto front” -> denotes", "questions_for_authors": "* “MODULI also employs a loss-weight trick, setting a higher weight for the next state s1 of the trajectory $x0(\\tau)$ to encourage more focus on it, closely related to action execution.”\nThis sentence is not clear, please elaborate on how this is implemented.\n\n* It is not clear how Equation 7 is implemented. In particular, how is each $(\\omega, g)$ pair constructed? Just before the Equation, $D_P$ is defined as a set of trajectories, but then the method is sampling pairs of weights and returns. How do you ensure that each $\\hat{g}$ in $D_P$ is actually the maximum return for each $\\omega$?\n\n* It is not clear how the Return Deviation (RD) metric is employed for the baseline algorithms. A different generalized return predictor is learned for each algorithm? When algorithm A has a better RD metric than an algorithm B, why does it imply better generalization?\n\n* “However, there are limitations, such as experiments conducted in structured state space and continuous actions.” \nWhat are structured state space and continuous actions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new algorithm called MODULI for offline multi-objective reinforcement learning. The proposed method uses a conditional diffusion model to represent the agent’s policy and leverages two new techniques—one to normalize multi-objective returns and another (a “slider adapter” neural network) to enable the diffusion model to better generalize to unseen, out-of-distribution preferences. The authors evaluated their method through extensive experiments on the D4MORL benchmark dataset and showed that MODULI outperforms several relevant baselines.

All reviewers agree that this is a well-written paper and that the authors clearly and convincingly presented, motivated, and justified their design choices and claims. Two reviewers noted that the experiments are comprehensive and demonstrate the method’s effectiveness across a wide range of qualitatively different tasks. Another reviewer commented that, although diffusion models have been extensively studied in offline RL, this work makes a substantial contribution through its novel normalization scheme and its “slider adapter” component—designed, as mentioned earlier, to support better generalization. They noted that these contributions enable significant performance improvements. Reviewers also pointed out that all theoretical results and proofs appear sound and correct. Finally, all reviewers agree that the paper’s key claims were adequately supported.

Three key concerns were raised during the discussion phase:

1. The paper’s novelty might be limited since using a diffusion-based planner could be viewed as functionally equivalent to deploying a Decision Diffuser;

2. The authors’ claim in Figure 3—that “*MODULI can significantly extend the Pareto front to both sides*”—does not appear to be accurate, and the performance improvements over the MORvS(P) baseline are not substantial;

3. The authors use only three random seeds in each experiment, which limits the statistical significance of the results, and a justification is needed for why this is sufficient to support their claims.

The authors provided a thorough rebuttal addressing all three concerns, and two reviewers increased their scores as a result.

Overall, reviewers agree that this paper contributes important insights that will benefit the ICML community and that the concerns raised during the discussion phase were not critical and can be addressed in a revised version. The reviewers encouraged the authors to improve the paper by incorporating the points raised in the reviews and discussion phase, and argued that doing so would substantially strengthen the quality and impact of the work.
Decision: Accept (poster)