Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Authors: Xudong Yu
Chenjia Bai
Haoran He
Changhong Wang
Xuelong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-task reinforcement learning
preference learning
conditional diffusion model
trajectory generation
alignment
offline RL
Primary area: reinforcement_learning
TL;DR: We propose multi-task preference representations and regularized conditional diffusion model to generate trajectories aligned with human preferences.

Abstract: Sequential decision-making can be formulated as a conditional generation process, with targets for alignment with human intents and versatility across various tasks. Previous return-conditioned diffusion models manifest comparable performance but rely on well-defined reward functions, which requires amounts of human efforts and faces challenges in multi-task settings. Preferences serve as an alternative but recent work rarely considers preference learning given multiple tasks. To facilitate the alignment and versatility in multi-task preference learning, we adopt multi-task preferences as a unified framework. In this work, we propose to learn preference representations aligned with preference labels, which are then used as conditions to guide the conditional generation process of diffusion models. The traditional classifier-free guidance paradigm suffers from the inconsistency between the conditions and generated trajectories. We thus introduce an auxiliary regularization objective to maximize the mutual info

Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Xudong Yu Harbin Institute of Technology hit20byu@gaill.cmm com
Chenjia Bai'
Institute of Artificial Intelligence (TeleAI,, China Telecom
baicj@chinatteeeco..
Haoran He
Changhong Wang
Hong Kong University of Science and Technology haoran. he@connect ust. hk
Harbin Institute of Technology cwangChit edd.. cn
Xuelong Li
Institue of Artificial Intelligence (TeleAI), China Telecom xuelong. li@iese.org
Abstract
Sequential Deeision-making can be formulated as a conditional generation process, with targets for alignment with human intents and versatility across various tasks. Previous return- ecturn-conditioned diffusion models manifest comparable performance but rely well-defined reward functions, which requires amounts of human efforts and faces challenges in multi-task settings. Preferences serve as an alternative but recent work rarely considers preference learning given multiple tasks. To facilitate the alignment and versatility in multi-task preference learning, we adopt multi-task preferences
unified framework. In this work, we propose learn preference representations aligned with preference labels, which are then used as conditions to guide the conditional generation process of diffusion models. The tra- ditional classifier-ff guidance paradigm suffers from the inconsistency between the conditions and generated trajectories. We thus introduce an auxiliary regu- larization objective to maximize the mutual information between conditions and corresponding generated trajectories, improving their alignment with preferences. Experiments in D4RI. and Meta- World demonstrate the effectiveness and favorable performance our method single- and multi-task scenarios.
Introduction
In sequential decision-making, agents are trained to accomplish fixed or varied human-designed goals by interacting with the environment learning from offline data. Two key objectives emerge during training: alignment, wherein agents are expected to take actions conforming to human intents expressed as crafted rewards or preferences, and versatility, denoting the capacity tackle multiple tasks and generalize to unseen tasks. A promising avenue involves framing sequential decision- making as sequence modeling problem via transformer [T or diffusion models 3 This paradigm uses expressive models to capture the trajectory distributions and prevents unstable value estimation in conventional offline Reinforcement Learning (RL) methods 4.. Particularly, utilizing return- conditioned or value-guie value-guided diffusion models to perform planning or trajectory generation achieves favorable performance offline RI. [5 and the multi-task variant 77.
"Correspondence Chenjia Bai (baicj@chhcc chaieeicccnaaeeeomccn)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite the great progress, applying diffusion models to sequential decision-madee still faces challenges. The condition generation process relies on a pre-defined reward function to pro- vide return conditions. However, developing multiple task-specific reward functions in multi- task settings requires significant efforts [9] and may cause unintended behaviors [0]] (ii) The condition generation process with classifier-free guidance [[]] often fails to ensure consistency between conditions and generations. As an example Figure [ the return-conditioned Deci- sion Diffuser [5] struggles to achieve effective alignment for generated trajectories. Inspired by recent works 9 12 13I. we find that preferences offer more versatile supervision across multi-tasks than scalar rewards. Specifically, the trajectories from the i-th task are preferred over the j-th task we set the i-th task as the can reverse the
target task. Conversely, preference label when setting the target task
Thun Raturns of Gererated Trajectories
Ideal Canditional Generation
Current Results of Decision Diffuser
to the j-th task. Therefore, ences rather than rewards to guide the condi- tional diffusion models in offline multi-task RL.
adopt prefer-
100
80
The generated trajectories are intended to align with preferences, prioritizing higher returns or specific tasks.
To establish aligned and versatile conditional generation, our proposition involves adopting multi-task preferences and constructing unified preference representation for both single- and multi-task scenarios, instead of learning scalar rewards from preference data [15 16). The ac- quired representations
20
0.3 0.4
0.9 1.0
The Condition
Figure 1: Illustration of the return-conditioned gen- eration of Decision Diffuser [5] in hopper-medium- expert task. Existing return-conditional diffusion models fail to align generated trajectories with the return condition, while the red line indicates the desired relationship between the return conditions and true returns of generated trajectories [14]
aligned with the pref- erence labels. The representations subsequently serve as conditions guide the conditional gen- eration of diffusion models. In this case, two key challenges arise: (i) aligning the representations with preferences, and (ii) aligning the generated trajectories with representation conditions.
To address the above challenges, we introduce Conditional Alignment via Multi-task Preference representations (CAMP) for multi-task preference learning. Specifically, define multi-task prefer- ences and extract preference representations from trajectory segments. (i) To align the representation with preferences, we propose triplet loss and a Kullback-Leibler (KL) divergence loss enable preferred and undesirable trajectories mapped into distinct zones in the representation space. The learned representations not only differentiate trajectories yielding higher returns but also discern trajectories across various tasks. Additionally, we learn an 'optimal' representation for each task to represent the trajectory with the highest preference. (ii) To align the generated trajectories with representation conditions in diffusion models, we introduce a Mutual Information (MI) regularization method. It augments pnrresentttioncconiiiion objective, maximizing the correlation between the conditions and the generated outputs. During the inference for specific task, we provide the 'optimal' representation for that task as the condition of the diffusion model, allowing the generated trajectories to adhere to desired preferences. Experiments on D4RL 17 and World [88] demonstrate the superiority of our method and the aligned performance in both multi-task and single-task scenarios.
diffusion models with an auxiliary MI optimization
2 Preliminaries
2.1 Diffusion Models for MDP
Diffusion models have emerged as effective generative models adept at learning high- diggesiimnsional data distribution p(x) 2
19 20 A diffusion model comprises a pre-defined forward process ax) and trainable reverse process po ( where l E [1, K] denotes the timestep index, and as R decides the variance schedule. By sampling Gaussian noise from p(æx) and iteratively employing the denoising step po( ( for K steps, diffusion models can generate Zo ~ p(z). Moreover, if additional conditions are introduced into
(( (x)-1) = N (Maz Vor*k-1,

---

the denoising process such that 24-1 Pe(xs-1l=x c), diffusion models can also estimate the conditional distribution p(z|c).
The Markov Decision Process (MDP) is defined by tuple (S, A, P. r. n), where and A are the state and action spaces, P is the transition function,, is the reward function, and y is discount factor. We consider an offline setting where the policy is learned from a given dataset D. For multi-task learning, we introduce an additional task space 7 that contains m discrete tasks. Diffusion models have been used for offline RL to overcome the distribution shift caused by Temporal-Difference (TD) learning 4 Specifically, diffusion model can formulate sequential conditional generative modeling problem by considering Zo as a state sequence ($t..... S(+h) ) for planning [5]. The condition e typically encompasses the return along the trajectory and is designed to generate trajectories with higher returns.
perision-makingg
Optimizing the conditional diffusion model involves maximizing the log likelihood log p(x) log J p(z c)p(c)de, where p(z]e) is the conditional likelihood for a specific c. Building on prior research [9 211. the optimization achieved by maximizing the Evidence Lower Bound (ELBO):
logp(xo) 2 (Zo, c) _ Byt (mo) [Bg [log pe (T1, c)]l DKL(qulp((e)
K
- DKI (xk.(a(ax><<)))lpeee
Le (oo) [gg [DKL (PKI 11l c))]
where == gu (c) represents the approximate variational posterior mapping zo to the condition and f g i (T,, The complete derivation provided in $A.1 In practice, this optimization problem can be addressed via score-matching objective 1 as,
Lo
El|l oo (2 - B)e + (20, k)]]3,
where is parameterized by a neural network predict the noise NO0,, I), the expectation is calculated w.r.t. E [1, K], E D, and 3 ~ Bernoull(pp).
2.2 Preference Learning for Conditional Generation
In decision-making human preferences are often applied on trajectory segments r= [8i, a1) (s)
[22 For trajectory pair (Ti, 72), human feedback yields a preference label y E (0, 1, 0.5) that indicates which segment human prefers. Here, y signifies 71 > T2. y signifies 7I 72, and 0.5 means that two trajectories have the same preference. Previous studies commonly employ the Bradley-Terry (BT) model [23] to derive a reward function from such preferences. Considering learning with offline preference data [16). and given a dataset D, = {(T1, 72, v)) f is learned by maximizing the following objective:
C Ep,
> 12) + (1 y) log P(re > TI]]
In what follows, we simplify the notation by omitting the label and denote the preference data as ((rt. }")) where we have + T Previous methods (24 [S] based on the BT model follow two-phase learning process, first deriving reward function from preference data and then training policy with RI. Nevertheless, this process hinges on the assumption that pairwise preferences can be substituted with reward model that can generalize to out-of -iitt--distritttinn data 25 Moreover, these
methods often require complex reward models [.5] when preferences are intricate 26 27
As result, we bypass the reward learning process and learn a preference-relevant representation that aligns well with trajectories in both single- and multi-task settings. Then the diffusion models can use these representations as conditions to generate trajectories that align with human preferences. In this setup, we regard Fo = r as the trajectory segments. and the condition representation of trajectories, denoted as = f(T). Therefore, the learning objective of diffusion model becomes log (T)),, and the loss function becomes Co Ellle Eo (kk (1 B)w + 30, [
the preference-relanee
3 Method
In this section, we give the definition of multi-task preferences and introduce how to extract preference representation from pairwise trajectories. Then, we present the conditional generation process and an auxiliary optimization objective align the generated trajectories with preferences.
3

---

3.1 Versatile Representation for Multi-Task Preferences
Multi-task preferences. In the context of multi-task settings, tasks exhibit distinct reward functions, making reward-relevant information insufficient for providing versatile preferences across tasks. For example, moving fast preferred and obtains high rewards in a 'running' task, while being unfavorable in 'yoga" task. To address this challenge, we extend single-task preferences that only contain reward-relevant information to multi-task settings. Specifically, we consider two kinds of preferences given trajectories from tasks. For a specific task E [m,] trajectories are assessed based on (i) the return of trajectories when they belong to the same task, i.e., 7 ž p' if ((((+) 2 R(r""), where R(.)) calculates the cumulative reward, and (ii) the task-relevas trajectories, i.e., T' > r with j o i. This means that trajectories from the target task are more preferred than any trajectories 7 from different task [m)).
of
Preference representations. Based on the multi-task preferences, we propose to learn tra- jectory representations aligning with the prefer- ence data, as shown in Figure 2 The learned representations integrate both the trajectory and preference information, serving the condition for subsequent trajectory generation. During learning representations, we also need to find the 'optimal' representations {w, heemm] that repre- sent the optimal trajectories (T") he(m) for each task, where r." is preferred over any offline tra- jectories in task i. The {wi hemm will be used for inference in diffusion models to generate de- sired trajectories for each task. Thus, we need to learn trajectory encoder
Multi- Masti--asskaa Preference Data
Representation Space for Tasks
[+'*]
f
(ef 3/sc
Figure Illustration of the representation space of trajectories in multi-task preference data, For each task i, the positive samples 7 consist of preferred trajectories r' from task i, while negative samples T include less preferred - from the same task, as well as T from other tasks. Trajectories from diverse tasks are expected to be differentiated in the representation space, and {wī heemi attempts to characterize the best trajectories for each task.
(T) that extracts greference-reeeaam information and the optimal representation {wf [m[]).
Furthermore, we model the representations from distributional perspective to cover their uncer- tainty. practice, the distribution of the optimal representation p(w") -V (u,, );) and the dis- tribution of representations given trajectory p(w[r) = ((u En) (T)) are both modeled as multivariate Gaussian distributions with diagonal covariance matrix, where d is parameterized by transformer manseed encoder. To summarize, the learnable parameters include vectors {uz Σ Hie[m] for { [w]] he(m] and trajectory encoder fa.
Loss functions for fu and w,: For each task i in training, we build the multi-task dataset D br' r5) to learn the representation space. The preference data are constructed by using the intra-task preference (i.e., -'+ x()) and the inter-task preference (i.e., -++ 75). We denote the representation distributions as
p(wt)= = (r+t), p(w7) = sm(T") or fo((T))
For simplicity, we denote p(u;t) )-= N(ut, Σt) and p(w,) ) NN ΣT) by omitting the parameter
v). These distributions are optimized using the following KL loss
(3) This loss function encourages the encoder to map trajectories with similar preferences to closer embeddings while distancing dissimilar trajectories. In practice,
C(b,, E;) (N (M Ef) + 1/ (N(K, (N
find that optimizing two sets of parameters (i.e., {ut, Eg} } and u simultaneously is unstable and leads to a trivial solution. Thus, we adopt an iterative optimizing process by using stop mop-gradient (SG) operator. Specifically, loss L(SG(1), HZ; 27) to optimize {er, E; J. and L(wb SG Σ()) to optimize the encoder fø-
use the
Furthermore, simply minimizing the KL loss cannot prevent the divergence of the unbounded term (A x, ([( )),), which may result in deviated representation distributions and unstable training. Hence, add triplet loss to leam representations, as
C k() Ep (p/mmx((o ((x( pf) d(p, ,Hi) + + 0)),
(4)

---

Preferencc dalaset
Taglet
fv
dtwr www.
w
d(w", wwn)
(1) Versatile Preference Representations
Generated Unajectories
stepss
Mutual Information Regularized Diffusion Model
Representation Guided Trajectory Generation
(T) and the optimal one W from trajectory segments which comprise positive samples -- and negative samples T (2) We augment the diffusion model with auxiliary mutual information term / (To; w) to ensure the alignment between and w. (3) During the inference process, the diffusion model conditioned on w can generate desired trajectories aligned with preferences.
Figure 3: Overview of our method. (1) We learn preference representations
where d is the Euclidean distance in the embedding space. The triplet loss calculates the similarity between wa and preferred representations w,",, and the similarity between w/ and unfavorable ones w; respectively. By regulating the distances between {ur H4, } and μ4; the training process is stabilized. Mcanwhile, minimizing the triplet loss also guarantees that the optimal embedding is more similar to wi and less similar to w; while wit and w,-" stay away from each other. We set as a margin between the two similarities and adopt the same iterative optimizing process for L(1,, pC)). The illustration in Figure 3 captures our learning process. For brevity, we have omitted details related the distributed form and multi-task learning components while retaining the core methodology.
Prompting with wf. By optimizing the triplet and KL loss, w gradually aligns with more preferred representations, converging to the optimal representations, After training, the diffusion model can be prompted by (nz, ))) for a specific task The model will then generate the optimal trajectory for task i with the guidance of conditions. The task can be novel task not present in the training set, and the diffusion model will try to generalize to the new task in the representation space.
3.2 Representation Aligned Trajectory Generation
Given preference-based versatile representations, we train diffusion model to generate trajectories that align with the representations, Prior work utilizes classifier-fre free guidance and aligns the generated samples with low-dimensional conditions, such as returns. However, we find this is insufficient to capture the complex relationship between conditions and trajectories. Using the representative method Decision Diffuser
5 as example, Figure I reveals the relationship between the return conditions and true returns of generated trajectories the Hopper-medium -xpoerr task. While it desirable to generate trajectories with higher returns as we increase the return condition, in practice, the attributes of generated samples do not exhibit a smooth transition with changes in the condition. Empirically, Decision Diffuser uses target return of 0.8-0.9 for the generation process. This difficulty becomes more severe our method because we cannot exclusively search the high-dimensional representation space and find an empirically effective representation. Therefore, it is critical to enhance the alignment between generated trajectories and previously learned preference representations.
MI regularization. Inspired by recent works on generative models [28] 29] 21), we introduce an auxiliary optimization objective aimed strengthening the association between representations w ~ qu (w(T) and the generated trajectories Tp- Specifically, we augment the learning objective from Eq. with regularization term based mutual information between To and Formally, we train the conditional diffusion model using the following combined objectives:
max By(o) [log p(To)] Ç - [(TO,
(5)
5

---

Algorithm Algorithm of CAMP
Require: Multi-task preferences D. trajectory encoder (+)), diffusion model ca(-). predictor g²(-). inverse dynamic model gw(+--). optimal representations w
1: // Training
2: for each batch { - TT
from D do 3: Update encoder fa with ( } (ref. Eq.
&
4: Update optimal wt," with ((SG())) unt) (ref. Eq.
Train MI-regularized diffusion model with Eg. (8)
6: Update inverse dynamics model 9., with Eq.
7: end for
8:
// Inference
9: for each step in one episode do
10: Given current state 8t and Iearned representations w
11: Generate trajectories 7 = {st, 8t+1,"" ... 8eth] after K denoising steps Po(Te-1/Te wz).
12: Predict actions {at, ,,thh where a, = 9w (St; 8t-1).
13: end for
where g(To) indicates the trajectory distribution of the offinn dataset DD and Ç is hyper-parameter controlling the strength of regularization. Eq. (5) encompasses processes such as sampling trajectories ((T)), obtaining corresponding representations (w(())) via f.p and the denoising process pe (e(Tk-117 Th: w)) to derive TO.
Tractable objective. In practice, sampling to from p(ro) maximize the MI objective requires the diffusion model to denoise K steps from TK- This process imposes huge computational burden and may lead to potential memory overflow due to the gradient propagation across multi-steps. Consequently, there arises need for an approximate objective, and an alternative approach is to replace the optimization (Too w) with I [Tk: w). This substitution is motivated by considering the relationship between To and The as described by the diffusion process т vā, + vI 0rEp faitfuse (TO). According to the data processing inequality [30] we have
I(To: w) 2 (faiifuse (Toi:: w) = [[TT w), As result, [TTs; w) serves as lower bound of I(To: w). thus maximizing [(Ta; w) also maximizes [ w). In this case, Eq. (5) can be rewritten as:
(6)
max E (et [lo p(r)) + Ç [(TE. w).
We note that the objective in Eq. 7 can be rewritten into an equivalent form that can be optimized efficiently (see $A.2 for detailed derivation).
Proposition 3.1. The optimization objective in Equation 7 can be transformed to (0 Φ) = (Fothoo
[Cotho w)) - Eep()r) [Dxi (Pss (w)))l
(8)
The first term resembles the ELBO term in Eq. (1) denoted as Lelbo (wo (2tg
To. C w) which is the same as standard conditional diffusion models in Decision Diffuser [5]. The ELBO term aims estimate the trajectory distribution via a conditional diffusion process, thus we adopt similar conditional score-matching objective to optimize it. The second term contributes to enhancing the alignment between To and we where Pyl (w)) is empirically averaged samples
D via f and p(Tk) x faituse(T)) with To sampling from q(T)). We can minimize the KL divergence to optimize go (w(
(T)). ), variational predictor to predict the condition w from the denoised sample Tj. In practice,
we instantiate it with a neural network represented by o. taking predicted noises (T(Tk) as inputs.
3.3 Algorithmic Description
The entire procedure is shown in Algorithm [ During training, iteratively update the representation encoder f/ and the optimal representation we to learn versatile preference representations. Then we update the parameters of the conditional diffusion model via the loss function in Eq. (8). To decode the actions from a predicted trajectory, an inverse dynamic model is learned by using a supervised loss from the dataset, as
min E, E. I|a Iw (s s')11

---

For planning in a specific task i, we use the optimal representation wi and the current state St as a condition to generate an optimal trajectory (t...... St+h]. Then the inverse dynamics model 4: (8,, 8t+1) is used to decode the action from two consecutive states.
4 Related Work
Diffusion Models in RL. Diffusion models exhibit significant advantages in generative modeling and characterizing complex distributions. On the one hand, they can be applied for modeling policies, with many research works [31]6]] suggesting that diffusion models are more effective at capturing multi-modal action distributions than other policy types like energy-based policies. On the other hand, diffusion models are adept at directly predicting trajectories via conditional generation, adopting conditions such as value gradients 37 77, returns [5], prompts [8] Our work also trains diffusion models to generate trajectories, but focuses on guidance from preference representations. Regarding alignment with human preferences, recent works include the integration with designed attributes for various behaviors [6], tinettuned policy with preferences [38] 39]. and preference augmentation techniques improve trajectory generation [40]. While these works focus single-task settings, our work seeks a versatile solution by considering multi-task preferences.
Learning From Human Preferences. Current methods of learning from preferences can be categorized into two groups: direct learning and indirect learning. Indirect learning methods involve learning a reward model and incorporating existing RL algorithms. They employ techniques like off-policy learning [24]. pseudo-labeling and data augmentation [41), iterative updates of reward models and policies 422 leveraging Transformer architectures to learn reward models [15]. On
the other hand, direct learning methods bypass reward model learning and directly incorporate human preferences into the learning process. This approach involves various strategies, such combining decision transformer styles with preference embeddings 44 mapping reward functions to optimal policies 44 45), or aligning models using extended Bradley-Terry comparisons
(46). This work falls under the category of offline direct learning approaches, framing it as a sequence modeling problem.
Our work considers challenging multi-task settings. While much prior research attempts to find Pareto optimal solutions considering the trade-offs between different preferences, these methods often require heuristic vector selection for unknown Pareto fronts 47 48), millions of pre-collected preference labels and further online queries [10), or combination with Gaussian processes to learn preference relations [49]. In contrast, approach does not require any heuristic methods and learns trajectory representations aligned with preferences from offline data.
5 Experiments
In this section, we will validate our approaches through extensive experiments. Our focus revolves around addressing the following key questions: (i) Can our approach demonstrate superior perfor- mance compared to existing approaches? (ii) Does the trajectory encoder discern different trajectories corresponding to multi-task preferences? And does [wi ke[m] align with the optimal trajectories? (iii) Can the diffusion model capture the trajectory distribution and generate trajectories aligned with preferences? (iv) How about the generalization ability of our method on unseen tasks? (v) To what extent are multi-dimensional representations and regularization crucial our approach?
5.1 Setups and Baselines
We conduct experiments on two benchmark datasets, Meta-World [18 for multi-task scenarios and D4RL [17] for single-task scenarios. Within evaluations MetaWorld, we consider two distinct datasets: near-optimal dataset, which comprises the entire experiences obtained during the training of SAC [50] agent for each task, and sub-optimal dataset, encompassing only the initial 50% of the replay buffer. More details about datasets and baselines are provided in $
Categorically, our baselines encompass three types of approaches: offline preference-based meth- ods, offline reward-based RL methods, and behavior cloning (BC). Within the realm of off wefeeeece-basee methods, our selections include: 1) PT [15. using transformer network to model reward functions, integrated with RI. algorithms: 2) OPRL [6), which employs ensemble-based reward functions; 3) OPPO 43 adopting one-step process to model offline trajectories and
7

---

Near Optimal Dataset
Sub Optimal Dataset
Aeorage Success Bate
80
800
60-
60
40-
40
20
20
MT-8C
MT4QL
MTDH!
MI-OPPO-p
MT-OPPG-W
Durss
Figure Average success rates in MT-10 benchmarks trained with different datasets, Orange bars are reward-based methods, while green bars represent preference- based methods. Detailed comparisons for each task can found D
Table 1: Performance comparison in D4RL benchmarks with scripted preferences. The subscript < indicates the bascline with access true reward functions, while and indicate the reported
scores and our re- re-implementation with default parameters, respectively.
Environments
BC
IQL.
PT
OPRL.
OPPO
OPPO
CAMP (Ours)
hallcheetah mediun mediunn
42.4 1 0.2
48.3 1 0.2
0..6..0.1
42.0 -: 2.8
43.4 0.2
42.5 0.2
45.0 0.3
halfcheetal-mmedee peplu
35.7 2.3
4.2
42
41.5 =.6 2.6
39.800
2.
40.5
2.0
halfcheetal maadld--ddiiiiinge expert
56.0 1 7.4
94.7
86.8
90.5 1 4.0
88.9 - 2.3
89.6 0,9
95.0
1:1 2.1
walker2d-medlum
63.3 16.2
80
16.
10.3=1111
85.029
11.5388
7399
21.8 10.2
3.0
75.7=339 39
53.3 62
71.7 11.7444
19.7 = 10.3
60.5
1.1
walker2d medium--cpprrt
99.0 16.0
111.7 0.9
1.0.4
5.6.4
105.0 2.4
19.1
104.8 3.0
hopper medium
53.5 1.8
3.8
25.7 12
45.6 0 3.5
86.3
3.2
48.7 4.2
59.3 0.8
hopper medium replay
10,9
hesper-mediuu-epper
$2.3 + 4.0
107.4 + 7.8
44.0 + 88
68.8 18.1
108.0
5.1
99.5 + 23.5
108 + 1.0
Average
50.4
81.6
65.7
61.4
79.7
59.2
71.6
preferences, avoiding reward learning. For methods enjoying access to true reward functions, our selection includes: 1) IQL [4) which performs in-distribution Q-learning and achieves significant performance; 2) MTDiff [8), a method leveraging diffusion models in multi-tat task scenarios. Since many baselines do not consider multiple tasks, we make modifications and mark them with 'MT-"
5.2 Performance Comparison
Multi-task Performance on Meta-World We assess the multi-task performance using MT-10 tasks and present the results in Figure 4 Our observations are as follows: 1) Given near-optimal datasets, CAMP outperforms MT-BC and MT-IQL trained with ground-truth rewards, with only small performance gap compared to MTDiff.. 2) Given sub-optimal datasets, CAMP demonstrates comparable performance to MT-IQL and surpasses other baselines, highlighting its robustness to dataset quality. 3) Two variations of OPPO fail to learn effective policies for multiple tasks, exposing the limitations of existing preference- based RL methods in handling multi-task scenarios. In comparison, CAMP achieves a performance improvement of nearly four times.
Single-task Performance on D4RL As shown Table T CAMP demonstrates superior per- formance across all D4RL Mujoco tasks compared to BC. Additionally, it exhibits comparable performance with IQL in medium- tasks. When compared to offline preference-bast such as PT or OPRL, CAMP showcases significant improvements, particularly in hopper tasks. While OPPO is an effective preference-based method, we observe its sensitivity to random seeds. In our re-implementation using default hyperparameters, its performance degrades in walker2d and hopper tasks. In contrast, CAMP provides aligned trajectory generation and favorable performance.
methods
5.3 Visualization
While showing superior performance, does CAMP learn meaningful representations perform desired conditional generation? this part, we map trajectory segments two-dimensional space via T-SNE [51] visualization and analyze properties of the trajectory encoder and the diffusion model.
Do representations w discern different trajectories? To assess the capabilities of discerning different trajectories and aligning with the best trajectories, we sample several trajectories from D and project them the latent space via fu. with subsequent T-SNE visualization. The results at different stages during training are illustrated in the left panel of Figure 5] With increasing training steps, we find a gradually clear classification among trajectories with difterent returns. Meanwhile, the optimal representations w/, represented as red dots, gradually approach trajectories with higher returns.
8

---

Esoch
Etech 40
Epoch 605
C
20
Ofline
310
40-
Visualization of trajectory representations MT-10
-20-
Beech
Epoch 40
Esech 100
-40-
23
800
-50
-40
-20
20
50
89
-20
23
(c) Visualization of generated trajectories
-12
-22
28
(b) Visualization of representations on Hopper- medium-expiettt
Figure 5: Left: Brighter dots indicate trajectories with higher returns. Red dots represent each dimension of w Black triangles in (b) mark trajectories with the highest return. fe can separate trajectories from different tasks and with different returns. w aligns with the optimal trajectories for each task. Right: Guided by w diffusion models can generate trajectories To that mainly lie around i (shown black circles), which represents better trajectories in offline data 70-
Does the diffusion model generate trajectories aligned with preferences? While w provides useful guidance, want to validate the generative ability of the conditional diffusion model. We map the generated trajectories to latent space and compare them with offline trajectories. As demonstrated in Figure ((c) the green points denote the generated trajectories, while the red dots represent w We observe that these generated trajectories for 6 tasks align closely with wy, which represent the more favorable trajectories within the dataset. This phenomenon showcases that generated trajectories guided by w align with preferred trajectories, validating our approach's effectiveness.
5.4 Analysis on Generalization Ability
To validate the generalization ability, evaluate generated trajectories guided by dे & {w; "w"hien that are learned from trajectories of unseen tasks. Wi from those new tasks are learned using the method described in Section 3.1 We hold the diffusion model trained on MT-10 tasks fixed and assess its performance when faced with unseen representation conditions. We compare its performance with that of MT-BC, MT-IQL, and MTDiff, all of which are trained using the same settings. As depicted in Table 2 our approach exhibits favorable performance on unseen tasks and outperforms baseline methods by a considerable margin. Further analyses are presented in H
Jie[m]
Table 2: Generalization performance five unseen tasks. CAMP exhibits superior performance.
Unseen Tasks
MT-BC
MT-IQL
MIDiff
CAMP
cutton-preess-aad wall-y2
0.0 0.0
21.0 21.0
16.0 1 17.3
22.0
333
button-preee button-press-toppee press-topdoom-wall--
21.0 + 19.0
33,0 + 19.0
72.0 8.6
66.8 5.0
handle press-vt
43.0 13.0
60.0 4.0
36.7 4: 12.3
76.8
7.2
hundle-press-siin press--ers---
0.0 0.0
1.0tt 1.0
2.8
11.6 1.5
peg unplug-side side-v2
0.0 4: 0.0
0.0 1: 0.0
2.7 1.9
1.6 1.6:1 1.5
Average
12.4
19.9
17.8
35.8
5.5
Ablation Study
This part delves into the impact of multi-dimensional representations and the MI regularization term. Due to space limitations, detailed implementations refer to SC] Here, we highlight key conclusions: Learning nuulti-immensionll representation and choosing suitable dimension are crucial. When the dimension is too low, such as |w] = 1, the representations are insufficient to capture preferences trajectories and provide effective guidance. Conversely, when the dimension too high, as in [w] 64, the representation space becomes challenging to learn, resulting in inferior performance. 2) The auxiliary loss of mutual information plays pivotal role our framework. Without this regularization term, method's performance on all MetaWorld and D4RL tasks degrades.

---

6 Conclusion
This paper introduces regularized conditional diffusion model for alignment with preferences in multi-task scenarios. Based on the versatile multi-task preferences, our method acquires preference representations that differentiate trajectories across tasks and with different returns, as well as an optimal representation aligning with the best trajectory for each task. By regularizing exiting diffusion models in RL with mutual information maximization between conditions and generated trajectories, our method can generate desired trajectories by conditioning on the optimal representation for each task, ensuring aligmment with preferences. Experimental validation demonstrates the favorable performance and generalization ability of our method. Future work may involve using faster sampling methods to enhance algorithm efficiency or extending to fine-tuning foundation models.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No. No.62306242) and the Yangfan Project of the Shanghai Municipal Science and Technology (Grant No. No.23YFI1 14412204)
References
[1] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning one big sequence modeling problem. Advances in neural information processing systems, 1273-121, 2021.
[2] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[3] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion flexible behavior synthesis. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 9902-9915. PMLR, 2022.
[4] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URI 1d=69n2s9ZJWFB
net/forum?
[5] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023. URL httpshtss:://// btttpe://openrevies: wett:/oorrrdddds1foooootoo.
[6] Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni. Yao Mu. Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model. CoRR, abs/2310.0200 2023.
[7] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 26087-26105. PMLR, 2023.
[8] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang. Bin Zhao, and Xuelong Diffuusio model is an effective planner and data synthesizer for multi-task reinforcement learning. In Thirty- Conference on Neural Information Processing Systems, 2023, URL http:tt///peererriiii net met/forum?ice fddM
[9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[10] Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human tohe--oooot--ooppp rl. In Conference on Robot Learning, pages 2014-2025. PMLR, 2023.
[11] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
[12] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27 277444, 2022.
10

---

[13] Dongyoung Go. Tomasz Korbak. Germán Kruszewski. Jos Rozen. Nahyeon Ryu, and Mare Dymetman. Aligning language models with preferences through f-divergence minimization. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 11546-1 11533. PMLR, 2023.
[14] Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36, 2024.
[15] Changyeon Kim, Jongjin Park, Jinwoo Shin. Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference trans- former: Modeling human preferences using transformers for RL. In The Eleventh International Conference on Learning Representations, 2023. URL https:s://ppereeeiieeeeee// wwt. www.:e/oorrourddeeeetttsxx
[16] Daniel Shin, Anca D. Dragan, and Daniel S. Brown. Benchmarks and algorithms for offline preference- based reward learning. Trans. Mach. Learn. Res., 2023, 2023.
[17] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucket, and Sergey Levine. D4rl: Datasets for deep data drivenriven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[18] Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn. and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages 1094-1100. PMLR. 2020.
[19] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208. 2022.
[20] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In ICML, volume 202 of Proceedings of Machine Learning Research, pages PMLR, 2023.
[21] Yingheng Wang. Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. InfoDiftusion: Representation learning using information maximizing diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors. Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 36336-36354. PMLR, 23-29 Jul 2023. URL. httpss//proceedingg.. mlr. pttps//ve2e/aaeegeeeeeeeeeeeeeee html
[22] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preferenc--base reinforcement learning methods. Journal of Machine Learning Research, 18(136): 46, 2017.
[23] Ralph Allan Bradlcy and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
[24] Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: feedback- interactive reinforcement learning via relabeling experience and unsupervised pre-training. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 6163. PMLR, 2021.
[25] Mohammad Gheshlaghi Azar, Mark Rowland. Bilal Piot, Daniel Guo. Daniele Calandriello, Michal Valko, and Rémi Munos. A general thcoretical paradigm to understand learning from human preferences. arXiv preprint 2rXi:23300. 2006,, 2023.
[26] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv: 10..0733, 2023.
[27] Jocy Hejna, Rafacl Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without arXiv preprint arXiv:2310.13633 2023.
[28] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovac: Information maximizing variational autoen- coders. arXiv preprint 1706. 100.0077..0222,, 2017.
[29] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[30] Normand J Beaudry and Renato Remner. An intuitive proof of the data processing inequality. Quantum Information & Computation, 12(5 12(5-))442-441, 2012.
[31] Zhendong Wang. Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations. 2023. URL httpss://operreeeee... wwt.:/eet/oooteee AHVFDP1--

---

[32] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
[33] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304.1000 10573, 2023.
[34] Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations. 2023. URL Jtttpttt:///pperrrrii
aet/forum?? 422s33a2k
[35] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgeseu, Sergio Val- carcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, het//oorumiid-Pv1GPPP
2023. _
[36] Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Large-scale actionless video pre-training via discrete diffusson for efficient policy learning. In Neural Information Processing Systems, 2024.
[37] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 20725-20745. PMLR, 2023.
[38] Chenyou Fan, Chenjia Bai, Zhao Shan, Haoran He, Yang Zhang, and Zhen Wang. Task-agnostic pre- training and task task-guided fine-tuning for versatile diffusion planner. arXiv preprint wwXiv22409.1944/ 2024.
[39] Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi, and Chenjia Bai. Forward kl regularized preference optimization for aligning diffusion policies. arXiv preprint arXiv:2409.05622 2024.
[40] Anonymous. Flow to hetter: Offline preference-hased reinforcement learning via preferred trajectory generation. In The Twelfth International Conference on Learning Representations. 2024. URL https
/oopenreview.coomm www. www//eeeuuuuneee www.foorrunndddnssssssee
[41] Jongjin Park, Younggyo Seo, Jinwoo Shin. Honglak Lee. Pieter Abbeel, and Kimin Lee. SURF: semi- supervised reward learning with data augmentation for feedback-effiiient preference-hased reinforcement learning. In ICLR. OpenReview.net, 2022.
[42] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-hased reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 22270 22284, 2022.
[43] Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preferenee- guided policy optimization. In ICML, volume 202 Proceedings of Machine Learning Research, pages 19753-15766 PMLR, 2023.
[44] Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL tattps://arxiss org/ abs/2305 2305 18290
[45] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-far-allll Multi-ohjective direct preference optimization. arXiv preprint arXiv:2110.037008 2023.
[46] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. / 17492, 2023.
[47] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In International Conference on Machine Learning, pages 6597-6607. PMLR, 2020.
[48] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advancex in neural information processing systems, 32, 2019.
[49] Adriana Birlutiu, Perry Groot, and Tom M. Heskes. Multi-task preference learning with gaussian processes. In The European Symposium on Artificial Neural Networks, 2009. URL https://api semanticscholar. (t://Coppsl:::9293:0
12

---

[50] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[51] Laurens Van der Maaten and Geoffryy Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[52] Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Prompt-tuning decision transformer with preference ranking, arXiv preprint arXiv:2305. 09648, 2023.
[53] Lili Chen, Kevin Lu, Aravind Rajeswaran. Kimin Lee. Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
[54] Scott Emmons. Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022. URL
https://openrevicce www.://wwwwe/eowmnneoeeee S174XXXII
[55] Peter Vamplew, Benjamin J Smith, Johan Källström, Gabriel Ramos, Roxana Rádulescu, Diederik M Roijers, Conor F Hayes, Fredrik Heintz, Patrick Mannion, Pieter JK Libin, et al. Scalar reward is not enough: A response to silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2):41, 2022.
[56] Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Zero-shot preference learning for offiine rl via optimal transport. arXiv preprint arXiv:2306.03615, 2023.
[57] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2020.
[58] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances Neural Information Processing Systems, 35:5775 -78777777777777 2022.
[59] Bingyi Kang. Xiao Ma. Chao Du, Tianyu Pang. and Shuicheng Yan. Efficient diffiusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[60] Wenxin Tai, Fan Zhou, Goce Trajcevski, and Ting Zhong, Revisiting denoising diffusion probabilistic models for speech enhancement: Condition collapse, efficiency and refinement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13627-13635, 2023.
[61]
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang. Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacca: Pareto alignment via preference adaptation for llms. arXiv preprint arXiv:2402.0200 2024.
[62] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.0897 2024.
[63] Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On diversified preferences of large language model alignment, 2024.
[64] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objecttve alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402. 10207, 2024.
[65] Yiju Guo, Ganqu Cui. Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, al. Controllable preference optimization: Toward controllable multi-objective alignment. Xiviv preprint arXiv:2402. 11922.9900 19085, 2024.
[66] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model, arXiv preprint arXiv:2311. 13231, 2023.
13

---

A Theoretical Analysis
A.1 Derivation of ELBO in Equation
Here we derive the ELBO of the diffusion model by considering conditional denoising process (0 e):
log (oo) = log p(c)p(xo.. c)dx1.xde
ple)ttraxxloollznxxxox (20 de
log
( zo (c(
p(c)p(zo.xle)
llog (o(
p(c)p(ro.klc)
log ((01,
log (2 c) (cc(20))((1111)))
-k Po (o 12
E (cl
Ilx=2 (( 2 12---111- 1,,
p(c) po (o
p(ax)
k== Po (3 1T c)
E (cl
log qu (clpa))
g00) (20)
k= (
log p(c)
p(xx)
k= po (1
1o)
(c
po (x
9v (c(
g [20)
k= q (T (
p(c)
p(xx)
R==2 pa (2 cz),
(cl
log
Po (0oo
UN (c(oa)
g( (2))
I (
log p(c)
p(ax)
log po (#k | C)
log po + log ( (x))
+
Iv (c(2a)
g 12 Fo)
log p(c)
log p(xxk)
(c(oo))
[E + (z())
Poogpooxolo111 c)]] ((
Iv (c((0)))
q(<x<< (20))
K
[log po 1 Fk e)
E [a 1
B la mo)
q Ik, 2o) ((0x-112x.20)
k=
(mx) (mo)
(e [log (2ol
DKL (q(xx (m))))((k))
DKL (co
reconstruction term
prior matching for ux
prior matching
E
(elzo [x( E
[KI (1(x
demoising matching term
(10)
Following previous work 19
21
the above ELBO provides interpretations for each term:
The reconstruction term resembles the part of the ELBO of vanilla variational autoencoder,
and can be optimized using Monte Carlo estimates 19 In 2 this term is learned using a separate decoder.
The prior matching term for Iæ indicates the discrepancy between the distribution of the noisy version of zo after K steps and the standard Gaussian prior. This term has no trainable parameters, and we ignore it during training.
The denoising matching term measures the discrepancy between the ground-truth denoising function / xo) and the approximated denoising transition function po Po c). It is minimized when the approximated denoising transition stays close to the ground-truth denoising transition step.
14

---

The prior matching term for indicates the discrepancy between the approximate posterior qu (c)oo) and the prior p(c), which can be standard Gaussian distribution.
A.2 Derivation of the loss function in Equation (
First, we give several explanations about Eq. 77 which provides a lower bound of Eq. ( This
relationship is built top of that [ w) 2 [TT,, w). Intuitively, 7 consists of more Gaussian noises than To. thus providing less information about w. In particular, when k K, we get a pure Gaussian noise so that [(TK; w) 0.
Then we analyze Eq. in two parts and derive them to two terms in Eq. (8. respectively. For By(ro) (log p(To)], rewrite Eq. (TO) by setting To To, c and average the terms over the data distribution g(To):
E Bg((a) [log] 2 E [(etbo(ro..))
By((o.ra) (wttro) (oggpu w)]]
Bytto) DRI. (g(((((TxTT)) (p)(T))
(11)
Bo (Bga()) DKL( (1o(Tra) [
g() [KiI (gp (1o(
According to previous work 19 this optimization problem can be simplified as:
(1 (1-a
arg min
[l|co co(Tie w. 11]]
(12)
" (k) (1 āg ax)a
where og is a function of a coefficients and 6g (e; 0, /) is the source noise that determines TM from To- For the mutual information regularization term / [Tk, w), it can be derived as follows:
I [(Ty w) = H (w) (k)
(ww) / p(w = w', logp(w w' TH ddw'dd drw
Bp(wllra) [log p(w'(Tx)
(w) + Ep(Tx)
(o( go (w (Tx)
(13)
(w) + Ep(ra)
DEL [p(w' (w lo
+ (p (w((mm))
2a
2 (w)) + Ep(r.) [m [log (w [m)]]]
We introduce a lemma derive the above inequality. Please refer lemma 5.1 of (2 for detailed proofs.
Lemma A.1. For random variables X, Y and function f(x, y) under suitable regularity conditions: Ezwx, Ja [2, y)] = Ex X..xXyy- a If v)].
By using lemma A.1 we can derive that
I I[Tk: w) 2 Hww) + Ep(r.) E p logg (w [T]]]]
H(w) + (w)) [Ep(Tu|n) wy) [E(("") (log qs [Tx)]]]
Emp((w) |- Po (w)) + (p)) [p(Tw) [log (w(())))
(14)
Epw(w) log qo (w((TT)
= -Ep(To)
(w)(w) = --Ep(ra) [DEL (p (w) (w(l)))
In Eq. 14
we omit the condition of p(Tx w) as P(Tx) because 7. comes from To D by adding
Gaussian noises. Combining Eqs. and (1). we can obtain the tractable objective in Eq.
15

---

B Implementation Details
In this part, we introduce some details about our methods and evaluation, including benchmarks, baselines, and implementations. We then provide specific comparisons with several related works, including Decision Diffuser, MTDiff, and OPPO.
B.1 Datasets
MetaWorlk MT-10 In this study, we employed MT-10 tasks from MetaWorld as benchmarks to assess multi-task performance. These tasks share similar dynamics, involving the manipulation of a Sawyer robot to interact with various objects to achieve diverse manipulation goals, as shown in Figure 6 Each task exhibits distinct state spaces and reward functions, presenting significant challenge for learning strategies. The primary evaluation metric we utilized is the success rate of task completion, with the attained return serving as secondary measure.
Considering this is focused on an offline learning setting, we followed the methodology of MTDiff 8 employing replay buffer during SAC [50) training as the offline dataset. For each task, trained an agent using SAC to progress from a random policy toward an expert policy. Additionally, categorized two distinct datasets: the near-optimal dataset and the sub-optimal dataset, differing in the number of expert trajectories included. The near-optimal dataset comprises all the replay buffer data, totaling 100 million transitions, while the sub-optimal dataset contains only the initial 50% of this data.
Following previous work 43 15 16,, We collect preference data with scripted teacher. In particular, for multi-tast preferences, we construct intra-task preferences among trajectories by comparing their returns and inter-task preferences among tasks based the task relevance, illustrated in Figure 2
button press
door open
drawer close
drower open
insert side
pick place
push
reach
window open
window close
Figure 6: MetaWold MT-10 tasks. tasks.
The goal is learn policy that can succeed diverse set of
D4RL To evaluate our methods on single-task settings, we conduct experiments on the Mujoco locomotion tasks from D4RL benchmarks. Three tasks Halfcheetah, Walker2d, and Hopper chosen, with three types of datasets, including medium, medium-replay and medium-cxpert, We obtain preferences from scripted teachers, following previous work [15] 43 [6]
B.2 Baselines
As outlined Section 5.1, the baselines we compare can categorized into three groups: offline preference-hased methods, offline reward-based RI. methods utilizing ground-truth reward functions, and behavior cloning. the following sections, we delve into the specifics and experimental details of these baselines to provide comprehensive comparative analysis.
PT PT utilizes transformer architecture learn scalar reward model, which is then used to optimize policies with the IQL algorithm. We use its official implementation and default parameters
OPRL OPRL combines IQL with reward functions from ensemble-based disagreement. We use its official implementation and default parameters
http:///iihtnccommmmmmmt--100//0 Transfffrr huaps://gthuub.om///ddddthhi///
16

---

OPPO & MT-OPPO-p & MT-OPPO-w OPPO shares a similar motivation with us for avoiding explicitly reward learning. It models offline trajectories and preferences in a one-step process. Specifically, it optimizes offline hindsight information matching objective for seeking conditional policy and a preference modeling objective for finding the optimal condition. We use its official implementation and default parameters | We observe unstable results over 5 random seeds, which not the same as the reported results in the paper. We speculate that OPPO may be sensitive to different random seeds.
To make it suitable for multi- task settings, we make two types of modifications to its original version, named 'MT-OPPO-p' and MT-OPPO-w'. For 'MT-OPPO-p', we add task id inputs for the condi- tional policy, so that the original policy [aass, 2) can be extended to ( 2, ID). For "MT-OPPO-N we modify OPPO using the same method as our approach, extending its representations to multiple dimensions, so that the policy becomes π Ta 8, Z, ID) and z is a multi-dimension representation. Following previous work 8 we project task IDs to latent variables via 3-layer ML..
BC & MT-BC Traditional behavior cloning learns direct mapping from states to actions r(ass). In our experiments, encode the scalar task ID to latent variable via MLP and concatenate the latent variable with the original states. Therefore, MTBC utilizes a conditional policy by conditioning on task IDs. We modify it based implementations from CORL
IQL & MT-IQL IQL is an effective offline RL method, which performs in-distribution Q-learning and expectile regression. We use the implementation from CORL and make similar modifications to MTBC.
MTDII MTDiff diffusion-basse method that combines Transformer backbones and prompt learning for generative planning in multi-task settings. We use its official implementation and default parameters
B.3
Our implementation
Our code is built on Decision Diffuser and OPPO We leverage their implementations of diffusion models and transformer-batt encoders while developing the representation learning process for multi-task scenarios and the auxiliary mutual information regularization.
7
Following Decision Diffuser, we use the temporal U-Net architecture to predict noise, where timesteps and representations w are separately mapped to 128-dimensional vectors by 2-layered MLPs.
We employ 200 denoising steps, consistent with previous work 8
The training details of the inverse dynamics model g are aligned with those of Decision Diffuser.
For the trajectory encoder, which projects trajectory segments 7 to latent representations w, we adopt similar Transformer architecture to that of OPPO, but we learn distributional representations.
The conditional guidance weight in diffusion models is set to 1.2 for most tasks and 1.5 for the
expertt task.
The learning rate of the diffusion model is 24 with the Adam optimizer.
to 6 in Meta tasks and l in D4RL tasks, with results averaged over multiple seeds. In Meta World benchmarks, as each environment has 50 random goals, evaluations averaged over these 50 random goals.
Training steps
The horizon h of trajectories is set 20 in the MT-10, halfcheetah, and walker2d tasks, and 100 in the hopper tasks.
Paunss/ggiibb.commiinntttttttttt
htaps//githbb.co//xkkbbbbbbooo
17

---

Batch size is set to 256 for halfcheetah and walker2d tasks, and 32 for hopper tasks and each task in MetaWorld MT-10 tasks (total batch size is 320).
The regularization coefficient Ç is set to 0.1 for the MT-10, halfcheetah, and hopper-mediuge tasks, 0.5 for the mulker2d-mediune and mallkerdd-meiium--ee experrt tasks, 0.01 for the hopper- medium-replay and walker20-mecium-rree medium-replay tasks, and 1.0 for the hopper-medium mediim--deepeeeeeee expeert task.
The dimension of preference representations is set to 16.
We conduct training on an NVIDIA GeForce RTX 3090. Training time varies with task complexity, approximately 30 hours for D4RL tasks and 59 hours for MT-10 tasks.
B.4 Difference with other methods
Decision Diffuser Decision Diffuser is an effective method for conditional generative modeling, with conditioning options including returns, constraints, skills. In particular, when using returns as conditions, Decision Diffuser utilizes normalized returns and sets the target return as 0.9 for most tasks during the generation process. In Figure []] we have revealed that the conditiond paradigm may not ensure alignment between specified return conditions and generated trajectories. Unlike Decision Diffuser, we propose regularization term for conditional diffusion models and obtain enhanced alignment between given representation conditions and generated trajectories. As shown in Figure 5 generated trajectories To under the guidance of w cluster around the region near wf- Our method also differs from Decision Diffuser in the utilization of preference data instead of reward labels and the versatility across multi-task scenarios.
MTDifT MTDiff employs diffusion models for modeling large-scale multi-task offline data. Similar to Decision Diffuser, MTDiff adopts classifier-fres guidance but introduces prompt learning for both modeling policies and trajectories. Specifically, it utilizes normalized cumulative returns and regards task-relevant information as prompts. By employing these task task-specific prompts as conditions, MTDiff distinguishes between different tasks and generates desired trajectories for each specific task. The prompts consist of expert demonstrations in the form of trajectory segments, akin to the approach in PromptDDT [52] In contrast, our method eliminates the need for expert demonstrations and instead extracts multi-dimensional representations from multi-task preferences. These representations serve conditions for the diffusion model. Experiments conducted on MetaWorld tasks demonstrate that our method achieves comparable performance to MTDiff, all without the necessity for reward labels or demonstration prompts.
OPPO In the realm of offline preference- reinforcement learning, OPPO models offline trajectories and preferences in one-step process, circumventing the need for reward modeling. Specifically, OPPO learns an optimal context using preference modeling objective and subsequently optimizes a contextual policy. During the learning of contexts, OPPO constructs positive and negative samples and uses the triplet loss to optimize the contexts. With these learned contexts, OPPO then develops a conditional policy, akin to DT ( and RvS [54]. However, our method utilizes the KL loss and the triplet loss to optimize representation distributions, which align with multi-task preferences. Moreover, our approach focuses on conditional diffusion models and the alignment for trajectory generation. It is noteworthy that our method demonstrates more stable performance and excels in multi-task settings, exhibiting successful generalization to unseen tasks. In contrast, OPPO is specifically designed for the single-task setting.
C Ablation Study
In this part, we aim to dissect and analyze the influences of two key elements: 1) the dimension of representations, and 2) the auxiliary mutual information optimization objective.
C.1 How does the dimension of w affect our method?
When the dimension of is reduced to 1, the learned representations in our method resemble those of vectorized reward models (55) or distributional rewards (56). Conversely, the dimension
too high, both learning representations and aligning conditional generation become challenging
18

---

80
40
Average
20
[M] -1
(W]
8
[4]
16
[w]]
32 64
Figure 7: Ablation on the dimension of w.
due to the increased complexity the representation space. We conduct an ablation study with |w] E 1, 8, 16, 32, 64, where |w] = and |w|| = 64 represent two extremes. As presented in Figure 7 the performance on MT-10 tasks significantly decreases when |w| = 1, and it exhibits performance degradation when the dimension of representations is too big. This implies that should adopt suitable dimension for preference representations, and we choose |w] 16 in our experiments.
C.2 Is the mutual information regularization critical?
Our approach employs a regularization term based on mutual information to enhance the alignment between preference representations and generated trajectories. this section, we investigate the impact of this regularization term on the final performance. Specifically, we set Ç = 0 for evaluations on MTM1T 10 and D4RL tasks. As illustrated in Table discarding this mutual information regularization term results in performance degradation in almost all tasks. This underscores the importance of the regularization term in enhancing the alignment between the representation conditions and generated trajectories.
Table 3: Ablation results on the MI regularization term.
Tasks
W/O MI regularization
With MI regularization (
MT-10
67.5 1.7
68.9
walker2d-medium--enee
79.9 22.6
104.8 = 3.0
malker2dd-mdiummeee
56.5 1 6.7
60.5 1.1
hopper-meedium-exper
93.8 + 6.8
108.9 = 1.0
hopper-mediumm-eelay
54.4 + 0.1
56.2 - 0.7
siichhetta--mdium--eee
94.4 + 1.4
95.0 +
Regarding that the performance improvement is not significant on some tasks, we would like sup- plement several analysis. The diffusion model first models the trajectory distribution represented by offline data and then ensures the generation of relatively better trajectories by controlling conditions.
For the MT- task, simple behavior cloning performs reasonably well (as shown in Figure 4), indicating that the basic trajectory distribution is relatively good. Therefore, controlling conditions may not significantly affect the final performance. Similarly, in some simple tasks and mepper-medium-repl.e the trajectories learned directly by the diffusion model exhibit weak dependence on conditions. In these cases, MI regularization provides limited improvement.
like halfcheetah-mmeiim--eeee
walker2d-medium-en
However, on tasks with more complex data modes, such as malker2d-mediine ceddium--xpprtt and hopper-medium expert, the conditional distribution learned by the diffusion model is more complex, leading to more pronounced effect of controlling conditions. In this case, we also observe a significant improvement conditional generation with the addition of MI regularization.
C.3 Ablation study on the influence of the number of tasks
We have also conducted ablation study the number of tasks K and compared the performance on five tasks. The results including average return and average success rate across tasks are shown below.
From these results, it can observed that as the number tasks K increases from to 5 to 10, both the average return and success rate in experiments gradually increase. This suggests that when K
19

---

Table 4: Ablations on the number of tasks.
Tasks
K=3
K=5
K=10
Reach-v2
172.9
1209.4
3625.7
Pick- Pick-Place-v2
2.8
2.7
564.8
Window- Mandow-Open-v2
130.2
1195.6
3188.7
Push-v2
32.2
1083.3
Drawer-Open-v2
586.2
2304.0
Average Success Rate Across 5 Tasks 0.0%
4.0%
51.2%
is not large, increasing the number of tasks K can enhance the performance of specific task. We attribute this to the speculation that as the number tasks increases, the representation learning for specific task improves. As mentioned in Section 3.1, method learns task-specific preference representation and an optimal representation by constructing positive and negative samples, where the negative samples include all trajectories from other tasks. When K increases, the number of negative samples significantly increases, which helps our method learn the preference representation under the given task. Therefore, learning from more tasks leads to better representations of specific task, resulting in improved performance and accuracy for that particular task.
It is also important to note that when K becomes too large, the learning process for the diffusion model becomes more challenging. This is because the diffusion model needs to fit the trajectory distributions of multiple tasks. Especially when the multi-task trajectory data are low-quality, the difficulty of fitting for the diffusion model increases.
D Supplementary Results of The Average Performance in MT-10 Tasks
In this section, we present the complete results of our evaluations. All results are obtained over multiple random seeds. All methods are trained with the same data and adopt the same evaluation metric. Tables 5 and 6 elaborate on the success rates on MT-10 tasks given near-optimal and sub- optimal datasets. A high success rate indicates the model's proficiency in consistently accomplishing the tasks in MT-10 10 benchmarks.
Table 5: Average success rates given optrmml datasets.
Environments
MTBC
MTIQL
MIDiff
MT-OPPO-p
MT-OPPO-w
CAMP
bullon-press-lonn press-lopdoown-22
69.0 1
78.0 0.0
20.5
:
0.0
76.0 12.0
door-open-v2
99.0 1 1.0
98.0 0.0
93.3.9..
0.0 0.0
0.0
92.0
drawer-close-v2
100.0
100.0
0.0
92.0
98.0
100.0
drawer-open-v2
20.0
83.3 83.3
0.0
0.0
27.0 19.0
peg-insert-side--
54.0
7.00
73.3 12.5
0.0
0.0
89.0 100
pick-place-ve
6.0
3.0
76.7 + 12.5
0.0
0.0
26.0
push-v2
9.0
1.
20.0
16.0
0.0
210
reach-v2
6.00
4.0
71.0
16.0
8.0
32.0
window-close-v2
1.0.0
94.0
1.0.0
0.0
30.0
100.0 0.0
window-open-v2
50.0 32,0
76.0 2.0
83.3 4.7
13.3 22.0 13.3
18.0 18.3
32.0 4.0
Average
62.2 022160 6.2
63.7 1.5
76.7 16.73.335 3.9
14.6 1: 4.6
6.7
68.9 2.7
Table 6: Average success rates given sub-optimal datasets.
Environments
MTBC
MTIQL
MTDiff
MT-OPPO-p
MT-OPPO-w
CAMP
button- preooo-pree--oooww-yy
50.0 6.0
70.0
53.3 9.4
0.0 0.0
0.0
87.2 1 12.9
dnor-open-v2
47.0 19.0
75.0
26.7 12.5
12.5
0.0 + 0.0
0.0 0.0
71.2 + 8.8
drawer-close-v2
100.0
9..0
100.0
100.0 0.0
88.0
100.0 0.0
drawer-open-v2
12
4.0
40.0 16.3
0.0
0.0
6.0 1 4,9
peg-insert-side-v2
2.0
59.0
2.73
0.0
0.0
61.6 26.1
pick-place-v2
1.0 1.0
2.0++2..0
0.0
4.0
0.0
0.8
push-v2
3.0
13.0
23.3+4
12.0
0.0
10.8
reach-v2
2..0
79.0
40.
10.9
2.0
window-close-v2
47.0
7..0
63.3
0.0
2.0 4.0
98.0
window-open-v22
2.00
90.0
70.0
18.0
14.0 18.5
81.6 12.1
Average
8.66
2..7
4.22
1..4
10.6
56.2 1,66
20

---

E Additional Explanations About The Problem Setting
Our method lies in the broad field of Offline Preference Learning, where the agent learns policies from preference data rather than a designed reward function. However, the preference labels defined by scripted teachers or humans are often tailored for a specific task, and learned policies can only align with this task. To solve this problem, we aim to provide unified preference representation for both Single- and Multi-Task Preference Learning. Based on the representation, we learn Multi-Tate Diffusion Policy via conditional trajectory generation by using the learned representation as a condition.
F Acceleration Sampling for Diffusion models
While multi-step denoising process the diffusion model generation is time-consuming, our contri- bution is orthogonal to those sampling acceleration methods for diffusion models, such as DDIM [57]. DPM-solver [58], EDP [59], and can be easily combined with them. fact, we have incorporated the implementation of the DPM-solver into our method.. This improvement has boosted the inference speed by a factor 8.4 compared to the previous method. Specifically, on an Nvidia RTX3090, the denoising time per step has been reduced from 15.2ms to 1.8ms. However, our initial results indicate a compromise in the quality of the generated trajectories, likely due to the complexity of generating continuous trajectories compared to mmae--gnneration tasks. We will explore further enhancements to balance performance and quality in our future work.
G Further Analysis About MI Regularization
The conditional diffusion model is not enough to provide alignment The remarkable success of Stable Diffusion and Midjourney has highlighted the potential of diffusion models to generate desired images given textual conditions. However, when diffusion models are applied to offline reinforcement learning and trajectory modeling. the situation appears different. Previous work on return-conditioned diffusion models [5] 14] generates trajectories by conditioning on target returns, but their conditions often hyperparameters adjusted for each environment. As shown in Figure 1, our tests with different conditions reveal that the relationship between the generation and the conditions is not expected. Given higher target return, the diffusion model cannot generate better trajectories. In fact, this phenomenon has been noted in other related works as well 66). Therefore, the current conditional diffusion models insufficient provide the necessary alignment between preference and trajectory in our setting. This is one of the main problems we aim to solve.
Why using MI regularization? Inspired by the work in the field of image generation, such as VAE (28). InfoGAN [29]. and InfoDiffusion (21), we propose the adoption of mutual information regularization. During the learning phase of the diffusion model, effectively estimates prior noise, with its posterior estimation conditioned on preference representation. In our debugging phase, we observed that noise estimates from previous works often disregard conditioning. leading to the diffusion model's inability to effectively learn the conditional distribution p(r]c). Therefore, in our work, we imposed a constraint of maximizing mutual information on the diffusion model, ensuring a tighter relationship between the posterior noise estimation and the condition. This guarantees that the noise estimation network can capture information about the condition. Consequently, during the denoising generation phase, the trajectories generated by the diffusion model can well correspond to the condition. Moreover, in response to the inefficiency of the multi-step denoising process in diffusion models, have introduced reasonable approximation that simplifies the implementation of regularization. The experimental results also demonstrate that enhancing this connection leads to better outcomes than not doing so.
H
Additional Analysis About The Generalization Ability
H.1 How to obtain Wi for a new task A:?
Since our method projects trajectory segments into representation space, obtaining wi of the new task A is akin to locating its position in the representation space. Assuming the optimal representations for
21

---

known tasks i and j as W and wž respectively, we can approximate wi with wi or use wi as starting point to estimate w when task l is very similar to task i, indicating that wi is close to w in the representation space. Similarly, when task k is composed of task i and task j. interpolation between "i and uy suffices to estimate w.. These methods are relatively straightforward and direct. However, if task k is distant from known tasks, we consider relearning wig from scratch using trajectory data of task k.
Learning the optimal representations wi for new tasks from scratch is consistent with learning representations for known tasks. We require trajectory data with preference pairs on the new task. By using the multi-task preference proposed in Section 3.1, we construct favorable trajectories as positive samples and unfavorable trajectories along with trajectories from other tasks as negative samples. We learn preference representations and optimal representations for task k in the same manner of learning wi. Several points should be noted:
When continuing training the already trained representation network fas the number of learning samples for the new task is significantly fewer, approximately only one-fourth.
Learning w for new tasks can be decoupled from learning the diffusion model. During inference new tasks, the trained wi can be inputted into the diffusion model to generate target trajectories for the new task.
Learning w does not require expert trajectories of task k and mixed data is often sufficient.
H.2 Dataset requirement?
Our learned diffusion model is designed to model the relationship between preference representations and trajectory segments, rather than directly mapping representations to complete trajectories. The preference representations we learn effectively map different trajectory segments into representation space and then generate trajectory segments based on the conditions of the representation. Therefore, as long as there are some favorable segments within near-optimal trajectories, the learned distribution will involve optimal trajectories, and we can condition the diffusion model on w" to generate optimal trajectory segments. In other words, our approach does not require expert trajectories. In fact, the experimental results in Figure 4 validate that our method performs well on both near-optimal and sub-optimal data. Nevertheless, If the quality of the offline datasets is extremely poor, for example, if they consist entirely of random data, it becomes challenging to derive informative preferences. Consequently, we cannot learn a representation space capable of clearly distinguishing the quality of trajectory segments, thereby limiting the generalization ability of the diffusion model. We note that poor performance and generalization on random data is also commonly encountered in offline RL. and offline preference learning methods [15 43
H.3
What enables the generalization to unseen tasks?
We attribute this generalization ability to two main factors: the representation space constructed from preference representations Iearned from multi-task preferences, and the mutual information regularization applied to the conditional diffusion model.
Firstly, by constructing multi-task preferences, we ensure that trajectory segments from different tasks map to distinct regions in the representation space, and segments of different qualities from the same task distribute smoothly in the representation space. This enables our representation space to effectively differentiate between trajectories of different qualities across tasks. Therefore, for a new task k; if it is highly similar to known tasks, we can approximate or interpolate w from the optimal representations of known tasks. Conversely, it is significantly diffcrent, we can reconstruct preference pairs and samples to map the trajectory segments of the new task to different regions in the representation space. Similarly, task can also be distinguished from other tasks.
Second, we have enhanced the existing conditional diffusion model by introducing MI regularization, strengthening the connection between generated trajectories and given conditions. MI regularization sharpens the correspondence between the representation space and the trajectory segment distribution. In other words, we can control which task the generated trajectories come from by switching the given conditions. When w for the new task 4o is obtained through various methods, including approximation, interpolation, or relearning, we can ensure that the diffusion model conditioned on ww generates the desired trajectories for task k.
22

---

I Space and Time Complexity
The computation in our method primarily involves two parts: 1) Training the MI Regularized Diffusion Model: Estimating the mutual information between the generated trajectory To and the preference representation w is computationally intensive, as generating To requires k-step denoising. To mitigate this, we apply approximations and replace I(To; w) with [(Tk; w), thus bypassing the denosing process during training. [(T5; w) can obtained from the estimated noise at each step, requiring only a two-layer MLP to align the dimensions with 2) Inference Stage: Sampling during the inference stage time-consumin consumons which we have discussed in Appendix F. To address this, we have experimented with DPM-solver [1] to reduce the sampling time of the diffusion model. This improvement has boosted the inference speed by a factor of 8.4 compared to the previous method. Specifically, on an Nvidia RTX3090, the denoising time per step has been reduced from 15.2ms to 1.8ms. Despite the improvement in efficiency, our initial results indicate a compromise in the quality of the generated trajectories, likely due to the complexity of generating continuous trajectories. We will explore further enhancements to balance efficiency and quality in our future work.
We have also conducted further comparison of the space complexity and time complexity of CAMP, with the diffusion model-based multitask learning method, MTDiff. According to the results Table 7 the runtime for each update in MTDiff is approximately half of that in CAMP. This is because, addition to optimizing the diffusion model, CAMP also optimizes the preference representation network and the optimal preference representation. It is worth noting that the learning of preference representation and the optimization of the diffusion model can be decoupled; if decoupled. our method might exhibit greater flexibility and efficiency. On the other hand, by comparing the GPU memory usage during algorithm execution, we found that the space required by CAMP is about one-third of that required by MTDiff. This indicates that, although CAMP is more computationally intensive, its memory space requirements are significantly lower. This reason may be that CAMP utilizes a U-Net based noise predictor, while MTDiff needs more complex Transformer-basee noise predictor.
Table 7: Comparison of MTDiff and CAMP
MTDiff
CAMP
Runtime per update (s)
32.0
62.5
Memory used (MB)
7322
2516
Broader Impacts
This paper presents a novel approach sequential decision- meckii, aiming to advance the field of machine learning. Our method addresses critical challenges in aligning decision ceciiioonmmaknng making processes with human intents and achieving versatility across various tasks. By adopting multi-task preferences as a unified condition for decision-making, we offer a framework that enhances both alignments with human preferences and versatility across tasks.
The societal consequences of our work are manifold. By improving the controllability of decision- making processes concerning human preferences, our approach holds promise for applications in diverse domains such as robotics, healthcare, and personalized recommendation systems. Additionally, by addressing limitations in existing methods regarding the formulation of reward functions, we pave the way for more robust and adaptable machine learning systems.
Further, we believe our work is beneficial for the broader research community in preference learning. For instance, in the alignment of large language models, researchers are increasingly recognizing the importance of diverse human preferences for multi-objective (eeferences[6]]55) Our work actually offers a perspective on this issue. For text-to-image or text-tor to---oovvdeeo generation, better alignment with human preferences is also critical consideration (66), and our work holds relevant implications for these fields.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions and scope have been introduced in the abstract and introduc- tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in the conclusion and Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
All theoretical results have corresponding proofs, as shown in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the implementation details are introduced in Section 5, Appendix and We use open-sourced datasets and provide the code for ce-implemennation.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer:
[Yes]
The dataset is open-sourced and the code is provided, with implementation details in Appendix B.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https iow.eooomisisiioiooooo.oo
//nips. //nips.cc/pil cc/public/gut
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have reported error bars for all comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details in Appendix B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided related discussions in Appendix I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We can consider the safeguards for generation in future research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers of the baselines and benchmarks.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided code implementations for reference.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "3h0uo8vzSY", "reviewer": "Reviewer_L2wb", "summary": "This paper proposes a regularized conditional diffusion model which aligns with preferences for multi-task scenarios. To achieve this, learnable representations for preferences are aligned with preference labels, which can be adopted as condition inputs to guide the generation process of diffusion models. Meanwhile, in order to keep consistency between conditions and generated trajectories, a mutual information regularization method is utilized to improve their alignment with preferences. To verify its effectiveness, extensive experiments have been conducted on two datasets D4RL & Meta-World which cover both single- and multiple-task scenarios, and the results demonstrate the superior performance and generalization ability of the proposed method in aligning preferences across different tasks.", "questions": "1. I wonder what factors contribute to this phenomenon and how we choose the optimal approach for a given task.\n2. I would like to know whether the method proposed in the paper receives a greater impact on efficiency as the type and number of tasks increase.\n3. Is the generalization ability of this approach feasible in practical applications?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and detailed. The authors provide sufficient technical principles and experimental details for readers to understand and be able to reproduce their work. \n2. This paper gives a novelty idea in human preferences alignment and combines it with diffusion models in multi-task scenarios, which can be conducive to further foundation models finetuning. \n3. Conducting extensive experiments on public datasets and comparing them with different state-of-the-art methods makes the proposed method more practical and convincing.", "weaknesses": "1. From Table 1, we can see a large variation in performance for different tasks with the same implementation method.\n2. The proposed method seems to have high computational complexity.\n3. From Table 2, although the generalization ability for unseen tasks is somewhat improved by this approach compared to the baseline methods, the average performance is still not very good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to provide feedback on our paper. We appreciate your valuable comments and would like to address each of your concerns.\n\n**1. Table 1 presents a large variation in performance for different tasks with the same implementation method. What factors contribute to this phenomenon, and how do we choose the optimal approach for a given task?**\n\nThanks for your questions. **The significant variation in performance for the same method across different tasks, as seen in Table 1, is primarily due to the varying quality of the offline data used during the learning process.** Specifically, the single-task performance is evaluated on the commonly used D4RL Gym Mujoco tasks for offline RL, which include halfcheetah, walker2d, and hopper tasks. We employed three offline datasets: medium, medium-replay, and expert, which are obtained by different behavior policies. The trajectory quality in the medium and medium-replay datasets is lower than that in the expert dataset. Therefore, each method's performance on the medium and medium-replay datasets is generally lower than on the expert dataset.\n\n\nAs for how to choose the optimal approach for a given task, it depends on which training data are given. According to Table 1, for single-task learning, CAMP is less effective than IQL which utilizes actual rewards. This is understandable because reward signals provide more detailed and accurate information compared to preference data, and **learning with actual rewards in single tasks can potentially yield better performance**. Despite this, only given preference data, **CAMP outperforms other preference-based methods**. Therefore, if the given task provides actual reward signals, reward-based methods are generally superior. However, if the given task only has preference data, CAMP is better than other preference-based methods.\n\n**2. The proposed method seems to have high computational complexity.**\n\nThanks for your question. Regarding computation complexity, we have presented specific results for space complexity and time complexity in our response to Question 3 from Reviewer 3Xmu. Please refer to that for details. Here, we analyze the parts of the method that are more time-consuming and introduce how we addressed these issues. The computation in our method primarily involves two parts: \n\n- **Training the MI Regularized Diffusion Model**: Estimating the mutual information (MI) between the generated trajectory $\\tau_0$ and the preference representation $w$ is computationally intensive, as generating $\\tau_0$ requires $k$-step denoising. To mitigate this, we apply approximations and replace $I(\\tau_0;w)$ with $I(\\tau_k;w)$, thus bypassing the denosing process during training. $I(\\tau_k;w)$ can be obtained from the estimated noise at each step, requiring only a two-layer MLP to align the dimensions with $w$.\n- **Inference Stage**: Sampling during the inference stage is time-consuming, which we have discussed in Appendix F. To address this, we have experimented with DPM-solver [1] to reduce the sampling time of the diffusion model . This improvement has boosted the inference speed by a factor of 8.4 compared to the previous method. Specifically, on an Nvidia RTX3090, the denoising time per step has been reduced from 15.2ms to 1.8ms. Despite the improvement in efficiency, our initial results indicate a compromise in the quality of the generated trajectories, likely due to the complexity of generating continuous trajectories. We will explore further enhancements to balance efficiency and quality in our future work.\n\n[1] Lu C, Zhou Y, Bao F, et al. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS 2022\n\n**3. Whether the proposed method receive a greater impact on efficiency as the type and number of tasks increase?**\n\nThanks for your question. This issue is quite similar to the first question raised by Reviewer Y24x, so **we have included it in the Global Rebuttal. More experimental results can be found in our response to Question 1 from Reviewer Y24x**. Here, we provide a brief conclusion: As the types and number of tasks increase, the learning of preference representations improves, but it becomes more challenging for the diffusion model to fit the trajectory distributions of multiple tasks. The final performance is influenced by both of these factors.\n\n**4. From Table 2, the average performance on the generalization ability for unseen tasks is still not very good.**\n\n\nThanks for your question. The average performance for unseen tasks is not very good due to the inherent challenge of generalization in MetaWorld, where the difference between the learned and unseen tasks is substantial. This discrepancy causes the preference representations of new tasks learned by CAMP to be distant from those of existing tasks in the representation space. Although we incorporate MI regularization to enable the diffusion model to generate trajectories corresponding to the task preferences and exhibit generalization in the representation space, this generalization ability is weakened when the preference representations of new tasks differ significantly from those of existing tasks. This leads to the performance in Table 2 not being particularly outstanding. Nevertheless, CAMP still demonstrates superior performance compared to existing methods.\n\n\n**5. Is the generalization ability of this approach feasible in practical applications?**\n\nThanks for your question regarding the feasibility of the generalization ability of this approach in practical applications. While the current results indicate that the generalization ability is not yet at an optimal level, there is potential for improvement. With sufficient high-quality data and a diverse set of tasks, the generalization ability of the model can be further enhanced, making it more feasible for practical applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDjbZg9tKL", "reviewer": "Reviewer_3Xmu", "summary": "The paper proposes a novel method called the Regularized Conditional Diffusion Model for multi-task preference alignment, addressing the challenge of generating trajectories that align with human preferences in multi-task settings. This method introduces preference-based representations and a mutual information regularization technique to improve the alignment between generated trajectories and preferences.", "questions": "Please see the weaknesses", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method effectively captures and utilizes versatile preference representations, allowing for better alignment with human intents across multiple tasks.\n2.\tThe introduction of a mutual information regularization objective enhances the consistency between conditions and generated trajectories, improving performance.\n3.\tThe diffusion model guided by learned preference representations generates trajectories that align closely with desired preferences.", "weaknesses": "1.\tThe authors’ proposed method shares a similar motivation with OPPO. In the appendix, they explain the differences between their method and other approaches like OPPO and MTDiff. Their method improves upon OPPO and MTDiff by incorporating Diffusion and other techniques such as regularization for multi-task preferences. It is recommended that the authors emphasize these distinctions in the main text and update the contribution list to highlight their unique contributions.\n2.\tFrom the comparative experimental results presented in the figures and tables, it is evident that the authors’ method does not achieve the best performance across all metrics. It is suggested that the authors provide a corresponding discussion instead of merely describing the contents of the figures and tables.\n3.\tThe authors’ method involves computationally intensive modules such as mutual information and diffusion models. It is advised to include a discussion on the computational costs of the model, such as space complexity and time complexity, to enable a comprehensive comparison with related models.\n4.\tGiven that the authors’ model addresses multi-task preferences, it would be beneficial to discuss the impact of the quality of the provided preference data on the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to provide feedback on our paper. We appreciate your valuable comments and would like to address each of your concerns.\n\n**1. It is recommended that the authors emphasize these distinctions and update the contribution list.**\n\n\nThanks for your suggestion. We will emphasize the distinctions between CAMP, OPPO, and MTDiff in the main text and update the contribution list in the next version. Specifically, our contributions can be summarized as follows:\n\n- We have introduced multi-task preferences and proposed to learn high-dimensional preference representations from trajectory segments, which extends the versatility of traditional preference-based methods like OPPO.\n- We have proposed a mutual information regularized diffusion model for multi-task trajectory generation by conditioning on preference representations, saving the need for actual rewards and expert prompts used in MTDiff.\n- We have conducted extensive experiments on MetaWorld, showing comparable performance to reward-based methods, superiority to baseline preference-based methods, and favorable generalization performance to unseen tasks.\n\n\n**2. The proposed method does not achieve the best performance across all metrics. It is suggested that the authors provide a corresponding discussion.**\n\nThanks for your question. We found that this issue is quite similar to the second question raised by Reviewer Y24x. Therefore, **we have provided a detailed and unified response in the Global Rebuttal**. Please refer to it for more information. Here, we can briefly summarize some conclusions: preference data is not as accurate and abundant as reward data. Hence, methods that learn from preference data are less likely to perform as well as reward-based methods. Our experiments in both multi-task and single-task settings aim to demonstrate that CAMP, using only preference data, can achieve performance comparable to the best reward-based methods, while significantly outperforming other preference-based methods.\n\n**3. It is advised to include a discussion on the computational costs of the model, such as space complexity and time complexity.**\n\nThanks for your insightful question. Regarding the concerns of the reviewer about the optimization of mutual information and diffusion models, we have proposed corresponding improvement methods. These include scaling and simplifying the original mutual information objective, as well as employing faster sampling methods for diffusion models. **Please refer to our response to Question 2 from Reviewer L2wb for more details**, where we have discussed which parts of our approach were more time-consuming and how CAMP was optimized to address these issues.\n\nHere, we have conducted a further comparison of the space complexity and time complexity of CAMP, with the diffusion model-based multitask learning method, MTDiff [1]. Both methods were run on an Nvidia RTX 4090 GPU.\n\n| | MTDiff | CAMP |\n|--| --|--|\n|Runtime per update (s)| 32.0| 62.5|\n|Memory used (MB)|7322|2516|\n|\n\nAccording to the above results, the runtime for each update in MTDiff is approximately half of that in CAMP. This is because, in addition to optimizing the diffusion model, CAMP also optimizes the preference representation network and the optimal preference representation. It is worth noting that the learning of preference representation and the optimization of the diffusion model can be decoupled; if decoupled, our method might exhibit greater flexibility and efficiency. On the other hand, by comparing the GPU memory usage during algorithm execution, we found that the space required by CAMP is about one-third of that required by MTDiff. This indicates that, although CAMP is more computationally intensive, its memory space requirements are significantly lower. This reason may be that CAMP utilizes a U-Net based noise predictor, while MTDiff needs a more complex Transformer-based noise predictor.\n\n[1] He H, Bai C, Xu K, et al. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. NeurIPS 2023\n\n**4. It would be beneficial to discuss the impact of the quality of the provided preference data on the results.**\n\nThanks for your suggestion. We would like to discuss the impact of the quality of preference data from two perspectives: noisy preferences and the quality of provided trajectory data. Firstly, noisy preferences can indeed influence the accuracy of learned preference representations. To make representation learning robust to noisy preferences, our method has incorporated two useful components: 1) inspired by contrastive learning methods [1], our method also constructs positive and negative samples and optimizes the representation network so that they can be separated in the representation space. This is more robust than learning a reward model aligned with preferences in traditional preference-based RL  methods. 2) Our method learns a representation distribution, which captures the uncertainty contained in the preference dataset [2]. \n\nSecondly, in multi-task learning experiments, we tested learning results using near-optimal and sub-optimal datasets, which vary in the quality of trajectory data. The near-optimal datasets, which contain more expert trajectories, have higher quality compared to sub-optimal datasets. Figure 4 illustrates that our method is robust to different qualities of trajectory data.\n\n[1] Shen W, Zhang X, Yao Y, et al. Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards. ArXiv:2403.07708.\n\n[2] Duan S, Yi X, Zhang P, et al. Negating negatives: Alignment without human positive samples via distributional dispreference optimization. ArXiv:2403.03419.\n\n**We hope that our responses above, as well as those in the global rebuttal, have addressed your concerns. If they have, we kindly ask you to consider improving our score. If not, we welcome further discussion.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTLWiBRzC0", "reviewer": "Reviewer_Y24x", "summary": "This paper presents a novel approach using a regularized conditional diffusion model to align with preferences in multi-task reinforcement learning (RL) scenarios. The proposed method generates preference representations that effectively distinguish between various task trajectories and returns. By maximizing mutual information between conditions and generated trajectories, the model regularizes existing diffusion frameworks, enabling it to produce optimal task-specific trajectories aligned with given preferences.", "questions": "- Could you provide an additional ablation study on the influence of the number of tasks? Specifically, does the representation for task A differ when learning tasks {A, B} together versus tasks {A, B, C} together? Additionally, does the number of tasks impact the accuracy of task A?\n- Could you highlight the best results in each row of Table 1? It appears that CAMP's performance is not consistently better than existing methods across these environments, and its average result is not particularly strong. Does this imply that CAMP struggles to learn a good task representation? Alternatively, do you have any other explanations for these observations?\n- Regarding the multi-task dataset case, Tables 4 and 5 indicate that CAMP performs well on some tasks but poorly on others. Could you explain why this is the case? Is it due to an imbalance in the training of the multi-task scenario?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear - generating preference representations for various tasks to differentiate them and taking them as the conditions in the diffusion model so that we could consider preference learning in the multi-task scenario.\n- The method and all the designs in this method are reasonable to me.", "weaknesses": "My main concern is about the results. It seems that CAMP doesn't perform better than existing methods. For details, please refer to the Questions part. Thanks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to provide feedback on our paper. We appreciate your valuable comments and would like to address each of your concerns.\n\n**1. Could you provide an additional ablation study on the influence of the number of tasks? Specifically, does the representation for task A differ when learning tasks {A, B} together versus tasks {A, B, C} together? Additionally, does the number of tasks impact the accuracy of task A?**\n\nThanks for your insightful suggestion. We have conducted an ablation study on the number of tasks $K$ and compared the performance on five tasks. The results including average return and average success rate across tasks are shown below. \n\n||Tasks| K=3| K=5| K=10|\n|--|--|--|--|--|\n||Reach-v2|172.9 |1209.4 |3625.7 |\n||Pick-Place-v2|2.8|2.7|564.8|\n|Average Return|Window-Open-v2|130.2|1195.6|3188.7|\n||Push-v2|-|32.2|1083.3|\n||Drawer-Open-v2|-|586.2|2304.0|\n|Average Success Rate across 5 tasks|-|0.0% | 4.0%| 51.2%|\n||\n\nFrom these results, it can be observed that as the number of tasks $K$ increases from 3 to 5 to 10, both the average return and success rate in experiments gradually increase. This suggests that when $K$ is not large, increasing the number of tasks $K$ can enhance the performance of a specific task. We attribute this to the speculation that as the number of tasks increases, the representation learning for a specific task improves. As mentioned in Section 3.1, our method learns a task-specific preference representation and an optimal representation by constructing positive and negative samples, where the negative samples include all trajectories from other tasks. When $K$ increases, the number of negative samples significantly increases, which helps our method learn the preference representation under the given task. Therefore, learning from more tasks leads to better representations of a specific task, resulting in improved performance and accuracy for that particular task.\n\nIt is also important to note that when $K$ becomes too large, the learning process for the diffusion model becomes more challenging. This is because the diffusion model needs to fit the trajectory distributions of multiple tasks. Especially when the multi-task trajectory data are low-quality, the difficulty of fitting for the diffusion model increases.\n\n\n**2. Could you highlight the best results in each row of Table 1? It appears that CAMP's performance is not consistently better than existing methods across these environments, and its average result is not particularly strong. Does this imply that CAMP struggles to learn a good task representation? Alternatively, do you have any other explanations for these observations?**\n\n\nThanks for your feedback. We will highlight the best results in each row of Table 1 to clarify the comparisons in the next version. We notice that your concern is similar to the second question raised by Reviewer 3Xmu. Therefore, **we have provided a detailed and unified response in the Global Rebuttal**. Here, we would like to briefly clarify that in Table 1, the reason CAMP does not appear to outperform all other methods is not due to CAMP's failure to learn a good representation. Rather, it is because we included the reward-based method IQL as a baseline. In the single-task setting, CAMP learns solely from preference data, while IQL learns from actual rewards, making it more challenging for CAMP to learn an optimal policy compared to IQL. However, when comparing CAMP with other preference-based methods like PT and OPPO, we find that CAMP still demonstrates significant advantages.\n\n**3. Regarding the multi-task dataset case, Tables 4 and 5 indicate that CAMP performs well on some tasks but poorly on others. Could you explain why this is the case? Is it due to an imbalance in the training of the multi-task scenario?**\n\n\nThank you for your question. Tables 4 and 5 show the performance of different methods on the MT-10 tasks with near-optimal and sub-optimal data. The inconsistency in CAMP's performance across the 10 tasks—where it performs well on some tasks but poorly on others—is not due to an imbalance in multi-task training. Instead, it reflects the inherent difficulty of the tasks. Some tasks within the MT-10 set are more challenging, while others are relatively easier. This variability in task difficulty explains the differences in performance observed. Additionally, many tasks in MetaWorld have the characteristic that even if the cumulative reward reaches 90% of the maximum reward, the success rate remains 0. This also contributes to the large variance observed in the results presented in Tables 4 and 5.\n\n**We hope that our responses above, as well as those in the global rebuttal, have addressed your concerns. If they have, we kindly ask you to consider improving our score. If not, we welcome further discussion.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

We appreciate all the reviewers for their dedication and recognition of our work, including acknowledging our motivation and novelty [L2wb, Y24x], the effectiveness and improvements of our method [L2wb, 3Xmu], and our presentation [L2wb, Y24x]. 

We have attempted to address each reviewer's concerns point by point. We notice that the reviewers have some similar questions. In addition to individual responses, we would like to provide unified responses here.

- **Why CAMP's performance is not consistently better than existing methods across all environments?**

Reviewers 3Xmu and Y24x both expressed confusion regarding the phenomenon observed in Figure 4 and Table 1, where our method does not appear to outperform all other methods. We would like to provide further explanations for the experimental results. Figure 4 and Table 1 compare the performance in multi-task and single-task scenarios, respectively. We select **reward-based** methods and **preference-based** methods as our baselines.

In Figure 4, the orange bars represent methods that learn from actual rewards, while the green bars represent methods that learn from preference data. Our aim is to show that in multi-task learning, whether given near-optimal datasets or sub-optimal datasets, our preference-based learning method achieves performance comparable to the best reward-based learning method. Additionally, our method demonstrates a clear advantage over other preference-based learning methods in multi-task scenarios, demonstrating the superiority of learning high-dimensional multi-task preference representations for trajectory generation.

In Table 1, we also distinguish between reward-based methods and preference-based methods. While our method outperforms other preference-based baseline methods, it is slightly weaker than the reward-based method IQL, which utilizes actual rewards for each state-action pair. The inferiority to IQL is because reward signals provide absolute evaluation of state-action pairs as scalars, offering more information compared to relative preferences. CAMP uses relative preferences between pairs of trajectories, which makes learning more challenging.

- **Computational Complexity**

Reviewers L2wb and 3Xmu both raised concerns about the computational complexity of our method. In our response to reviewer L2wb, we analyze two main time-consuming components of CAMP and explain the adjustments we have made to each part, including simplifications to the original MI regularization objective and applications of sampling acceleration methods on the diffusion model. In our response to reviewer 3Xmu, we present additional results and compare CAMP and MTDiff in terms of both space complexity and time complexity. Although CAMP requires a longer update time due to the consideration of representation learning, it needs significantly less GPU memory.

- **What about the performance when the number of tasks increases?**

Reviewers L2wb and Y24x both raised concerns about whether the performance of our method improves as the number of tasks increases, including the accuracy and efficiency of representation learning. This issue can be considered from two perspectives. 

On the one hand, when the type and number of tasks increase, the representation learning for each task indeed becomes more efficient. As explained in Section 3.1, our method learns preference representations by constructing positive and negative samples. The positive samples are high-quality trajectories of a specific task, while the negative samples consist of all trajectories from other tasks. As the type and number of tasks increase, the number of negative samples significantly increases, thereby greatly enhancing the efficiency of representation learning. This can be validated by the supplementary results in our response to Question 1 from Reviewer Y24x. When the number of tasks increased from 3 to 5 to 10, the performance of specific tasks indeed improved. 

On the other hand, as the number of tasks increases, especially when the trajectory data provided for each task is of low quality, the diffusion model may struggle to fit the trajectory distribution for each task well. In other words, as the number of tasks increases, the learning process for the diffusion model becomes more challenging. This can potentially affect the trajectory generation for each task. The generated trajectories may still perform poorly, even if the preference representations for different tasks are accurate.

Meta-review: Post rebuttal, all reviewers have agreed to accept the paper, with their primary concern regarding non-SOTA performance addressed. After thoroughly reviewing the content and rebuttals, I concur that this paper proposes a valuable regularized conditional diffusion model for multi-task preference alignment. Therefore, I recommend acceptance. Please ensure that the necessary revisions are incorporated into the final version.
Decision: Accept (poster)