Regularized Conditional Diffusion Model for Multi-Task Preference Alignment

Authors: xudong  yu
chenjia  bai
haoran  he
changhong  wang
xuelong  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 14:50:24

Abstract: Sequential decision-making can be formulated as a conditional generation process, with targets for alignment with human intents and versatility across various tasks. Previous return-conditioned diffusion models manifest comparable performance but rely on well-defined reward functions, which requires amounts of human efforts and faces challenges in multi-task settings. Preferences serve as an alternative but recent work rarely considers preference learning given multiple tasks. To facilitate the alignment and versatility in multi-task preference learning, we adopt multi-task preferences as a unified framework. In this work, we propose to learn preference representations aligned with preference labels, which are then used as conditions to guide the conditional generation process of diffusion models. The traditional classifier-free guidance paradigm suffers from the inconsistency between the conditions and generated trajectories. We thus introduce an auxiliary regularization objective to maximize the mutual info

Main paper: 
 introduction: 
 In sequential decision-making, agents are trained to accomplish fixed or varied human-designed goals by interacting with the environment or learning from offline data. Two key objectives emerge during training: alignment, wherein agents are expected to take actions conforming to human intents expressed as crafted rewards or preferences, and versatility, denoting the capacity to tackle multiple tasks and generalize to unseen tasks. A promising avenue involves framing sequential decisionmaking as a sequence modeling problem via transformer [1] or diffusion models [2, 3] . This paradigm uses expressive models to capture the trajectory distributions and prevents unstable value estimation in conventional offline Reinforcement Learning (RL) methods [4] . Particularly, utilizing returnconditioned or value-guided diffusion models to perform planning or trajectory generation achieves favorable performance in offline RL [5, 6] and the multi-task variant [7, 8] .Despite the great progress, applying diffusion models to sequential decision-making still faces challenges. (i) The condition generation process relies on a pre-defined reward function to provide return conditions. However, developing multiple task-specific reward functions in multitask settings requires significant efforts [9] and may cause unintended behaviors [10] . (ii) The condition generation process with classifier-free guidance [11] often fails to ensure consistency between conditions and generations. As an example in Figure 1 , the return-conditioned Decision Diffuser [5] struggles to achieve effective alignment for generated trajectories. Inspired by recent works [9, 12, 13] , we find that preferences offer more versatile supervision across multi-tasks than scalar rewards. Specifically, the trajectories from the i-th task are preferred 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0The Return Condition Figure 1 : Illustration of the return-conditioned generation of Decision Diffuser [5] in hopper-mediumexpert task. Existing return-conditional diffusion models fail to align generated trajectories with the return condition, while the red line indicates the desired relationship between the return conditions and true returns of generated trajectories [14] .over the j-th task if we set the i-th task as the target task. Conversely, we can reverse the preference label when setting the target task to the j-th task. Therefore, we adopt preferences rather than rewards to guide the conditional diffusion models in offline multi-task RL.The generated trajectories are intended to align with preferences, prioritizing higher returns or specific tasks.To establish aligned and versatile conditional generation, our proposition involves adopting multi-task preferences and constructing a unified preference representation for both single-and multi-task scenarios, instead of learning scalar rewards from preference data [15, 16] . The acquired representations are aligned with the preference labels. The representations subsequently serve as conditions to guide the conditional generation of diffusion models. In this case, two key challenges arise: (i) aligning the representations with preferences, and (ii) aligning the generated trajectories with representation conditions.To address the above challenges, we introduce Conditional Alignment via Multi-task Preference representations (CAMP) for multi-task preference learning. Specifically, we define multi-task preferences and extract preference representations from trajectory segments. (i) To align the representation with preferences, we propose a triplet loss and a Kullback-Leibler (KL) divergence loss to enable preferred and undesirable trajectories mapped into distinct zones in the representation space. The learned representations not only differentiate trajectories yielding higher returns but also discern trajectories across various tasks. Additionally, we learn an 'optimal' representation for each task to represent the trajectory with the highest preference. (ii) To align the generated trajectories with representation conditions in diffusion models, we introduce a Mutual Information (MI) regularization method. It augments representation-conditioned diffusion models with an auxiliary MI optimization objective, maximizing the correlation between the conditions and the generated outputs. During the inference for a specific task, we provide the 'optimal' representation for that task as the condition of the diffusion model, allowing the generated trajectories to adhere to desired preferences. Experiments on D4RL [17] and Meta-World [18] demonstrate the superiority of our method and the aligned performance in both multi-task and single-task scenarios.the denoising process such that x k-1 ∼ p θ (x k-1 |x k , c), diffusion models can also estimate the conditional distribution p(x|c).The Markov Decision Process (MDP) is defined by a tuple (S, A, P, r, γ), where S and A are the state and action spaces, P is the transition function, r is the reward function, and γ is a discount factor. We consider an offline setting where the policy is learned from a given dataset D. For multi-task learning, we introduce an additional task space T that contains m discrete tasks. Diffusion models have been used for offline RL to overcome the distribution shift caused by Temporal-Difference (TD) learning [4] . Specifically, a diffusion model can formulate sequential decision-making as a conditional generative modeling problem by considering x 0 as a state sequence (s t , . . . , s t+h ) for planning [5] . The condition c typically encompasses the return along the trajectory and is designed to generate trajectories with higher returns.Optimizing the conditional diffusion model involves maximizing the log likelihood log p(x) = log p(x|c)p(c)dc, where p(x|c) is the conditional likelihood for a specific c. Building on prior research [19, 21] , the optimization is achieved by maximizing the Evidence Lower Bound (ELBO):EQUATIONwhere q ψ := q ψ (c|x 0 ) represents the approximate variational posterior mapping x 0 to the condition c, and q x k-1 = q(x k-1 |x k , x 0 ). The complete derivation is provided in §A.1. In practice, this optimization problem can be addressed via a score-matching objective [2, 11] as,EQUATIONwhere ϵ θ is parameterized by a neural network to predict the noise ϵ ∼ N (0, I), the expectation is calculated w.r.t. k ∈ [1, K], τ ∈ D, and β ∼ Bernoulli(p). 
preference learning for conditional generation: 
 In decision-making, human preferences are often applied on trajectory segments τ = [s i , a i ] i∈[1,h] [22] . For a trajectory pair (τ 1 , τ 2 ), human feedback yields a preference label y ∈ {0, 1, 0.5} that indicates which segment a human prefers. Here, y = 1 signifies τ 1 ≻ τ 2 , y = 0 signifies τ 1 ≺ τ 2 , and y = 0.5 means that two trajectories have the same preference. Previous studies commonly employ the Bradley-Terry (BT) model [23] to derive a reward function r from such preferences. Considering learning with offline preference data [16] , and given a dataset D τ = {(τ 1 , τ 2 , y)}, r is learned by maximizing the following objective:L r = E Dτ [y log P [τ 1 ≻ τ 2 ] + (1 -y) log P [τ 2 ≻ τ 1 ]] .In what follows, we simplify the notation by omitting the label y and denote the preference data as {(τ + , τ -)}, where we have τ + ≻ τ -. Previous methods [24, 15] based on the BT model follow a two-phase learning process, first deriving a reward function from preference data and then training a policy with RL. Nevertheless, this process hinges on the assumption that pairwise preferences can be substituted with a reward model that can generalize to out-of-distribution data [25] . Moreover, these methods often require complex reward models [15] when preferences are intricate [26, 27] .As a result, we bypass the reward learning process and learn a preference-relevant representation that aligns well with trajectories in both single-and multi-task settings. Then the diffusion models can use these representations as conditions to generate trajectories that align with human preferences. In this setup, we regard x 0 = τ as the trajectory segments, and the condition c as the preference-related representation of trajectories, denoted as w = f (τ ). Therefore, the learning objective of diffusion model becomes log p(τ 0 ), and the loss function becomesL θ = E[∥ϵ -ϵ θ (τ k , (1 -β)w + β∅, k)∥ 2 ]. 
method: 
 In this section, we give the definition of multi-task preferences and introduce how to extract preference representation from pairwise trajectories. Then, we present the conditional generation process and an auxiliary optimization objective to align the generated trajectories with preferences.Multi-task preferences.In the context of multi-task settings, tasks exhibit distinct reward functions, making reward-relevant information insufficient for providing versatile preferences across tasks. For example, moving fast is preferred and obtains high rewards in a 'running' task, while being unfavorable in a 'yoga' task. To address this challenge, we extend single-task preferences that only contain reward-relevant information to multi-task settings. Specifically, we consider two kinds of preferences given trajectories from m tasks. For a specific task i ∈ [m], trajectories are assessed based on (i) the return of trajectories when they belong to the same task, i.e., τ i+ ⪰ τ i- if R(τ i+ ) ≥ R(τ i-), where R(•) calculates the cumulative reward, and (ii) the task-relevance of trajectories, i.e., τ i ≻ τ j with j ̸ = i. This means that trajectories from the target task i are more preferred than any trajectories τ j from a different task j ∈ [m].𝜏 ! = 𝜏 "! , 𝜏 # #$" 𝜏 % = {𝜏 "% } 𝑓 &Representation Space for Tasks 𝑤 " * Figure 2 : Illustration of the representation space of trajectories in multi-task preference data. For each task i, the positive samples τ + consist of preferred trajectories τ i+ from task i, while negative samples τ -include less preferred τ i-from the same task, as well as τ j from other tasks. Trajectories from diverse tasks are expected to be differentiated in the representation space, and {w * i } i∈[m] attempts to characterize the best trajectories for each task.Based on the multi-task preferences, we propose to learn trajectory representations aligning with the preference data, as shown in Figure 2 . The learned representations integrate both the trajectory and preference information, serving as the condition for subsequent trajectory generation. During learning representations, we also need to find the 'optimal' representations {w * i } i∈[m] that represent the optimal trajectories {τ * i } i∈[m] for each task, where τ * i is preferred over any offline trajectories in task i. The {w * i } i∈[m] will be used for inference in diffusion models to generate desired trajectories for each task. Thus, we need to learn a trajectory encoder w = f ψ (τ ) that extracts preference-relevant information and the optimal representation{w * i } i∈[m]. Furthermore, we model the representations from a distributional perspective to cover their uncertainty. In practice, the distribution of the optimal representation p(w * i ) = N (µ * i , Σ * i ) and the distribution of representations given a trajectory p(w|τ ) = N (µ ψ (τ ), Σ ψ (τ )) are both modeled as multivariate Gaussian distributions with a diagonal covariance matrix, where ψ is parameterized by a transformer-based encoder. To summarize, the learnable parameters include vectors {µ * i ,Σ * i } i∈[m] for {w * i } i∈[m]and a trajectory encoder f ψ .Loss functions for f ψ and w * i . For each task i in training, we build the multi-task dataset D = {τ i+ , τ i-, τ j } to learn the representation space. The preference data are constructed by using the intra-task preference (i.e., τ i+ ⪰ τ i-) and the inter-task preference (i.e., τ i+ ≻ τ j ). We denote the representation distributions asp(w + i ) = f ψ (τ i+ ), p(w - i ) = f ψ (τ i-) or f ψ (τ j ). For simplicity, we denote p(w + i ) = N (µ + i , Σ + i ) and p(w - i ) = N (µ - i , Σ - i )by omitting the parameter ψ. These distributions are optimized using the following KL loss,EQUATIONThis loss function encourages the encoder to map trajectories with similar preferences to closer embeddings while distancing dissimilar trajectories. In practice, we find that optimizing two sets of parameters (i.e., {µ * i , Σ * i } and ψ) simultaneously is unstable and leads to a trivial solution. Thus, we adopt an iterative optimizing process by using a stop-gradient (SG) operator. Specifically, we use the loss L(SG(ψ), µ * i , Σ * i ) to optimize {µ * i , Σ * i }, and L(ψ, SG(µ * i , Σ * i )) to optimize the encoder f ψ . Furthermore, simply minimizing the KL loss cannot prevent the divergence of the unbounded termD KL (N (µ - i , Σ - i )∥N (µ * i , Σ * i )), which may result in deviated representation distributions and unstable training. Hence, we add a triplet loss to learn representations, as where d is the Euclidean distance in the embedding space. The triplet loss calculates the similarity between w * i and preferred representations w + i , and the similarity between w * i and unfavorable ones w - i , respectively. By regulating the distances between {µ + i , µ - i } and µ * i , the training process is stabilized. Meanwhile, minimizing the triplet loss also guarantees that the optimal embedding is more similar to w + i and less similar to w - i , while w + i and w - i stay away from each other. We set δ as a margin between the two similarities and adopt the same iterative optimizing process for L(ψ, µ * i ). The illustration in Figure 3 captures our learning process. For brevity, we have omitted details related to the distributed form and multi-task learning components while retaining the core methodology.EQUATIONPrompting with w * i . By optimizing the triplet and KL loss, w * i gradually aligns with more preferred representations, converging to the optimal representations. After training, the diffusion model can be prompted by (µ * i , Σ * i ) for a specific task i. The model will then generate the optimal trajectory for task i with the guidance of conditions. The task i can be a novel task not present in the training set, and the diffusion model will try to generalize to the new task in the representation space.Given preference-based versatile representations, we train a diffusion model to generate trajectories that align with the representations. Prior work utilizes classifier-free guidance and aligns the generated samples with low-dimensional conditions, such as returns. However, we find this is insufficient to capture the complex relationship between conditions and trajectories. Using the representative method Decision Diffuser [5] as an example, Figure 1 reveals the relationship between the return conditions and true returns of generated trajectories in the Hopper-medium-expert task. While it is desirable to generate trajectories with higher returns as we increase the return condition, in practice, the attributes of generated samples do not exhibit a smooth transition with changes in the condition. Empirically, Decision Diffuser uses a target return of 0.8-0.9 for the generation process. This difficulty becomes more severe in our method because we cannot exclusively search the high-dimensional representation space and find an empirically effective representation. Therefore, it is critical to enhance the alignment between generated trajectories and previously learned preference representations.Inspired by recent works on generative models [28, 29, 21] , we introduce an auxiliary optimization objective aimed at strengthening the association between representations w ∼ q ψ (w|τ ) and the generated trajectories τ 0 . Specifically, we augment the learning objective from Eq. ( 2) with a regularization term based on mutual information between τ 0 and w. Formally, we train the conditional diffusion model using the following combined objectives: where q(τ 0 ) indicates the trajectory distribution of the offline dataset D, and ζ is a hyper-parameter controlling the strength of regularization. Eq. ( 5) encompasses processes such as sampling trajectories τ ∼ q(τ 0 ), obtaining corresponding representations w ∼ q ψ (w|τ ) via f ψ , and the denoising process p θ (τ k-1 |τ k , w)) to derive τ 0 .EQUATIONTractable objective. In practice, sampling τ 0 from p(τ 0 ) to maximize the MI objective requires the diffusion model to denoise K steps from τ K . This process imposes huge computational burden and may lead to potential memory overflow due to the gradient propagation across multi-steps. Consequently, there arises a need for an approximate objective, and an alternative approach is to replace the optimization on I(τ 0 ; w) with I(τ k ; w). This substitution is motivated by considering the relationship between τ 0 and τ k , as described by the diffusion processτ k = √ ᾱt τ 0 + √ 1 -ᾱt ϵ 0 := f diffuse (τ 0 ).According to the data processing inequality [30] , we haveEQUATIONAs a result, I(τ k ; w) serves as a lower bound of I(τ 0 ; w), thus maximizing I(τ k ; w) also maximizes I(τ 0 ; w). In this case, Eq. ( 5) can be rewritten as:EQUATIONWe note that the objective in Eq. ( 7) can be rewritten into an equivalent form that can be optimized efficiently (see §A.2 for a detailed derivation). Proposition 3.1. The optimization objective in Equation (7) can be transformed toEQUATIONThe first term resembles the ELBO term in Eq. ( 1) denoted as L elbo (x 0 = τ 0 , c = w), which is the same as standard conditional diffusion models in Decision Diffuser [5] . The ELBO term aims to estimate the trajectory distribution via a conditional diffusion process, thus we adopt a similar conditional score-matching objective to optimize it. The second term contributes to enhancing the alignment between τ 0 and w, where p ψ (w) is empirically averaged on samples τ ∈ D via f ψ , and p(τ k ) ∝ f diffuse (τ 0 ) with τ 0 sampling from q(τ 0 ). We can minimize the KL divergence to optimize q ϕ (w|τ k ), a variational predictor to predict the condition w from the denoised sample τ k . In practice, we instantiate it with a neural network represented by ϕ, taking predicted noises ϵ θ (τ k ) as inputs.The entire procedure is shown in Algorithm 1. During training, we iteratively update the representation encoder f ψ and the optimal representation w * i to learn versatile preference representations. Then we update the parameters of the conditional diffusion model via the loss function in Eq. ( 8). To decode the actions from a predicted trajectory, an inverse dynamic model is learned by using a supervised loss from the dataset, asEQUATIONFor planning in a specific task i, we use the optimal representation w * i and the current state s t as a condition to generate an optimal trajectory [s t , . . . , s t+h ]. Then the inverse dynamics model a t = g ω (s t , s t+1 ) is used to decode the action from two consecutive states. 
related work: 
 Diffusion Models in RL. Diffusion models exhibit significant advantages in generative modeling and characterizing complex distributions. On the one hand, they can be applied for modeling policies, with many research works [31] [32] [33] [34] [35] [36] suggesting that diffusion models are more effective at capturing multi-modal action distributions than other policy types like energy-based policies. On the other hand, diffusion models are adept at directly predicting trajectories via conditional generation, adopting conditions such as value gradients [3, 37, 7] , returns [5] , or prompts [8] . Our work also trains diffusion models to generate trajectories, but focuses on guidance from preference representations. Regarding alignment with human preferences, recent works include the integration with designed attributes for various behaviors [6] , fine-tuned policy with preferences [38, 39] , and preference augmentation techniques to improve trajectory generation [40] . While these works focus on single-task settings, our work seeks a versatile solution by considering multi-task preferences.Learning From Human Preferences. Current methods of learning from preferences can be categorized into two groups: direct learning and indirect learning. Indirect learning methods involve learning a reward model and incorporating existing RL algorithms. They employ techniques like off-policy learning [24] , pseudo-labeling and data augmentation [41] , iterative updates of reward models and policies [42] , or leveraging Transformer architectures to learn reward models [15] . On the other hand, direct learning methods bypass reward model learning and directly incorporate human preferences into the learning process. This approach involves various strategies, such as combining decision transformer styles with preference embeddings [43] , mapping reward functions to optimal policies [44, 45] , or aligning models using extended Bradley-Terry comparisons [46] . This work falls under the category of offline direct learning approaches, framing it as a sequence modeling problem.Our work considers challenging multi-task settings. While much prior research attempts to find Pareto optimal solutions considering the trade-offs between different preferences, these methods often require heuristic vector selection for unknown Pareto fronts [47, 48] , millions of pre-collected preference labels and further online queries [10] , or combination with Gaussian processes to learn preference relations [49] . In contrast, our approach does not require any heuristic methods and learns trajectory representations aligned with preferences from offline data. 
experiments: 
 In this section, we will validate our approaches through extensive experiments. Our focus revolves around addressing the following key questions: (i) Can our approach demonstrate superior performance compared to existing approaches? (ii) Does the trajectory encoder discern different trajectories corresponding to multi-task preferences? And does {w * i } i∈[m] align with the optimal trajectories? (iii) Can the diffusion model capture the trajectory distribution and generate trajectories aligned with preferences? (iv) How about the generalization ability of our method on unseen tasks? (v) To what extent are multi-dimensional representations and regularization crucial to our approach?We conduct experiments on two benchmark datasets, Meta-World [18] for multi-task scenarios and D4RL [17] for single-task scenarios. Within evaluations on MetaWorld, we consider two distinct datasets: near-optimal dataset, which comprises the entire experiences obtained during the training of a SAC [50] agent for each task, and sub-optimal dataset, encompassing only the initial 50% of the replay buffer. More details about datasets and baselines are provided in §B.Categorically, our baselines encompass three types of approaches: offline preference-based methods, offline reward-based RL methods, and behavior cloning (BC). Within the realm of offline preference-based methods, our selections include: 1) PT [15] , using a transformer network to model reward functions, integrated with RL algorithms; 2) OPRL [16] , which employs ensemble-based reward functions; 3) OPPO [43] , adopting a one-step process to model offline trajectories and preferences, avoiding reward learning. For methods enjoying access to true reward functions, our selection includes: 1) IQL [4] , which performs in-distribution Q-learning and achieves significant performance; 2) MTDiff [8] , a method leveraging diffusion models in multi-task scenarios. Since many baselines do not consider multiple tasks, we make modifications and mark them with 'MT-'.Multi-task Performance on Meta-World We assess the multi-task performance using MT-10 tasks and present the results in Figure 4 . Our observations are as follows: 1) Given near-optimal datasets, CAMP outperforms MT-BC and MT-IQL trained with ground-truth rewards, with only a small performance gap compared to MTDiff. 2) Given sub-optimal datasets, CAMP demonstrates comparable performance to MT-IQL and surpasses other baselines, highlighting its robustness to dataset quality. 3) Two variations of OPPO fail to learn effective policies for multiple tasks, exposing the limitations of existing preference-based RL methods in handling multi-task scenarios.In comparison, CAMP achieves a performance improvement of nearly four times.Single-task Performance on D4RL As shown in Table 1 , CAMP demonstrates superior performance across all D4RL Mujoco tasks compared to BC. Additionally, it exhibits comparable performance with IQL in medium-expert tasks. When compared to offline preference-based methods such as PT or OPRL, CAMP showcases significant improvements, particularly in hopper tasks. While OPPO is an effective preference-based method, we observe its sensitivity to random seeds. In our re-implementation using default hyperparameters, its performance degrades in walker2d and hopper tasks. In contrast, CAMP provides aligned trajectory generation and favorable performance.While showing superior performance, does CAMP learn meaningful representations or perform desired conditional generation? In this part, we map trajectory segments to two-dimensional space via T-SNE [51] visualization and analyze properties of the trajectory encoder and the diffusion model.To assess the capabilities of discerning different trajectories and aligning with the best trajectories, we sample several trajectories from D and project them to the latent space via f ψ , with subsequent T-SNE visualization. The results at different stages during training are illustrated in the left panel of Figure 5 . With increasing training steps, we find a gradually clear classification among trajectories with different returns. Meanwhile, the optimal representations w * i , represented as red dots, gradually approach trajectories with higher returns. Does the diffusion model generate trajectories aligned with preferences? While w * i provides useful guidance, we want to validate the generative ability of the conditional diffusion model. We map the generated trajectories to latent space and compare them with offline trajectories. As demonstrated in Figure 5 (c), the green points denote the generated trajectories, while the red dots represent w * i . We observe that these generated trajectories for 6 tasks align closely with w * i , which represent the more favorable trajectories within the dataset. This phenomenon showcases that generated trajectories guided by w * i align with preferred trajectories, validating our approach's effectiveness.To validate the generalization ability, we evaluate generated trajectories guided by ŵ * k / ∈ {w * i } i∈[m] that are learned from trajectories of unseen tasks. ŵ * k from those new tasks are learned using the same method as described in Section 3.1. We hold the diffusion model trained on MT-10 tasks fixed and assess its performance when faced with unseen representation conditions. We compare its performance with that of MT-BC, MT-IQL, and MTDiff, all of which are trained using the same settings. As depicted in Table 2 , our approach exhibits favorable performance on unseen tasks and outperforms baseline methods by a considerable margin. Further analyses are presented in §H. This part delves into the impact of multi-dimensional representations and the MI regularization term. Due to space limitations, detailed implementations refer to §C. Here, we highlight key conclusions: 1) Learning a multi-dimensional representation and choosing a suitable dimension are crucial. When the dimension is too low, such as |w| = 1, the representations are insufficient to capture preferences in trajectories and provide effective guidance. Conversely, when the dimension is too high, as in |w| = 64, the representation space becomes challenging to learn, resulting in inferior performance.2) The auxiliary loss of mutual information plays a pivotal role in our framework. Without this regularization term, our method's performance on all MetaWorld and D4RL tasks degrades. 
conclusion: 
 This paper introduces a regularized conditional diffusion model for alignment with preferences in multi-task scenarios. Based on the versatile multi-task preferences, our method acquires preference representations that differentiate trajectories across tasks and with different returns, as well as an optimal representation aligning with the best trajectory for each task. By regularizing exiting diffusion models in RL with mutual information maximization between conditions and generated trajectories, our method can generate desired trajectories by conditioning on the optimal representation for each task, ensuring alignment with preferences. Experimental validation demonstrates the favorable performance and generalization ability of our method. Future work may involve using faster sampling methods to enhance algorithm efficiency or extending to fine-tuning foundation models.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please refer to Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have reported error bars for all comparisons.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the details in Appendix B.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have provided related discussions in Appendix I.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: We can consider the safeguards for generation in future research.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original papers of the baselines and benchmarks.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Derivation of ELBO in Equation (1)Here we derive the ELBO of the diffusion model by considering a conditional denoising process p θ (x k-1 |x k , c):log p(x 0 ) = log p(c)p(x 0:K |c)dx 1:K dc = log p(c)p(x 0:K |c)q(x 1:K |x 0 )q ϕ (c|x 0 ) q(x 1:K |x 0 )q ϕ (c|x 0 ) dx 1:K dc = log E q(x 1:K |x0) E q ψ (c|x0) p(c)p(x 0:K |c) q(x 1:K |x 0 ))q ψ (c|x 0 )) ≥ E q(x 1:K |x0) E q ψ (c|x0) log p(c)p(x 0:K |c) q(x 1:K |x 0 ))q ψ (c|x 0 )) = E q(x 1:K |x0) E q ψ (c|x0) log p(c)p(x K )p θ (x 0 |x 1 , c) K k=2 p θ (x k-1 |x k , c) q ψ (c|x 0 )q(x 1 |x 0 ) K k=2 q(x k |x k-1 , x 0 ) = E q(x 1:K |x0) E q ψ (c|x0) log p(c) q ψ (c|x 0 ) • p θ (x 0 |x 1 , c) • p(x K ) q(x 1 |x 0 ) • K k=2 p θ (x k-1 |x k , c) K k=2 q(x k |x k-1 , x 0 ) = E q(x 1:K |x0) E q ψ (c|x0) log p(c) q ψ (c|x 0 ) • p θ (x 0 |x 1 , c) • p(x K ) q(x 1 |x 0 ) • K k=2 p θ (x k-1 |x k , c)q(x k-1 |x 0 ) K k=2 q(x k-1 |x k )q(x k |x 0 ) = E q(x 1:K |x0) E q ψ (c|x0) log p(c) q ψ (c|x 0 ) • p θ (x 0 |x 1 , c) • p(x K ) q(x K |x 0 ) • K k=2 p θ (x k-1 |x k , c) K k=2 q(x k-1 |x k , x 0 ) = E q(x 1:K |x0) E q ψ (c|x0) log p(c) q ψ (c|x 0 ) + log p θ (x 0 |x 1 , c) + log p(x K ) q(x K |x 0 ) + K k=2 log p θ (x k-1 |x k , c) q(x k-1 |x k , x 0 ) = E q ψ (c|x0) log p(c) q ψ (c|x 0 ) + E q(x1|x0) E q ψ (c|x0) [log p θ (x 0 |x 1 , c)] + E q(x K |x0) log p(x K ) q(x K |x 0 ) + K k=2 E q(x k-1 ,x k |x0) E q ψ (c|x0) log p θ (x k-1 |x k , c) q(x k-1 |x k , x 0 ) = E q(x1|x0) E q ψ (c|x0) [log p θ (x 0 |x 1 , c)] reconstruction term -D KL (q(x K |x 0 )||p(x K ))prior matching term for x K -D KL (q ψ (c|x 0 )||p(c))prior matching term for cEQUATION)Following previous work [19, 21] , the above ELBO provides interpretations for each term:• The reconstruction term resembles the part of the ELBO of a vanilla variational autoencoder, and can be optimized using Monte Carlo estimates [19] . In [2] , this term is learned using a separate decoder.• The prior matching term for x K indicates the discrepancy between the distribution of the noisy version of x 0 after K steps and the standard Gaussian prior. This term has no trainable parameters, and we ignore it during training.• The denoising matching term measures the discrepancy between the ground-truth denoising function q(x t-1 |x t , x 0 ) and the approximated denoising transition function p θ (x t-1 |x t , c).It is minimized when the approximated denoising transition stays close to the ground-truth denoising transition step.• The prior matching term for c indicates the discrepancy between the approximate posterior q ψ (c|x 0 ) and the prior p(c), which can be a standard Gaussian distribution.A.2 Derivation of the loss function in Equation (8)First, we give several explanations about Eq. ( 7), which provides a lower bound of Eq. ( 5). This relationship is built on top of that I(τ 0 ; w) ≥ I(τ k , w). Intuitively, τ k consists of more Gaussian noises than τ 0 , thus providing less information about w. In particular, when k = K, we get a pure Gaussian noise so that I(τ K ; w) = 0.Then we analyze Eq. ( 7) in two parts and derive them to two terms in Eq. ( 8), respectively. For E q(τ0) [log p(τ 0 )], we rewrite Eq. ( 10) by setting x 0 = τ 0 , c = w and average the terms over the data distribution q(τ 0 ):EQUATIONAccording to previous work [19] , this optimization problem can be simplified as:EQUATIONwhere σ q is a function of α coefficients and ϵ 0 ∼ N (ϵ; 0, I) is the source noise that determines τ k from τ 0 . For the mutual information regularization term I(τ k , w), it can be derived as follows:EQUATIONWe introduce a lemma to derive the above inequality. Please refer to lemma 5.1 of [29] for detailed proofs.Lemma A.1. For random variables X, Y and function f (x, y) under suitable regularity conditions:E x∼X,y∼Y |x [f (x, y)] = E x∼X,y∼Y,x ′ ∼X|y [f (x ′ , y)].By using lemma A.1, we can derive thatEQUATIONIn Eq. ( 14), we omit the condition of p(τ k |w) as p(τ k ) because τ k comes from τ 0 ∼ D by adding Gaussian noises. Combining Eqs. ( 11) and ( 14), we can obtain the tractable objective in Eq. ( 8).In this part, we introduce some details about our methods and evaluation, including benchmarks, baselines, and implementations. We then provide specific comparisons with several related works, including Decision Diffuser, MTDiff, and OPPO.MetaWorld MT-10 In this study, we employed MT-10 tasks from MetaWorld as benchmarks to assess multi-task performance. These tasks share similar dynamics, involving the manipulation of a Sawyer robot to interact with various objects to achieve diverse manipulation goals, as shown in Figure 6 . Each task exhibits distinct state spaces and reward functions, presenting a significant challenge for learning strategies. The primary evaluation metric we utilized is the success rate of task completion, with the attained return serving as a secondary measure.Considering this is focused on an offline learning setting, we followed the methodology of MTDiff [8] , employing a replay buffer during SAC [50] training as the offline dataset. For each task, we trained an agent using SAC to progress from a random policy toward an expert policy. Additionally, we categorized two distinct datasets: the near-optimal dataset and the sub-optimal dataset, differing in the number of expert trajectories included. The near-optimal dataset comprises all the replay buffer data, totaling 100 million transitions, while the sub-optimal dataset contains only the initial 50% of this data.Following previous work [43, 15, 16] , We collect preference data with a scripted teacher. In particular, for multi-task preferences, we construct intra-task preferences among trajectories by comparing their returns and inter-task preferences among tasks based on the task relevance, as illustrated in Figure 2 . D4RL To evaluate our methods on single-task settings, we conduct experiments on the Mujoco locomotion tasks from D4RL benchmarks. Three tasks Halfcheetah, Walker2d, and Hopper are chosen, with three types of datasets, including medium, medium-replay, and medium-expert. We obtain preferences from scripted teachers, following previous work [15, 43, 16] .As outlined in Section 5.1, the baselines we compare can be categorized into three groups: offline preference-based methods, offline reward-based RL methods utilizing ground-truth reward functions, and behavior cloning. In the following sections, we delve into the specifics and experimental details of these baselines to provide a comprehensive comparative analysis.PT PT utilizes transformer architecture to learn a scalar reward model, which is then used to optimize policies with the IQL algorithm. We use its official implementation and default parameters 2 .OPRL OPRL combines IQL with reward functions from ensemble-based disagreement. We use its official implementation and default parameters 3 .OPPO & MT-OPPO-p & MT-OPPO-w OPPO shares a similar motivation with us for avoiding explicitly reward learning. It models offline trajectories and preferences in a one-step process. Specifically, it optimizes an offline hindsight information matching objective for seeking a conditional policy and a preference modeling objective for finding the optimal condition. We use its official implementation and default parameters 4 . We observe unstable results over 5 random seeds, which is not the same as the reported results in the paper. We speculate that OPPO may be sensitive to different random seeds.To make it suitable for multi-task settings, we make two types of modifications to its original version, named 'MT-OPPO-p' and 'MT-OPPO-w'. For 'MT-OPPO-p', we add task id inputs for the conditional policy, so that the original policy π(a|s, z) can be extended to π(a|s, z, ID). For 'MT-OPPO-w', we modify OPPO using the same method as our approach, extending its representations to multiple dimensions, so that the policy becomes π(a|s, z, ID) and z is a multi-dimension representation. Following previous work [8] , we project task IDs to latent variables via a 3-layer MLP.BC & MT-BC Traditional behavior cloning learns a direct mapping from states to actions π(a|s).In our experiments, we encode the scalar task ID to a latent variable via MLP and concatenate the latent variable with the original states. Therefore, MTBC utilizes a conditional policy by conditioning on task IDs. We modify it based on implementations from CORL 5 .IQL & MT-IQL IQL is an effective offline RL method, which performs in-distribution Q-learning and expectile regression. We use the implementation from CORL and make similar modifications to MTBC.MTDiff MTDiff is a diffusion-based method that combines Transformer backbones and prompt learning for generative planning in multi-task settings. We use its official implementation and default parameters 6 .Our code is built on Decision Diffuser7 and OPPO 8 . We leverage their implementations of diffusion models and transformer-based encoders while developing the representation learning process for multi-task scenarios and the auxiliary mutual information regularization.• Following Decision Diffuser, we use the temporal U-Net architecture to predict noise, where timesteps and representations w are separately mapped to 128-dimensional vectors by 2-layered MLPs.• We employ 200 denoising steps, consistent with previous work [5, 8] .• The training details of the inverse dynamics model g are aligned with those of Decision Diffuser.• For the trajectory encoder, which projects trajectory segments τ to latent representations w, we adopt a similar Transformer architecture to that of OPPO, but we learn distributional representations.• The conditional guidance weight in diffusion models is set to 1.2 for most tasks and 1.5 for the halfcheetah-medium-expert task.• The learning rate of the diffusion model is 2e -4 with the Adam optimizer.• Training steps are set to 2e 6 in MetaWorld tasks and 1e 6 in D4RL tasks, with results averaged over multiple seeds. In MetaWorld benchmarks, as each environment has 50 random goals, evaluations are averaged over these 50 random goals.• The horizon h of trajectories is set to 20 in the MT-10, halfcheetah, and walker2d tasks, and 100 in the hopper tasks.• Batch size is set to 256 for halfcheetah and walker2d tasks, and 32 for hopper tasks and each task in MetaWorld MT-10 tasks (total batch size is 320).• The regularization coefficient ζ is set to 0.1 for the MT-10, halfcheetah, and hopper-medium tasks, 0.5 for the walker2d-medium and walker2d-medium-expert tasks, 0.01 for the hoppermedium-replay and walker2d-medium-replay tasks, and 1.0 for the hopper-medium-expert task.• The dimension of preference representations is set to 16.• We conduct training on an NVIDIA GeForce RTX 3090. Training time varies with task complexity, approximately 30 hours for D4RL tasks and 59 hours for MT-10 tasks.Decision Diffuser Decision Diffuser is an effective method for conditional generative modeling, with conditioning options including returns, constraints, or skills. In particular, when using returns as conditions, Decision Diffuser utilizes normalized returns and sets the target return as 0.9 for most tasks during the generation process. In Figure 1 , we have revealed that the return-conditioned paradigm may not ensure alignment between specified return conditions and generated trajectories. Unlike Decision Diffuser, we propose a regularization term for conditional diffusion models and obtain enhanced alignment between given representation conditions and generated trajectories. As shown in Figure 5 (c), generated trajectories τ * 0 under the guidance of w * i cluster around the region near w * i . Our method also differs from Decision Diffuser in the utilization of preference data instead of reward labels and the versatility across multi-task scenarios.MTDiff MTDiff employs diffusion models for modeling large-scale multi-task offline data. Similar to Decision Diffuser, MTDiff adopts classifier-free guidance but introduces prompt learning for both modeling policies and trajectories. Specifically, it utilizes normalized cumulative returns and regards task-relevant information as prompts. By employing these task-specific prompts as conditions, MTDiff distinguishes between different tasks and generates desired trajectories for each specific task. The prompts consist of expert demonstrations in the form of trajectory segments, akin to the approach in PromptDT [52] . In contrast, our method eliminates the need for expert demonstrations and instead extracts multi-dimensional representations from multi-task preferences. These representations serve as conditions for the diffusion model. Experiments conducted on MetaWorld tasks demonstrate that our method achieves comparable performance to MTDiff, all without the necessity for reward labels or demonstration prompts. OPPO In the realm of offline preference-based reinforcement learning, OPPO models offline trajectories and preferences in a one-step process, circumventing the need for reward modeling. Specifically, OPPO learns an optimal context using a preference modeling objective and subsequently optimizes a contextual policy. During the learning of contexts, OPPO constructs positive and negative samples and uses the triplet loss to optimize the contexts. With these learned contexts, OPPO then develops a conditional policy, akin to DT [53] and RvS [54] . However, our method utilizes the KL loss and the triplet loss to optimize representation distributions, which align with multi-task preferences. Moreover, our approach focuses on conditional diffusion models and the alignment for trajectory generation. It is noteworthy that our method demonstrates more stable performance and excels in multi-task settings, exhibiting successful generalization to unseen tasks. In contrast, OPPO is specifically designed for the single-task setting.In this part, we aim to dissect and analyze the influences of two key elements: 1) the dimension of representations, and 2) the auxiliary mutual information optimization objective.When the dimension of w is reduced to 1, the learned representations in our method resemble those of vectorized reward models [55] or distributional rewards [56] . Conversely, if the dimension is too high, both learning representations and aligning conditional generation become challenging due to the increased complexity of the representation space. We conduct an ablation study with |w| ∈ 1, 8, 16, 32, 64, where |w| = 1 and |w| = 64 represent two extremes. As presented in Figure 7 , the performance on MT-10 tasks significantly decreases when |w| = 1, and it exhibits performance degradation when the dimension of representations is too big. This implies that we should adopt a suitable dimension for preference representations, and we choose |w| = 16 in our experiments.Our approach employs a regularization term based on mutual information to enhance the alignment between preference representations and generated trajectories. In this section, we investigate the impact of this regularization term on the final performance. Specifically, we set ζ = 0 for evaluations on MT-10 and D4RL tasks. As illustrated in Table 3 , discarding this mutual information regularization term results in performance degradation in almost all tasks. This underscores the importance of the regularization term in enhancing the alignment between the representation conditions and generated trajectories. Regarding that the performance improvement is not significant on some tasks, we would like to supplement several analysis. The diffusion model first models the trajectory distribution represented by offline data and then ensures the generation of relatively better trajectories by controlling conditions.• For the MT-10 task, simple behavior cloning performs reasonably well (as shown in Figure 4 ), indicating that the basic trajectory distribution is relatively good. Therefore, controlling conditions may not significantly affect the final performance. Similarly, in some simple tasks like halfcheetah-medium-expert or walker2d-medium-replay and hopper-medium-replay, the trajectories learned directly by the diffusion model exhibit weak dependence on conditions. In these cases, MI regularization provides limited improvement. • However, on tasks with more complex data modes, such as walker2d-medium-expert and hopper-medium-expert, the conditional distribution learned by the diffusion model is more complex, leading to a more pronounced effect of controlling conditions. In this case, we also observe a significant improvement in conditional generation with the addition of MI regularization.We have also conducted an ablation study on the number of tasks K and compared the performance on five tasks. The results including average return and average success rate across tasks are shown below.From these results, it can be observed that as the number of tasks K increases from 3 to 5 to 10, both the average return and success rate in experiments gradually increase. This suggests that when K is not large, increasing the number of tasks K can enhance the performance of a specific task. We attribute this to the speculation that as the number of tasks increases, the representation learning for a specific task improves. As mentioned in Section 3.1, our method learns a task-specific preference representation and an optimal representation by constructing positive and negative samples, where the negative samples include all trajectories from other tasks. When K increases, the number of negative samples significantly increases, which helps our method learn the preference representation under the given task. Therefore, learning from more tasks leads to better representations of a specific task, resulting in improved performance and accuracy for that particular task.It is also important to note that when K becomes too large, the learning process for the diffusion model becomes more challenging. This is because the diffusion model needs to fit the trajectory distributions of multiple tasks. Especially when the multi-task trajectory data are low-quality, the difficulty of fitting for the diffusion model increases.In this section, we present the complete results of our evaluations. All results are obtained over multiple random seeds. All methods are trained with the same data and adopt the same evaluation metric. Tables 5 and 6 elaborate on the success rates on MT-10 tasks given near-optimal and suboptimal datasets. A high success rate indicates the model's proficiency in consistently accomplishing the tasks in MT-10 benchmarks. Our method lies in the broad field of Offline Preference Learning, where the agent learns policies from preference data rather than a designed reward function. However, the preference labels defined by scripted teachers or humans are often tailored for a specific task, and learned policies can only align with this task. To solve this problem, we aim to provide a unified preference representation for both Single-and Multi-Task Preference Learning. Based on the representation, we learn Multi-Task Diffusion Policy via conditional trajectory generation by using the learned representation as a condition.While multi-step denoising process in the diffusion model generation is time-consuming, our contribution is orthogonal to those sampling acceleration methods for diffusion models, such as DDIM [57] , DPM-solver [58] , EDP [59] , and can be easily combined with them. In fact, we have incorporated the implementation of the DPM-solver into our method.. This improvement has boosted the inference speed by a factor of 8.4 compared to the previous method. Specifically, on an Nvidia RTX3090, the denoising time per step has been reduced from 15.2ms to 1.8ms. However, our initial results indicate a compromise in the quality of the generated trajectories, likely due to the complexity of generating continuous trajectories compared to image-generation tasks. We will explore further enhancements to balance performance and quality in our future work.The conditional diffusion model is not enough to provide alignment The remarkable success of Stable Diffusion and Midjourney has highlighted the potential of diffusion models to generate desired images given textual conditions. However, when diffusion models are applied to offline reinforcement learning and trajectory modeling, the situation appears different. Previous work on return-conditioned diffusion models [5, 14] generates trajectories by conditioning on target returns, but their conditions are often hyperparameters adjusted for each environment. As shown in Figure 1 , our tests with different conditions reveal that the relationship between the generation and the conditions is not as expected. Given a higher target return, the diffusion model cannot generate better trajectories. In fact, this phenomenon has been noted in other related works as well [60] . Therefore, the current conditional diffusion models are insufficient to provide the necessary alignment between preference and trajectory in our setting. This is one of the main problems we aim to solve.Why using MI regularization? Inspired by the work in the field of image generation, such as InfoVAE [28] , InfoGAN [29] , and InfoDiffusion [21] , we propose the adoption of mutual information regularization. During the learning phase of the diffusion model, it effectively estimates prior noise, with its posterior estimation conditioned on preference representation. In our debugging phase, we observed that noise estimates from previous works often disregard conditioning, leading to the diffusion model's inability to effectively learn the conditional distribution p(τ |c). Therefore, in our work, we imposed a constraint of maximizing mutual information on the diffusion model, ensuring a tighter relationship between the posterior noise estimation and the condition. This guarantees that the noise estimation network can capture information about the condition. Consequently, during the denoising generation phase, the trajectories generated by the diffusion model can well correspond to the condition. Moreover, in response to the inefficiency of the multi-step denoising process in diffusion models, we have introduced a reasonable approximation that simplifies the implementation of regularization. The experimental results also demonstrate that enhancing this connection leads to better outcomes than not doing so.H.1 How to obtain w * k for a new task k?Since our method projects trajectory segments into representation space, obtaining w * k of the new task k is akin to locating its position in the representation space. Assuming the optimal representations for known tasks i and j as w * i and w * j respectively, we can approximate w * k with w * i or use w * i as a starting point to estimate w * k when task k is very similar to task i, indicating that w * k is close to w * i in the representation space. Similarly, when task k is composed of task i and task j, interpolation between w * i and w * j suffices to estimate w * k . These methods are relatively straightforward and direct. However, if task k is distant from known tasks, we consider relearning w * k from scratch using trajectory data of task k.Learning the optimal representations w * k for new tasks from scratch is consistent with learning representations for known tasks. We require trajectory data with preference pairs on the new task. By using the multi-task preference proposed in Section 3.1, we construct favorable trajectories as positive samples and unfavorable trajectories along with trajectories from other tasks as negative samples. We learn preference representations and optimal representations for task k in the same manner of learning w * i . Several points should be noted:• When continuing training on the already trained representation network f ψ , the number of learning samples for the new task is significantly fewer, approximately only one-fourth.• Learning w * k for new tasks can be decoupled from learning the diffusion model. During inference on new tasks, the trained w * k can be inputted into the diffusion model to generate target trajectories for the new task.• Learning w * k does not require expert trajectories of task k, and mixed data is often sufficient.Our learned diffusion model is designed to model the relationship between preference representations and trajectory segments, rather than directly mapping representations to complete trajectories. The preference representations we learn effectively map different trajectory segments into a representation space and then generate trajectory segments based on the conditions of the representation. Therefore, as long as there are some favorable segments within near-optimal trajectories, the learned distribution will involve optimal trajectories, and we can condition the diffusion model on w * to generate optimal trajectory segments. In other words, our approach does not require expert trajectories. In fact, the experimental results in Figure 4 validate that our method performs well on both near-optimal and sub-optimal data. Nevertheless, If the quality of the offline datasets is extremely poor, for example, if they consist entirely of random data, it becomes challenging to derive informative preferences. Consequently, we cannot learn a representation space capable of clearly distinguishing the quality of trajectory segments, thereby limiting the generalization ability of the diffusion model. We note that poor performance and generalization on random data is also commonly encountered in offline RL [4] and offline preference learning methods [15, 43] .We attribute this generalization ability to two main factors: the representation space constructed from preference representations learned from multi-task preferences, and the mutual information regularization applied to the conditional diffusion model.Firstly, by constructing multi-task preferences, we ensure that trajectory segments from different tasks map to distinct regions in the representation space, and segments of different qualities from the same task distribute smoothly in the representation space. This enables our representation space to effectively differentiate between trajectories of different qualities across tasks. Therefore, for a new task k, if it is highly similar to known tasks, we can approximate or interpolate w * k from the optimal representations of known tasks. Conversely, if it is significantly different, we can reconstruct preference pairs and samples to map the trajectory segments of the new task to different regions in the representation space. Similarly, task k can also be distinguished from other tasks.Second, we have enhanced the existing conditional diffusion model by introducing MI regularization, strengthening the connection between generated trajectories and given conditions. MI regularization sharpens the correspondence between the representation space and the trajectory segment distribution. In other words, we can control which task the generated trajectories come from by switching the given conditions. When w * k for the new task k is obtained through various methods, including approximation, interpolation, or relearning, we can ensure that the diffusion model conditioned on w * k generates the desired trajectories for task k.The computation in our method primarily involves two parts: 1) Training the MI Regularized Diffusion Model: Estimating the mutual information between the generated trajectory τ 0 and the preference representation w is computationally intensive, as generating τ 0 requires k-step denoising.To mitigate this, we apply approximations and replace I(τ 0 ; w) with I(τ k ; w), thus bypassing the denosing process during training. I(τ k ; w) can be obtained from the estimated noise at each step, requiring only a two-layer MLP to align the dimensions with w. 2) Inference Stage: Sampling during the inference stage is time-consuming, which we have discussed in Appendix F. To address this, we have experimented with DPM-solver [1] to reduce the sampling time of the diffusion model. This improvement has boosted the inference speed by a factor of 8.4 compared to the previous method. Specifically, on an Nvidia RTX3090, the denoising time per step has been reduced from 15.2ms to 1.8ms. Despite the improvement in efficiency, our initial results indicate a compromise in the quality of the generated trajectories, likely due to the complexity of generating continuous trajectories. We will explore further enhancements to balance efficiency and quality in our future work.We have also conducted a further comparison of the space complexity and time complexity of CAMP, with the diffusion model-based multitask learning method, MTDiff. According to the results in Table 7 , the runtime for each update in MTDiff is approximately half of that in CAMP. This is because, in addition to optimizing the diffusion model, CAMP also optimizes the preference representation network and the optimal preference representation. It is worth noting that the learning of preference representation and the optimization of the diffusion model can be decoupled; if decoupled, our method might exhibit greater flexibility and efficiency. On the other hand, by comparing the GPU memory usage during algorithm execution, we found that the space required by CAMP is about one-third of that required by MTDiff. This indicates that, although CAMP is more computationally intensive, its memory space requirements are significantly lower. This reason may be that CAMP utilizes a U-Net based noise predictor, while MTDiff needs a more complex Transformer-based noise predictor. This paper presents a novel approach to sequential decision-making, aiming to advance the field of machine learning. Our method addresses critical challenges in aligning decision-making processes with human intents and achieving versatility across various tasks. By adopting multi-task preferences as a unified condition for decision-making, we offer a framework that enhances both alignments with human preferences and versatility across tasks.The societal consequences of our work are manifold. By improving the controllability of decisionmaking processes concerning human preferences, our approach holds promise for applications in diverse domains such as robotics, healthcare, and personalized recommendation systems. Additionally, by addressing limitations in existing methods regarding the formulation of reward functions, we pave the way for more robust and adaptable machine learning systems.Further, we believe our work is beneficial for the broader research community in preference learning. For instance, in the alignment of large language models, researchers are increasingly recognizing the importance of diverse human preferences for multi-objective preferences [61] [62] [63] [64] [65] . Our work actually offers a perspective on this issue. For text-to-image or text-to-video generation, better alignment with human preferences is also a critical consideration [66] , and our work holds relevant implications for these fields.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.https://github.com/csmile-1006/PreferenceTransformerhttps://github.com/danielshin1/oprlhttps://github.com/bkkgbkjb/OPPOhttps://github.com/tinkoff-ai/CORLhttps://github.com/tinnerhrhe/MTDiffhttps://github.com/anuragajay/decision-diffuser/https://github.com/bkkgbkjb/OPPO 


Decision: Post rebuttal, all reviewers have agreed to accept the paper, with their primary concern regarding non-SOTA performance addressed. After thoroughly reviewing the content and rebuttals, I concur that this paper proposes a valuable regularized conditional diffusion model for multi-task preference alignment. Therefore, I recommend acceptance. Please ensure that the necessary revisions are incorporated into the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a regularized conditional diffusion model which aligns with preferences for multi-task scenarios. To achieve this, learnable representations for preferences are aligned with preference labels, which can be adopted as condition inputs to guide the generation process of diffusion models. Meanwhile, in order to keep consistency between conditions and generated trajectories, a mutual information regularization method is utilized to improve their alignment with preferences. To verify its effectiveness, extensive experiments have been conducted on two datasets D4RL & Meta-World which cover both single- and multiple-task scenarios, and the results demonstrate the superior performance and generalization ability of the proposed method in aligning preferences across different tasks.
strength_weakness: strength_weakness: 1. From Table 1, we can see a large variation in performance for different tasks with the same implementation method.
2. The proposed method seems to have high computational complexity.
3. From Table 2, although the generalization ability for unseen tasks is somewhat improved by this approach compared to the baseline methods, the average performance is still not very good.
questions: questions: 1. I wonder what factors contribute to this phenomenon and how we choose the optimal approach for a given task.
2. I would like to know whether the method proposed in the paper receives a greater impact on efficiency as the type and number of tasks increase.
3. Is the generalization ability of this approach feasible in practical applications?
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes a novel method called the Regularized Conditional Diffusion Model for multi-task preference alignment, addressing the challenge of generating trajectories that align with human preferences in multi-task settings. This method introduces preference-based representations and a mutual information regularization technique to improve the alignment between generated trajectories and preferences.
strength_weakness: strength_weakness: 1.	The authors’ proposed method shares a similar motivation with OPPO. In the appendix, they explain the differences between their method and other approaches like OPPO and MTDiff. Their method improves upon OPPO and MTDiff by incorporating Diffusion and other techniques such as regularization for multi-task preferences. It is recommended that the authors emphasize these distinctions in the main text and update the contribution list to highlight their unique contributions.
2.	From the comparative experimental results presented in the figures and tables, it is evident that the authors’ method does not achieve the best performance across all metrics. It is suggested that the authors provide a corresponding discussion instead of merely describing the contents of the figures and tables.
3.	The authors’ method involves computationally intensive modules such as mutual information and diffusion models. It is advised to include a discussion on the computational costs of the model, such as space complexity and time complexity, to enable a comprehensive comparison with related models.
4.	Given that the authors’ model addresses multi-task preferences, it would be beneficial to discuss the impact of the quality of the provided preference data on the results.
questions: questions: Please see the weaknesses
limitations: limitations: n/a
paper_summary: paper_summary: This paper presents a novel approach using a regularized conditional diffusion model to align with preferences in multi-task reinforcement learning (RL) scenarios. The proposed method generates preference representations that effectively distinguish between various task trajectories and returns. By maximizing mutual information between conditions and generated trajectories, the model regularizes existing diffusion frameworks, enabling it to produce optimal task-specific trajectories aligned with given preferences.
strength_weakness: strength_weakness: My main concern is about the results. It seems that CAMP doesn't perform better than existing methods. For details, please refer to the Questions part. Thanks.
questions: questions: - Could you provide an additional ablation study on the influence of the number of tasks? Specifically, does the representation for task A differ when learning tasks {A, B} together versus tasks {A, B, C} together? Additionally, does the number of tasks impact the accuracy of task A?
- Could you highlight the best results in each row of Table 1? It appears that CAMP's performance is not consistently better than existing methods across these environments, and its average result is not particularly strong. Does this imply that CAMP struggles to learn a good task representation? Alternatively, do you have any other explanations for these observations?
- Regarding the multi-task dataset case, Tables 4 and 5 indicate that CAMP performs well on some tasks but poorly on others. Could you explain why this is the case? Is it due to an imbalance in the training of the multi-task scenario?
limitations: limitations: N/A

Comments: title: Looking forward to your feedback
Looking forward to your feedback
Looking forward to your feedback
Response to the author
comment: We are glad that our response has addressed your concerns, and we commend your effort in evaluating our paper and raising the score. Your valuable suggestions are greatly appreciated and very helpful in improving the quality of our paper. We will incorporate discussions on contributions, experiments, and computational complexity in the next version based on your suggestions.
Thank you very much for the further clarifications. I will increase my score.
Thanks for your valuable feedback. The construction of positive and negative samples is illustrated in Figure 2. Here, we provide a further explanation.

In the $m$ tasks, before calculating the loss and updating the gradients each time, a task $i \in [m]$ is randomly sampled. Then, from the trajectory data of task $i$, two batches of data with batch size are sampled. These are divided into two groups based on their preferences, with one group having higher preference and the other having lower preference. The batch with higher preference is taken as the positive samples, and the batch with lower preference as the negative samples. Additionally, trajectory data with batch size are sampled from all other tasks $j \in [m], j \neq i$, and all these trajectories are considered as negative samples in this gradient update. In summary, for each update, $(m+1)\times batchsize$ trajectories are sampled, where the batch size trajectories are positive samples from the sampled task $i$, and $m\times batchsize$ trajectories are negative samples from task $i$ and other tasks $j$.

We hope that our responses have addressed your concerns.
Thanks for your responses and clarifications. It looks good to me. May I ask a further question? Could you please provide more details about constructing positive and negative samples for this work? Thanks.
Dear Reviewer Y24x,

We first would like to thank the reviewer's efforts and time in reviewing our work. We were wondering if our responses have resolved your concerns. Since the discussion period is ending soon, we would like to express our sincere gratitude if you could check our reply to your comments. We will be happy to have further discussions with you if there are still some remaining questions! We sincerely look forward to your kind reply!

Best regards,

The authors
Dear Reviewer 3Xmu,

We first would like to thank the reviewer's efforts and time in reviewing our work. We were wondering if our responses have resolved your concerns. Since the discussion period is ending soon, we would like to express our sincere gratitude if you could check our reply to your comments. We will be happy to have further discussions with you if there are still some remaining questions! We sincerely look forward to your kind reply!

Best regards,

The authors
Dear Reviewer L2wb,

We first would like to thank the reviewer's efforts and time in reviewing our work. We were wondering if our responses have resolved your concerns. Since the discussion period is ending soon, we would like to express our sincere gratitude if you could check our reply to your comments. We will be happy to have further discussions with you if there are still some remaining questions! We sincerely look forward to your kind reply!

Best regards,

The authors
Thanks for your valuable feedback.

**A1**: We do not intend to suggest that IQL is excessively strong. Instead, it serves as a representative reward-based offline RL method that can be extended to multi-task scenarios.  The performance discrepancy between IQL and CAMP primarily arises from their different utilization of reward signals and preference data. We compare CAMP with IQL to demonstrate that our approach, which relies solely on preference data, can achieve performance comparable to reward-based methods in multi-task scenarios. 

**A2**: Thanks for your correction. The varying difficulty levels among tasks indeed lead to training imbalances in MTL. In our current work on using diffusion models for multi-task trajectory generation, we did not account for conflicts between tasks, which could result in uneven performance in MetaWorld. To address this problem, previous MTL methods have employed strategies such as weighted summation of multi-task losses, gradient manipulation of each loss , and Pareto optimal solutions. These strategies optimize multiple losses simultaneously and may prevent performance degradation due to task conflicts. However, these approaches mainly focus on vision tasks and have been barely investigated in diffusion models or trajectory generation for decision-making. Our method can be combined with these strategies to further enhance multi-task decision-making performance. Recently, several methods, including uncertainty weighting [1] and PCgrad [2], have been integrated to mitigate the negative transfer problem in image generation using diffusion models [3]. These approaches can provide valuable guidance for future work and be extended to our mutli-task decision-making scenarios, mitigating the imbalance in our conditional trajectory generation.

[1] Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, CVPR 2018.

[2] Yu T, Kumar S, Gupta A, et al. Gradient surgery for multi-task learning, NeurIPS 2020.

[3] Go H, Lee Y, Lee S, et al. Addressing negative transfer in diffusion models, NeurIPS 2023.
Thanks for the response. 
Q1: Can I understand that the non-significant superiority of CAMP in Table is because the baseline IQL is too strong? If so, why do we just use IQL?
Q2: Do we have any solution for the different difficulties of the tasks in MTL? It's indeed the imbalance problem in MTL and should be solved carefully.