AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
Authors: Zibin Dong
Yifu Yuan
Jianye HAO
Fei Ni
Yao Mu
YAN ZHENG
Yujing Hu
Tangjie Lv
Changjie Fan
Zhipeng Hu
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement learning; Diffusion models; RLHF; Preference aligning
Primary area: reinforcement learning

Abstract: Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose AlignDiff, a novel framework that leverages RLHF to quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi-perspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabeling behavioral datasets with relative strengths, we proceed to train an attribute-conditioned diffusion model, which serves as a planner with the attribute strength model as a director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared to other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-AI collaboration. More visualization videos are released on https://aligndiff.github.io/.

Published as a conference paper at ICLR 2024
ALIGNDIFF: ALIGNING DIVERSE HUMAN PREFERENCES VIA CALAVIOR--USTOMISAAAN
DIFFUSION MODEL
Zibin Dong"1, Yifu Yuan" 1, Jianye Haol 1, Fei Ni', Yao Mu², Yan Zheng', Yujing Hu?, Tangjie Lv², Changjie Fan?, Zhipeng Hu² Cooleege of Intelligence and Computing. Tianjin University. Fuui Al Lab, Netease Inc., Hangzhou, China, 'The University of Hong Kong
ABSTRACT
Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose Align- Diff. novel framework that leverages RLHF quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero- shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi--erspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabel- ing behavioral datasets with relative strengths, we proceed train attribute- conditioned diffusion model, which serves as a planner with the attribute strength model director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-Al collaboration. More visualiza- tion videos are released on https://aligndiff. htt.aligginiiggggub.o.oooo///
INTRODUCTION
One of the major challenges in building versatile RL agents is aligning their behaviors with human preferences (Han et al., 2021; Guan et al., 2023). This primarily due to the inherent abstract- ness and mutability of human preferences. The abstractness makes it difficult to directly quantify preferences through hand-designed reward function (Hadfield-Mnnell et al., 2017: Guan et al., 2022), while the mutability makes it challenging to design a one mme-size-itts-ll solution since prefer- ences vary among individuals and change over time (Pearce al., 2023). Addressing these two challenges can greatly enhance the applicability and acceptance of RI. agents in real life.
The abstractness of human preferences makes manual-designed reward functions not available (Ve- cerik et al., 2018; Hadfield-Menell et al., 2017). Other sources of information such as image goals or video demonstrations are being incorporated to help agents understand preferred behav- iors (Andrychowicz et al., 2017; Pathak et al., 2018: Ma et al., 2022). However, expressing prefer- ences through images or videos is inconvenient for users. Expressing through natural language is a more user-friee uiernffiendly option. Many studies have investigated how to map natural languages reward signals for sagguugg--gidee learning (Goyal et al., 2019; Arumugam et al., 2017: Misra et al., 2017). However, due language ambiguity, grounding agent behaviors becomes difficult, leading to lim- ited effective language instructions. One promising approach is to deconstruct the agent's 's behavior into combinations of multiple Relative Behavioral Attributes (RBA) varying levels of strength (Guan et al., 2023), allowing humans to express preferences in terms of relative attribute strengths.
'These authors contributed equally to this work. 1 Corresponning author; Jianye Hao (jianee.ha@@ (ww.oour.w..u..nn.

---

Published as a conference paper at ICLR 2024
The primary limitation of this approach is that it merely refines attribute evaluation into myopic, single-step state-action reward model, ignoring the impact on overall trajectory performance.
The mutability of human preferences emphasizes the need for zero-shot behavior customizing. Training agents using an attribute-conditione reward model fails to overcome this issue (Hu et al., 2020; Guan et al., 2023), it requires constant fine-tuning or even retraining whenever the user's intention changes, resulting in a poor user experience. To bridge this gap, we need a model to fit diverse behaviors and support retrieving based on the relative attribute strengths without repetitive training. Although standard choices like conditional behavior cloning can achieve similar goals, they are limited by poor expressiveness and fail to capture diverse human preferences (Ross al., 2011: Razavi et al.. 2019; Orsini et al., 2021). Therefore, we focus on powerful conditional gen- erative models, specifically diffusion models, which have demonstrated excellent expressiveness in dealing with complex distributions and superior performance in decision-making tasks with complex dynamics (Janner et al., 2022; Ajay et al., 2023; Dong et al., 2024).
In this paper, we introduce AlignDiff, novel framework that leverages RLHF to quantify hu- man preferences, covering abxtractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mit- tability. We created multi-perspective hu- man feedback datasets containing comparisons for the attributes on diverse behaviors, which were used train a transformer-basee attribute strength model. This model captures the rela- tive strength of attributes on the trajectory level. We then used the attribute strength model to annotate behavioral datasets and trained a dif- fusion model for planning. Within AlignDifff, agents can accurately match user-customized behaviors and efficiently switch from one to another. Its name, AlignDiff, represents both Aligmment Diffusion and aligning to bridge the difference between human preferences and agent behaviors. We summarize the main contributions of Align follows:
TIIE 01
seed 42 1P O.1
sperd 6.7
Junp
Janq Vigke
Juap Bouey
and Abowe
C.S.
QUAN 2.1
QHHE-OA lamarans
KINL E.
Tas
5ped
Spenl
AlignDiff achieves zero-shot human prefer- ences aligning. Here, three robots continuously switch their behaviors based on human instructions, where o represents no effect.
Figure 1:
We introduce AlignDiff: a novel framework that leverages RLHF technique to quantify human preferences and utilizes them to guide diffusion planning for zero-shot behavior customizing. We establish reusable multi-perspective human feedback datasets through crowdsourcing, which contains diverse human judgments on relative strengths of pre-defined attributes for various tasks. By making our dataset repositories publicly available, we aim to contribute to the wider adoption of human preference aligning.
We design set of metrics to evaluate an agent's preference matching, switching, and covering capability, and evaluate Diff on various locomotion tasks. The results demonstrate its supe- rior performance in all these aspects. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-Al collaboration.
RELATED WORKS
2.1 REINFORCHMENT LEARNING FROM HUMAN FEEDBACK
Reinforcement Learning from Human Feedback (RLHF) is powerful technique that can speed up AI training and enhance AI capabilities by leveraging human feedback. (Pilarski et al., 201 1; Akrour et al., 201 1; Wilson et al., 2012; Akrour et al., 2012; Wirth & Fürnkranz, 2013; Zhou et al., 2024; Liu et al., 2024; Yuan et al., 2024). There are various forms of human feedback (Bıyik et al., 2022; Cabi et al., 2019; Lee et al., 2021), in which collecting pairwise comparison over decision trajecto- ries is a common approach. This feedback is used to learn reward models for RL training, which significantly improves training efficiency and performance, especially in environments with sparse or ill def-ned rewards (Christiano et al., 2017; Lee et al., 2021; Park et al., 2022; Liang et al., 2023a; Zhou al., 2020). However, their limitation lies in the focus optimizing single objective. In
2

---

Published as a conference paper at ICLR 2024
recent years, RLHF has also been applied to fine-tune large language models (LLMs) by leveraging human preferences to improve truthfulness and reduce toxic outputs (Stiennon et al., 2020; Ouyang et al., 2022). These methods highlight its potential for human preference quantification. RBA leverages RLHF to distill human understanding of abstract preferences into an reward model, which achieves simple and effective quantification. However, this approach has limi- lations. It refines the evaluation of attributes into single step and ignores the impact on the overall trajectory. Furthermore, it requires retraining whenever new preferences emerge.
2.2 DIFFUSION MODELS FOR DECISION MAKING
Diffusion models, a type of score matching-based generative model (Sohl-Dickstein et al., 2015; Ho et al., 2020), initially gained popularity in the field of image generation (Ramesh et al., 2021; Galatolo al., 2021; Saharia et al., 2022; Rombach et al., 2022). Their strong conditional generation capabilities have led to success in various domains (Peng et al., 2023; Rasul et al., 2021; Reid et al., 2023; Liu al., 2023), including decision making (Janner et al., 2022; Liang al., 2023b; Chen et al., 2023: Wang et al., 2023; Pearce et al., 2023; Hegde et al., 2023; Dong et al., 2024). One common approach employing diffusion models to generate decision trajectories conditioned on rewards or other auxiliary information, which are then used for planning (Ajay et al., 2023; Ni et al., 2023). However, these reward-conditioned generations only utilize a small portion of the learned distribution. We propose using human preferences to guide diffusion models to discover and combine diverse behaviors, ensuring the full utilization of the learned trajectory distribution.
3 PRELIMINARIES
We consider the scenario where human users may want to customize an agent's behavior at any given time. This problem can be framed as a reward-fres frre Markov Decision Process (MDP) denoted as M = AS P. ax). Here, S represents the set of states, A represents the set of actions, P S x A x 5 - [0, is the transition function, and a {ar.... (rs) represents a set of k predefined attributes used to characterize the agent's behaviors. Given a state-only tra- jectory r'
Problem setup:
.t-1). (811)) we assume the existence of an attribute strength function that maps the trajectory to relative strength vector (7') va = [o"),... baa] E [0, 1Ja. Each ele- ment of the vector indicates the relative strength of the corresponding attribute. A value of 0 for 2 implies the weakest manifestation of attribute a; while value of 1 represents the strongest manifestation. We formulate human preferences as a pair of vectors (Wingg m²), where viay repre- sents the target relative strengths, and m E {0, 1}th is a binary mask indicating which attributes are of interest. The objective is find policy a = ( m)) that minimizes the LI norm 11 - (E, o m=//11 where o denotes the Hadamard product. We learn human prefer- ences from an unlabeled state-action dataset D = (r), which contains multiple behaviors.
Preference-basee Reinforcement Learning (PbRL) (Christiano et al., 2017) is a framework that leverages human feedback to establish reward models for RL training. In PbRL, researchers typi- cally collect pairwise feedback from annotators on decision trajectories ( Feedback labels represented as y E O). (0, 1), (0.5, 0.5) where (1, indicates that Tι performs better, (0, 1) indicates that 72 performs better, and (0.5, 0.5) indicates comparable performance. The collected feedback dataset is denoted as D,- = [T1, 72,9}
} and can be used to train reward models.
Denoising Diffusion Implicit Models (DDIM) (Song et al., 2021) are a type of diffusion model that consists of a non Mon-Maakovian forward process and Markovian reverse process for learning the data distribution g (x)). The forward process q 20) is non Mor-Markovian but designed
ensure that (0,, mn NV(EE (1 T) The reverse process (a (xx;) = N (( (æ;, 1). E,) is trained to approximate the inverse transition kernels. Starting with Gaussian noise कт (O, I). a sequence of latent variables (IT-1,-.. , .x1) is generated iteratively through series of reverse steps using the predicted noise. The noise predictor (x, t)), parameterized by a deep neural network, estimates the noise c ~ NN(0, I) added to the dataset sample Ty produce the noisy sample Tt, Using DDIM, we can choose a short subsequence R of length S to perform the reverse process, significantly enhancing sampling speed.
Classifier-free guidance (CFG) (Ho & Salimans, 2021) is a guidance method for conditional gen- eration, which requires both conditioned noise predictor 6ø ( 1, e) and unconditioned one
3

---

Published as a conference paper at ICLR 2024
① Multi-perspecivv Human Feedback
Attribute Model Training
3 Diffusion Training
state-only trajectory
enahedding
Anmotated
|| ca
Velority
Dataset
(r",
Step sire
Humanness-
Trunsformer Encader
Reward-free
Bik,p) m
AlignDitt
Step
Attribute Mapping Layer
Humanness
Trajectory Annotating
Uniluru
Artribute yf
4 AlignDiff Inference
AlignDiff Architecture
Language Command
Tix
Mapping
Self-
Timestop
Linear LayerNorm
AliguDitf
Environment
Attributt
Attention
MLP
Discwvtbe Embedding
Denoising
Mask m
Figure 2: Overview of AlignDiff. We begin by collecting human feedback through crowdsourcing, which then used to train an attribute strength model Ç Relabeled by it, an annotated dataset is then used to train an attribute- coabbutte-oniiiiooned diffusion model ég. With these two components, we can use AlignDiff to conduct preference alignment planning.
c (2,, t), where e the condition variable. By setting a guidance scale w and giving condition c, use Eo (ar, 1 c) (1 w)e wes (Ir,,) () predict noise during the reverse process.
METHODOLOGY
We propose AlignDiff, novel framework that leverages RLHF to quantify human preferences and utilizes them to guide diffusion planning for zero-shot behavior customizing. The process gen- eralization is depicted in Fig. 2, and the framework consists of four parts. Firstly, collect multi- perspective human feedback through crowdsourcing. Secondly, we use this feedhack to train an attribute strength model, which then to relabel the behavioral datasets. Thirdly, train a diffusion model on the annotated datasets, which can understand and generate trajectories with various attributes. Lastly, we can use AlignDiff for inference, aligning agent behaviors with human preferences at any time. Details of each part are discussed in the following sections.
4.1 MULTI-PERSPECTIVE HUMAN FEEDBACK COLLECTION
We extract small subset of pairs ((T1, T2)) from the behavioral datasets and ask human annota- tors to provide pairwise feedback. Instead of simply asking them to choose better one between two videos, we request analyzing the relative performance of each attribute a,, resulting in a feedback dataset DD {(71, T2, Vaur) where Vaur (V1,.... Mk... For example (see Fig. 2, part D), to eval- uate a bipedal robot with attributes (speed, stride, humanness) the annotators would be provided with two videos and asked to indicate which one has higher speed/bigger stride/better hu- manness, respectively. This process is crucial for creating a model that can quantify relative attribute strength, many attributes, such humanned Even for more specific and measurable attributes, such as speed, using human labels helps the model produce more distinct behaviors, shown by our experiments.
only be evaluated through human intuition.
4.2 ATTRIBUTE STRENGTH MODEL TRAINING
After collecting the feedback dataset, we can train the attribute strength model by optimizing modified Bradley-Terry objective (Bradley Terry, 1952). We define the probability that human annotators perceive to exhibit stronger performance on ag follows:
exp Eor (T)
pad [
To) Eje(1.) exp (T,
(I)
4

---

Published as a conference paper at ICLR 2024
where (T) indicates the i-th element of L (r). To approximate the attribute strength function, we optimize the following modified Bradley-Terry objective:
((())
Σ
E
yr(1) pa [T > T2l y1( p IT2 > m
(2)
(TL:72-BurrEDD JED (E(1,-
It's worth noting that there are significant differences between learning the attribute strength model and the reward model in RLHF. For instance, one-step state-action pairs cannot capture the attribute strengths, so G must be designed as mapping concerning trajectories. Additionally, we aim for Ço accommodate variable-length trajectory inputs. Therefore, we use transformer encoder as the structure (see Fig. 2, part @2), which includes an extra learnable embedding concatenated with the input. The output corresponding to this embedding is then passed through linear layer to map it to the relative strength vector va After training, we can partition the dataset 'D into fixed- length trajectories of length H. These trajectories are then annotated using &a resulting in a dataset Da {(T", )) for diffusion training.
4.3 DIFFUSION TRAINING
We use relative strength values va and at- tribute masks m conditioning inputs for the diffusion model, resulting in a conditioned noise predictor Co(2,, va, m.) and uncon- ditioned one eo (xt). According to our defini- tion of m (xx)) equivalent
Trained locomotioo AligaDilf deployed
complete unseen tasks in zero-shot manner.
a condi- tioned noise predictor with a mask where all values are O. Therefore, we only need one network to represent both types of noise pre- dictors simultaneously. However, the network structure must meet two requirements: 1) ma should eliminate the influence of nonrequested attributes the model while preserving the ef- fect the interested attributes, and 2) va can- not be simply multiplied with m² and fed into the network, as a value of in v" still car- ries specific meanings. To meet these require- ments, we design an attribute-oriented encoder. First, we discretize each dimension of the rela- tive strength vector into V selectable tokens as follows:
stride.
stride.
up.
Figure 3: AlignDiff, trained solely locomotion dataset, demonstrates the capability to accomplish un- seen downstream tasks through human instructions.
o clip( ,0, 6)
6) V I (i- 1=1,-....
(3)
where 5 small slack variable. This ensures that each of the possible cases for each attribute assigned a unique token. As a result, the embedding layer outputs a vector that contains information about both attribute category and strength. This vector is then multiplied with the mask, passed through multi-head self-attention layer and a linear layer, and used as conditioning input to noise predictor. Furthermore, due to our requirement of large receptive field to capture the backbone, DiT (Peebles & Xie,
attributes on the trajectory level, we employ transformer-basee 2023), for our noise predictor, instead of the commonly used UNet (Ronneberger et al., 2015). To adopt DiT for understanding relative attribute strengths and guiding decision trajectory generation, we made several structural modifications. (See Fig. 2, part AlignDiff Architecture). Combining all the above components, we train diffusion model with the noise predictor loss, where m is sampled from a binomial distribution B (k,, p), and represents the no-masking probability.
(( Be Pege.pe")--Pe..uuunnmmomemee...
es (x n" mm
(4)
4.4 ALIGNDIFF INFERENCE
With an attribute strength model ?g and a noise predictor 6g, we can proceed to plan with AlignDifff. Suppose state 8t, given preference (u m DDIM sampler with subsequence is of
5

---

Published as a conference paper at ICLR 2024
Hopger-5
Walker
Humansid
Hopper-H
Walke-H
Humamid-H
MA2 Twebelo
Tembedd
The vertical axis represents the MAE threshold, which is presented using logarithmic scale. horizontal axis represents the percentage samples below MAE threshold. Each point (z, y) on the curve indicates that the algorithm has a probability of to achieve an MAE hetween the agent's 's relative attribute strength and the desired value below æ. A larger area enclosed by the curve and the axes indicate better performance in matching human preferences.
Figure 4:
length iteratively generate candidate trajectories.
xx Ex.... Dx; VI En, (1FK)) V EM... - oh Ex( + on, Ew:
(5)
VEn
During the process, fix the current state it is always known. Each in the candidate trajectories satisfies human preference (v m)) priori. Then utilize ?g criticize and select the most aligned one to maximize the following objective:
J(r) = 11 (T)) m
(6)
The first step of the chosen plan is executed in the environment. For the convenience of readers, summarize the training and inference phases of AlignDiff in Algorithm and Algorithm 2. Additionally, to make interaction easier, we add natural language control interface. By keeping an instruction corpus and using Centence-BERT (Reimers & Gurevych, 2019) calculate the similarity between the instruction and the sentences in the corpus, we can find the intent that matches the closest and modify the attributes accordingly.
EXPERIMENTS
We conduct experiments various locomotion tasks from MuJoCo (Todorov al., 2012) and DMControl (Tunyasuvunakool et al., 2020a) to evaluate the preference aligning capability of the algorithm. Through the experiments,
aim answer the following research questions (RQs):
Matching (RQ1): Can Aiig better align with human preferences compared to other baselines? Switching (RQ2): Can AlignDiff quickly and accurately switch between different behaviors?
Covering (RQ3): Can AlignDiff cover diverse behavioral distributions in the dataset?
Robustness (RQ4): Can AlignDiff exhibit robustness to noisy datasets and limited feedback?
5.1 EXPERIMENTAL SETUP
Benchmarks: We select hopper, walker, and humanoid locomotion tasks the benchmarks (See Appendix A for more details). Predefined attributes are presented in Table 1. each experiment, we compare synthetic la- bels generated by scripts (denote as S) and human labels collected by crowdsourcing (denote as H) separately to demonstrate the generalizability of AlignDiff.
Table 1: Predetined attributes for each task.
Environment
Attributes
Speed
Hopper
Jump height
Speed
Torso height
Walker
Stride length
Left-right leg preference
Baselines: There are several existing paradigms for constructing policies conditioned on human preferences, which we use as baselines in our experiments (See Ap- pendix C for more details):
Humanness
Speed
Humanoid
Head height
Humanness

---

Published as a conference paper at ICLR 2024
Table 2: Area enclosed by the MAE curve. larger value indicates better alignment performance. Perfor- mance on the humanness attribute is reported a separate row.
Type/Attribute
Environment
GCrro ondtiiees
SMisequane Studding
Tarmng
AlignDiff
Hopper
0.285 1: 0.009
0.573 i 0.008
0.408 1 0.030
0.628 + 0.026
Synthetic
Walker
0.319 + 0.005
0.333 + 0.007
0.445 + 0.012
0.621 + 0.023
Humanoid
0.252 + 0.016
0.258 + 0.030
0.258 + 0.004
0.327 - 0.016
Average
0.285
0.388
0.370
0.525
Hopper 0.305 + 0.013
0.387 0.031
0.300 + 0.024
0.485 + 0.013
Human
Walker 0.322 1: 0.011
0.341 + 0.009
0.435 1: 0.005
0.597 + 0.018
Humanoid
0.262 + 0.026
0.272 = 0.029
0.262 + 0.027
0.313 + 0.015
Average
0.296
0.333
0.332
0.465
Walker
0.334 1: 0.035
0.448 1 0.003
0.510 1 0.006
0.615 + 0.022
Humanness
Humanoid
0.285 + 0.020
0.320 + 0.021
0.257 + 0.003
0.396 + 0.020
Average
0.310
0.384
0.384
0.510
Goal conditioned behavior clone (GC) leverages supervised learning to train goal conditioned policy T( m²) for imitating behaviors that match human preferences. We implement this baseline following RvS (Emmons et al., 2022).
Sequence modeling (SM) is capable of predicting the optimal action based on historical data and prompt tokens, allowing it to leverage the simplicity and scalability of Transformer architectures. We adopt the structure of Decision Transformer (DT) (Chen et al., 2021) and incorporate the preference (v m²) as additional token in the input sequence.
TD Learning (TDL) is a classic RL paradigm for learning optimal policies. TD3BC (Fujimoto & Gu, 2021) is one such algorithm designed for offiin RL settings. Since reward model is required to provide supervised training signals for policy optimization, we distill an
reward model from 20 to train TD3BC. This serves as an improved version of RBA.
Throughout the experiments, all algorithms share the same attribute strength model to ensure fair- ness. The source of human feedback datasets can be found in Appendix B.
5.2 MATCHING (RQ1)
Evaluation by attribute strength model: We conducted multiple trials to collect the mean ab- solute error (MAE) between the evaluated and target relative strengths. For each trial, we sample an initial state so, a target strengths win.. and a mask m as conditions for the execution of each algorithm. Subsequently, the algorithm runs for T steps, resulting in the exhibited relative strengths va evaluated by Čo. We then calculated the percentage of samples that fell below pre-designed thresholds to create the MAE curves presented in Fig, 4. The area enclosed by the curve and the axes were used to define a metric, which is presented in Table 2. A larger metric value indicates better performance in matching. Experiments show that AlignDiff performs significantly better than other baselines on the Hopper and Walker benchmarks. On the Humanoid benchmark, AlignDiff is the only one that demonstrates preferences aligning capability, while the other baselines fail to learn useful policies. We also find that AlignDiff exhibits slightly better performance on synthetic labels, which may be attributed the script evaluation being inherently rational without any noise.
Evaluation by humans We further conducted a questionnaire-base evaluation. Specifically, we instructed the algorithm to adjust an attribute to three different levels (corresponding 0.1, 0.5, and 0.9), resulting in three video segments. The order of the videos was shuffled, and human evaluators were asked to sort them. A total of 2.160 questionnaires were collected from 424 human evaluators. The sorting accuracy results are reported in Table 3, in which a higher accuracy indicates better per- formance in matching human preferences. We observe that AlignDiff performs significantly better than other baselines across different environments. This leads us to conclude that AlignDiff suc- cessfully utilizes the powerful conditional generation abilities of diffusion models to exhibit notable differences in the specified attributes. Additionally, we find that sorting accuracy is much higher when using human labels compared to synthetic labels. This suggests that human labels provide a level of intuition that synthetic labels cannot, resulting in better alignment with human prefer- ences. To ensure impartiality, we have also conducted a t-test on the evaluator groups We refer to Appendix D for more information the evaluation process.

---

Published as a conference paper at ICLR 2024
Table 3: Accuracy of videos sorting by human evaluators. A higher accuracy indicates better performance in matching human preferences. Performance on humanness attribute is reported in separate row.
Label/Attribute
Environment
GCori [C)
Seguass Modiing)
TDmmmtamnn!
AlignDiff
Hopper
0.67
80.86
36.23
85.45
Synthetic
Walker
5.86
9.72
49.49
70.23
Humanoid
19.88
19.88
4.02
65.97
Average
8.80
38.47
29.91
73.88
Hopper
0.00
32.78
15.56
95.24
Human
Walker
4.81
2.55
59.00
93.75
Humanoid
21.82
27.66
0.00
66.10
Average
7.06
23.12
24.85
85.03
Walker
2.88
1.02
64.00
93.33
Humanness
Humanoid
21.82
27.66
0.00
72.88
Average
12.35
14.34
32.00
83.11
We conducted a specific human evaluation to assess the humanness attribute, and the results reported in a separate of both Table 2 and Table 3. In terms of humanness, we observe that Alig performs significantly better than other baselines. With the aid of human labels, AlignDiff is able effectively capture the features of human motion patterns and exhibit the most human-like behavior.
Investigation of the humanness attribute:
5.3 SWITCHING (RQ2)
Track the changing target attributes: To evaluate the ability of the learned model to switch between different behaviors, we conducted an attribute-trakking experiment on the Walker bench- mark. Starting from the same initial state, we ran each algorithm for 800 steps, modifying the target attributes Vaagg at steps (0, 200, 400, 600), and recorded the actual speed and torso height of the robot The tracking curves are presented in Fig. We observe that AlignDiff quickly and ac- curately tracked the ground truth, whereas the other baselines showed deviations from it, despite demonstrating trend in attribute changes.
Complete unseen tasks by attribute instructions: In addition, we tested AlignDiff's zero-shot capability under human instructions by deploying it to unseen downstream tasks. By adjusting tributes such speed, torso height, and stride length by the human instructor, the walker robot, which was only trained on locomotion datasets, successfully completed the gap-crossing and ob- stacle avoidance tasks from Bisk benchmark (Gehring et al., 2021). Fig. 3 presents selected key segments this test, highlighting the promising potential AlignDiff for human-A Al collaboration.
5.4
COVERING (RQ3)
GoundPruth
AlignOfff
GC
SM
TDL
0
0.0
1.0
1.2
0.5
0.2
2.0
1.-
12-
1.2
Veen core
0.0
0.
Figure 5: Distribution plats of attribute strength values and actual attributes are shown. The horizontal axis represents the attribute strength value v, while the vertical axis represents the corresponding actual attribute p(u,v) v) denotes the probability that, given target attribute strength value the algorithm can produce trajectory with the actual attribute u. The color gradient in the distribution plot represents the probability p(u, v), ranging from dark to light.
To align with complex and variable human intention, AlignDiff requires the capability to cover a diverse range of behaviors within the offline datasets. This investigation aims determine: Can
8

---

Published as a conference paper at ICLR 2024
Spood ma))
Tew to
1.2
700
400
600
2013
200
400
620
800
Time
Time
AlianDuit
SM
TDL
GroandTruii
Figure 6: Visualization of behavior switching. The target attributes of speed and torso height at steps 200, 400, and 600 are set [0.5, 0.75, 0.875, 0.3875] and [0.8, 0.4, 0.7, 0.35|, respectively. We obtain the actual attributes corresponding to the target from the dataset distribution in Fig. 5 as the ground truth.
Table 4: Area enclosed by the MAE curve of AlignDiff trained on noisy datasets.
Environment
GC coeatoeedd
SM
TDLan
AlignDiff
oracle
0.319 + 0.007
0.338 + 0.008
0.453 + 0.012
0.634 = 0.020
20% noise
0.320 1: 0.014
0.327 + 0.013
0.323 :: 0.031
0.559 + 0.041
50% noise
0.300 + 0.022
0.281 + 0.014
0.305 = 0.028
0.384 + 0.022
AlignDiff produce the most probable dynamics within the datasets, and even combine different at- tributes to produce unseen behaviors? We compare the distribution of p(u, v), which represents the likelihood that the algorithm produces trajectories with the actual attribute u given the target strength v, between the algorithm and the datasets. Due intractability, Monte Carlo methods are used to approximate it, as detailed in Appendix E. We focus on the speed and torso height attributes defined in the Walker benchmark, as the corresponding physical quantities can be directly obtained from the MuloCo engine. As shown in Fig. 5, we report the distributions corresponding to each al algorithm and observe that AlignDiff not only covers the behaviors in the datasets but also fills the "disconnected regions" in the ground truth distribution, indicating the production of unseen behaviors.
5.5 ROBUSTNESS (RQ4)
We evaluate the robustness of our algorithm from two as- pects: 1) Robustness to dataset noise. We established two additional datasets for the Walker benchmark, where random decision trajectories are mixed into the original datasets proportions of 20% and 50%, respectively. We
Table 5: Performance of AlignDiff trained with different number of feedback labels.
Number labels
Area
10,000
0.628 + 0.026
2,000 (80%L)
0.621 + 0.013 (1.11%))
train algorithms on the original datasets and these two noisy datasets. The performance is shown in Table 4. The
(95%))
1 0.029 (16.2%))
experiment shows that AlignDiff has the best robustness to noise. 2) Robustness to the number of feedback labels. We compare the performance of AlignDiff trained on 10k, 2k, and 500 synthetic labels the Hopper benchmark, shown in Table 5. The experiment shows that the performance does not decrease significantly when the number of feedback labels decreases from 10k to 2k, and only some performance loss is observed when the number decreases 500. This result indicates that A Difg can achieve good performance with fewer feedback labels.
CONCLUSION
In this paper, we introduce AlignDiff, novel framework that achieves zero-shot human preference aligning. Our framework consists of two main components. The first component utilizes RLHF technology to quantify human preferences, addressing the abstractness of human preferences. The second component includes a match desired behaviors and efficiently switch from one another, addressing the mutability of human preferences. We conducted various experiments to evaluate the algorithm's 's capability of preference matching, switching, and covering. The results demonstrate that AlignDiff outperforms other strong baselines with exceptional performance. However, like other diffusion- rithms (Hegde et al., 2023), DifgnDiff is slow at inference time due to the iterative sampling process. Performing faster sampling may mitigate the issue. The ability of AlignDiff to accomplish unseen tasks under human instructions showcases its potential to combine skills and complete complex tasks under the command of higeer-level models (e.g. LLMs), which leave as future work.
diffusion model, which can plan to accurately
RL algo-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported by the National Key R&D Program of China (Grant No. 2022ZD01 16402), the National Natural Science Foundation of China (Grant Nos. 92370132) and the Xiaomi Young Talents Program of Xiaomi Foundation. We thank YouLing Crowdsourcing for their assistance in annotating the dataset.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? The Eleventh Interna- tional Conference on Learning Representations, ICLR, 2023.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML. PKDD, 2011.
Riad Akrour, Marc Schoenauer, and Michèle Sebag. April: Active preference learning-based inforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 116-131, 2012.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS, 2017.
Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson Wong, and Stefanie Tellex. Ac- curately and efficiently interpreting human-robot instructions of varying granularities. In Pro- ceedings of Robotics: Science and Systems, 2017.
Erdem BIyIk, Dylan P Losey, Malayandi Palan, Nicholas C LLnddlfi, Gleb Shevehuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrat- ing demonstrations and preferences. The International Journal of Robotics Research, pp. 45-67, 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pp. 324-345, 1952.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, al. Scaling data- driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv:1909. 12200, 2019.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, ICLR. 2023.
Lili Chen, Kevin La, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning 2021.
via sequence modeling. arXiv preprint
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, NIPS, 30, 2017.
Zibin Dong. Jianye Hao, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and Yan Zheng. Diffuserlite: Towards real-time diffusion planning. arXiv preprint arXiv:2401. /5443, 2024.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, ICL., 2022.
10

---

Published as a conference paper at ICLR 2024
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, NIPS, 2021.
Federico Galatolo, Mario Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via CLIP- -guiedd generative latent space search. Proceedings of the International Con- ference Image Processing and Vision Engineering, 2021.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. In Advances in Neural Information Processing Systems, NIPS, 2021.
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. Using natural language for reward shaping in reinforcement learning. Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI, 2019.
Lin Guan, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging approximate symbolic mod- cls for reinforcement learning via skill diversity. arXiv preprint arXiv:2202.088 022888.00.8 2022.
Lin Guan, Karthik Valmeekam, and Subbarao Kambhampati. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, NIPS, 2017.
Shengman Han, Eugene Kelly, Shahrokh Nikou, and Eric-Oluf Svee. Aligning artificial intelligence with human values: reflections from a phenomenological perspective. Al and SOCIETY, 37, 2021.
Shashank Hegde, Sumeet Batra, K. R. Zentner, and Gaurav S. Sukhatme. Generating behaviorally diverse policies with latent diffusion models. arXiv preprint arXiv:2305.18738, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistie models. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NIPS, 2020.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: new approach of reward shaping. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS, 2020.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning. ICML, 2022.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback- interactive reinforcement learning via relabeling experience and unsupervised pre- pra-train In International Conference on Machine Learning, ICML, 2021.
Hebin Liang, Zibin Dong, Yi Ma, Xiaotian Hao, Yan Zheng, and Jianye Hao. A hierarchical im- itation learning-based decision framework for autonomous driving. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM, 2023a.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models adaptive self-evolving planners. In International Conference Machine Learning, JCML, 2023b.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. Interna- fional Conference on Machine Learning, ICML, 2023.

---

Published as a conference paper at ICLR 2024
Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprint 14%44 2024.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In NeurlPS 2022 Foundation Models for Decision Making Workshop. 2022.
Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, 2017.
Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl, 2023.
Manu Orsini, Anton Raichuk, Leonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? Advances Neural Information Processing Systems, NIPS, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in Neural Information Processing Systems, pp- 27730-27744, 2022.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee. Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-bar reinforcement learning. In International Conference on Learning Representations, ICLR, 2022.
Deepak Pathak, Parsa Mahmoudich*, Michael Luo*, Pulkit Agrawal*, Dian Chen, Fred Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. In International Conference on Learning Representations, ICLR, 2018
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Im- itating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 22Xi::2212.09748.. 2023.
Xingang Peng, Jiaqi Guan, Qiang Liu, and Jianzhu Ma. MolDiff: Addressing the bond incon- sistency problem in 3D molecule diffusion generation. In Proceedings of the 40th International Conference on Machine Learning, ICML, pp. 27611-27622 2023.
Patrick Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In 2011 IEEE international conference on rehabilitation robotics, pp. 2011.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 2021.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. International Conference on Machine Learning, ICML, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2 Advances in neural information processing systems, NIPS, 2019.
12

---

Published as a conference paper at ICLR 2024
Machel Reid, Vincent Josua Hellendoorn, and Graham Neubig. DiffusER: Diffusson via edit-based reconstruction. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- image segmentation. arXiv preprint arXiv: 04597, 2015.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international ference on artificial intelligence and statistics, pp. Proceedings, 2011.
JMLR Workshop and Conference
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistie text-to- image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, NIPS, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0677, 2017.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, In Proceedings of the 32nd International Conference on International Conference on Machine Learning, ICML, pp. 2256-2265 2265, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. In Interna- tional Conference on Learning Representations, ICLR, 2021.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, NIPS, 33:3008-3021, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, pp. 100022, 2020a.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020b.
Mel Vecerik. Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 2018
Zhen Wang. Marko Jusup, Rui-Wn Wang. Lei Shi, Yoh Iwasa, Yamir Moreno, and Jürgen Kurths. Onymity promotes cooperation in social dilemma experiments. Science Advances, 2017.
Zhen Wang. Marko Jusup, Lei Shi. Jounghun Lee, Yoh Iwasa, and Stefano Boccaletti. Exploiting a cognitive bias promotes cooperation in social dilemma experiments. Nature Communications, 2018.
13

---

Published as a conference paper at ICLR 2024
Zhen Wang. Marko Jusup, Hao Guo, Lei Shi, Sunčana Geček, Madhur Anand, Matjaž Perc, Chris T. Bauch, Jürgen Kurths, Stefano Boccaletti, and Hans Joachim Schellnhuber. Communicating senti- ment and outlook reverses inaction against collective risks. Proceedings of the National Academy of Sciences, 2020.
Zhen Wang, Chunjiang Mu, Shuyue Hu, Chen Chu, and Xuelong Li. Modelling the dynamics of regret minimization large agent populations: master equation approach. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. The Eleventh International Conference on Learning Representations, ICLR, 2023.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, NIPS, 2012.
Christian Wirth and Johannes Fürnkranz. Preference-based reinforcement learning: A preliminary survey. In Proceedings of the ECML/PKDI-11 13 Workshop on Reinforcement Learning from Gen- eralized Feedback: Beyond Numeric Rewards, 2013.
Yifu Yuan, Jianye HAO, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and YAN ZHENG. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In The Twelfth International Conference on Learning Representa- tions, ICLR, 2024.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassan- zadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmel- lat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving, 2020.
Xinglin Zhou, Yifu Yuan, Shaofu Yang, and Jianye Hao. Mentor: Guiding hierarchical re- inforcement learning with human feedback and dynamic distance constraint, arXiv preprint (4t.:22402.1222444 2024.
Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via offline objti-oeeective rl. In International Conference on Learning Representations, ICLR, 2023.
14

---

Published as a conference paper at ICLR 2024
A TASKS AND DATASETS
In this section, we will provide detailed description of three locomotion tasks that serve as perimental benchmarks, along with an explanation of how we collected the corresponding offline datasets.
Hopper. Hopper is single-legged robot selected from the Gym-MuJoCo locomotion tasks. We use pre-trained PPO (Schulman et al., 2017) agents provided by PEDA (Zhu et al., 2023) to collect total of million time steps for our datasets. The relative magnitudes of speed and height provided by PEDA used to generate synthetic labels.
Walker. Walker is a simplified humanoid bipedal robot selected from the deepmind control suite benchmarks (Tunyasuvunakool et al., 2020b). To collect an offline dataset, modify the reward function of the original Walker benchmark and randomly select target velocities and heights. We train 32 policies with SAC (Haarnoja et al., 2018). These policies are used to collect total of 3.2 million time steps for our datasets, To generate synthetic labels, we retrieve the corresponding phys- ical quantities from the MuloCo engine to compute and compare the attributes of two trajectories. However, humanness can only be obtained through human feedback, making it infeasible to employ the synthetic labels setting.
Humanoid. Humanoid is 3D bipedal robot designed to simulate human. To collect offline dataset, we modify the reward function of original Humanoid benchmark and randomly select target velocities and heights. We train 40 policies with SAC. These policies are used to collect total of million time steps for our datasets. To generate synthetic labels, we retrieve the corresponding physical quantities from the MuJoCo engine to compute and compare the attributes of two trajec- tories. However, humanness can only be obtained through human feedback, making it infeasible to employ in the synthetic labels setting.
B HUMAN FEEDBACK COLLECTION DETAILS
Human preferences are influenced by various internal and external factors (Wang al., 2017: 2022; 2018; 2020). Hence, in this section, we provide our human feedback collection details. We recruited total of 100 crowdsoureing workers annotate 4,000 feedback labels for each environment. Each worker was assigned to annotate 120 labels. They were given video pairs that lasted for about sec- onds (100-time steps, generating videos at 30 frames per second) and were asked indicate which showed stronger performance based on pre-defined attributes. We made sure that each worker was compensated fairly and provided them with a platform that allowed them stop working at any time. There is no reason to believe that crowdsourcing workers experienced any physical or mental risks in the course of these studies. The task description provided crowdsoure- ing workers is as follows:
their progress and
Hopper
Question: For each attribute, which side of the video pair shows a stronger performance?
Options: (Left Side, Equal, Right Side)
Definitions of pre-defined attributes:
Speed: The speed at which the agent moves to the right. The greater the distance moved the right, the faster the speed.
Jump height: If the maximum height the agent can reach when jumping is higher, stronger performance should be selected. Conversely, weaker performance should selected if the maximum height is lower. If it is difficult to discer which is higher, select equal.
Walker
Question: For each attribute, which side of the video pair shows stronger performance? Options: (Left Side, Equal, Right Side)
Definitions of pre-defined attributes:
15

---

Published as a conference paper at ICLR 2024
Speed:
The speed at which the agent moves the right. The greater the distance moved to the right, the faster the speed.
The maximum distance between the agent's feet. When the agent exhibits abnor- mal behaviors such falling, shaking, unstable standing, weaker performance should be selected.
Stride:
Leg Preference: If the agent prefers the left leg and exhibits a walking pattern where the left leg drags the right leg, stronger performance should be selected. Conversely, weaker performance should be selected. the agent walks with both legs in a normal manner, equal should be selected.
Torso height:
The torso height of the agent. If the average height of the agent's torso during movement is higher, stronger performance should be selected. Conversely, weaker performance should be selected if the average height is lower. If it is difficult to discern which is higher, select equal.
The similarity between agent behavior and humans. When the agent is closer to human movement, stronger performance should be selected. When the agent exhibits abnormal behaviors such as falling, shaking, or unstable standing, weaker performance should be selected.
Humanness:
Humanoid
Question: For each attribute, which side of the video pair shows stronger performance? Options: (Left Side, Equal, Right Side)
Definitions pre-defined attributes:
Speed: The speed which the agent moves the right. The greater the distance moved
to the right, the faster the speed.
The head height of the agent. the average height the agent's head during movement is higher, stronger performance should be selected. Conversely, weaker performance should be selected if the average height is lower. which is higher, select equal.
Head height:
difficult to discern
Humanness:
The similarity between agent behavior and humans. When the agent is closer to human movement, stronger performance should be selected. When the agent exhibits abnormal behaviors such falling, shaking, unstable standing, weaker performance should be selected.
C BASELINES DETAILS
In this section, we introduce the implementation details of each baseline in our experiments, the reasonable and necessary modifications compared to the original algorithm, and the reasons for the modifications.
Table 6: Hyperparameters of GC(Goal conditioned behavior clone).
Hyperparameter
Value
Hidden layers
2
Layer width
1024
Nonlinearity Batch size
ReLU
32
Optimizer Learning rate
Adam
10-3
x 10 Hopper/Walker
Gradient steps
10 Humanoid
16

---

Published as a conference paper at ICLR 2024
Table 7: Hyperparameters of SMeequuence modeling).
Hyperparameter
Value
Number of layers
Number of attention heads
Embedding dimension Nonlinearity
128
ReLU
Batch size Optimizer
32
Adam
Learning rate
10-4 10
Weight decay
Dropout
0.1
32 Walker
Context length
100 Hopper/Humanoid
5 10 Hopper/Walker
Gradient steps
10 Humanoid
Table 8: Hyperparameters of distilled reward model.
Hyperparameter
Value
Hidden layers
2
Layer width
512
Nonlinearity
ReLU
Batch size
256
Optimizer
Adam
Learning rate
10-4
Length of trajectories
100
Gradient steps
5000
C.1 GC(GOAL CONDITIONED BEHAVIOR CLONE)
GC leverages supervised learning to train goal conditioned policy TT(a st, va m We imple- ment this baseline based RvS (Emmons al., 2022) and choose human preferences (r m²) as w. Following the best practice introduced RvS, we implement GC policy as a 3-layer MLP, which formulates truncated Gaussian distribution. Given dataset DG {7 va optimize the policy to maximize:
Σ
Σ
Bmn~Bik, [log (aele v", ml
(+ wn JEDa ISISH
The hyperparameters are presented in Table The selection is nearly identical the choices pro- vided in Emmons et al. (2022).
C.2 SM(EQUUENCE MODELING)
SM is capable of predicting the optimal action based on historical data and prompt tokens, allowing it to leverage the simplicity and scalability of Transformer architectures. Following the structure introduced DT (Chen al., 2021), we design the structure of the SM baseline as causal Trans- former. In comparison to DT, where reward is present as signal, our task only involves target strength vector x and an attribute mask m Therefore, we remove the RTG tokens and instead set the first token the embedding of (o m)) During the training phase, SM is queried to predict the current action a based on (v m) historical trajectories (s<),a=)), and the current state s, We optimize SM to minimize:
Σ (HH, 15tsH
Σ
Bk a 0m -ael? 2
(8)
17

---

Published as a conference paper at ICLR 2024
Table 9: Hyperparameters of TDL,(D learning..
Hyperparameter
Value
Hidden layers
2
Layer width
512
Number of Q functions
2
Nonlinearity Discount Tau
ReLU
0.99
0.005
Policy noise
0.2
Noise clip
0.5
Policy frequency
2
Alpha
2.5
Batch size
32
Optimizer
Adam
Learning rate
3 10-4
5 10 Hopper/Walker
Gradient steps
10 Humanoid
During the inference phase, we use f(sci, azt, v m)) as our policy and continuously store state-action pair into the historical trajectory. The hyperparameters are presented in Table The selection is nearly identical to the choices provided in Chen et al. (2021).
C.3 TDL(TD LEARNING)
TD learning is classic RL paradigm for learning optimal policies. Since reward model is needed provide supervised training signals, first need distill &g into reward model. Follow- ing the approach in RBA, we train an altribuee-strenttt-conittt
reward model, denoted as ra (St, at, va, m) We begin by using the reward model to define an attribute proximity proba- bility:
exp E. (s a/; va m.)
Plm > T2 T2/00 m
(9)
E Eie(1.2) exp E, ro(s), oj va m²)
This equation represents the probability that attribute strength of T is more aligned with (v m) compared 72. Subsequently, obtain pseudo-labels through the attribute strength model:
(TT F2, v (1, 0), if (l(((( v²) o m S (1(59(72) v²) m
(10)
(0, 1), if ((((((())) - aa) o m > (1(5"(12 va) m
With these pseudo-labels, train the distilled reward model by optimizing cross-entropy loss: log P[Ti > 12|00, m ((2) > n,/,,,="
-Epo BinelEm=-Biie B(.
(11)
Intuitively, if executing action a, under state s; leads trajectory that exhibits attribute strength closer to (v m)), the corresponding reward ro(st, da m2) will be larger; conversely, if the attribute strength is farther from (v m)), the reward will be smaller.
Since the reward model now conditions the attribute strength, we need to define the attribute- strength-conddtioned Q function Q(s, a, va, m²) and the policy ((s, va m²). With all these com- ponents, we can establish TDL baseline on top of TD3BC and optimize the policy by maximizing:
Σ
Σ E Em==B(k.p) (AQ(st: T (8t, va m m)), xa m²) - (ITs,, xa m²) - a) (12)
(r JDD ISISH
The hyperparameters are presented in Table 8 and Table 9, The selection is nearly identical to the choices provided in Fujimoto & Gu (2021).
D HUMAN EVALUATION DETAILS
D.I EVALUATION PROCESS
18

---

Published as a conference paper at ICLR 2024
Human evaluation is conducted in the form of questionnaires, which include multiple attribute strength ranking questions. The flow chart of the human evaluation process is sum- marized in Fig. 7. We collected a to- tal of 270 sets of videos for each al- gorithm, resulting in a total of 2,160 questionnaires. And we invite a tal of 424 human evaluators to partic- ipate in the experiment. Before the questionnaire, each evaluator is asked to provide basic demographic infor- mation help understand the dis- tribution of evaluators. Demographic information included gender, age, education, and experience with AI. The distribution of evaluators for each task is presented in Fig. 8. In each questionnaire, in addition to the relevant videos men- tioned Section 5.2, provide evaluators with task description of the environment and detailed explanations of the attributes being evaluated. The textual descriptions are as follows:
Head 0.1
Head betglts
Head Intaht
each algorithm
Miserical vrajemary
aljused atrilute:
Head height
description: provided attribute historcaal uajectury set to three different levels. Inter the correponding attnbbute value each video and fill the blank.
Figure 7:
The flowchart of the human evaluation process intro- duced in Section 5.2. The information visible to human assessors enclosed within gray boxes.
Hopper
Description: Hopper is single-legged robot that adjusts its movement based predefined attributes. Each attribute, such as Movement Speed, ranges from 0 to 1, with higher val- ues indicating stronger attributes (e.g.. a value of represents the fastest speed). In each task, we will set one attribute to three different levels: 0, I, and (corresponding to attribute strengths of 0.1, 0.5, and 0.9 respectively) for seconds of continued movement, in random order. You will see 3 videos with modified attributes. Your task is to infer the corresponding attribute value for each video. For example, if video exhibits the strongest attribute perfor- mance, you should select 2 from the options below. Each option can only be chosen once. you find it challenging to determine an attribute for a particular video, you can mark it as None. The None option be selected multiple times.
Attribute explanation:
The speed of movement, with higher attribute values indicating faster speed. Generally, the faster the movement the floor grid texture, the faster the speed.
Movement Speed:
Jump Height: The maximum height that can be reached by jumping, with higher attribute values corresponding to higher jump heights. Jump height can be determined by the jump posture or the size of the texture of the floor grid.
Walker
Description: Walker is bipedal robot that adjusts its movement based predefined at- tributes. Each attribute, such as Movement Speed, ranges from 0 to 1, with higher values indicating stronger attributes (e.g., value of 1 represents the fastest speed). In each task, you'll be shown 3-second video of Walker's past movements and informed about an at- tribute along with its current strength value. Next, will set the attribute to three different levels: 0, 1, and 2 (corresponding to attribute strengths of 0.1, 0.5, and 0.9 respectively) for the following 5 seconds of continued movement, in random order. You will see videos with modified attributes. Your task is to infer the corresponding attribute value for each video. For example, if a video exhibits the strongest attribute performance, you should select from the options below. Each option can only be chosen once. If you find it challenging to determine an attribute for particular video, you can mark as None. The None option can be selected multiple times.
Attribute explanation:
Movement Speed: The speed of movement, with higher attribute values indicating faster speed. Generally, the faster the movement of the floor grid texture, the faster the speed.
19

---

Published as a conference paper at ICLR 2024
The maximum distance between the feet, with higher attribute values responding to larger strides. Stride length can be determined by the number of grid squares crossed in single step.
Stride Length:
Legt-Right Leg Preference: A higher attribute value indicates Walker's tendency left leg to initiate right leg movement; a lower value indicates a preference for the right leg initiating left leg movement; when the attribute value is close to 0.5, Walker tends to use both legs simultaneously.
the
The height of the torso, with higher attribute values indicating greater torso height. Generally, the smaller the angle between the legs, the higher the torso; the larger the angle between the legs, the lower the torso.
Torso Height:
Humanness: The degree of similarity between Walker's behaviors and human behaviors, with higher attribute values indicating greater similarity.
Humanoid
Humanoid bipedal robot that adjusts its movement based on predefined attributes. Each attribute, such as Movement Speed, ranges from 0 to 1, with higher val- ues indicating stronger attributes (e.g.. a value of represents the fastest speed). In each task, we will set one attribute to three different levels: 0, and (corresponding to attribute strengths of O. 0.5, and 0.9 respectively) for seconds continued movement, in random order. You will see 3 videos with modified attributes. Your task is to infer the corresponding attribute value for each video. For example, a video exhibits the strongest attribute perfor- mance, you should select 2 from the options below. Each option can only be chosen once. If you find it challenging to determine an attribute for a particular video, you can mark it as None. The None option be selected multiple times.
Description:
Attribute explanation:
The speed of movement, with higher attribute values indicating faster speed. Generally, the faster the movement of the floor grid texture, the faster the speed.
Movement Speed:
Head Height:
The height of the head, with higher attribute values indicating greater head
height.
The degree of similarity between Humanoid's behaviors and human behav- iors, with higher attribute values indicating greater similarity.
Humanness:
D.2 T-TEST FOR HUMAN EVALUATORS
Table 10:
T-test result for human evaluation.
Classification
t-statistic
p-value
Male/Female
0.2231
0.8234
No experreenee/Experieneed
0.1435
0.8859
To ensure the reliability of our designed human evaluation, we conducted a t-test on the group of evaluators analyze whether there was any bias in the questionnaire. We divide the participants into two groups based on gender and whether they have AI learning experience, respectively. We set the hypothesis that there is no significant difference the average accuracy of the questionnaire evalu- ation between the two groups. If the hypothesis is accepted, we can conclude that the questionnaire design is unbiased and the experiment is reliable. the hypothesis rejected, we conclude that there is bias in the questionnaire design, which may cause certain groups make biased judgments, rendering the experimental design unreliable. The results of the t-test for the two groups under the two classification methods presented in Table 10. The results show that the p-values are higher than the significance level (0.05). Therefore, we can accept the null hypothesis and conclude that the experimental results are reliable.
20

---

Published as a conference paper at ICLR 2024
E DISTRIBUTION APPROXIMATION DETAILS
For each algorithm, we uniformly sample target strength value from the interval [0, 1]. Then we conduct each algorithm to obtain the trajectories and their corresponding actual attribute u. These values constitute the set ((,,.)- Let Umax and Umin denote the maximum and minimum values in this set, respectively. Next, divide the attribute strength interval and the actual attribute interval into equidistant segments, resulting in K cells: E [0, UkejK oj and E [umin Umax] = USer d' We can then use the following equation to obtain the approximation:
Pi)(u,)= (((,,)) E , c [D(w,))
/ Joz
p(u, du de
(13)
F VIDEO CLIPS OF VARIOUS BEHAVIORS PRODUCED BY ALIGNDIFF
In this section, aim to showease additional behaviors generated by AlignDiff in the form of video segments. Specifically, we employ AlignDiff trained with human labels to generate behaviors. We focus on five attributes: movement speed, torso height, left-right leg preference, humanness defined in the Walker domain, and humanness defined in the Humanoid domain. For each attribute, we provide three video clips starting from a random initial state, with the corresponding attribute values adjusted to [0.9, 0.5, 0.1], respectively.
F.1 WALKER: TORSO HEIGHT
As illustrated in Fig. when the strength attribute value is 0.9, the walker exhibits minimal knee flexion to maintain a relatively higher torso position. At a value of 0.5, the walker moves in a normal manner. However, when the attribute value is 0.1, the walker adopts a posture close kneeling, moving near the ground.
F.2 WALKER: LEFT RIGHT LEG PREFERENCE
As illustrated in Fig. 10, when the strength attribute value is 0.9, the walker primarily relies the left leg for movement, while the right leg is scarcely utilized. At value of 0.5, the walker alternates steps between the left and right legs. However, 0.1, the walker predominantly relies on the right leg for movement, with minimal use of the left leg.
F.3 WALKER: HUMANNESS
As illustrated in Fig. 11, when the strength attribute value is 0.9, the walker exhibits running pattern similar to that of a human. At value of 0,5, the walker displays reliance on a single leg, resembling the movement of person with injured right leg. However, 0.1, the walker engages in minimal leg movement, taking small and fragmented steps, deviating significantly from human-like walking.
F.4 HUMANOID HUMANOID-HUMANNES HUMANNESS
As illustrated Fig. 12, when the strength attribute value 0.9, the humanoid is capable of walking with both legs, resembling human locomotion. At strength value of 0.5, the humanoid can only perform single-leg movements, resembling a person with an injured right leg. When the strength value is 0.1, the humanoid will collapse and exhibit convulsive movements.
G IMPLEMENTATION DETAILS
In this section, we provide implementation details of AlignDifff
21

---

Published as a conference paper at ICLR 2024
Table 11: Hyperparameters of the attribute strength model.
Hyperparameter
Value
Embedding dimension
128
Number of attention heads
Number of layers Dropout
2
0.1
Number of ensembles
Batch size Optimizer
256
Adam
Learning rate
10
Weight decay
10-4
Gradient steps
3000
Table 12: Hyperparameters of the diffusion model.
Hyperparameter
Value
384 Hopper/Walkee
Embedding dimension of DiT
512 Humanoid
6 Hopper/Walker
Number of attention heads of DiT
8 Humanoid
12 Hlopper/Walker
Number of DiT blocks
14 Humanoid
Dropout
0.1
32 Walker
Planning horizon
100 Hopper/Huumnoiid
Batch size Optimizer
32 Adam 2 x 10-4 10-4
Learning rate
Weight decay
5 x 10t Hopper/Walker 10 Humanoid
Gradient steps
G.1 ATTRIBUTE STRENGTH MODEL
The attribute strength model utilizes Transformer Encoder architecture without employing causal masks. We introduce an additional learnable embedding, whose corresponding output then mapped to relative attribute strengths through linear layer. The structural parameters of the Transformer and the training hyperparameters are presented in Table 11.
We train total of three ensembles, and during inference, the average output of the ensembles is taken as the final output.
The direct output of the attribute strength model is real number without range constraints, but the attribute strengths we define are within the range of to 1. Therefore, we extract the maximum and minimum values of each attribute strength from the dataset of each task and normalize the output attribute values to be within the range of 0 to 1.
G.2 DIFFUSION MODEL.
The AiignDiff architecture consists of a DiT structure comprising 12 DiT Blocks, The specific structural and training parameters be found in Table 12.
For the Walker benchmark, we use a planning horizon of 32, while for Hopper and Humanoid, 100. We find that the length of the planning horizon should be
the planning horizons
22

---

Published as a conference paper at ICLR 2024
chosen to adequately capture the behavioral attributes of the agent, and this principle is followed selecting the planning horizons for the three tasks our experiments.
We use 200 diffusion steps, but for Hopper and Walker, we only sample a subsequence of 10 steps using DDIM, and for Humanoid, use subsequence of 20 steps.
We employ a guide scale of 1.5 for all tasks. We observed that smaller guide scales result in slower attribute switching but more reliable trajectories, while larger guide scales facilitate faster attribute switching but may lead unrealizable trajectories.
Algorithm AlignDiff training
Require: Annotated Dataset De. epsilon estimator Eg. unmask probability p
while not done do
(En, x ~ DG
t ~ Uniform Uniform([1,... .T))
<~ NO
m p(k,p)
Update Es to minimize Eq. (4)
end while
Algorithm AlignDiff planning
Require:
epsilon estimator ca. attribute strength model Ço target attribute strength v attribute mask m S length sampling sequence K, guidance scale while not done do
Observe state s;; Sample N noises from prior distribution Tag NN0, I)
for i = S , do
Fix 9; for Ix.
Eg + (1 + w (Ix, Ki va . wes (#x, Ki)
-x + Denoise (0) // Eq. (5)
end for
T f arg min ||(0 20 ) o m
Extract or from Execute 0t end while
H EXTENSIVE ABLATION EXPERIMENTS
H.1 ACCURACY OF THE ATTRIBUTE STRENGTH MODEL TRAINED ON HUMAN LABELS
Table 13: Success rate of the attribute strength model trained on human foedback datasets.
Size of training sets
Speed
Torso height
Stride length Left-right leg preference Humanness
3,200
92.48
92.68
83.01
$2.42
87 60
1,600
91.09
90.54
83.22
$1.32
$4.22
91.41
82.62
79.30
The accuracy the attribute strength model trained on human labels can help us understand the ability of the model for attribute alignment. Therefore, we conducted an additional experiment. We randomly selected 800 out of 4,000 human feedback samples from the Walker-H task as test set. From the remaining 3,200 samples, we collected 3,200/ 1,200//,600//80 ,600/800 samples as training sets, respectively, to train the attribute strength model. Subsequently, we recorded the highest prediction success rate of the model in the test set during the training process; see Table 13. We observed that only the "Humanness" attribute exhibited a significant decrease in prediction accuracy as the number of training samples decreased. This could be attributed to the fact that "Humanness" is the most abstract attribute among the predefined ones, making more challenging for the model to learn discriminative patterns with limited feedback labels.
23

---

Published as a conference paper at ICLR 2024
Table 14: Percentage of human labels that agree with the ground truth (annotated 'gt') and with other annotators (annotated as 'inter'). "Masked agreement" indicates the agreement calculation after excluding samples labeled as "cquivalent performance".
Type
Speed
Torso height
Stride length
Left-right leg preference
Humanness
Agrecment (gt)
85.55
84.25
Masked agreemeent (gt)
96.96
98.16
Agreement (inter)
84
77
81
79
72
Masked agreement (inter)
99
91
93
97
86
Table 15: Relationship between inference time and performance.
Sample steps
Inference time per action (seconds)
Performance
10
0.286 = 0.003
0.628 = 0.026
0. 144 1 0.002
0.621 1 0.023
0.089 = 0.002
0.587 + 0.019
H.2 AGREEMENT OF THE ANNOTATORS As annotator alignment is always a challenge in the RLHF community, we aimed to analyze the level of agreement achieved by the human labels collected through crowd-soureing. Since the assigned samples for each annotator were completely different during our data collection, it was not possible to calculate interannotator agreement for this portion of the data. To address this, we selected two attributes, velocity and height, from the Walker task, for which ground truth values could be obtained from the MuJoCo engine. We calculated the agreement between the annotators and the ground truth for these attributes. And, to further assess inter-annotator agreement, we reached out to some of the annotators involved in the data collection process. Three annotators agreed to participate again, and we randomly selected 100 pairs of videos for them to annotate. We calculate the agreement among the three annotators (considering an agreement when all three annotators provided the same feedback, excluding cases where they selected "equivalent performance"). The results are presented in Table 14. We observed high level of agreement among the annotators for attributes other than 'Humanness', which may be attributed to the relatively straightforward nature of these attributes and the provision of detailed analytical guidelines, as demonstrated in Appendix B. The agreement on the "Humanness" attribute was relatively lower, which can be attributed to its abstract nature and the varying judgment criteria among individuals. This observation is also supported by the phenomenon highlighted in Appendix H.I, where the attribute strength model showed increased sensitivity to the number of training labels for the "Humanness" attribute.
H.3 THE EFFECT OF DIFFUSION SAMPLE STEPS
AlignDiff, like other diffusion planning methods such as Diffuser (Janner et al., 2022) and De- cision Diffuser (DD) (Ajay et al., 2023), requires multiple steps of denoising to generate plans, which can lead to long inference times. However, AlignDiff' has made efforts to mitigate this issue by taking advantage of the DDIM sampler (Song et al., 2021), which allows plan generation with fewer sampling steps. To investigate the relationship between inference time and AlignDiff per- formance, we conducted an additional experiment. Specifically, we evaluated different numbers of sampling steps on the Walker-S task, measuring the MAE area metric and the average time per step for decision- makinng see Table 15. We observed that reducing the sampling steps from 10 (as used in our experiments) to 5 did not lead to significant decline in performance, but halved the inference time. Notably, a noticeable performance decline occurred only when reducing the sampling steps to 3. Compared to Diffuser's use of 20 sampling steps and DD's use of 100 sampling steps, have already minimized the inference time to the best extent possible within the diffusion planning approach. In future work, will consider inference time as crucial optimization objective and strive to further reduce it.
H.4 THE EFFECT OF THE ATTRIBUTE-ORIENTEE ENCODER
Table 16: Ablation of the usage of an strribuee-orennte encoder.
Label
Environment
AlignDiff (no enc.)
AlignDiff
Synthetic
Walker
0.544 1 0.081
0.621 1. 0.023
24

---

Published as a conference paper at ICLR 2024
As mentioned in Section 4.3, setting the attribute strength value to 0 carries practical significance, as it represents the weakest manifestation of an attribute. Therefore, directly masking an attribute to indicate that it is not required may lead to confusion between "weakest manifestation" and "not required" by the network. In other words, providing an expected attribute value close to O might make it difficult for the network differentiate whether it should exhibit weak manifestation of the attribute or not require it. However, these scenarios are hypothetical, and we aim to supplement this observation with an ablation experiment demonstrate this phenomenon quantitatively. We replace the attribute encoder of AlignDiff with a simple MLP and apply mask (i.e., using 0 to indicate the absence of required attribute) to the "not needed" attributes (denoted as AlignDiff (no enc.)). As shown in Table 16, the results on Walker-S reveal noticeable performance drop. This suggests that the current masking approach in AlignDiff effective.
H.5 THE EFFECT OF THE NUMBER OF SELECTABLE TOKENS
Table 17: Area metric table of AlignDiff using different numhers of selectable tokens.
Label
Environment
AlignDiff (V=10)
AlignDiff (V=50)
AlignDiff (V=100)
Synthetic
Hopper
0.597 0.009
0.626 0.025
0.628 +: 0.026
In the AlignDiff atrribute-oriented encoder, discretize the strength values of all attributes, with each attribute assigned to one of the V selectable tokens. As expected, reducing the value of V can lead an increase in the quantization error and a decrease in the precision of the preference alignment. To investigate the relationship between the performance of AlignDiff and the number of selectable tokens V, we conducted an additional experiment on the Hopper-S task. We tested the performance of AlignDiff with three different values of V (10/50/100) and the results are shown in Table 17. We observed that when V was reduced from 100 to 50, AlignDiff did not show a significant performance drop. However, a significant performance decline was observed when V was reduced to 10. From Fig. 4, we can observe that only around 10% of the samples in the Hopper- S task achieved an MAE below 0.02 (the quantization eπror with V=50), while approximately 90% of the samples achieved an MAE below 0.1 (the quantization error with V=10). This suggests that can strike balance between parameter size and performance by analyzing the MAE curve to select an appropriate value for V.
I HOW LANGUAGE MODEL USED IN THE PIPELINE
In the AlignDiff pipeline. the Sequence- Bert (Reimers & Gurevych, 2019) is used as an intermediary for transforming "natural language" to "attributes". Specifically, keep instruction corpus, where each element can be represented as a triplet ((emb, attr, dir) where emb represents the Sentence-Ber embedding of a given language instruction (e.g., "Please run faster"), attr represents the attribute that the language instruction intends to modify (c.g., "speed"), and dir represents the direction of attribute change, either "increase" or "decrease."
When a human user provides a language instruction, it is first transformed into an embedding by Sentence-Bert Then, cosine similarity is calculated between the embedding and all emb elements in the instruction set. The instruction with the highest similarity is considered as the user's 's intention, allowing us to determine whether the user wants to increase or decrease a specific attribute. For example, if the user wants to increase the "speed" attribute, we set popeed to (v-ped + 1)/2, and wants to decrease it, we set speeed to eppeed/ /2. The current approach may be relatively simple, but it is still sufficient to capture some human intentions. For instance, Figure 6 page 9 of the paper demonstrates full control solely based on natural language instructions. In the left image, provided instructions ['Please move faster.', 'Keep increasing your speed.', 'You slow down your pace now.'] at 200/400/600 steps, respectively. In future work, we would try to introduce LLM (I (Language Learning Model) to assist in recognizing more complex language instructions.
25

---

Published as a conference paper at ICLR 2024
Table 18: Area metric table of AlignDiff and other two RBA-hased methods.
Label
Environment
RBA
RBA+
AlignDiff
Synthtic
Walker
0.293 +: 0.026
0.445 +: 0.012
0.621 + 0.023
J COMPARISON WITH RBA
Walker-S
1.0
0.8
0.6
0.4
AligaDid
RBA+
RBA
Figure 13: MAE curve of AlignDiff' and other two RBA-based methods.
AlignDiff and RBA are two fundamentally different approaches. AlignDiff is decision model that aligns behaviors via planning, whereas RBA mainly trains a reward model to represent hu- man preferences, without direct decision-making capabilities. Even comparing the reward models, AlignDiff's s attribute strength model has unique properties, such as evaluating variable-lengle tra- jectory attributes and supporting masks for uninterested data, which RBA's per-step reward function lacks. To fully illustrate the differences, we briefly revisit the RBA methodology (focusing on RBA- Global, which most similar AlignDiff's RLHF part).
RBA first trains an atrribute-oonditionedd reward model using the same method as described in Ap- pendix C.3, Eqs. (9) to (11), to support downstream policy learning for alignment. The original RBA paper uses a edecision-making approach that does not require separate training, as noted in their Github repository': As mentioned in the paper, the current implementation optimizes the reward simply by sampling large set of rollouts with the scripts or policies that we used to syn- thesize behavior dataset. Specifically, the authors construct a policy library {π;} from all policies used to synthesize the behavior dataset. For given state s, they search this library to find the ac- tion a = arg (f.e)) TRBA a;) that maximizes the RBA reward model. This approach is impractical, since constructing sufficiently comprehensive policy library is difficult, and action optimization via search can be extremely tomecconummngg
To construct proper RBA baseline for comparison, we desire an approach that can align with human preferences whenever they change without separate training, like AlignDiff. So, as mentioned in Section 5.1, we use offline stribute-conditiones TD learning algorithm (a modified TD3BC) to maximize the RBA reward model. Unlike RBA's search over a limited library, this approach is more practical (no need for a large policy library) and can generalize across diverse policies. Therefore, instead of lacking an RBA comparison, we actually compare against a refined version of RBA.
In this section, to quantitatively illustrate the difference, further reconstruct the exact RBA de- cision process using the SAC policies, which are used to collect the Walker-S dataset, as the policy library, comparing it with AlignDiff and TDL (referred to as RBA+ here to emphasize as an im- proved RBA). The resulting MAE curve and area table are presented in Fig. 13 and Table 18. The experiments confirm our expectations: AlignDiff outperforms RBA+, and RBA+ outperforms RBA due to its greater robustness. We offer an observation on RBA's poor performance: our specialized SAC policies behave erratically outside their specialized motion modes, general RL issue. Since we use random initial states to test stability when behaviors must change, this is very challenging for a seebrary-searc-baaae method RBA.
hhtps://gituhh.cmm// sitrractingwiith-en--ssere
26

---

Published as a conference paper at ICLR 2024
Gender
Age
Education
Experience
57.5%
26.0%
47.9%
34.2%
Hopper-s
8.2%
8.2%
2.7%
32.9%
28.8%
16.4%
39.7%
13.7%
35.6%
24.7%
52.9%
26.5%
47.1%
33.3%
6.9%
Walker-s
7.8%
2.0%
33.3%
9.8%
8.8%
17.6%
45.1%
13.7%
35.3%
Humanold-s
40.9%
15.9%
15.9%
.888%
27.3%
18.2%
4,5%
27.3%
4..5%
31.8%
11.4%
52.3%
25.0%
56.8%
43.2%
52.6%
17.9%
15.4%
23.1%
Hopper-H
34.6%
5.1%
3.8%
53.8%
9.0%
10.3%
42.3%
14.1%
14.1%
30.8%
56.5%
17.6%
Wa Hoer-H
14.1%
21.2%
20.0%
3.5%%
51.8%
8.2%
11.8%
15.3%
38.8%
18.8%
32.9%
61.9%
9.5%
7%
19.0%
Humanoid-H
23.8%
7.1%
9.5%
14.3%
19.0%
9.0%
7.1%
14.3%
21.4%
28.6%
61.9%
47.6%
38.1%
Undergraduate exp 1-3y More than
Fernale Others
Undergraduate
1y
Graduate
24 and above
Figure 8: Distribution of participants involved in the human assessment experiment is presented. Prior to receiving the questionnaire, each participant is requested to provide basic information in- cluding gender, age, education, and AI experience. We present the distribution of these information categories the form of pie charts.
27

---

Published as a conference paper at ICLR 2024
Figure 9: The behaviors generated by AlignDiff' when only adjusting the torso height attribute. From the first row to the last row, they respectively represent attribute strength values [0.9, 0.5, 0.1].
Figure 10: The behaviors generated by AlignDiff' when only adjusting the left-right leg preference attribute. From the first row the last row, they respectively represent attribute strength values of [0.9, 0.5, 0.1].
Figure 11: The behaviors generated by AlignDiff when only adjusting the humanness attribute. From the first
row the last row, they respectively represent attribute strength values of [0.9, 0.5, 0.11.
Figure 12: The behaviors generated by AlignDiff when only adjusting the humanness attribute. From the first
row the last row, they respectively represent attribute strength values of [0.9, 0.5, 0.1].
28


Reviews:
[{"review_id": "sc8F0XakwM", "reviewer": "Reviewer_SGJ2", "summary": "This paper utilizes the diffusion model conditioned on attributes for planning. Inspired by RLHF, this paper first fits a reward model trained to predict preference over trajectories given human preference, followed by using it to predict the attribute strengths given the episode. This is then used to label the unlabelled episodes with the corresponding attributes, which are then used to condition the diffusion model, trained in a classifier-free guidance sense.  \n\nThe paper shows the advantages of the proposed technique over existing baselines for learning the policy including on human preferences, and tests its robustness when the strengths are suddenly changed in between. Lastly, an ablation of label pool size is done to understand its impact on learning the reward model.", "questions": "Refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow in most places, and I like the experiments on the robustness and label efficiency of the reward function.", "weaknesses": "As someone who is not well versed in the empirical reinforcement community, my weak comments would be high level and mostly some of my confusion throughout the paper. \n\n- Can authors provide the inter-annotator correlation when they're labeling the reward model? Annotator alignment is a problem in the RLHF community when it comes to LLMs and even RLHF for diffusion models in image generation (See [1]) therefore it would be good to see some numbers on annotator agreement. \n\n- Can authors provide the accuracy of the reward model after training it based on human preference? Does the Area metric in Table 5 correspond to that? Moreover, what is the accuracy of random guessing? I am assuming it would be 50%? \n\n- Are there any ablations done in case one does not apply masking in the way current AlignDiff is applying? What if one just used 0s in the strength where it is not needed? I think having that result would further showcase the utility of masking in the current way. \n\n- How exactly is BERT used in the pipeline? How is the mapping from BERT representations to strength and mask learned? \n\n- How is the performance affected by the precision of discretization? \n\n- $\\mathcal{B}(k, p)$ is not defined. \n\n- How is the performance of the reward function (that predicts strengths) affected by the length of episodes, when varied during training and inference time (zero-shot say)? \n\n\n[1] Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8MvFp7XPMM", "reviewer": "Reviewer_t4Ww", "summary": "This paper presents a method for aligning agent behaviors with human preferences. Firstly, it introduces multi-perspective human feedback datasets. Secondly, it trains an attribute-conditioned diffusion model, referred to as AlignDiff, to act as a director for preference alignment during the inference phase. AlignDiff utilizes Reinforcement Learning from Human Feedback (RLHF) to quantify human preferences, allowing it to match user behaviors and seamlessly transition between different preferences.", "questions": "1. What is the inference time?\n2. How about the comparison with the state-of-the-art methods, such as RBA.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed diffusion-based framework demonstrates exceptional performance in decision-making scenarios involving complex dynamics.\n2. The method presented in this paper is capable of effectively matching user-customized behaviors and seamlessly transitioning between different preferences.\n3. Additionally, this paper introduces a valuable contribution in the form of a multi-perspective human feedback dataset. This dataset has the potential to facilitate the wider adoption of human preference aligning techniques.\n4. The proposed method leverages a multi-stage diffusion process, which effectively simulates the reinforcement learning (RL) process.", "weaknesses": "1. Both the proposed method and RBA utilize RLHF for aligning agent behaviors with human preferences. The novelty is unclear.\n2. The diffusion model usually achieves the best result in the final step. How does the diffusion model guarantee the best human preference at each step? Does the proposed method obtain a plan with T diffusion steps? If so, how about the inference time?\n3. The proposed method only did some ablation studies and has not compared with the state-of-the-art methods, such as RBA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYkNoRE1yG", "reviewer": "Reviewer_WtmM", "summary": "This paper focuses on the issue of consistency between agent behavior and human preferences in RLHF, and proposes an alignment method based on the diffusion model. The authors construct a multi-perspective human feedback dataset and train an attribute model, which is then used to relabel the dataset. A diffusion model is utilized as a planner and it's trained on the preference-aligned dataset. In this way, the authors achieve preference aligning between different human. Both quantitative and qualitative experimental results demonstrate the effectiveness of this method.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors revisit the impact of inherent human annotator preferences on reinforcement learning training, which is illuminating.\n\n2. The proposed method is innovative and achieves relatively good results, which is demonstrated by the experiments.\n\n3. The visualizations and supplementary material provided in the website support the paper and make it easier for readers to understand.\n\n4. The paper is clearly written and well organized.", "weaknesses": "1. The paper does not include ablation experiments on attribute model training, so the actual effect of attribute alignment is not easy to measure.\n\n2. The explanation of some details of the method is not clear enough. For example, the meaning of equation (5) and \"inpainting manner\" needs further clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fvZoXJGYpZ", "reviewer": "Reviewer_pwJk", "summary": "This paper proposes a diffusion model with RLHF to train an RL agent that follows human preferences and instructions. A attribute strength model is trained on a newly built human feedback datasets, which is leveraged to annotate the behavior dataset. Extensive experiments are conducted on the proposed method in terms of the preference matching, switching, and covering. All achieves superior performance compared to baselines.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using RLHF to align human preference is reasonable and insightful.  \n2. The experiments are extensive and verify the effectiveness of the proposed method. \n3. The design of the attribute strength and the corresponding datasets could be helpful to many relative future works.", "weaknesses": "Could the authors offer more clarifications and analysis to demonstrate the extent to which the proposed attribute strength can encompass a broad spectrum of human preferences and instructions? \nHow accurate is the language model to find the correct attribute strength that match user's intent?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)