Text-Aware Diffusion for Policy Learning
Authors: Calvin Luo
Mandy He
Zilai Zeng
Chen Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
reinforcement learning
Primary area: reinforcement_learning
TL;DR: Pre-trained, frozen diffusion models generate dense zero-shot reward signals for text-conditioned policy learning.

Abstract: Training an agent to achieve particular goals or perform desired behaviors is often accomplished through reinforcement learning, especially in the absence of expert demonstrations.  However, supporting novel goals or behaviors through reinforcement learning requires the ad-hoc design of appropriate reward functions, which quickly becomes intractable. To address this challenge, we propose Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a pretrained, frozen text-conditioned diffusion model to compute dense zero-shot reward signals for text-aligned policy learning.  We hypothesize that large-scale pretrained generative models encode rich priors that can supervise a policy to behave not only in a text-aligned manner, but also in alignment with a notion of naturalness summarized from internet-scale training data.  In our experiments, we demonstrate that TADPoLe is able to learn policies for novel goal-achievement and continuous locomotion behaviors specified by natural language, in both Humanoid and Dog environments. The behaviors are learned zero-shot without ground-truth rewards or expert demonstrations, and are qualitatively more natural according to human evaluation. We further show that TADPoLe performs competitively when applied to robotic manipulation tasks in the Meta-World environment, without having access to any in-domain demonstrations.

Text-Aware Diffusion for Policy Learning
Calvin Luo
Mandy He"
Zilai Zeng" Chen Sun
Brown University
{calvin luo, maliiinyllmmmdddyy...... mel.iilai. zeng, cwensuu)abrrom.ecoom
Abstract
Training an agent to achieve particular goals or perform desired behaviors of- ten accomplished through reinforcement learning, especially in the absence of expert demonstrations. However, supporting novel goals or behaviors through reinforcement learning requires the ad-hoc design of appropriate reward fune- tions, which quickly becomes intractable. To address this challenge, we propose Text-Aware Diffusion for Policy Learning (TADPoLe), which uses pretrained, frozen text-conditioned diffusion model to compute dense zero-shot reward sig- nals for text-aligned policy learning. We hypothesize that large-scale pretrained generative models encode rich priors that can supervise a policy to behave not only in text-aligned manner, but also in alignment with notion of naturalness summarized from internet-scale training data. In our experiments, we demonstrate that TADPoLe able learn policies for novel goal-achievement and continuous locomotion behaviors specified by natural language, in both Humanoid and Dog environments. The behaviors are learned zero-shot without ground-trut rewards or expert demonstrations, and are qualitatively more natural according to human evaluation. We further show that TADPoLe performs competitively when applied to robotic manipulation tasks in the Meta-World environment, without having access to any in-domain demonstrations.
Introduction
Can train reinforcement learning agents that drive humanoids in virtual environment 33 to stably stand? How about standing with hands on hips, kneeling, or doing splits? While state-of- the-art algorithms have shown success the former scenario (e.g. [14]), the latter (illustrated in Figure [ remains challenging due to the need for carefully (and often manually) crafted reward functions to specify the desired behaviors. The dependence ad-hoc designed reward functions renders inscalable the learning of ever-increasing amounts of novel behaviors, which are required in applications ranging from character animation
robotic manipulation 42
Our work looks towards natural language as a powerful interface through which humans can flexibly specify desired goals or behaviors of interest. We therefore investigate how construct zero-shot text-conditioned reward signal, replacing the need for ad-hoc designs, through which text-aligned policies can be learned. We present Text-Aware Diffusion for Policy Learning (TADPoLe), which utilizes a large-scale pretrained, frozen text-conditioned diffusion model to generate a dense reward signal for policy learning. We hypothesize that generative diffusion models, which pretrained on internet-scale datasets to produce text-aligned, natural-looking images 32 29 and videos B 10 [5],
be utilized to automatically craft multimodal reward signal that encourages agent behave both faithfully with respect to text conditioning and naturally with respect to human perception. Our method is novel in its reward computation, well its utilization of domain-agnosti generative model, rather than one trained from environment-specffic or task-specific video demonstrations, as used in prior work 9 24
20
19
Equual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

person
person doing
dog standing"
dog chasing robot the black safe"
"a pushing soccer ball net"
position'
splits"
Figure I: Our proposed Text-Aware Diffusion for Policy Learning (TADPoLe) framework leverages frozen, pretrained text-aware diffusion models to automatically craft dense text-conditioned rewards for policy learning. Here we visualize TADPoLe achieving diverse text-conditioned goals in the Humanoid, Dog, and Meta- World environments.
TADPoLe is motivated by the insight that reinforcement learning policy can be viewed as agent- centric implicit video representation when operating within an environment with visual rendering capabilities. As illustrated in Figure[]]lett of actions following a policy тe. and the conversion of the action sequence into video subsequences through the environment's ss rendering function. policy can therefore be seen iteratively generating frames conditioned on the actions it selects; on the other hand, a text-to-imaae diffusion model can also be seen as generating static image frames, but conditioned on natural language instead. A connection can then be established between policy and diffusion model, where the frame or video segment "generated" by the policy can be critiqued by evaluating how likely a text-conditioned diffusion model would generate the same visuals, thus providing dense text-aligned reward signals to guide policy learning (Figureßrigh) right). Our work is inspired by DreamFusion 25 where text- conditioned 3D model is learned through rendered views, and where volumetric raytracing ensures spatial consistency. Here, we seek learn a rext-conditioned policy through rendered frames or subsequences, where the environment naturally ensures temporal continuity and consistency with respect to notion of physics instantiated by the environment.
agent's video generation process involves the selection
Concretely, TADPoLe achieves rext-conditioned policy learning by using generative diffusion model in discriminative manner. computes the reward signal weighted combination of reward terms, which aim to measure the alignment between the rendered observation and text conditioning, and the naturalness of the agent's is behaviors, respectively. In this way, we can in effect "distill" the natural visual and motion priors as well as vision-text alignment understanding captured within the diffusion model into a policy. By default, TADPoLe uses text-to- to-image diffusion model [3]] to densely compute reward signal solely from the immediate subsequent frame after each action. We then generalize the framework to Video-TADPPLe,, which uses a text-to-video diffusion model [I2] to calculate dense rewards as a function of sliding context window of past as well as future frames achieved. The agent is thus trained to select actions such that arbitrary consecutive subsequences of frames are well well-aligned with text as well as natural video (e.g. motion) priors.
We highlight TADPoLe as the first approach to leverage domain -anaiiii-nnosiicc visual generative models for policy learning. Through quantitative and human evaluations on Humanoid [39] Dog [3 and Meta-World 42 environments, we demonstrate that TADPoLe enables the learning of novel, zero- shot policies that are flexibly and accurately conditioned on natural language inputs, across multiple robot configurations and environments, for both goal-achievement and continuous locomotion tasks.
TADPoLe therefore provides two main benefits simultaneously: performant approach towards zero-shot policy learning, where complex reward functions no longer need to be manually specified per task, and promising path towards distilling priors summarized from large-scale pretraining into policies, ultimately resulting in the learning of more cituurally-alignee behavior within arbitrary environments. Visualizations and code are provided at diiiiuiion supervision. wtt.riiiiiiiiiuubiooooooooo github.iotadpole/
Related Work
Diffusion models B3 35 16 36
have recently demonstrated amazing generative modeling capa-
bilities, particularly in the domain of text-conditioned image generation [9 32 31]. Notably, guidance 36 17 has been shown to be a critical component in producing visual outputs aligned
with textual data, enabling the generation of images that accurately match desired text caption, especially when the models are scaled to utilize large foundation models such as CL.IP (2 or T5 (2).
and trained on massive image-text datasets [33] Our work is inspired by DreamFusion [25). where

---

"[tert prompp))
Diffusion Model
render env (f, To)
V
Figure 2: policy He that interacts with an environment can be treated an agent-centric implicir video representation, where the arrow of time is actuated by the agent's 's actions and the pixels are rendered by the environment. The rendered behaviors can then be evaluated by text-aware diffusion model to produce dense rewards, thereby providing text-conditioned update signals to the policy,
pretrained, frozen text-to-image diffusion model is able to supervise the learning of zero-shot 3D models conditioned on text. We propose leveraging pretrained diffusion models to supervise the learning of flexible, lext-conditioned policies in a zero-shot manner over the time dimension.
There are numerous works that investigate how interactive agents can learn to perform behaviors specified by textual inputs. SayCan []] grounds the knowledge of complex, high-level behaviors within an LLM to the context of robot through pretrained behaviors. This then enables an LLM instruct and guide a robot, through combining low-level behaviors, to perform complex temporally extended behaviors. LangLiP proposes method for incorporating free-form natural language conditioning into imitation learning by first associating goal images with text captions and training policy follow either language or image goals. but only conditioning on natural language during test time inference 4 The Text-Conditioned Decision Transformer learns a causal transformer to autoregressively produce actions conditioned on both text tokens as well as state and action tokens [26]. Similarly, Hiveformer proposes unified multimodal Transformer model for robotic manipulation that conditions on natural language instructions, camera views, as well as past actions and observations [II]. However, LangLfP, Text- coxt-oondtio Decision Transformer, and Hiveformer, all require training on datasets of trajectories that have been labelled with natural language. In contrast, TADPoLe enables the learning of text-conditioned policies irrespective of visual environment, and without requiring any pretraining dataset of demonstrations or labeling.
Similar to our work, UniPi [6] treats the sequential -erision-making problem as text-conditioned video generation problem. The authors propose training video diffusion model to produce future visual plan for the agent: the subsequent frames are then converted to actions by means of an inverse dynamics model. VLP 7 utilizes text-to-vides generative models for planning and goal generation for an agent. Both methods require the video generative models to be trained ad-hoc on the target environments, whereas we directly use frozen general-purpont generative models. Mahmoudieh et al. [21] propose framework that uses CLIP generate a reward signal from text description of goal state and raw pixel observations from the environment, which is then used to learn a task policy. In VLM-RM 30), the authors also explore utilizing CLIP as the reward model for training humanoids to accomplish complex goal-reaching tasks. In our work, we investigate locomotion tasks on top of goal-achievement, and explore how using diffusion models produce a reward signal can outperform CLiP-based approaches. Although not conditioned on text, VIPER S also aims to harness recent advancements in generative modeling by employing video prediction model's likelihoods reward signal. However, VIPER does not enable the learning of policies conditioned on text, and requires in-domain expert videos for ad-hoc training the video model. Finally, Diffusion Reward [I8] also extracts a reward from a diffusion model to train policies; however, it requires training an ad-hoc video model expert trajectories, the collection of which cannot always be assumed to be trivial, and does not enable text-conditioned policy learning.
3 Method
We propose Text-Aware Diffusion for Policy Learning (TADPoLe) to learn text-aligned policies by leveraging frozen, pretrained text-conditioned diffusion models. An overview of the framework can be found in Figure

---

"[tert prompt)
render
O
Environment
add
T aa
St+1
Eo
Diffusion Model
0.
Tt
π
EΦ
predict
Figure 3: An illustration of the TADPoLe pipeline, which computes text-conditioned rewards for policy learning through a pretrained, frozen diffusion model. At each timestep, the subsequent frame rendered through the environment is corrupted with a sampled Gaussian source noise vector eg- The pretrained eext-conditioned diffusion model then predicts the source noise that was added. The reward designed to be large when the selected action produces frames well-aligned with the text prompt.
3.1 Text-Aware Diffusion for Policy Learning
We first describe how TADPoLe produces text-conditioned rewards from image observations. At each timestep t, reward r; is computed as score between rendered subsequent image 0++1 and the provided text caption describing the behavior of interest, denoted by y. using frozen, pretrained text- to-image diffusion model. We begin by corrupting the rendered image 0t+1 with a sampled Gaussian source noise vector Eu (N(e; 0, I) to produce noisy observation Ot+1, and use the diffusion model to make an unconditional prediction (ōr++) tnoise) as well as a conditional prediction (O Unoise: 3). Here (-) is neural network that predicts the source noise given 0 the level of noise corruption tpaise, and optionally the text prompt y: we overload notation to have ép represent the source noise prediction Figure 3 We then compute the mean squared error (MSE) between the
two predictions as a reward signal ri to be maximized:
align
T
aligo
Umoise: v) - (o.+i troise) )lo
B.3 we empirically observe that r allan plays a crucial role on the success of T. We hypothesize that for an appropriately-sseectee noise corruption level tnoise. this term measures the alignment between the environmental observation and the text prompt. Intuitively, for unconditional prediction (ō:+1 Enoise). the model is incentivized only to bring the noisy input to any arbitrary cleaner image, and makes minimal edits by moving towards the closest clean mode in data space. On the other hand, if the model recognizes visual features in the noisy image aligned with the text prompt, conditional prediction Es (ō Enoise y) is incentivized to do "extra work" and bring it closer to the specific mode described by the text conditioning. We thus expect the MSE to be larger for well-aligned text conditioning. For an unaligned text prompt, the model may have more difficulty in recognizing relevant visual features in the corrupted image, and therefore generally has lower computed rallinn signal. Therefore maximizing rž aliggn is a tractable proxy for maximizing the alignment between the rendered observation 0,+1 and the provided text prompt
As investigated in Appendix
We also wish encourage behaviors that are natural to human perception (e.g. humanoid should walk similar to how a typical pedestrian would walk). We approximate the naturalness of a behavior by how accurately the diffusion model is able predict the exact source noise vector that was applied. Intuitively, if it voluntarily predicts the exact noise vector with informative text conditioning, thereby perfectly reconstructing the query image, then the diffusion model believes the original rendered frame is reasonably natural (according to the priors captured by the diffusion model). We would therefore like to minimize (ā mnoise y) Eu 2|-- We would also like this term to be comparatively closer to the source noise vector than the unconditional prediction is, further reaffirming the benefit of the text conditioning. We therefore seek to also maximize a comparative reconstruction term as below:
p (1E6(ō1415 Enoise) -Eol23 [1ēp(ō1+1; tnoise y) Eoll?

---

Algorithm Text-Aware Diffusion for Policy Learning (TADPoLe)
1: prompt (atpooleecciioophaaess_
2: To initialize(f)
3: D D+- {
4: while not converged:
5:
((s))
6: for r (nneeeppiode__enggil
7:
To 4 8t)
8:
S141 P( S;, a,)
9:
0,
10:
0+41 P(0 8141)
11:
0,+1 f- noisiiy(Or+)) 60, tnoise)
12:
7 = tadpole_revard Eo, prompt)
13:
T
(z a, 8(+1)
14:
- D T
15:
loss policy_loss(]
16:
grads = gradient (loss, 0)
17:
opt. app..aply_graieents (grads, ()
Ultimately, we compose these terms into final reward signal v, exposed to the policy during training. We scale each of the individual terms with tunable hyperparameter constants, and apply symlog B3 transformation operation:
Tr symlog(w) Fr
align + symlog(w) * rF²))
The choice using symlog as a reward normalization technique is thoroughly studied in Section 4.5 TADPoLe is agnostic to the specific choices of policy network architecture and optimization objectives. pseudocode of the method is provided in Algorithm [T. is worth emphasizing that Unoise and subscript-less refer to different notions of time; indexes the timestep of the agent in the environment, whereas naose determines the level of noise to corrupt the raw observed image.
3.2 TADPoLe with Text-to-Video Diffusion Models
Conceptually, there exist fundamental limitations to using text-to-image model to provide a reward signal. As each image is evaluated statically and independently, we are unable expect the text- to-image diffusion model to be able to accurately understand and supervise an agent in learning notions of speed, or in some cases, direction, as such concepts require evaluating multiple consecutive timesteps deduce. We therefore propose signal is calculated over sliding windows consecutive frames through a pretrained text-to-video diffusion model. We extend and generalize the reward formulation from TADPoL thusly.
where dense text-conditioned reward
We can compute reward terms for arbitrary start index i and end index j inclusive, for j. by considering the sequence of subsequently rendered frames 0/++1:j+1]- We once again utilize source noise vector en ~ (c; 0, I, 1 to produce noisy observation 0[++s++1]- Then, we can compute a batch of alignment reward terms through one inference step of the text-to-video diffusion model
align [e
& (+++1:+11] Enoise y) Exp (0 tnoise
and batch of reconstruction reward terms
(0 tooke) - =ol 2 noose: v) - Eoll
For desired context window of size n, we then calculate the reward each timestep & utilizing each context window that involves achieved observation 0,+1:
r; symlog (1 * "f--it ( 11) + symlog (12 * t 11)
align
Intuitively, we seek to calculate an overall reward for an action based off how well the resulting rendered frame aligns with text-conditioning at the beginning of motion sequence, the end of one, and arbitrarily inbetween. For window size 1, this recreates TADPoLe behavior, but using textooovvieoo model; for > 1, we make the computation tractable through dynamic programming.
5

---

4 Experiments
We now demonstrate the effectiveness of TADPoLe on goal achievement, continuous locomotion, and robotic manipulation tasks. All results are achieved without access to in-domain demonstrations.
4.1 Experimental Setup and Evaluation
Benchmarks: We present our main results using the Dog and Humanoid environments from the DeepMind Control Suite [39], and robotic manipulation tasks from WetadWorld [42]. Dog and Humanoid are known to be challenging due to their large action space, complex transition dynamics, and lack of task- task-specific priors (such termination conditions). We update the environments by modifying the terrain rendered by MuloCo (38) to have green grass and blue sky. We also limit the number of environment timesteps to be 300, which is sufficient demonstrate successful learning of a behavior, rather than the default 1000. The agent's 's initialized joint configurations are also fixed, as focus learning rext-conditioned capabilities rather than robustness initialization conditions. Worl- was initially designed for multi-task and meta-reinforcement learning, and was later adopted evaluate language-conditioees imitation learning algorithms [3 22
37 We select a suite of tasks which are balanced for diversity and complexity, and pair each task with a text prompt (see tasks and their corresponding prompts in Appendix C). Following prior design [8] for Kora-World, we also add a sparse success signal to the dense text-conditioned reward signals.
Implementation: We use TD-MPC [4] the reinforcement learning algorithm for all tasks. It is the first documented model to solve Dog tasks when ground truth rewards are available for walking. We fix the hyperparameters the default ones recommended by the TD-MPC authors (see Table in Appendix) for all experiments unless otherwise mentioned. We train Humanoid and Dog agents for 2M steps, and Meta-World agents for 700K steps. For Meta-World experiments, we scale the sparse success signal by 2. Visualizations and quantitative evaluations are reported using the last checkpoint achieved the end of training. We use StableDiffussion 2.1 (3I the text-to-image diffusion model (~1.3 parameters), and AnimatedDiff [2] v2 (~1.B parameters) as the text-to-videe diffusion model. AnimatedDiff is implemented on top of StableDiffusion 1.5. We fix the reward weights w1= 2000 and W2 = 200 based on Humanoid standing and walking performance, and study their impact in Appendix B.3 Selection of noise level is discussed in Appendix A All experiments are performed on single NVIDIA V100 GPU.
Baselines: We compare TADPoLe against other text-to-rewaan approaches, including VLM-RM (30 LIV 120]. and Text2Reward [41], on top of the same underlying TD-MPC architecture, hyperpa- rameters, and optimization scheme for fair comparison. For LIV, use their provided based checkpoint finetuned on robotic demonstration videos. For VLM-RM, we utilize the ViT-bigG-14 CLIP checkpoint (~1.3B parameters), reported as the best performing in their work. We follow a prompt template provided in the Text2Reward paper to generate reward functions for the Dog and Humanoid agent, interfaced through vanilla ChatGPT using GPT-3.5. Whereas VLM-RM and LIV provide a multimodal reward signal, and are more directly comparable to TADPoLe, it is of note that Texr2 generates a rext-conditioned reward function purely as the output of a pretrained language model. However, Texr2Reward does have access to underlying sensor data such as speed and direction in real-time, whereas the visual interface approaches, including TADPoLe, do not.
We benchmark all text-conditioned methods with a corresponding standard- ized prompt for fair comparison, and report both quantitative as well qualitative comparisons. We use cumulative ground-truth rewards as quantitative evaluation metrics for Dog and Humanoid when it is available. We note this is a naturally wnfavorable comparison for methods that provide a text-conditioned reward signal purely through a visual interface, as the reward the agent receives has no access to the underlying sensors (such as ones that measure speed and energy usage) that the ground-truth reward function uses to evaluate performance. For example, the ground-truit reward function may have an arbitrary threshold on a speed sensor that needs be hit constitute successful "walking", and a separate threshold for "running"; however the detailed characteristics of and even existence of such sensor, as well as any thresholds surrounding it, are hidden for policies supervised only through vision and language feedback. Nonetheless, it offers a standardized, numerical compari- son across all methods. For Meta- Worl-, we report the "success rate" evaluation metric, computed as the proportion of evaluation rollouts in which the agent successfully completes the given task.
Evaluation Protocols:

---

Table 1: Results for goal-achievement experiments on DeepMind Control Suite Dog and Humanoid environments. For rows with an associated ground-truth reward function, numerical results are listed; for performant approaches, text-conditioned behavior learning, checkmarks denote if the resulting policy is aligned with the provided text prompt according to human evaluation.
report mean and standard deviation across 5 seeds. For novel zero-shot
Environment
Prompt
VLM-RM
LIV
Text2Reward
TADPoLe (Ours)
Ground-Truth
Humanoid
person standing"
247.05 16.90)
11.27
10.50
254.43 (1 8.76)
287.68 (1 4.64)
Humanoid
"a person in lotus position" "a person doing splits" "a person knecling" "a dog standing"
X
X
Humanoid
Humanoid Dog
X
X
x
Prompt
TADPoLe Naturalness (1)
"a dog standing"
87.5%
"a person in lotus position"
62.5%
"a person doing splits"
62.5%
person kneeling"
70.8%
"a person walking"
84.0%
standing"
standing hands
above heedd"
"a dog walking"
76.0%
Figure 4: TADPoLe demonstrates sensitivity to subtle variations to the input prompt, learning to stand in different positions with only slight modifications to the text conditioning.
Table Qualitative study: percentages denote user preference for the naturalness of the result- ing motion produced by TADPoLe over VLM- RM (goal-achievement) and Tideo-TADPoL. over ViCLIP-RM (continuous locomotion).
A main benefit of utilizing a reward signal conditioned flexibly on text is the ability to learn policies with behaviors beyond those defined by existing ground-truth reward functions. As these have no corresponding ground-truth reward functions, quantitative comparison across different text- conditioned methods is challenging: we therefore appeal to qualitative user study. We perform paid study through the Prolific platform, with total of 25 anonymous random participants without prior training to estimate a general response from the human population. For a video demonstration from each trained model, selected the last timestep of policy training without seerkiig, each participant is asked if it sufficiently aligns with the text prompt it was conditioned on. These results depicted in tables checkmarks (V) and x-marks (X), where checkmark denotes if majority of participants believe it text-aligned. In A7 we provide the fine-grained user study results on what percentage of the users believe the video achieved by the policy is appropriately text-aligned. We then proceed with a user study regarding naturalness. Given a video produced by VLM-RM and TADPoLe,
given choice to which they believed to be the more natural motion or pose. This seeks to approximate how naturally the resulting Humanoid and Dog policies behave, according human belief over how people and dogs naturally move in the real world.
4.2 Goal Achievement
For text-conditioned goal-achievement, the objective is to learn policy to consistently achieve particular pose described by text prompt; as the emphasis for every frame to match a fixed goal pose rather than performing continuous motion, it is natural to apply TADPole with text-to-image diffusion models. We set the noise level tuoise ~ (4000, 500), with intuition provided in Appendix A
In the Humanoid environment, there is ground-truth reward function that measures standing performance, as a function of the straightness of the agent's spine. We therefore compare all text- conditioned methods using the provided reward function as quantitative metric, with standard prompt of "a person standing"; these results are shown in the first row of Table TADPoLe and VLM-RM achieve competitive quantitative performance with an agent trained on the ground-truth reward function. The following rows show that according to user study, TADPoLe consistently achieves text-aligned behaviors beyond making the Humanoid stand, whereas other approaches often fail. Table[[shows shows that users consistently found TADPoLe to produce more natural-looking motions and poses when compared head-to-hear thad with VLM-RM.

---

Table 3: Results for continuous locomotion experiments. For with associated ground-truth reward function, numerical results are listed; for performant approaches, we report mean and standard deviation across 5 seeds. Evaluation for text-alignment is also reported. -ideo-TADPoLe greatly outperforms ViCLIP-RM on both Humanoid and Dog.
Kavironment
Prompp
LIV
Text2Reward
Video-TADPoLe (Ours)
VICLIP-RM
Ground
Humanoid
"a person walking"
0.65
3.35
145.60 (1 48.20)
25.51 (L 40.45)
275.06 (1 9.21)
Dog
"a dog walking"
16.86
63.15
60.20 (# 8.82)
14.67 (# 9.84)
280.07 (# 3.07)
Humanoid
"a person walking"
x
Dog
"a dog walking"
x
x
x
We then investigate whether or not TADPoLe is sensitive to subtle variations of the input prompt. We change the text prompt from "a person standing" to "a person standing with hands above head" and "a person standing with hands on hips". In Figure we visually verify that the resulting Humanoid policy can indeed learn distinct behaviors that respect the different hand placement specifications. We take this as evidence that TADPoLe is capable of respecting fine- fine-grained details and subtleties of the input prompts when learning text-conditioned policies.
4.3 Continuous Locomotion
We further explore the ability of TADPoLe to learn continuous locomotion behaviors conditioned on natural language specifications. Such tasks often difficult to learn purely from static external description, as there is no canonical pose or goal frame that if reached, would denote successful achievement of the task. This is challenging for approaches that statically select a canonical goal- frame to achieve, such as CLIP or LIV, and we propose Taloo,, which leverages large-scale pretrained text-to-video generative models, as a promising direction forward.
We utilize a noise level tnoise ~ U500, 600) in our continuous locomotion experiments. We perform scarch over context windows of size {1, 2, 4, 4,8), and report the best configuration per task. We observe that when the context window is too high (c.g. or higher), the agent has consistently lower performance, and that although the agent learns coherent motion and repeats it, the pose is less text-aligned. For fair comparison against a text-video alignment model trained in a contrastive manner, extend VLM-RM to ViCLIP-RA where ViCLIP- checkpoint 40 finetuned from ViT-L-14 CLIP is used to compute dense, text-conditioned rewards. At each timestep f, we compute dense rewards as cosine similarity between the encoded representations of video observation up to 1++ 1 and the text prompt. We ask VICLIP to encode 8 video frames at a time, which is adopted by its authors for zero-shot experiments.
For the Humanoid task, we find that Video-TADPoLe achieves the best results amongst methods trained purely from visual and/or language feedback in Table On the other hand, ViCLIP-RM, indeed learns to take steps, but does so sideways while maintaining an unnaturally lopsided pose. Meanwhile, LIV and Text2Reward fail to learn meaningful behaviors.
For the Dog task, we notice that the policy learned via ViCLIP-RM collapses; it learns to strike a particular pose and maintain it for perpetuity. Texr2Reward, which does not have access any visual information, but does have access to ground-truth state information for the Dog including speed, direction, and joint positions, achieves reward of 63.15. Ultimately, Video-TADPol.. le achieves comparable result using a context window of 4, while also distinguishing itself as the most natural- looking policy in qualitative terms as the Dog agent appears to perform step-taking motions, rather than remain stationary. Table 2 further showcases a higher preference for the naturalness of the learned policies for continuous Tocomotion achieved by Vidco-TADPoLe compared to ViCLIP-RM, in both Dog and Humanoid environments.
In Figure 5 we visualize episode return curves achieved by Video-TADPoLe for Humanoid agent with the prompt "a person walking". We visualize how during training, the computed Video-TADPoLe reward shares positive corrllttin with the reward computed by ground-truth function for walking over all episodes, lending confidence to as a coherent, well-defined reward function. We also visualize the ground-trute truth evaluation curve. Figure A4 offers another example for "a dog walking".
8

---

Video-TADPoLe (Train)
Ground-Truth (Train)
Ground-Truth (Evaluation)
Episode Return (Humanoia Walk)
200
200
150
150
220
200
100
50
160
0
0.0
0.5
1.5
0.0
0.5
1.5
2
0.0
0.5
1.5
Environment Steps (1e6)
Environment Steps (1e6)
Environment Steps (1e6)
Figure 5: Episode return curves for a Humanoid agent trained with Video- Paloo,, using the prompt "a person walking". We observe that the Video-TADPoLe reward signal (left) positively correlated with the agent's performance as measured with ground-truth reward during training (middle) and evaluation (right). Shaded regions denote the standard deviation across five random seeds.
Table 4: Average success rate for robotic manipulation tasks in Meta-World 42 rollouts. We compare between TADPoLe and VLM-RM, both approaches that do not utilize in- domain data demonstrations, and find TADPoLe significantly outperforms VLM-RM. We report
over 30 evaluation
mean performance and standard deviation across 10 seeds for each task.
Success Rate (%)
Door Open
Door Close
Drawer Open
Drawer Close
Window Open
Window Close
VLM-RM
0
79.7 (2: 39.8) 100.0 (+
10.0 30.0)
100.0 0)
(1:29.0)
TADPoLe
40.0 (+ 49.0)
45.3 46.6)
100.0 (# 0)
74.0 (= 37.9)
30.0 45.8)
Success Rate (%)
Coffee Push
Button Press
Soccer
Lever Pull
Average
VLM-RM
4.0 (+ 12.0)
30.0 (= 45.8) 73.0 (= 38.5)
5.3 (# 11.6)
11.3 (= 18.3)
25.0
TADPoLe
18.6 (#: 20.5)
25.0 (#: 15.4)
0 (2 0)
50.6
4.4
Robotic Manipulation
We further investigate how well TADPoLe can be applied learn robotic manipulation tasks through dense text-conditioned feedback. We do so by replacing the manually-designed ground-truth dense reward for each Meta-World task with TADPoLe's text-conditioned reward. Since TADPoLe aims to leverage domain-agnostie diffusion models for policy learning, we focus our evaluation to compare with baseline methods that also do not utilize in-domain (expert) demonstrations for the robotic manipulation tasks. We note that most of the prior methods which report performance on Meta- World rely on (often expert-producees video demonstrations from similar domain or the target environment directly for representation learning 23 reward learning [18], or both (20). They are thus not directly comparable TADPoLe.
We perform thorough comparisons between TADPoLe and VIM-RM by evaluating them on a diverse set of selected Meta- World tasks. Both models reuse the setup in Section 4.2 without Meta-World modification, with training performed for 700k steps. In Table 4 we report the final success rate for each manipulation task averaged over 30 evaluation rollouts. We highlight that TADPoLe achieves high success rates across a variety of tasks, and significantly exceeds VLM-RM in terms of average overall performance. We take this as positive signal that TADPoLe can meaningfully provide dense text-conditioned rewards that replace dense ground- -trtt hand hand-designed feedback. We also highlight how TADPoLe able to successfully supervise the learning of policies within the wynthetic-lookien visual environment of Meta- World without finetuning the pretrained text-to-image diffusion model, despite the visual attributes (such as the appearance of the robotic arm, or the quality of the renderings) being quite dissimilar from the style of images StableDiffusion was trained on.
4.5 Normalization Study
Of interest what reward normalization technique is most performant for adjusting the raw computed alignment and reconstruction terms into final reward used for policy learning. We investigate

---

Table 5: Quantitative results for TADPoLe and Video-TADPoLe with and without the symlog normalization operation, averaged over seeds. Hyperparameters such as weights and noise level were kept the same across all experiments. We discover that not only does using the symlog transformation enable the highest empirical rewards, it also facilitates the reuse of hyperparameters across tasks, environments, and other diffusion models.
Environment
Method
Prompt
SymLog
Direct Scaling
SymExp
Min-Max
Standardization
Humanoid
TADPoLe
"a person standing"
267.23
239.74
241.49
256.81
236.59
Humanoid
person walking"
226.29
4.58 35.30
3.68
61.31
134.66
Dog
Mido-TADPoLe
"a dog walking"
81.22
9.46
6.15
5.05
variety of normalization strategies in a quantitative manner in Table reusing parameters w1 = 2000
and 102 = 200 across experiments, the selection of which is detailed in Section [B. and Table
Apart from the symlog transformation, we also compare against using no additional normalization (denoted as "Direct Scaling"), ") and using symexp. We also compare against empirical normalization techniques. This includes min-max rescaling, where empirically estimated minimum value is subtracted from the achieved reward, which is then divided by an empirically calculated min-max range, and rescaled to [.. This also includes standardization, which subtracts an empirically estimated mean from the achieved reward and then divides it by an empirically estimated standard deviation. We apply these techniques across Humanoid and Dog environments, for both TADPoLe and Video-TADPol that achieves the best empirical results across robotic configurations, visual environments, and desired tasks, while reusing the same hyperparameter settings. We hypothesize that it helps to normalize the raw computed reward signals across diffusion models and environments to be roughly on the same scale. Indeed, we showcase how removing the symlog transformation, as well as using other normalization techniques, reduces consistent policy learning performance across environments and diffusion models with the same fixed hyperparameters. We further note that it does not require empirically estimated values, unlike min-max max and standardization.
We discover that the symlog operation is the reward normalization strategy
5 Conclusion and Future Work
We present Text-Aware Diffusion for Policy Learning (TADPoLe), a framework that optimizes a policy according provided natural language prompt through a pretrained text-conditioned diffusion model. TADPoLe enables novel behaviors to be learned in a zero-shot manner purely from text conditioning, and also offers a promising angle to train policies to behave accordance with natural priors summarized from large-scale pretraining. TADPoLe can be applied across visual environments and different robotic states without modification, and we experimentally demonstrate that TADPoLe is able to learn novel goae-nchievement as well as continuous locomotion behaviors conditioned only on text, across Humanoid, Dog, and Meta- World environments.
Limitations: An observed limitation of TADPoLe is that it is difficult to explicitly control the weight each individual word of an input text prompt has on the reward provided to the agent. For certain prompts, TADPoLe could potentially cause the agent to remain stationary since it may focus on alignment with the noun in the phrase rather than details of the goal. How to provide fine-grained control over the text-conditioning is an interesting direction to explore in future work. Further interesting future work includes utilizing multiple camera views simultaneously to compute the dense reward, as environments generally allow flexible rendering from arbitrary angles. Another observed limitation is that TADPoLe depends on a highly stochastic operation, namely, repeatedly resampling Gaussian source noise vector at each timestep. The behavior of the resulting policy, after training for many iterations, can therefore vary for the same input text prompt, and potentially cause high variance in both visual and quantitative performance. How to control the stability of convergence to consistent policy across repeated runs is an interesting future direction for exploration.
Acknowledgements. We would like to thank Amil Merchant, Daniel Ritchie, David Bau, Ben Poole, Ting Chen, Nate Gillman, and Yilun Du for helpful discussions and feedback. This work is supported by Samsung Advanced Institute of Technology, NASA, and a Richard B. Salomon Faculty Research Award for Chen Sun. Our research was conducted using computational resources at the Center for Computation and Visualization at Brown University.
10

---

References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint (rXiv:2204.0169) 2022.
[2] Mazen Al Borno, Martin Lasa, and Aaron Hertzmann. Trajectory optimization for full-body movements with complex contacts. IEEE transactions on visualization and computer graphics, 1905)11405-1414, 2012.
[3] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprini urXiv:2401. 12945, 2024.
[4] Lynch Corey and Sermanet Pierre. Language conditioned imitation learning over unstructured data. In Robotics: Science and Systems Conference (RSS), 2020.
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 14:877008794 2021.
[6] Yilun Du, Sherry Yang, Bo Dai. Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via ext--guiedd video generation. In Conference on Neural Information Processing Systems (NeurIPS). 2023.
[7] Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video language planning. In International Conference on Learning Representations (ICLR), 2024.
[8] Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
[9] Torabi Faraz, Warnell Garrett, and Stone Peter. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, 2018.
[10] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to- video generation by explicit image conditioning. In European Conference Computer Vision (ECCV), 2024.
[11] Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning, pages 175-187. PMLR, 2023.
[12] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations, 2024.
[13] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 2.30/.04044 104, 2023.
[14] Nicklas Hansen. Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), 2022.
[15] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXhv:2210.023033 2022.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[17] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:220..12598, 2022.

---

[18] Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. European Conference on Computer Vision (ECCV), 2024.
[19] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
[20] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: language-image representations and rewards for robotic control. International Conference on Machine Learning (ICML), 2023.
[21] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning, pages 14743- 14752. PMLR, 2022.
[22] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding. Jun Jin, Bin Wang. Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain thought. Advances in Neural Information Processing Systems, 36, 2024.
[23] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning, 2022.
[24] Xue Bin Peng, Ze Ma. Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1-20, 2021.
[25] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2023.
[26] Aaron I. Putterman, Kevin Lu, Igor Mordatch, and Pieter Abbeel. Pretraining for language conditioned imitation with transformers. In Offline Reinforcement Learning Workshop at Neural Information Processing Systems, 2021.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, ((1):5485-555) 2020.
[29] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXivv2204.06125, 2022.
[30] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. In International Conference Learning Representations (ICLR), 2024.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[32] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Conference Neural Information Processing Systems (NeurIPS), 2022.
[33] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 400m: Open dataset of flltefitered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
12

---

[34] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[35] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[36] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[37] Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Biyik, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
(38) Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[39] Saran Tunyauuvuakkool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. mom_cotrol: Software and tasks for continuous control. Software Impacts, 6: 101000 2020.
[40] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohuí Wang, et al. Internvid: A large-scale video-text dataset for multi- modal understanding and generation. In The Twelfth International Conference on Learning Representations, 2023.
[41] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang. and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In International Conference on Learning Representations (ICI.R) 2024.
[42] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages (094-1100 PMLR, 2020.
13

---

A Intuition Regarding A Reasonable Noise Level Range
We desire reward signal that is high when the text prompt is well-aligned with the frame rendered following a well-selected action. However, we find that using too high or too low a noise level will cause the reward computation to ignore discount the provided text prompt.
For sufficiently low noise value, the conditional and unconditional noise prediction is similar for a particular text prompt and rendered frame that already has high cross-modal alignment. Being already close in data space to the desired mode described by the text prompt, the unconditional prediction will also cheaply seek to denoise the input towards the original input. Therefore, too low a noise value may cause the computed reward signal to decrease as cross-modal alignment increases, which runs counter to what is desirable. This is supported by looking at the left tails of the two graphs depicted in Figure AI] where the difference in computed TADPoLe rewards between well-aligned paired inputs and misaligned paired inputs is small.
Similarly, choosing a sufficiently high noise value may also cause unconditional and conditional predictions to be similar. For given noisy input with virtually all of the spatial structure perturbed, the pretrained denoising model intuitively makes denoising predictions that fill in general structure to the image irrespective of text-conditioning This is intuitively similar to the behavior of an unconditional prediction. Once again, we can visually observe this the right tails of both graphs in Figure AT for high noise values, both misaligned and well-aligned paired inputs have similar computed TADPoLe rewards.
Intuitively, TADPoLe is able to meaningfully quantify text-conditioned alignment with rendered observations into a reward signal for noise level range approximately the middle. For balanced level of noise corruption on text-aligned rendered observation, there is enough existing visual structure to help the denoising model make a meaningful rext-conditioned prediction close to the original input, whereas an unconditional prediction may not do so accurately. In Figure A we observe that for a noise range in the middle, TADPoLe is able to substantially favor the well-aligned pair over the misaligned pair, respecting changes in both text prompt as well as rendered observations. We verify in our experiments that using a noise level between 400 and 500 works well for TADPoLe, across both Dog and Humanoid environments.
Promot Comparise
person standing head'
02-
person standing hands above
0.5
personn stareding with hands on hipe"
60
Figure A1: Noise range intuition for fixed image but two distinct prompts (left), and for fixed prompt but two distinct images (right). Through visualization, we verify that U(400, 500) is a reasonable range from which to sample noise levels that meaningfully distinguish vision-text alignment for arbitrarily rendered frames.
A.1 Noise Level for Video-TADPoLe
We discover that Video-TADPe PoLe, and use a range of 500 to 600 in our experiments. We hypothesize that being able to observe multiple frames at once enables the space-time U-Net to exploit uncorrupted (or cesser-)orrupted) portions from adjacent frames across the context window to inform how each individual frame should be denoised coherently. Because each frame in a context window has distinct source noise applied to it, structural information can be leaked from randomly uncorrupted portions of the other frames in the context window when making predictions. Therefore, to make the denoising prediction more challenging, thereby forcing it to rely on the text-conditioning and learned motion priors rather than what is readily extractable from the provided context, generally a higher noise level acceptable compared to TADPoLe.
achieves better performance with a higher noise level than TAD-
14

---

B Detailed Hyperparameters
In our experiments, we place less emphasis on the underlying architecture and optimization scheme, instead comparing method against other lext-conditioned reward functions. We keep the same reinforcement learning architecture, optimization scheme, and ext-to-imaaeefeextoo-viiee generative model in all environments, unless otherwise noted. We provide detailed hyperparameters about these existing components below.
B.1 Pretrained Text-Conditioned Diffusion Models
We utilize StableDiffusion v2.1 checkpoint for TADPoLe experiments. We provide the sizes of the components in Table For Video-TADPoLA experiments, we use a pretrained AnimateDiff checkpoint, and provide relevant details in Table We do update or modify either checkpoint during our training, utilizing them purely for inference.
B.2 TD-MPC
We include the default hyperparameters from the TD-MPC implementation in Table A3 for complete- ness. We do not modify the default recommended settings for both Humanoid and Dog environments, as well the Meta- Woold experiments.
Table Al: StableDiffusion Components. For completeness, we list sizes of the components of the StableDiffusion v2.1 checkpoint used in TADPoLe experiments. The checkpoint is used purely for inference, and is not modified or up- dated any way. Note that the VAE Decoder not utilized in our framework.
Table A3: TD-MPC hyperparameters. We the official implementation TD-MPC [14] with no adjustments to the hyperparameters, but list it below for completeness. However, we do set the number of training steps to million for all experiments using TD-MPC.
Hyperparameter
Value
Component
# Parameters (Millions)
Discount factor (7)
0.99
Seed steps
5,000
VAE (Encoder)
34.16
Replay buffer size
Unlimited
VAE (Decoder)
49.49
Sampling technique
PER 0.6, 0.4)
Planning borizon (H)
U-Net
865.91
Initial parameters (2,, ()
[0.2)
Text Encoder
340.39
Population size
512
Elite fraction
64
lirrttion
12 (Humanoid)
(Dog)
Policy fraction Number of particles
5%
Momentum coeflicient Temperature (r) hidden size
Table A2: AnimateDiff Components. For completeness, we list sizes of the components of the AnimateDiff checkpoint used in Video- TADPoLe experiments. The checkpoint is used purely for inference, and is not modified or up- dated in any way. Note that the VAE Decoder is not utilized in our framework.
0.5
MLP activation
ELU (Humanoid, Dog)
Latent dimension
Learning rate
3c-4 (Dog)
1e-3 (Humanoid)
Optimizer (9)
Adam 0.9, (1399)
Temporal coefficient (A)
0.5
Reward coefficient (e.)
0.5
Value loss coefficient (cy)
0.1
Consistency coeflicient (ca)
Component
Parameters (Millions)
Exploration schedule (e)
0.5 -+ 0.05 (25k steps)
Planning harizon
steps)
VAE (Encoder) 34.16
Batch size
2048 (Dog)
VAE (Decoder)
49.49
(Humanoid)
U-Net
1312.73
Momentum coefficient (C)
0.99
Steps per gradient update 8 update fraquency
Text Encoder
123.06
B.3 Selecting wi and u22
We select hyperparameters and w2, which scale the alignment term align and the reconstruction term soe respectively, firstly such that the resulting computed reward r has value roughly between 0 and at any arbitrary timestep. This is visualized in Figure Al where over all noise levels, the
15

---

computed reward stays roughly between 0 and 1. Because the same pretrained rext-conditioned diffusion model used across all environments without modification, these hyperparameters can be generally reused to achieve the same kind of reward scale across environments. We found that the values of W1 2000 and w2 = 200 through a light hyperparameter sweep, reported in Table (4 and indeed verify that they work well without modification across both Humanoid and Dog environments in our main experiments.
Table A4: Ablation over wi and w2 on Humanoid Stand and Walk
W1
W2
Humanoid Stand
Humanoid Walk
200
2000
249.82
2.72
1000
2000
240.05
121.20
2000
200
262.22
226.29
2000
1000
195.87
152.32
2000
2000
218.22
90.20
C Meta- World Tasks
We visualize the complete suite of selected WetadWorld tasks in Figure A2 along with the official names of the tasks. In Table AS we also list the corresponding prompts consistently utilized across all text-conditioned approaches. In Table AA we list the average success for 12 robotic manipulation tasks. As add sparse "success" signal to text-conditioned approaches such as VLM-RM and TADPoLe, following prior experimental design [8 for Meta-World, we also list the performance of utilizing the sparse signal only.
We first notice that no approach is able solve two selected tasks, PPeg Insert Side" and "Shelf Place"; these were therefore omitted in Table[4] Furthermore, we observe that utilizing sparse reward signal only appears to have strong default performance, and achieves higher overall average success rate than TADPoLe or VLM-RM. This showcases the inherent power of sparse reward in solving Meta-World tasks. However, there are two additional takeaways; firstly, VLM-RM surprisingly achieves a substantial decrease in performance from the default sparse reward signal, highlighting TADPoLe as a more preferable dense coxt-condtioned reward provider. Secondly. TADPoLe is able to solve more overall tasks than other approaches. In particular, TADPoLe can solve the "Door Open" task, which is completely unable to solved by VLM-RM or using a sparse reward only. We therefore highlight TADPoLe promising eoxt-conittioeed reward signal in replacing ground-truth hand-designed feedback, particularly as the text-to-image diffusion model utillized was pretrained in general manner, without finetuning explicitly on World demonstrations.
Table A5: Meta-Worke Task- Pairs
Task
Text Prompt
Door Open
opening door closing door
Door Close
Drawer Open
opening drawer closing a drawer opening window closing a window pushing white mug towards the coffee machine pressing a button
Drawer Close
Window Open
Window Close
Coffee Push
Button Press
Soccer
pushing soccer ball into the net inserting peg into the slot placing an object on the shelf pulling a lever
Peg Insert Side
Shelf Place
Lever Pull
16

---

Door Open
Door Close
Drawer Open
Drawer Close
Window Open
Window Close
Coffee Push
Button Press
Soccer
Peg Insert Side
Shelf Place
Lever Pull
Figure A2: Meta-World Tasks. We select robotic arm tasks from Meta-World suite as our evaluation task set, balanced in terms of diversity and complexity.
Table A6: Average success rate for robotic manipulation tasks in Meta-World evaluation rollouts. We compare between TADPoLe and VLM-RM, both approaches that do not utilize in-domain data or demonstrations, and find TADPoLe significantly outperforms VLM-RM.
42 over 30
We report mean performance and standard deviation across 10 seeds for each task.
Success Rate (%)
Door Open
Door Close
Drawer Open
Drawer Close
Window Open
Window Close
Coffee Push
Sparse
(1L 0)
99.7 1.0)
74.7 (1. 28.7)
99.3 (1: 2.0)
(4 (1. 5.8)
93.0 16.9)
46.3 11.3)
VLM-RM
(
79.7 (+ 39.8)
10.0 (+ 30.0)
100 (4
9.7 (+-29.0)
0
(- 12.0)
TADPoLe
40.0 (+: 49.0)
100 (= 0)
45.3 (t: 46.6)
100 (# 0)
74.0 (:: 37.9)
30.0 (= 45.8)
18.6 (# 20.5)
Success (%) Button Press
Soccer
Peg Insert
Shelf
Average
Sparse
86.7 (4 29.1)
12.7 (+ 14.1)
0 0)
0 ()
(++ D)
50.6
VLM-RM
30.0 (ct 45.8)
5.3 (= 11.6)
0 0)
( 0)
11.3 (#: (8.3)
20.8
TADPoLe
(3 38.5)
25.0 15.4)
(2
0)
42.2
D Training Curves
In our experiments, we compare against other methods that provide text-conditioned rewards. As each of these methods are formulated differently, the difference in scales naturally prevent easy direct comparison. However, plotting the training curves for TADPoLe does yicld insights into its speed of convergence, and investigating the visual performance at intermediate steps be interesting, In Figure A3 we showcase the TADPoLe training curves for a variety of novel text-conditioned policies. We highlight the performance at steps 500k, IM, 1.5M, and 2M in red; we further visualize the policy achieved at these intermediate steps by showcasing the last frame of the achieved video.
In the case of the policy learned for the prompt "a person standing with hands on hips", for example, we see that whereas the initial policies fail, the policy first learns to stand up by step 1.5M. Then, by the last training step, the policy has learned to correctly place its hands conspicuously on its hips. Similarly, for the prompt "a person kneeling", ", the policy first learns kneel on all fours; by the end, the policy learns to successfully kneel on knee.
17

---

person stardingg with hands sbove head ITACRLL
persoo stand Fn th hands hipe" CRUPPLL
010
259
120
Siop
pernan knweling! (HADNL
chening the bal ITADNGU
340
180
300
1.00
6.79
0.89
425
Figure A3: TADPoLe training curves for variety of text-conditionings. with intermediate visual- izations. The frames displayed are always the last frame achieved by the policy, at that particular training step.
Video TADPoLe (Train)
Ground-Truth (Train)
Ground Truth (Evaluation)
Episode Return (Dog Walk)
180
80
80
160
60
140
40
40
120
20
20
1.5 Environment Steps (1e6)
0.5
1.5 Enviranment Steps (1e6)
0.0
1.5 Environment Steps (1e6)
Figure A4: Episode return curves of Dog Walk agent trained with Tideo-TADPoLe, using the prompt "a dog walking". From left to right: the cumulative Vidco-TADPon rewards achieved throughout training, the cumulative ground-truth rewards achieved throughout training, and the cumulative ground-truth rewards achieved during evaluation. Shaded regions denote the standard deviation across five random seeds.
18

---

Table A7: Numerical results from the user study text-alignment. Out of 25 anonymous users, for each method, we report the percentage that believe the video achieved by a policy at the end of training is aligned with the provided text prompt. Entries above 50% are displayed in Table I as checkmarks (V); below 50% are displayed as x-marks (x).
Environment
Prompt
VLM-RM
LIV
Text2Reward
TADPoLe (Ours)
Humanoid
"a person in lotus position"
52%
28%
4%
60%
Humanoid
"a person doing splits"
64%
88 %
0%
84%
Humanoid
"a person kneeling" "a dog standing"
48%
4%
0%
64%
Dog
16%
0%
40%
84%
E
Motivating Visualizations
For further intuition on TADPoLe's behavior, we visualize the complete denoising results for query video achieved through Dog policy and corrupted with some level noise, conditioned on consistent text prompt of "a dog walking". Note that this is purely for visualization purposes; in practice, when training policies, we not visually denoise over multiple steps but instead extract reward signal from components of a one-step denoising prediction. However, the multi-step denoising visualizations can offer us a glimpse of intuition into the behavior of the pretrained diffusion model and the properties of its predicted components at one single denoising step.
We first find that StableDiffusion, used in TADPoLe, is able to reconstruct well-aligned policy rollout (a video of the Dog agent actually walking) relatively accurately frame-by by-frame, for a noise level of 500 (Figure A5). However, the predictions can differ substantially for misaligned policy rollout (a video of the Dog agent falling over rather than walking); for frames where the dog has collapsed on the floor, the StafleDiifusionn model still tries to respect the specified input prompt and attempts to predict dogs standing upright or walking, as depicted in Figure A6 resulting in more noticeable difference between the achieved frame and predicted frame. This difference can be exploited to distinguish between well-aligned and misaligned policy rollouts, lending confidence to StableDiffusion as a supervisory signal for text-conditioned policy learning. For a higher noise level, StableDiffusion quickly forms hallucinatory final predictions (depicted in Figures A7 A8). but preserves more of the structure the original video when the achieved video and provided text prompt is aligned.
We also visualize reconstruction predictions using a text-to-video model, namely an AnimateDiff v2 checkpoint, in Figures | to reconstruct the Dog with a coherent texture and pose over time. Figures [AT]] A12 we highlight how text-to-vides model is more robust in reconstructing the achieved video by the policy at high noise levels compared with StableDiffusion (Figures A7 A8 This supports the hypothesis outlined in Section A.I
and We find that the motion prior is indeed meaningful, enabling it
19

---

Figure A5: Visualizing the denoising of good query trajectory from a noise level of 500 to completion using per-frame StableDiffusion.
Figure A6: Visualizing the denoising of a failed query trajectory from a noise level of 500 to completion using per-frame StableDiffusion.
Figure A7: Visualizing the denoising of good query trajectory from a noise level of 700 to completion using per-frame StableDiffusion.
Figure A8: Visualizing the denoising of a failed query trajectory from noise level of 700 to completion using per-frame StableDiffusion.
20

---

Figure A9: Visualizing the denoising of good query trajectory from a noise level of 500 to completion using AnimateDiff.
Figure A10: Visualizing the denoising of a failed query trajectory from a noise level of 500 to completion using AnimateDiff.
Figure All: Visualizing the denoising of good query trajectory from noise level of 700 to completion using AnimateDiff.
Figure A12: Visualizing the denoising of a failed query trajectory from noise level of 700 to completion using AnimateDiff.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In both the abstract and introduction, we state our main contribution we propose Text-Aware Diffusion for Policy Learning (TADPoLe) as the first approach to leverage domain-agnostic visual generative models for policy learning. TADPoLe uses large-scale pretrained text-conditioned diffusion model a discriminative manner to enable the learning of novel, zero-shot policies that are flexibly and accurately conditioned on natural language inputs, across multiple robot configurations and environments, for both goal-achievement and continuous locomotion tasks. We conduct thorough qualitative and quantitative evaluations across various benchmarks to experimentally support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our work in the concluding section of the main paper, where we highlight the lack of fine nhe-grained control over text conditioning and the high variance of the policy performance as two observed shortcomings of our proposed method that are promising future directions to explore.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
22

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not include any theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly describe our approach in the Method section (Section 3 and provide implementation details and evaluation protocols the Experiments section (Section 4 In addition, we provide detailed hyperparameters in the Appendix (B) Our experiments are performed using publicly available pretrained diffusion models and algorithms, and our codebase will be released publicly to support full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
23

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our codebase will be publicly released, along with detailed instructions to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https /nipps.ccppubll. guwigegieeeeeeeege CCodeSumbeeiscccc
detodeuuSS//o commiissionn ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettalls
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include training and evaluation details in the Experiments section (Sec- tion D. along with detailed hyperparameters for all components of our method in the Appendix (Section By. Furthermore, for user study experiments, we have included the relevant decisions and details surrounding the collection procedure.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
24

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars for goal achievements, continuous locomotion, and robotic manipulation experimental results, and explicitly state the number of seeds utilized.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Experiments section (Section 4 we report that all of our experiments are performed on single NVIDIA V100 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpe:///////is//// cc/ppublic/tthhccccuddedl....
Answer: [Yes]
Justification:
The research conducted in this paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
25

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no extremely noteworthy positive or negative societal consequences of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We do not release data or models with a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
26

---

Answer: [Yes]
Justification: We properly credit all publicly available codebases and models used in the paper. Their licenses and terms of use are properly respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Along with the release of our codebase, we will provide well-documented instructions for reproducibility.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Although we queried anonymous humans for user study of our achieved results, there was no direct research on or using human subjects. The anonymous humans in the user study were paid higher than the minimum wage of the country of the data collector.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
27

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Although we queried anonymous humans for user study of our achieved results, we do not perform any direct research with or on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "x1SP5xyWcb", "reviewer": "Reviewer_z3di", "summary": "The paper introduces Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a large-scale pretrained text-conditioned diffusion model to provide zero-shot reward signals for training agents without expert demonstrations or manually designed reward functions. TADPoLe enables agents to learn behaviors and achieve goals specified by natural language in various tasks, demonstrating its effectiveness.", "questions": "1. Even with sparse rewards in MetaWorld, the state-of-the-art results are better than those presented by the authors. \n2. Is the naturalness in Table 2 determined through user surveys? I believe that videos would provide more compelling evidence.\n3. How do different lengths and forms of prompts affect the rewards? What impact would slight adjustments to the prompts during training have?", "limitations": "The authors discuss limitations in the paper about how to control the weight of each individual word in the prompt. However, I believe that the primary limitations lie in the efficiency of the diffusion model. This paper has many aspects that require further discussion and improvement.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea is straightforward and easy to follow; the authors have adopted a clear approach to present their insights.\n\n2. Using text-conditioned diffusion models to provide rewards is novel.", "weaknesses": "1. Although the text-aware diffusion reward is novel, both text-aware rewards and diffusion rewards have been proposed by prior works. The paper lacks an apple-to-apple comparison, making it difficult to discern the specific advantages of using diffusion models.\n\n2. I find the experiments to be quite limited. For example, the authors only use TD-MPC as the algorithm backbone, which is just one model-based RL algorithm and does not have a significant advantage in visual RL. Additionally, the curve shown in Figure 5 is quite odd—why not place the baseline and proposed method on the same graph and include curves for other prompts? Furthermore, the comparisons in the Metaworld experiments are minimal, making the experimental results less convincing.\n\n3. The absence of real-world experiments makes it hard to assess the paper's contribution to the community.\n\n4. The authors use a diffusion model to generate rewards at each step, but there is no detailed analysis of the computational cost and its impact on speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer z3di for their comments and feedback on our work; we are happy to hear that the reviewer appreciates the novelty of our approach, and we seek to address their concerns below.\n\n**On comparisons:** we performed *apple-to-apple* comparisons with three recent text-aware rewards, namely VLM-RM (ICLR 24), LIV (ICML 23), and Text2Reward (ICLR 24). We utilize the exact same underlying model architecture, hyperparameters and optimization procedure, where only the text-aware reward is changed across methods, to make it as comparable as possible (Table 1, 3). We highlight that using diffusion models has superior or comparable quantitative performances against other text-aware rewards under an apple-to-apple comparison, while having superior text-alignment and naturalness benefits (Table 2). To the best of our knowledge, there is not a diffusion reward baseline which we can compare with in an apple-to-apple setup. A concurrent work (to be published at ECCV 24), Diffusion-Reward, is the only work that leverages diffusion models for reward modeling. However, it does not condition on natural language, and thus does not enable text-conditioned policy learning, which is a central focus of our work. Furthermore, Diffusion-Reward relies on in-domain expert video demonstrations, while TADPoLe does not.\n\n**On the use of TD-MPC:** we utilize TD-MPC as a standardized backbone to evaluate different text-aware reward methods in a comparable setting, to focus on text-conditioned reward quality. We choose TD-MPC due to it being the first model able to solve the complex Dog environment (given ground-truth rewards for walking and standing), with strong performance on Humanoid as well. Given its powerful modeling capability, we select it as a testbed for text-aware reward comparison.\nOn Figure 5: we place the baseline reward provided by the ground truth function and the TADPoLe reward on separate plots because they have different scales and are not directly comparable. However, we show that side-by-side, the trends across the two rewards are positively correlated, which is desirable. For TADPoLe reward plots across novel prompts, which do not have associated ground truth reward functions, additional examples can be found in Figure A11 of the Appendix.\n\n**On Metaworld comparisons:** Most prior methods which report performance on MetaWorld rely on (often expert-produced) video demonstrations from a similar domain or the target environment directly. We therefore focus our comparison on other methods that utilize a large-scale pretrained model for zero-shot (no in-domain demonstrations) text-conditioned policy learning, namely VLM-RM. Other work (namely Diffusion-Reward) utilize in-domain diffusion models trained explicitly on expert data and do not explore text-conditioned policy learning, which is the focus of our work. Furthermore, most performant state-of-the-art methods on MetaWorld were trained directly on dense ground-truth reward functions, whereas we evaluate the ability to recreate such behavior from dense text-conditioned rewards in a zero-shot manner. For further comparison, however, we include additional LIV results, as well as results from an adapted version of the Diffusion-Reward reward computation that is conditioned on natural language rather than historical frames, and uses a general text-to-video diffusion model in Table A8 of the rebuttal Appendix page. TADPoLe maintains superior performance in aggregate across the task suite.\n\n**On real-world experiments:** we agree that real-world experimentation would be nice to have. However, our focus is on exploring text-to-policy capabilities through diffusion models, which is a novel approach. We demonstrate our approach not only on the difficult Humanoid and Dog environments, which have high action spaces and complex transition dynamics, but also on Metaworld. In reference, prior work (VLM-RM) that used vision-language model supervision also did not apply it to real-world environments. However, in terms of future work, we do agree with the reviewer that deploying on real-world robotics is an exciting direction for further exploration.\nOn Computational Cost and Speed: the computational cost of TADPoLe is simply one denoising forward step of a pretrained diffusion model to generate each dense reward. In comparison, Diffusion-Reward requires multiple denoising steps to generate one singular reward (in practice, 10 steps are used). TADPoLe is therefore computationally cheaper and faster, when the same diffusion model is utilized, in terms of reward computation.\n\n**On naturalness via user surveys:** Evaluations on naturalness were performed through user surveys (25 random participants), using videos achieved by policies at the last step of training. These are provided on the [submission website](https://sites.google.com/view/tadpols/home), for reviewers to visualize.\n\n**On the effect of prompts on rewards:** We refer the reviewer to Figure (4), where we find that different lengths and slight adjustments to the prompt during training indeed impact the final performance of the policy. To begin with, we extend a prompt in length from “a person standing” to “a person standing with hands above head”, and observe that the longer description can be respected. Furthermore, when we perform slight adjustments to the prompt, with “a person standing with hands on hips”, we confirm that the policy learned is able to respect subtle details such as placement of the arms. We anticipate extensive prompt tuning to produce further benefits; but for the scope of our work (demonstrating a rich text-conditioned reward from large-scale pretrained diffusion models) we leave tuning the most performant prompt for a specific desired task to the enthusiast.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iO39uopVsd", "reviewer": "Reviewer_iiYV", "summary": "The paper presents Text-Aware Diffusion for Policy Learning (TADPoLe), a framework that leverages pretrained text-conditioned diffusion models to generate dense, zero-shot reward signals for policy learning in reinforcement learning tasks. The approach aims to address the limitations of manually designed reward functions by utilizing large-scale generative models to encode rich priors that guide policies in a natural and text-aligned manner. Experiments demonstrate TADPoLe’s effectiveness in learning policies for novel goals and continuous locomotion behaviors in various environments, including humanoid, dog, and robotic manipulation tasks, without ground-truth rewards or expert demonstrations.", "questions": "### Questions\n\n1. **Reward Scaling:** How does TADPoLe handle the scaling of reward signals across different tasks and environments to ensure consistent policy learning?\n2. **Text Prompt Complexity:** Can the method be extended to handle more complex and detailed text prompts? What are the limitations in terms of prompt complexity and specificity?\n3. **Long-Term Dependencies:** How does TADPoLe perform in tasks requiring long-term planning and dependencies? Are there any mechanisms to ensure temporal coherence in learned behaviors?\n4. **Generalization:** How well does the approach generalize to unseen environments or tasks that are significantly different from the training scenarios?", "limitations": "Same as the above section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Strengths:**\n\n- **Innovative Reward Generation:** TADPoLe introduces a novel approach to reward signal generation using pretrained diffusion models, reducing the need for manually crafted reward functions.\n- **Zero-Shot Learning:** The framework supports zero-shot policy learning, enabling the agent to learn new tasks and behaviors from natural language descriptions without prior demonstrations.\n- **Diverse Applications:** Demonstrates versatility across different environments and tasks, including humanoid and dog locomotion, and robotic manipulation in the Meta-World environment.\n- **Human Evaluation:** Qualitative assessments show that the policies learned by TADPoLe are perceived as more natural and aligned with the provided text prompts by human evaluators.", "weaknesses": "**Weaknesses:**\n\n- **Evaluation Metrics:** While the paper provides qualitative evaluations, the reliance on human judgment for assessing the naturalness and alignment of behaviors could introduce subjectivity.\n- **Scalability of Text Prompts:** The approach may struggle with very complex or ambiguous text prompts, and the extent to which it can handle highly detailed or context-specific instructions is not fully explored.\n- **Comparative Baselines:** The comparisons with other text-to-reward methods are limited, and it would be beneficial to include more diverse baselines to better understand the relative performance of TADPoLe.\n- **Computational Overhead:** The approach involves significant computational overhead due to the use of large-scale diffusion models, which could limit its practicality in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer iiYV for their helpful feedback and suggestions. We try to address their concerns below:\n\n**Qualitative evaluation:** We report quantitative comparisons whenever available (Tables 1, 3, 4), and we agree that human evaluation may inevitably introduce subjectivity. However, they are necessary when mere quantitative metrics are not sufficient to capture the performance difference, for example when evaluating the novel text-conditioned behaviors. We attempt to make such evaluation as impartial as possible; we use the videos of policy behavior at the last timestep of training without cherry-picking, and query 25 random participants without prior exposure to the task through an anonymized platform to estimate a general response from the human population. We provide details of our qualitative evaluation procedure in Section 4.1, and also additional fine-grained user study results in our updated rebuttal Appendix page (Table A7).\n\n**On text prompt scalability and complexity:** we agree that exploring text prompt scalability and sensitivity is important. We refer the reviewer to Figure (4), where we report preliminary investigations on how sensitive the method is when text prompts are extended with details (“a person standing” -> “a person standing with hands above head”), and when subtle details are modified in the text conditioning (“a person standing with hands above head” -> “a person standing with hands on hips”). However, despite showing that TADPoLe indeed demonstrates a level of sensitivity and scalability to long, detailed, text prompts, we agree that extremely complex prompts, such as multiple sequential instructions over time, may still be a challenge - developing this is a worthwhile future direction. We also note that our method is generally applicable to both image and video generative models, and therefore provides a way to handle text prompts based on both visual appearance as well as motion. As the modeling power of base text-to-video diffusion models improves, so too do we expect our approach to scale to respect more complex text prompts accurately. \n\n**On comparative baselines:** we highlight that text-conditioned policy learning through foundation models is a recent and active area of exploration. We compare against other recent work on text-aware rewards in VLM-RM (ICLR 24), LIV (ICML 23), and Text2Reward (ICLR 24), while also proposing novel baselines (ViCLIP-RM). We further highlight the diversity of our baselines: Text2Reward uses a language-only approach to create a text-conditioned reward function, whereas VLM-RM, LIV, and ViCLIP-RM compute rewards on-the-fly in a multimodal manner. In addition, in the new Table A8 of our rebuttal Appendix page, we further compare with LIV and Diffusion-Reward (a concurrent work, to be published at ECCV 24) on Meta-World, where TADPoLe enjoys the best performance.\n\n**On computational overhead:** Our reward computation only uses one denoising step to generate text-conditioned rewards, which avoids the overhead of multiple iterations of denoising steps usually performed with generating data through diffusion models in vanilla inference. Furthermore, our reward computation is general across diffusion model implementations, and the complexity of the diffusion model can be adjusted (such as through distillation) to fit desired resource constraints while still utilizing our proposed reward computation.\n\n**On reward scaling:** we discover that the symlog operation helps to normalize the raw computed reward signals across tasks and environments to be roughly on the same scale. We are therefore able to reuse hyperparameter settings (such as noise level, weights $w_1$ and $w_2$) across tasks and environments and even other diffusion models (e.g. Video Diffusion models) for consistent policy learning. We demonstrate this in a quantitative manner in Table A6 of the updated rebuttal Appendix page, by showcasing how removing the symlog normalization reduces consistent policy learning across environments and diffusion models with the same hyperparameter settings.\n\n**On long-term dependencies:** exploring how TADPoLe performs in tasks requiring long-term planning and dependencies is an interesting and worthwhile direction for future work. Due to the flexibility of text-conditioning, describing a task with long-term dependencies can be done in a way that is more like providing a sparse reward (e.g. describing just the desired goal) versus a manner that is more akin to providing a dense reward through multiple subtasks (e.g. providing detailed instructions). We anticipate the following considerations to potentially help solve long-term dependencies through TADPoLe: choosing appropriate underlying learning techniques (e.g. one that performs exploration, which can discover how to solve tasks that requires long-term dependencies), performing prompt tuning to discover the best prompts for a task, and learning to attend on different portions of a detailed text prompt (e.g. instructions) based off of progress of the policy’s performance.\n\n**On task generalization:** we would like to clarify that from the perspective of TADPoLe, all environments are unseen and novel; we utilize generally-pretrained diffusion model checkpoints, with no tasks or even examples of the downstream visual environment observed during pretraining. Policy supervision is computed purely from priors captured within diffusion models with large-scale pretraining and no in-domain examples are used to update the diffusion model. We believe that applying TADPoLe across Humanoid, Dog, and MetaWorld already demonstrates strong task generalization capability, as the agents and visual characteristics of each environment differ greatly from each other. Furthermore, we demonstrate how text-conditioned policies can be learned across these distinct environments without minimal modification to its hyperparameters, even generalizing across which pretrained diffusion model is used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X3Iscg3Wvy", "reviewer": "Reviewer_TWnR", "summary": "The paper introduces Text-Aware Diffusion for Policy Learning (TADPoLe), a method for reinforcement learning that leverages pretrained text-conditioned diffusion models to compute dense reward signals. This approach allows agents to learn text-aligned behaviors without the need for expert demonstrations or handcrafted reward functions. The authors demonstrate the effectiveness of TADPoLe in various environments, including Humanoid, Dog, and Meta-World, achieving zero-shot policy learning with natural language inputs.", "questions": "1. Clarification of Component Integration: Why is the 'symlog' transformation necessary? How does it and 'noise level range' specifically contribute to the performance improvements observed? More detailed explanations and justifications are needed.\n\n2. Source of Ground-Truth: Where do the ground-truth rendered subsequent images come from? Does the method rely on pre-existing well-rendered videos for each environment? This aspect should be clarified to understand the method's applicability.\n\n3. Experimental Expansion: To provide a more comprehensive evaluation, expand the range of tasks and include comparisons with the Diffusion-Reward method. This will help to position TADPoLe's performance in a broader context.\n\n4. Ablation Study: Conduct a more detailed ablation study to isolate the effects of the 'symlog' transformation and the specific weightings used. This will help to understand the contribution of each part to the overall performance and provide a clearer justification for their inclusion.\n\n5. Scalability and Practicality: Address the scalability issue regarding the need for well-rendered videos in environments lacking existing well-performed videos. How does the method handle such cases, and what are the implications for its practicality and scalability?", "limitations": "The author has addressed a limitation. It's also recommended to add the unsolvable questions to the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper provides a novel approach to reinforcement learning by using large-scale pretrained generative models to generate reward signals based on natural language descriptions. This method removes the need for manually crafted reward functions, which are often a bottleneck in RL tasks. The approach is well-motivated, leveraging the rich priors encoded in generative models trained on vast datasets. The experimental results show the ability of TADPoLe to learn diverse behaviors across different environments and tasks.", "weaknesses": "1. Presentation Issues: The paper's presentation is problematic. The integration of components such as the 'symlog' of the reward function and the 'noise level range' is not clearly explained. It is difficult to understand the necessity and utility of these components and how much performance improvement depends on them. Additionally, the paper lacks clarity on the source of ground-truth for rendered subsequent images and whether the method requires pre-existing well-rendered videos for each environment for TADPoLe training.\n\n2. Alignment with Motivation: The motivation to leverage text-conditioned diffusion models for reward generation is clear; however, the demand for well-rendered videos for each environment does not fully align with the goal of making reinforcement learning more practical and scalable. Some environments donot have existing well-performed video for the agent, so will this method require for a pre-trained policy to collect the video data?\n\n3. Experimental Insufficiency: The experiments are not comprehensive. There is a need to expand the range of tasks to provide a more thorough evaluation of the method. Additionally, the paper had better include comparisons with the Diffusion-Reward method [1] to highlight the advantages of TADPoLe. \n\n4. The ablation study is also insufficient. It should further explore the design of the 'symlog' component and the selection of the two weights, as these operations lack detailed justification. This analysis is crucial to understand the contribution of each part to the overall performance and to provide a clearer justification for their inclusion.\n\n5. Overclaim. The writing of the paper is dense and lacks clarity in several key areas. The explanations of the methodology and components are not sufficiently detailed, making it challenging for readers to grasp the full picture of how TADPoLe functions and why certain design choices were made. Improving the clarity and coherence of the writing would significantly enhance the paper's readability and accessibility.\n\n[1] Huang, T., Jiang, G., Ze, Y., & Xu, H. (2023). Diffusion Reward: Learning Rewards via Conditional Video Diffusion. arXiv preprint arXiv:2312.14134.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer TWnR for their detailed comments and thorough questions. We seek to address their concerns below:\n\n**On pre-existing well-rendered videos:** we would like to clarify a potential misunderstanding - our method does not require any pre-existing well-rendered videos for environments TADPoLe is applied to, and therefore there is no source of ground-truth image/video demonstrations for arbitrary environments (no offline videos or demonstrations are collected, and pre-existing expert policies are not needed). Rather, we utilize the rendering **function** at each timestep **on-the-fly** to generate a dense text-conditioned reward, as computed by a large-scale generally-pretrained diffusion model. We do not update the diffusion model with in-domain examples whatsoever. We therefore showcase how large-scale pretraining over natural images, videos, and text-alignment can directly transfer to the *zero-shot optimization* of policies that behave in alignment with text-conditioning, as well as natural-looking goal poses or motions.\n\n**On alignment with motivation:** we appreciate that the reviewer believes our motivation on using text-conditioned diffusion models for reward generation is clear; in light of our clarification that TADPoLe can be applied to arbitrary environments without demanding well-rendered video examples a priori (and therefore no pre-trained policies are needed either), we believe that our final proposed approach is well-aligned with the goal of making reinforcement learning more practical and scalable. Indeed, TADPoLe offers an avenue for supervising policies that behave not only in alignment with natural language, but also in alignment with natural-looking poses or motions captured from large-scale pretraining on natural images and videos.\n\n**On ablations:** the reviewer has highlighted three design decisions of interest: the *symlog* operation, the *noise level range*, and the selection of the two weights $w_1$ and $w_2$. We utilize the *symlog* operation to normalize the scales of the computed reward (in conjunction with $w_1$ and $w_2$ terms); indeed, we observe in Figure A1 where over all noise levels, the final computed reward stays roughly between 0 and 1, despite changes to the visual input or text prompt. Furthermore, this consistent normalization enables the reuse of hyperparameter settings (such as noise level, weights $w_1$ and $w_2$) across tasks and environments and even other diffusion models (e.g. Video Diffusion models) for consistent policy learning. We demonstrate this in a quantitative manner in Table A6 of the updated rebuttal Appendix page, by showcasing how removing the symlog normalization reduces consistent policy learning across environments and diffusion models with the same hyperparameter settings. For the *noise level range* used, we have provided much justification in the Appendix of our original paper submission through a discussion (Section A, A.1) as well as graphical Figures (Figure A.1). Similarly, in our original submission we have also reported that the values of the two weights $w_1$ and $w_2$ were selected through a hyperparameter sweep on Humanoid standing and walking performance (Section 4.1, Implementation), which we provide in Table A4, along with a discussion on their effects in Appendix C.3.\n\n**On experimentation:** the focus of our work is evaluating zero-shot optimization of policies conditioned on natural language. We highlight that text-conditioned policy learning through foundation models is a recent and active area of exploration. We compare against other recent work in VLM-RM (ICLR 24), LIV (ICML 23), and Text2Reward (ICLR 24), while also proposing novel baselines (ViCLIP-RM). We further highlight the diversity of our baselines to position TADPoLe in a broader context: Text2Reward uses a language-only approach to create a text-conditioned reward function, whereas VLM-RM, LIV, and ViCLIP-RM compute rewards on-the-fly in a multimodal manner. Furthermore, our range of tasks includes not only Dog and Humanoid (Table 1, 3), but also robotic manipulation tasks from MetaWorld (Table 4). We extend our range of tasks to include multiple novel prompts, as well as those with subtle details (Figure 4).\n\n**On comparison with Diffusion-Reward:** our work focuses on learning text-conditioned policies in a zero-shot manner through a large-scale generally-pretrained diffusion model. Diffusion-Reward (concurrent work, to be published at ECCV 24) is not a natural comparison for our approach, as it does not enable text-conditioned policy learning (their diffusion model is conditioned only on a history of frames), and their method explicitly requires expert video demonstrations from the environment to train their in-domain diffusion model. On the other hand, TADPoLe (along with listed baselines like VLM-RM, LIV, Text2Reward) requires no in-domain or expert demonstrations, and is text-conditioned. However, we have adapted a version of Diffusion-Reward for comparison purposes, where rewards are conditioned not on history but on a natural language input, and reported it in Table A8 of the rebuttal Appendix page. We demonstrate that TADPoLe outperforms the adapted Diffusion-Reward implementation in terms of overall success rate aggregated over all tasks in the suite. We further note that Diffusion-Reward requires multiple denoising steps for each dense reward computation (in practice they use 10 steps), whereas TADPoLe only requires one denoising step per reward computation, thus substantially improving in computation speed and complexity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "557UTAiTJ0", "reviewer": "Reviewer_KsL9", "summary": "The paper proposed a reward generating pipeline leveraging a text-conditioned diffusion model with a text prompt to master RL tasks described by the prompt.\nThe pipeline compares the difference of the generated image with/without the prompt and the original image to calculate a dense reward, which also makes sense intuitively.", "questions": "- Is $\\hat{\\epsilon}_\\phi$ a neural network (Sec 3.1) or a noise (Fig.3)?\n- What is the cons of replacing the diffusion model in your method with a CLIP-like model to access the alignment level of image and text? One I can think of is the naturalness cannot be calculated in such case.\n- Why is VLM-RM outperforming the proposed method in the standing Humanoid environment? Is it due to the training method of VLM-RM or because it is easier?\n- Is it possible to give a concrete example of the failure mode mentioned in the limitation section?\n- It would be interesting if the authors can try difficult tasks that even the diffusion model does not know how to generate and see the corresponding results of the RL training and evaluation.", "limitations": "See weakness 1. The paper concluded most of its limitation in the main text and Appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors cleverly use a conditional diffusion model to estimate the alignment level of an image to the text prompt with a hypothesis that  diffusion models capture naturalness and world knowledge.\n- The authors expand the proposed method from image-text space to video-text space.\n- As it is hard to quantitatively evaluate the performance of benchmarks, the paper conducted user studies to compare benchmarks by human volunteers.", "weaknesses": "- The proposed method requires more hyperparameter-tuning of the noise level and the overall reward function. Thus, more investigation is needed before training is conducted, such as Appendix A.\n- The proposed method heavily relies on the capability of the external diffusion model to obtain world knowledge. It is a natural sin due to the complexity of the tasks the paper is solving.\n- Another minor weakness is that some details of the user study are not presented, such as the variance of ✔ and ✘ and scores (instead of giving the majority and avg).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer KsL9 for their thoughtful comments and helpful feedback on our work. Below, we seek to both address the reviewer’s listed weaknesses and answer the posed questions:\n\n**On hyperparameter tuning:** we concur with the reviewer that our approach benefits from initial tuning of the noise level, and reward computation terms such as $w_1$ and $w_2$. However, we demonstrate that such hyperparameter settings generally transfer across environments and robotic states with minimal modification. For example, the Humanoid, Dog, and MetaWorld experiments were conducted with the same TADPoLe noise level and $w_1, w_2$ terms. The same $w_1, w_2$ terms were used for Video-TADPoLe, and were also kept consistent between Humanoid and Dog environments - only the noise level range changed (with justification and discussion in Appendix A.1). Therefore, although there are more terms to tune, tuning does not generally need to occur ad hoc but can be preserved flexibly across environments with decent effect.\n\n**On relying on external diffusion models for world knowledge:** we agree with the reviewer’s analysis that the diffusion model is utilized to provide world knowledge to a policy during optimization. As demonstrated in the paper, this offers a natural way for priors summarized from large-scale pretraining to benefit policy learning, enabling the learning of policies that act and behave according to “natural” poses and motions summarized from natural image and video datasets, as well as enabling flexible text-conditioning. As diffusion models improve in encoding world knowledge, stemming from efforts on understanding better physics, dynamics, etc. we anticipate the quality of policies learned through TADPoLe to directly improve as well.\n\n**On user study details:** In Table A7 of our rebuttal Appendix page, we provide the raw user study results on text-alignment, which correspond to the checkmark and x-marks in Table 1. Among 25 anonymous and randomly selected online participants, we deem a prompt to be correctly learned when more than 50% of the participants vote for text-behavior alignment (denoted by a checkmark). We observe that TADPoLe generally has the highest positive vote ratios.\n\n**Clarifying $\\mathbf{\\hat\\epsilon}_{\\boldsymbol{\\phi}}$:** we thank the reviewer for highlighting the notational discrepancy; indeed, we overload $\\mathbf{\\hat\\epsilon}_{\\boldsymbol{\\phi}}$: as a standalone it represents a noise prediction, and when utilized as a function it represents the neural network that produces the prediction. We will clarify this overloaded notation in the caption of Figure 3 in the final draft of the paper.\n\n**On replacing the diffusion model with CLIP-style:** indeed, the reviewer’s suggestion is what VLM-RM is doing, which we compare against in our experiments. Utilizing a diffusion model over a CLIP-based approach has conceptual benefits aligned with the intuition of the reviewer. As a CLIP-based alignment model is trained on clean pairs of image and text, during inference it therefore assumes that the input for the image is always clean. During ordinary retrieval, from a set of natural images, this is not an issue. During policy optimization, however, it essentially has the flexibility to search for the most text-aligned image amongst potentially unnatural-looking poses. Having a base assumption that every query image is “natural”, it may therefore cause the policy to learn an unnatural-looking pose but the model deems highly-aligned with the text prompt. On the other hand, a diffusion model is trained to not only respect text-alignment, but also understand what is a natural-looking image at all; as a generative model, it seeks to generate a natural-looking image and therefore must learn a notion of \"naturalness\" from its pretraining data. Therefore, an approach like TADPoLe can result in more natural-looking policies learned (as evidenced by the qualitative results reported in Tables 1, 2, and 3 in comparison to VLM-RM).\n\n**On VLM-RM standing performance:** another distinguishing factor between using a CLIP-style model and a diffusion model in our method is that CLIP provides a deterministic reward signal. In other words, for the same query image and query text prompt pair, the CLIP model will always provide the same alignment reward score. When utilized for policy optimization, for a fixed text prompt, the policy can essentially be seen as taking actions to search for the singular frame that will achieve the highest deterministic alignment score and then maintaining it for perpetuity. Therefore, goal-achievement tasks like Humanoid Stand are very stable for CLIP-based approaches to learn; it learns to achieve a standing position and maintain it. However, there are two drawbacks: having a singular deterministic score which encourages goal-achievement does not translate well to continuous locomotion, where there is no canonical pose to achieve, and the pose that it does achieve may not be aligned with a notion of naturalness as mentioned above.\n\n**On failure modes:** In our updated submission website, we provide Video-TADPoLe results for the prompts “a person walking to the left” and “a person walking to the right”. Whereas there are seeds of the Humanoid successfully walking with respect to the provided prompt, there are also seeds where the person walks in the opposite direction. How to provide fine-grained control over the text-conditioning to focus on key words such as direction (beyond simply walking) is interesting to explore for future work; furthermore, a potential way to remedy this limitation is with improvements to the underlying text-conditioned diffusion model itself, such as with motion adapters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive feedback, and are glad that TADPoLe was recognized as “novel”, “easy to follow”, and “demonstrates versatility across different environments and tasks”.

We have identified common points raised by the reviewers, which we summarize and respond to below: 

**Comparison with other diffusion-based rewards:** To the best of our knowledge, there is no prior work on text-aware diffusion rewards for an apple-to-apple comparison; as pointed out by reviewer TWnR, Diffusion-Reward is a relevant work (and also concurrent, as it is to be published at ECCV 24).  Furthermore, Diffusion-Reward does not take text conditioning, and requires the diffusion model to be trained with expert video demonstrations. TADPoLe, in comparison, is zero-shot and directly computes text-aware rewards through a generally pre-trained diffusion model without using any in-domain examples.  For the sake of comparison, however, we adapt Diffusion-Reward to be text-conditioned, and use the same diffusion model as TADPoLe for an apple-to-apple comparison on Meta-World.  We report the results in Table A8, found in the attached rebuttal Appendix page, and we discover that TADPoLe still outperforms Diffusion-Reward in overall task performance.

**Qualitative evaluation:**  Reviewers were interested in details of our qualitative evaluation, given the potential subjectivity introduced by this evaluation scheme.  We tried our best to make the human evaluation as impartial as possible; we performed our user study with 25 random participants through an anonymized random platform (Prolific) without prior training to estimate a general response from the human population.  We also use the videos of policy behavior at the last timestep of training without cherry-picking. In the attached rebuttal Appendix page, we provide the fine-grained user study results on what percentage of the users believe the video achieved by the policy is appropriately text-aligned.  We find that TADPoLe consistently achieves superior text-alignment preferences.
Robustness of hyperparameters: We highlight that although there are additional hyperparameters, which may require tuning, these hyperparameters can generally be shared across environments (Humanoid, Dog, MetaWorld), tasks (standing, kneeling, walking, etc.), and even diffusion models without modification (TADPoLe, Video-TADPoLe).  The selection of noise level, as well as hyperparameters $w_1$ and $w_2$ have been previously justified in the Appendix of the original submission.  We offer a new study on the symlog operation, and its effect on performance across tasks and environments, in Table A6 of the attached rebuttal Appendix page, and invite the reviewers’ interest towards it.

**Computational Cost:** we clarify that the computational cost of TADPoLe is simply one denoising forward step of a pretrained diffusion model to generate each dense reward. In comparison, Diffusion-Reward requires multiple denoising steps to generate one singular reward (in practice, 10 steps are used). TADPoLe is therefore computationally cheaper and faster, when the same diffusion model is utilized, in terms of reward computation.

We thank the reviewers for their time and consideration,

The Authors

Meta-review: This paper has mixed scores of (4,4,6,7), with a variety of points made by the reviewers about its strengths and weaknesses. Overall, I have some doubts about some of the crucial points made by a few of the low-scoring reviews for a few reasons, finding that the main concerns were either resolved or not in conflict with the main contributions of the paper. I generally agreed with the remaining points made (positives and negatives alike). Taken together, I recommend acceptance, and for the authors to ensure that they address the remaining concerns of the reviewers. Below, I summarize some points made by reviewers that I used to arrive at this recommendation.

## Reviewer z3di (4)
This reviewer was unconvinced by the author responses, and stated: 
> I don't think the videos shown by the authors are convincing ... The results using DrM on Adroit are also confusing.

and

> The authors said that 'prior work on MetaWorld mainly use dense ground-truth rewards, in-domain expert demonstrations, or both'. However, they mention DrM in their other replies, which uses sparse rewards and achieves much better performance."

I share these concerns, but find them minor, because they are not in conflict with the main claims of the paper ("our method is novel in its reward computation, as well as its utilization of a domain-agnostic generative model" ... "first approach to leverage domain-agnostic visual generative models for policy learning."). Furthermore, earlier in the review process, this reviewer stated

> As for the reason for real-world experiments, diffusion-reward actually added a lot of real-world experiments. Without real-world experiments, I think the contribution to the community is very limited. So I decide to maintain my score.

The authors responded to this criticism by pointing out "the referenced paper does not train any policies for real-world robots; the only policies they train are in simulation, which we also do in this work." I therefore consider this concern addressed, which was a major factor why this reviewer maintained their (4) score when they stated this concern.

## Reviewer iiYV (6)
This reviewer stated "Although I think this work still requires heavy prompt engineering and the generalization ability is weak, I would raise the score to 6."

## Reviewer TWnR (4)
This reviewer, when nudged for their final evaluation, stated there are still "questions about their work's performance and the true source of the observed performance improvement." This concern stems from the ablation study that demonstrated the importance of the 'symlog' transformation. However, this finding is consistent with the paper's claims ("our method is novel in its reward computation, as well as its utilization of a domain-agnostic generative model" ... "first approach to leverage domain-agnostic visual generative models for policy learning.") and in my opinion shouldn't count significantly against it. One issue I have with this finding, though, is that the current presentation of the paper does not make clear that it was discovered that the symlog operation was important as shown by Table A6 in the author's response, so it needs to be updated to reflect this finding. Furthermore, Table A6 is not a complete ablation across all environments and prompts. A more complete Table A6 would offer more evidence of this finding, and I strongly recommend the authors perform a more complete ablation. Whether additional ablations would support or undermine this finding it is irrelevant to the paper's main claims, and thus I discount this criticism a fair bit.

## Reviewer KsL9 (7)
This reviewer stated a mostly positive assessment with few minor weaknesses.
Decision: Accept (poster)