Text-Aware Diffusion for Policy Learning

Authors: calvin  luo
mandy  he
zilai  zeng
chen  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:12:35

Abstract: Training an agent to achieve particular goals or perform desired behaviors is often accomplished through reinforcement learning, especially in the absence of expert demonstrations.  However, supporting novel goals or behaviors through reinforcement learning requires the ad-hoc design of appropriate reward functions, which quickly becomes intractable. To address this challenge, we propose Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a pretrained, frozen text-conditioned diffusion model to compute dense zero-shot reward signals for text-aligned policy learning.  We hypothesize that large-scale pretrained generative models encode rich priors that can supervise a policy to behave not only in a text-aligned manner, but also in alignment with a notion of naturalness summarized from internet-scale training data.  In our experiments, we demonstrate that TADPoLe is able to learn policies for novel goal-achievement and continuous locomotion behaviors specified by natural language, in both Humanoid and Dog environments. The behaviors are learned zero-shot without ground-truth rewards or expert demonstrations, and are qualitatively more natural according to human evaluation. We further show that TADPoLe performs competitively when applied to robotic manipulation tasks in the Meta-World environment, without having access to any in-domain demonstrations.

Main paper: 
 introduction: 
 Can we train reinforcement learning agents that drive humanoids in a virtual environment [39] to stably stand? How about standing with hands on hips, kneeling, or doing splits? While state-ofthe-art algorithms have shown success on the former scenario (e.g. [14] ), the latter (illustrated in Figure 1 ) remains challenging due to the need for carefully (and often manually) crafted reward functions to specify the desired behaviors. The dependence on ad-hoc designed reward functions renders inscalable the learning of ever-increasing amounts of novel behaviors, which are required in applications ranging from character animation [2] to robotic manipulation [42] .Our work looks towards natural language as a powerful interface through which humans can flexibly specify desired goals or behaviors of interest. We therefore investigate how to construct a zero-shot text-conditioned reward signal, replacing the need for ad-hoc designs, through which text-aligned policies can be learned. We present Text-Aware Diffusion for Policy Learning (TADPoLe), which utilizes a large-scale pretrained, frozen text-conditioned diffusion model to generate a dense reward signal for policy learning. We hypothesize that generative diffusion models, which are pretrained on internet-scale datasets to produce text-aligned, natural-looking images [32, 29] and videos [3, 10, 15] , can be utilized to automatically craft a multimodal reward signal that encourages an agent to behave both faithfully with respect to text conditioning and naturally with respect to human perception. Our method is novel in its reward computation, as well as its utilization of a domain-agnostic generative model, rather than one trained from environment-specific or task-specific video demonstrations, as used in prior work [9, 24, 6, 8, 20, 19] . "a dog chasing its tail" "a dog standing" "a robot arm is closing the door of a black safe" "a robot arm is pushing a soccer ball into the net" TADPoLe is motivated by the insight that a reinforcement learning policy can be viewed as an agentcentric implicit video representation when operating within an environment with visual rendering capabilities. As illustrated in Figure 2 (left), an agent's video generation process involves the selection of actions following a policy π θ , and the conversion of the action sequence into video subsequences through the environment's rendering function. A policy can therefore be seen as iteratively generating frames conditioned on the actions it selects; on the other hand, a text-to-image diffusion model can also be seen as generating static image frames, but conditioned on natural language instead. A connection can then be established between a policy and a diffusion model, where the frame or video segment "generated" by the policy can be critiqued by evaluating how likely a text-conditioned diffusion model would generate the same visuals, thus providing dense text-aligned reward signals to guide policy learning (Figure 2 right). Our work is inspired by DreamFusion [25] , where a textconditioned 3D model is learned through rendered views, and where volumetric raytracing ensures spatial consistency. Here, we seek to learn a text-conditioned policy through rendered frames or subsequences, where the environment naturally ensures temporal continuity and consistency with respect to a notion of physics instantiated by the environment.Concretely, TADPoLe achieves text-conditioned policy learning by using a generative diffusion model in a discriminative manner. It computes the reward signal as a weighted combination of two reward terms, which aim to measure the alignment between the rendered observation and text conditioning, and the naturalness of the agent's behaviors, respectively. In this way, we can in effect "distill" the natural visual and motion priors as well as vision-text alignment understanding captured within the diffusion model into a policy. By default, TADPoLe uses a text-to-image diffusion model [31] to densely compute a reward signal solely from the immediate subsequent frame after each action. We then generalize the framework to Video-TADPoLe, which uses a text-to-video diffusion model [12] to calculate dense rewards as a function of a sliding context window of past as well as future frames achieved. The agent is thus trained to select actions such that arbitrary consecutive subsequences of frames are well-aligned with text as well as natural video (e.g. motion) priors.We highlight TADPoLe as the first approach to leverage domain-agnostic visual generative models for policy learning. Through quantitative and human evaluations on Humanoid [39] , Dog [39] , and Meta-World [42] environments, we demonstrate that TADPoLe enables the learning of novel, zeroshot policies that are flexibly and accurately conditioned on natural language inputs, across multiple robot configurations and environments, for both goal-achievement and continuous locomotion tasks.TADPoLe therefore provides two main benefits simultaneously: a performant approach towards zero-shot policy learning, where complex reward functions no longer need to be manually specified per task, and a promising path towards distilling priors summarized from large-scale pretraining into policies, ultimately resulting in the learning of more naturally-aligned behavior within arbitrary environments. Visualizations and code are provided at diffusion-supervision.github.io/tadpole/. 
related work: 
 Diffusion models [34, 35, 16, 36] have recently demonstrated amazing generative modeling capabilities, particularly in the domain of text-conditioned image generation [5, 29, 32, 31] . Notably, guidance [36, 5, 17] has been shown to be a critical component in producing visual outputs aligned with textual data, enabling the generation of images that accurately match a desired text caption, especially when the models are scaled to utilize large foundation models such as CLIP [27] or T5 [28] , and trained on massive image-text datasets [33] . Our work is inspired by DreamFusion [25] , where a pretrained, frozen text-to-image diffusion model is able to supervise the learning of zero-shot 3D models conditioned on text. We propose leveraging pretrained diffusion models to supervise the learning of flexible, text-conditioned policies in a zero-shot manner over the time dimension.There are numerous works that investigate how interactive agents can learn to perform behaviors specified by textual inputs. SayCan [1] grounds the knowledge of complex, high-level behaviors within an LLM to the context of a robot through pretrained behaviors. This then enables an LLM to instruct and guide a robot, through combining low-level behaviors, to perform complex temporally extended behaviors. LangLfP proposes a method for incorporating free-form natural language conditioning into imitation learning by first associating goal images with text captions and training a policy to follow either language or image goals, but only conditioning on natural language during test time inference [4] . The Text-Conditioned Decision Transformer learns a causal transformer to autoregressively produce actions conditioned on both text tokens as well as state and action tokens [26] . Similarly, Hiveformer proposes a unified multimodal Transformer model for robotic manipulation that conditions on natural language instructions, camera views, as well as past actions and observations [11] . However, LangLfP, Text-Conditioned Decision Transformer, and Hiveformer, all require training on datasets of trajectories that have been labelled with natural language. In contrast, TADPoLe enables the learning of text-conditioned policies irrespective of visual environment, and without requiring any pretraining dataset of demonstrations or labeling.Similar to our work, UniPi [6] treats the sequential decision-making problem as a text-conditioned video generation problem. The authors propose training a video diffusion model to produce a future visual plan for the agent; the subsequent frames are then converted to actions by means of an inverse dynamics model. VLP [7] utilizes text-to-video generative models for planning and goal generation for an agent. Both methods require the video generative models to be trained ad-hoc on the target environments, whereas we directly use frozen general-purpose generative models. Mahmoudieh et al. [21] propose a framework that uses CLIP to generate a reward signal from a text description of a goal state and raw pixel observations from the environment, which is then used to learn a task policy.In VLM-RM [30] , the authors also explore utilizing CLIP as the reward model for training humanoids to accomplish complex goal-reaching tasks. In our work, we investigate locomotion tasks on top of goal-achievement, and explore how using diffusion models to produce a reward signal can outperform CLIP-based approaches. Although not conditioned on text, VIPER [8] also aims to harness recent advancements in generative modeling by employing a video prediction model's likelihoods as a reward signal. However, VIPER does not enable the learning of policies conditioned on text, and requires in-domain expert videos for ad-hoc training the video model. Finally, Diffusion Reward [18] also extracts a reward from a diffusion model to train policies; however, it requires training an ad-hoc video model on expert trajectories, the collection of which cannot always be assumed to be trivial, and does not enable text-conditioned policy learning.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We have discussed the limitations of our work in the concluding section of the main paper, where we highlight the lack of fine-grained control over text conditioning and the high variance of the policy performance as two observed shortcomings of our proposed method that are promising future directions to explore.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We propose Text-Aware Diffusion for Policy Learning (TADPoLe) to learn text-aligned policies by leveraging frozen, pretrained text-conditioned diffusion models. An overview of the framework can be found in Figure 3 .Figure 3 : An illustration of the TADPoLe pipeline, which computes text-conditioned rewards for policy learning through a pretrained, frozen diffusion model. At each timestep, the subsequent frame rendered through the environment is corrupted with a sampled Gaussian source noise vector ϵ 0 . The pretrained text-conditioned diffusion model then predicts the source noise that was added. The reward is designed to be large when the selected action produces frames well-aligned with the text prompt.We first describe how TADPoLe produces text-conditioned rewards from image observations. At each timestep t, reward r t is computed as a score between rendered subsequent image o t+1 and the provided text caption describing the behavior of interest, denoted by y, using a frozen, pretrained textto-image diffusion model. We begin by corrupting the rendered image o t+1 with a sampled Gaussian source noise vector ϵ 0 ∼ N (ϵ; 0, I) to produce noisy observation õt+1 , and use the diffusion model to make an unconditional prediction εϕ (õ t+1 ; t noise ) as well as a conditional prediction εϕ (õ t+1 ; t noise , y). Here εϕ (•) is a neural network that predicts the source noise given õt+1 , the level of noise corruption t noise , and optionally the text prompt y; we overload notation to have εϕ represent the source noise prediction in Figure 3 . We then compute the mean squared error (MSE) between the two predictions as a reward signal r align t to be maximized:r align t = ∥ε ϕ (õ t+1 ; t noise , y) -εϕ (õ t+1 ; t noise )∥ 2 2 .As investigated in Appendix B.3, we empirically observe that r align t plays a crucial role on the success of TADPoLe. We hypothesize that for an appropriately-selected noise corruption level t noise , this term measures the alignment between the environmental observation and the text prompt. Intuitively, for unconditional prediction εϕ (õ t+1 ; t noise ), the model is incentivized only to bring the noisy input to any arbitrary cleaner image, and makes minimal edits by moving it towards the closest clean mode in data space. On the other hand, if the model recognizes visual features in the noisy image aligned with the text prompt, conditional prediction εϕ (õ t+1 ; t noise , y) is incentivized to do "extra work" and bring it closer to the specific mode described by the text conditioning. We thus expect the MSE to be larger for well-aligned text conditioning. For an unaligned text prompt, the model may have more difficulty in recognizing relevant visual features in the corrupted image, and therefore generally has a lower computed r align t signal. Therefore maximizing r align t is a tractable proxy for maximizing the alignment between the rendered observation o t+1 and the provided text prompt y.We also wish to encourage behaviors that are natural to human perception (e.g. a humanoid should walk similar to how a typical pedestrian would walk). We approximate the naturalness of a behavior by how accurately the diffusion model is able to predict the exact source noise vector that was applied. Intuitively, if it voluntarily predicts the exact noise vector with informative text conditioning, thereby perfectly reconstructing the query image, then the diffusion model believes the original rendered frame is reasonably natural (according to the priors captured by the diffusion model). We would therefore like to minimize ∥ε ϕ (õ t+1 ; t noise , y) -ϵ 0 ∥ 2 2 . We would also like this term to be comparatively closer to the source noise vector than the unconditional prediction is, further reaffirming the benefit of the text conditioning. We therefore seek to also maximize a comparative reconstruction term as below:r rec t = ∥ε ϕ (õ t+1 ; t noise ) -ϵ 0 ∥ 2 2 -∥ε ϕ (õ t+1 ; t noise , y) -ϵ 0 ∥ 2 2 .Algorithm 1 Text-Aware Diffusion for Policy Learning (TADPoLe)1: prompt = sample(action_phrase) 2: π θ = initialize(θ) 3: D ← {} 4: while not converged:5:s 0 ∼ p(s 0 ) 6:for t in range(episode_length):7:a t ∼ π θ (a t | s t ) 8: s t+1 ∼ P (s t+1 | s t , a t ) 9:ϵ 0 ∼ N (ϵ; 0, I) opt.apply_gradients(grads, θ)Ultimately, we compose these two terms into a final reward signal r t exposed to the policy during training. We scale each of the individual terms with tunable hyperparameter constants, and apply a symlog [13] transformation operation:r t = symlog(w 1 * r align t ) + symlog(w 2 * r rec t ). The choice of using symlog as a reward normalization technique is thoroughly studied in Section 4.5. TADPoLe is agnostic to the specific choices of policy network architecture and optimization objectives. A pseudocode of the method is provided in Algorithm 1. It is worth emphasizing that t noise and subscript-less t refer to different notions of time; t indexes the timestep of the agent in the environment, whereas t noise determines the level of noise to corrupt the raw observed image.Conceptually, there exist fundamental limitations to using a text-to-image model to provide a reward signal. As each image is evaluated statically and independently, we are unable to expect the textto-image diffusion model to be able to accurately understand and supervise an agent in learning notions of speed, or in some cases, direction, as such concepts require evaluating multiple consecutive timesteps to deduce. We therefore propose Video-TADPoLe, where a dense text-conditioned reward signal is calculated over sliding windows of consecutive frames through a pretrained text-to-video diffusion model. We extend and generalize the reward formulation from TADPoLe thusly.We can compute reward terms for arbitrary start index i and end index j inclusive, for i ≤ j, by considering the sequence of subsequently rendered frames o [i+1:j+1] . We once again utilize source noise vector ϵ 0 ∼ N (ϵ; 0, I j-i+1 ) to produce noisy observation õ[i+1:j+1] . Then, we can compute a batch of alignment reward terms through one inference step of the text-to-video diffusion model as:r align [i:j] = εϕ (õ [i+1:j+1] ; t noise , y) -εϕ (õ [i+1:j+1] ; t noise ) 2 2, and a batch of reconstruction reward terms as:r rec [i:j] = εϕ (õ [i+1:j+1] ; t noise ) -ϵ 0 2 2 -εϕ (õ [i+1:j+1] ; t noise , y) -ϵ 0 2 2 .For a desired context window of size n, we then calculate the reward at each timestep t utilizing each context window that involves achieved observation o t+1 :r t = 1 n n i=1 symlog w 1 * r align [t-i+1:t-i+n] [i -1] + symlog w 2 * r rec [t-i+1:t-i+n] [i -1] .Intuitively, we seek to calculate an overall reward for an action based off how well the resulting rendered frame aligns with text-conditioning at the beginning of a motion sequence, the end of one, and arbitrarily inbetween. For window size n = 1, this recreates TADPoLe behavior, but using a text-to-video model; for n > 1, we make the computation tractable through dynamic programming.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: We do not include any theoretical results in this work.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We now demonstrate the effectiveness of TADPoLe on goal achievement, continuous locomotion, and robotic manipulation tasks. All results are achieved without access to in-domain demonstrations.Benchmarks: We present our main results using the Dog and Humanoid environments from the DeepMind Control Suite [39] , and robotic manipulation tasks from Meta-World [42] . Dog and Humanoid are known to be challenging due to their large action space, complex transition dynamics, and lack of task-specific priors (such as termination conditions). We update the environments by modifying the terrain rendered by MuJoCo [38] to have green grass and blue sky. We also limit the number of environment timesteps to be 300, which is sufficient to demonstrate successful learning of a behavior, rather than the default 1000. The agent's initialized joint configurations are also fixed, as we focus on learning text-conditioned capabilities rather than robustness to initialization conditions. Meta-World was initially designed for multi-task and meta-reinforcement learning, and was later adopted to evaluate language-conditioned imitation learning algorithms [23, 22, 37] . We select a suite of tasks which are balanced for diversity and complexity, and pair each task with a text prompt (see tasks and their corresponding prompts in Appendix C). Following prior design [18] for Meta-World, we also add a sparse success signal to the dense text-conditioned reward signals.Implementation: We use TD-MPC [14] as the reinforcement learning algorithm for all tasks. It is the first documented model to solve Dog tasks when ground truth rewards are available for walking. We fix the hyperparameters to the default ones recommended by the TD-MPC authors (see Table A3 in Appendix) for all experiments unless otherwise mentioned. We train Humanoid and Dog agents for 2M steps, and Meta-World agents for 700K steps. For Meta-World experiments, we scale the sparse success signal by 2. Visualizations and quantitative evaluations are reported using the last checkpoint achieved at the end of training. We use StableDiffusion 2.1 [31] as the text-to-image diffusion model (∼1.3B parameters), and AnimatedDiff [12] Baselines: We compare TADPoLe against other text-to-reward approaches, including VLM-RM [30] , LIV [20] , and Text2Reward [41] , on top of the same underlying TD-MPC architecture, hyperparameters, and optimization scheme for fair comparison. For LIV, we use their provided CLIP-based checkpoint finetuned on robotic demonstration videos. For VLM-RM, we utilize the ViT-bigG-14 CLIP checkpoint (∼1.3B parameters), reported as the best performing in their work. We follow a prompt template provided in the Text2Reward paper to generate reward functions for the Dog and Humanoid agent, interfaced through vanilla ChatGPT using GPT-3.5. Whereas VLM-RM and LIV provide a multimodal reward signal, and are more directly comparable to TADPoLe, it is of note that Text2Reward generates a text-conditioned reward function purely as the output of a pretrained language model. However, Text2Reward does have access to underlying sensor data such as speed and direction in real-time, whereas the visual interface approaches, including TADPoLe, do not.We benchmark all text-conditioned methods with a corresponding standardized prompt for fair comparison, and report both quantitative as well as qualitative comparisons.We use cumulative ground-truth rewards as quantitative evaluation metrics for Dog and Humanoid when it is available. We note this is a naturally unfavorable comparison for methods that provide a text-conditioned reward signal purely through a visual interface, as the reward the agent receives has no access to the underlying sensors (such as ones that measure speed and energy usage) that the ground-truth reward function uses to evaluate performance. For example, the ground-truth reward function may have an arbitrary threshold on a speed sensor that needs to be hit to constitute successful "walking", and a separate threshold for "running"; however the detailed characteristics of and even existence of such a sensor, as well as any thresholds surrounding it, are hidden for policies supervised only through vision and language feedback. Nonetheless, it offers a standardized, numerical comparison across all methods. For Meta-World, we report the "success rate" evaluation metric, computed as the proportion of evaluation rollouts in which the agent successfully completes the given task. A main benefit of utilizing a reward signal conditioned flexibly on text is the ability to learn policies with behaviors beyond those defined by existing ground-truth reward functions. As these have no corresponding ground-truth reward functions, quantitative comparison across different textconditioned methods is challenging; we therefore appeal to a qualitative user study. We perform a paid study through the Prolific platform, with a total of 25 anonymous random participants without prior training to estimate a general response from the human population. For a video demonstration from each trained model, selected as the last timestep of policy training without cherry-picking, each participant is asked if it sufficiently aligns with the text prompt it was conditioned on. These results are depicted in tables as checkmarks (✔) and x-marks (✘), where a checkmark denotes if a majority of participants believe it is text-aligned. In Table A7 , we provide the fine-grained user study results on what percentage of the users believe the video achieved by the policy is appropriately text-aligned.We then proceed with a user study regarding naturalness. Given a video produced by VLM-RM and TADPoLe, users are given a choice as to which they believed to be the more natural motion or pose. This seeks to approximate how naturally the resulting Humanoid and Dog policies behave, according to human belief over how people and dogs naturally move in the real world.For text-conditioned goal-achievement, the objective is to learn a policy to consistently achieve a particular pose described by a text prompt; as the emphasis is for every frame to match a fixed goal pose rather than performing continuous motion, it is natural to apply TADPoLe with text-to-image diffusion models. We set the noise level t noise ∼ U (400, 500), with intuition provided in Appendix A.In the Humanoid environment, there is a ground-truth reward function that measures standing performance, as a function of the straightness of the agent's spine. We therefore compare all textconditioned methods using the provided reward function as a quantitative metric, with a standard prompt of "a person standing"; these results are shown in the first row of Table 1 . TADPoLe and VLM-RM achieve competitive quantitative performance with an agent trained on the ground-truth reward function. The following rows show that according to the user study, TADPoLe consistently achieves text-aligned behaviors beyond making the Humanoid stand, whereas other approaches often fail. Table 2 shows that users consistently found TADPoLe to produce more natural-looking motions and poses when compared head-to-head with VLM-RM. We then investigate whether or not TADPoLe is sensitive to subtle variations of the input prompt. We change the text prompt from "a person standing" to "a person standing with hands above head" and "a person standing with hands on hips". In Figure 4 , we visually verify that the resulting Humanoid policy can indeed learn distinct behaviors that respect the different hand placement specifications.We take this as evidence that TADPoLe is capable of respecting fine-grained details and subtleties of the input prompts when learning text-conditioned policies.We further explore the ability of TADPoLe to learn continuous locomotion behaviors conditioned on natural language specifications. Such tasks are often difficult to learn purely from static external description, as there is no canonical pose or goal frame that if reached, would denote successful achievement of the task. This is challenging for approaches that statically select a canonical goalframe to achieve, such as CLIP or LIV, and we propose Video-TADPoLe, which leverages large-scale pretrained text-to-video generative models, as a promising direction forward.We utilize a noise level t noise ∼ U (500, 600) in our continuous locomotion experiments. We perform a search over context windows of size n = {1, 2, 4, 8}, and report the best configuration per task. We observe that when the context window is too high (e.g. 8 or higher), the agent has consistently lower performance, and that although the agent learns coherent motion and repeats it, the pose is less text-aligned. For fair comparison against a text-video alignment model trained in a contrastive manner, we extend VLM-RM to ViCLIP-RM, where a ViCLIP-L-14 checkpoint [40] finetuned from ViT-L-14 CLIP is used to compute dense, text-conditioned rewards. At each timestep t, we compute dense rewards as cosine similarity between the encoded representations of video observation up to t + 1 and the text prompt. We ask ViCLIP to encode 8 video frames at a time, which is adopted by its authors for zero-shot experiments.For the Humanoid task, we find that Video-TADPoLe achieves the best results amongst methods trained purely from visual and/or language feedback as in Table 3 . On the other hand, ViCLIP-RM indeed learns to take steps, but does so sideways while maintaining an unnaturally lopsided pose. Meanwhile, LIV and Text2Reward fail to learn meaningful behaviors.For the Dog task, we notice that the policy learned via ViCLIP-RM collapses; it learns to strike a particular pose and maintain it for perpetuity. Text2Reward, which does not have access to any visual information, but does have access to ground-truth state information for the Dog including speed, direction, and joint positions, achieves a reward of 63.15. Ultimately, Video-TADPoLe achieves a comparable result using a context window of 4, while also distinguishing itself as the most naturallooking policy in qualitative terms as the Dog agent appears to perform step-taking motions, rather than remain stationary. Table 2 further showcases a higher preference for the naturalness of the learned policies for continuous locomotion achieved by Video-TADPoLe compared to ViCLIP-RM, in both Dog and Humanoid environments.In Figure 5 , we visualize episode return curves achieved by Video-TADPoLe for a Humanoid agent with the prompt "a person walking". We visualize how during training, the computed Video-TADPoLe reward shares a positive correlation with the reward computed by a ground-truth function for walking over all episodes, lending confidence to it as a coherent, well-defined reward function. We also visualize the ground-truth evaluation curve. Figure A4 offers another example for "a dog walking". Table 4 : Average success rate for robotic manipulation tasks in Meta-World [42] over 30 evaluation rollouts. We compare between TADPoLe and VLM-RM, both approaches that do not utilize indomain data or demonstrations, and find TADPoLe significantly outperforms VLM-RM. We report mean performance and standard deviation across 10 seeds for each task. We further investigate how well TADPoLe can be applied to learn robotic manipulation tasks through dense text-conditioned feedback. We do so by replacing the manually-designed ground-truth dense reward for each Meta-World task with TADPoLe's text-conditioned reward. Since TADPoLe aims to leverage domain-agnostic diffusion models for policy learning, we focus our evaluation to compare with baseline methods that also do not utilize in-domain (expert) demonstrations for the robotic manipulation tasks. We note that most of the prior methods which report performance on Meta-World rely on (often expert-produced) video demonstrations from a similar domain or the target environment directly for representation learning [23] , reward learning [18] , or both [20] . They are thus not directly comparable to TADPoLe.We perform thorough comparisons between TADPoLe and VLM-RM by evaluating them on a diverse set of selected Meta-World tasks. Both models reuse the setup in Section 4.2 without modification, with training performed for 700k steps. In Table 4 , we report the final success rate for each manipulation task averaged over 30 evaluation rollouts. We highlight that TADPoLe achieves high success rates across a variety of tasks, and significantly exceeds VLM-RM in terms of average overall performance. We take this as a positive signal that TADPoLe can meaningfully provide dense text-conditioned rewards that replace dense ground-truth hand-designed feedback. We also highlight how TADPoLe is able to successfully supervise the learning of policies within the synthetic-looking visual environment of Meta-World without finetuning the pretrained text-to-image diffusion model, despite the visual attributes (such as the appearance of the robotic arm, or the quality of the renderings) being quite dissimilar from the style of images StableDiffusion was trained on.Of interest is what reward normalization technique is most performant for adjusting the raw computed alignment and reconstruction terms into a final reward used for policy learning. We investigate a A4 .Apart from the symlog transformation, we also compare against using no additional normalization (denoted as "Direct Scaling"), and using symexp. We also compare against empirical normalization techniques. This includes min-max rescaling, where an empirically estimated minimum value is subtracted from the achieved reward, which is then divided by an empirically calculated min-max range, and rescaled to [-1, 1]. This also includes standardization, which subtracts an empirically estimated mean from the achieved reward and then divides it by an empirically estimated standard deviation. We apply these techniques across Humanoid and Dog environments, for both TADPoLe and Video-TADPoLe. We discover that the symlog operation is the reward normalization strategy that achieves the best empirical results across robotic configurations, visual environments, and desired tasks, while reusing the same hyperparameter settings. We hypothesize that it helps to normalize the raw computed reward signals across diffusion models and environments to be roughly on the same scale. Indeed, we showcase how removing the symlog transformation, as well as using other normalization techniques, reduces consistent policy learning performance across environments and diffusion models with the same fixed hyperparameters. We further note that it does not require empirically estimated values, unlike min-max and standardization.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We clearly describe our approach in the Method section (Section 3) and provide implementation details and evaluation protocols in the Experiments section (Section 4. In addition, we provide detailed hyperparameters in the Appendix (Section B). Our experiments are performed using publicly available pretrained diffusion models and algorithms, and our codebase will be released publicly to support full reproducibility. 
conclusion and future work: 
 We present Text-Aware Diffusion for Policy Learning (TADPoLe), a framework that optimizes a policy according to a provided natural language prompt through a pretrained text-conditioned diffusion model. TADPoLe enables novel behaviors to be learned in a zero-shot manner purely from text conditioning, and also offers a promising angle to train policies to behave in accordance with natural priors summarized from large-scale pretraining. TADPoLe can be applied across visual environments and different robotic states without modification, and we experimentally demonstrate that TADPoLe is able to learn novel goal-achievement as well as continuous locomotion behaviors conditioned only on text, across Humanoid, Dog, and Meta-World environments.Limitations: An observed limitation of TADPoLe is that it is difficult to explicitly control the weight each individual word of an input text prompt has on the reward provided to the agent. For certain prompts, TADPoLe could potentially cause the agent to remain stationary since it may focus on alignment with the noun in the phrase rather than details of the goal. How to provide fine-grained control over the text-conditioning is an interesting direction to explore in future work. Further interesting future work includes utilizing multiple camera views simultaneously to compute the dense reward, as environments generally allow flexible rendering from arbitrary angles. Another observed limitation is that TADPoLe depends on a highly stochastic operation, namely, repeatedly resampling a Gaussian source noise vector at each timestep. The behavior of the resulting policy, after training for many iterations, can therefore vary for the same input text prompt, and potentially cause high variance in both visual and quantitative performance. How to control the stability of convergence to a consistent policy across repeated runs is an interesting future direction for exploration. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in this paper adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There are no extremely noteworthy positive or negative societal consequences of our work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not release data or models with a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We desire a reward signal that is high when the text prompt is well-aligned with the frame rendered following a well-selected action. However, we find that using too high or too low a noise level will cause the reward computation to ignore or discount the provided text prompt.For a sufficiently low noise value, the conditional and unconditional noise prediction is similar for a particular text prompt and rendered frame that already has high cross-modal alignment. Being already close in data space to the desired mode described by the text prompt, the unconditional prediction will also cheaply seek to denoise the input towards the original input. Therefore, too low a noise value may cause the computed reward signal to decrease as cross-modal alignment increases, which runs counter to what is desirable. This is supported by looking at the left tails of the two graphs depicted in Figure A1 , where the difference in computed TADPoLe rewards between well-aligned paired inputs and misaligned paired inputs is small.Similarly, choosing a sufficiently high noise value may also cause unconditional and conditional predictions to be similar. For a given noisy input with virtually all of the spatial structure perturbed, the pretrained denoising model intuitively makes denoising predictions that fill in general structure to the image irrespective of text-conditioning. This is intuitively similar to the behavior of an unconditional prediction. Once again, we can visually observe this in the right tails of both graphs in Figure A1 ; for high noise values, both misaligned and well-aligned paired inputs have similar computed TADPoLe rewards.Intuitively, TADPoLe is able to meaningfully quantify text-conditioned alignment with rendered observations into a reward signal for a noise level range approximately in the middle. For a balanced level of noise corruption on a text-aligned rendered observation, there is enough existing visual structure to help the denoising model make a meaningful text-conditioned prediction close to the original input, whereas an unconditional prediction may not do so accurately. In Figure A1 , we observe that for a noise range in the middle, TADPoLe is able to substantially favor the well-aligned pair over the misaligned pair, respecting changes in both text prompt as well as rendered observations. We verify in our experiments that using a noise level between 400 and 500 works well for TADPoLe, across both Dog and Humanoid environments.Figure A1 : Noise range intuition for a fixed image but two distinct prompts (left), and for a fixed prompt but two distinct images (right). Through visualization, we verify that U (400, 500) is a reasonable range from which to sample noise levels that can meaningfully distinguish vision-text alignment for arbitrarily rendered frames.We discover that Video-TADPoLe achieves better performance with a higher noise level than TAD-PoLe, and use a range of 500 to 600 in our experiments. We hypothesize that being able to observe multiple frames at once enables the space-time U-Net to exploit uncorrupted (or lesser-corrupted) portions from adjacent frames across the context window to inform how each individual frame should be denoised coherently. Because each frame in a context window has a distinct source noise applied to it, structural information can be leaked from randomly uncorrupted portions of the other frames in the context window when making predictions. Therefore, to make the denoising prediction more challenging, thereby forcing it to rely on the text-conditioning and learned motion priors rather than what is readily extractable from the provided context, generally a higher noise level is acceptable compared to TADPoLe.In our experiments, we place less emphasis on the underlying architecture and optimization scheme, instead comparing our method against other text-conditioned reward functions. We keep the same reinforcement learning architecture, optimization scheme, and text-to-image/text-to-video generative model in all environments, unless otherwise noted. We provide detailed hyperparameters about these existing components below.We utilize a StableDiffusion v2.1 checkpoint for our TADPoLe experiments. We provide the sizes of the components in Table A1 . For Video-TADPoLe experiments, we use a pretrained AnimateDiff checkpoint, and provide relevant details in Table A2 . We do not update or modify either checkpoint during our training, utilizing them purely for inference.We include the default hyperparameters from the TD-MPC implementation in Table A3 for completeness. We do not modify the default recommended settings for both Humanoid and Dog environments, as well as the Meta-World experiments. We select hyperparameters w 1 and w 2 , which scale the alignment term r align and the reconstruction term r rec respectively, firstly such that the resulting computed reward r has value roughly between 0 and 1 at any arbitrary timestep. This is visualized in Figure A1 , where over all noise levels, the computed reward stays roughly between 0 and 1. Because the same pretrained text-conditioned diffusion model is used across all environments without modification, these hyperparameters can be generally reused to achieve the same kind of reward scale across environments. We found that the values of w 1 = 2000 and w 2 = 200 through a light hyperparameter sweep, reported in Table A4 , and indeed verify that they work well without modification across both Humanoid and Dog environments in our main experiments. We visualize the complete suite of selected Meta-World tasks in Figure A2 along with the official names of the tasks. In Table A5 we also list the corresponding prompts consistently utilized across all text-conditioned approaches. In Table A6 , we list the average success for all 12 robotic manipulation tasks. As we add a sparse "success" signal to text-conditioned approaches such as VLM-RM and TADPoLe, following prior experimental design [18] for Meta-World, we also list the performance of utilizing the sparse signal only.We first notice that no approach is able to solve two selected tasks, "Peg Insert Side" and "Shelf Place"; these were therefore omitted in Table 4 . Furthermore, we observe that utilizing a sparse reward signal only appears to have strong default performance, and achieves a higher overall average success rate than TADPoLe or VLM-RM. This showcases the inherent power of a sparse reward in solving Meta-World tasks. However, there are two additional takeaways; firstly, VLM-RM surprisingly achieves a substantial decrease in performance from the default sparse reward signal, highlighting TADPoLe as a more preferable dense text-conditioned reward provider. Secondly, TADPoLe is able to solve more overall tasks than other approaches. In particular, TADPoLe can solve the "Door Open" task, which is completely unable to be solved by VLM-RM or using a sparse reward only. We therefore highlight TADPoLe as a promising text-conditioned reward signal in replacing ground-truth hand-designed feedback, particularly as the text-to-image diffusion model utillized was pretrained in a general manner, without finetuning explicitly on Meta-World demonstrations. Table A6 : Average success rate for 12 robotic manipulation tasks in Meta-World [42] over 30 evaluation rollouts. We compare between TADPoLe and VLM-RM, both approaches that do not utilize in-domain data or demonstrations, and find TADPoLe significantly outperforms VLM-RM. We report mean performance and standard deviation across 10 seeds for each task. In our experiments, we compare against other methods that provide text-conditioned rewards. As each of these methods are formulated differently, the difference in scales naturally prevent easy direct comparison. However, plotting the training curves for TADPoLe does yield insights into its speed of convergence, and investigating the visual performance at intermediate steps can be interesting. In Figure A3 we showcase the TADPoLe training curves for a variety of novel text-conditioned policies. We highlight the performance at steps 500k, 1M, 1.5M, and 2M in red; we further visualize the policy achieved at these intermediate steps by showcasing the last frame of the achieved video.In the case of the policy learned for the prompt "a person standing with hands on hips", for example, we see that whereas the initial policies fail, the policy first learns to stand up by step 1.5M. Then, by the last training step, the policy has learned to correctly place its hands conspicuously on its hips. Similarly, for the prompt "a person kneeling", the policy first learns to kneel on all fours; by the end, the policy learns to successfully kneel on one knee. Table A7 : Numerical results from the user study on text-alignment. Out of 25 anonymous users, for each method, we report the percentage that believe the video achieved by a policy at the end of training is aligned with the provided text prompt. Entries above 50% are displayed in Table 1 as checkmarks (✔); below 50% are displayed as x-marks (✘). For further intuition on TADPoLe's behavior, we visualize the complete denoising results for a query video achieved through a Dog policy and corrupted with some level of noise, conditioned on a consistent text prompt of "a dog walking". Note that this is purely for visualization purposes; in practice, when training policies, we do not visually denoise over multiple steps but instead extract a reward signal from components of a one-step denoising prediction. However, the multi-step denoising visualizations can offer us a glimpse of intuition into the behavior of the pretrained diffusion model and the properties of its predicted components at one single denoising step.We first find that StableDiffusion, used in TADPoLe, is able to reconstruct a well-aligned policy rollout (a video of the Dog agent actually walking) relatively accurately frame-by-frame, for a noise level of 500 (Figure A5 ). However, the predictions can differ substantially for a misaligned policy rollout (a video of the Dog agent falling over rather than walking); for frames where the dog has collapsed on the floor, the StableDiffusion model still tries to respect the specified input prompt and attempts to predict dogs standing upright or walking, as depicted in Figure A6 , resulting in a more noticeable difference between the achieved frame and predicted frame. This difference can be exploited to distinguish between well-aligned and misaligned policy rollouts, lending confidence to StableDiffusion as a supervisory signal for text-conditioned policy learning. For a higher noise level, StableDiffusion quickly forms hallucinatory final predictions (depicted in Figures A7, A8 ), but preserves more of the structure of the original video when the achieved video and provided text prompt is aligned.We also visualize reconstruction predictions using a text-to-video model, namely an AnimateDiff v2 checkpoint, in Figures A9 and A10 . We find that the motion prior is indeed meaningful, enabling it to reconstruct the Dog with a coherent texture and pose over time. In Figures A11, A12 we highlight how a text-to-video model is more robust in reconstructing the achieved video by the policy at high noise levels compared with StableDiffusion (Figures A7, A8 ). This supports the hypothesis outlined in Section A.1. Justification: In both the abstract and introduction, we state our main contribution -we propose Text-Aware Diffusion for Policy Learning (TADPoLe) as the first approach to leverage domain-agnostic visual generative models for policy learning. TADPoLe uses a large-scale pretrained text-conditioned diffusion model in a discriminative manner to enable the learning of novel, zero-shot policies that are flexibly and accurately conditioned on natural language inputs, across multiple robot configurations and environments, for both goal-achievement and continuous locomotion tasks. We conduct thorough qualitative and quantitative evaluations across various benchmarks to experimentally support our claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. Justification: In the Experiments section (Section 4) we report that all of our experiments are performed on a single NVIDIA V100 GPU.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 


Decision: This paper has mixed scores of (4,4,6,7), with a variety of points made by the reviewers about its strengths and weaknesses. Overall, I have some doubts about some of the crucial points made by a few of the low-scoring reviews for a few reasons, finding that the main concerns were either resolved or not in conflict with the main contributions of the paper. I generally agreed with the remaining points made (positives and negatives alike). Taken together, I recommend acceptance, and for the authors to ensure that they address the remaining concerns of the reviewers. Below, I summarize some points made by reviewers that I used to arrive at this recommendation.

## Reviewer z3di (4)
This reviewer was unconvinced by the author responses, and stated: 
> I don't think the videos shown by the authors are convincing ... The results using DrM on Adroit are also confusing.

and

> The authors said that 'prior work on MetaWorld mainly use dense ground-truth rewards, in-domain expert demonstrations, or both'. However, they mention DrM in their other replies, which uses sparse rewards and achieves much better performance."

I share these concerns, but find them minor, because they are not in conflict with the main claims of the paper ("our method is novel in its reward computation, as well as its utilization of a domain-agnostic generative model" ... "first approach to leverage domain-agnostic visual generative models for policy learning."). Furthermore, earlier in the review process, this reviewer stated

> As for the reason for real-world experiments, diffusion-reward actually added a lot of real-world experiments. Without real-world experiments, I think the contribution to the community is very limited. So I decide to maintain my score.

The authors responded to this criticism by pointing out "the referenced paper does not train any policies for real-world robots; the only policies they train are in simulation, which we also do in this work." I therefore consider this concern addressed, which was a major factor why this reviewer maintained their (4) score when they stated this concern.

## Reviewer iiYV (6)
This reviewer stated "Although I think this work still requires heavy prompt engineering and the generalization ability is weak, I would raise the score to 6."

## Reviewer TWnR (4)
This reviewer, when nudged for their final evaluation, stated there are still "questions about their work's performance and the true source of the observed performance improvement." This concern stems from the ablation study that demonstrated the importance of the 'symlog' transformation. However, this finding is consistent with the paper's claims ("our method is novel in its reward computation, as well as its utilization of a domain-agnostic generative model" ... "first approach to leverage domain-agnostic visual generative models for policy learning.") and in my opinion shouldn't count significantly against it. One issue I have with this finding, though, is that the current presentation of the paper does not make clear that it was discovered that the symlog operation was important as shown by Table A6 in the author's response, so it needs to be updated to reflect this finding. Furthermore, Table A6 is not a complete ablation across all environments and prompts. A more complete Table A6 would offer more evidence of this finding, and I strongly recommend the authors perform a more complete ablation. Whether additional ablations would support or undermine this finding it is irrelevant to the paper's main claims, and thus I discount this criticism a fair bit.

## Reviewer KsL9 (7)
This reviewer stated a mostly positive assessment with few minor weaknesses.

Reviews: review: paper_summary: paper_summary: The paper introduces Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a large-scale pretrained text-conditioned diffusion model to provide zero-shot reward signals for training agents without expert demonstrations or manually designed reward functions. TADPoLe enables agents to learn behaviors and achieve goals specified by natural language in various tasks, demonstrating its effectiveness.
strength_weakness: strength_weakness: 1. Although the text-aware diffusion reward is novel, both text-aware rewards and diffusion rewards have been proposed by prior works. The paper lacks an apple-to-apple comparison, making it difficult to discern the specific advantages of using diffusion models.

2. I find the experiments to be quite limited. For example, the authors only use TD-MPC as the algorithm backbone, which is just one model-based RL algorithm and does not have a significant advantage in visual RL. Additionally, the curve shown in Figure 5 is quite odd—why not place the baseline and proposed method on the same graph and include curves for other prompts? Furthermore, the comparisons in the Metaworld experiments are minimal, making the experimental results less convincing.

3. The absence of real-world experiments makes it hard to assess the paper's contribution to the community.

4. The authors use a diffusion model to generate rewards at each step, but there is no detailed analysis of the computational cost and its impact on speed.
questions: questions: 1. Even with sparse rewards in MetaWorld, the state-of-the-art results are better than those presented by the authors. 
2. Is the naturalness in Table 2 determined through user surveys? I believe that videos would provide more compelling evidence.
3. How do different lengths and forms of prompts affect the rewards? What impact would slight adjustments to the prompts during training have?
limitations: limitations: The authors discuss limitations in the paper about how to control the weight of each individual word in the prompt. However, I believe that the primary limitations lie in the efficiency of the diffusion model. This paper has many aspects that require further discussion and improvement.
paper_summary: paper_summary: The paper presents Text-Aware Diffusion for Policy Learning (TADPoLe), a framework that leverages pretrained text-conditioned diffusion models to generate dense, zero-shot reward signals for policy learning in reinforcement learning tasks. The approach aims to address the limitations of manually designed reward functions by utilizing large-scale generative models to encode rich priors that guide policies in a natural and text-aligned manner. Experiments demonstrate TADPoLe’s effectiveness in learning policies for novel goals and continuous locomotion behaviors in various environments, including humanoid, dog, and robotic manipulation tasks, without ground-truth rewards or expert demonstrations.
strength_weakness: strength_weakness: **Weaknesses:**

- **Evaluation Metrics:** While the paper provides qualitative evaluations, the reliance on human judgment for assessing the naturalness and alignment of behaviors could introduce subjectivity.
- **Scalability of Text Prompts:** The approach may struggle with very complex or ambiguous text prompts, and the extent to which it can handle highly detailed or context-specific instructions is not fully explored.
- **Comparative Baselines:** The comparisons with other text-to-reward methods are limited, and it would be beneficial to include more diverse baselines to better understand the relative performance of TADPoLe.
- **Computational Overhead:** The approach involves significant computational overhead due to the use of large-scale diffusion models, which could limit its practicality in resource-constrained environments.
questions: questions: ### Questions

1. **Reward Scaling:** How does TADPoLe handle the scaling of reward signals across different tasks and environments to ensure consistent policy learning?
2. **Text Prompt Complexity:** Can the method be extended to handle more complex and detailed text prompts? What are the limitations in terms of prompt complexity and specificity?
3. **Long-Term Dependencies:** How does TADPoLe perform in tasks requiring long-term planning and dependencies? Are there any mechanisms to ensure temporal coherence in learned behaviors?
4. **Generalization:** How well does the approach generalize to unseen environments or tasks that are significantly different from the training scenarios?
limitations: limitations: Same as the above section.
paper_summary: paper_summary: The paper introduces Text-Aware Diffusion for Policy Learning (TADPoLe), a method for reinforcement learning that leverages pretrained text-conditioned diffusion models to compute dense reward signals. This approach allows agents to learn text-aligned behaviors without the need for expert demonstrations or handcrafted reward functions. The authors demonstrate the effectiveness of TADPoLe in various environments, including Humanoid, Dog, and Meta-World, achieving zero-shot policy learning with natural language inputs.
strength_weakness: strength_weakness: 1. Presentation Issues: The paper's presentation is problematic. The integration of components such as the 'symlog' of the reward function and the 'noise level range' is not clearly explained. It is difficult to understand the necessity and utility of these components and how much performance improvement depends on them. Additionally, the paper lacks clarity on the source of ground-truth for rendered subsequent images and whether the method requires pre-existing well-rendered videos for each environment for TADPoLe training.

2. Alignment with Motivation: The motivation to leverage text-conditioned diffusion models for reward generation is clear; however, the demand for well-rendered videos for each environment does not fully align with the goal of making reinforcement learning more practical and scalable. Some environments donot have existing well-performed video for the agent, so will this method require for a pre-trained policy to collect the video data?

3. Experimental Insufficiency: The experiments are not comprehensive. There is a need to expand the range of tasks to provide a more thorough evaluation of the method. Additionally, the paper had better include comparisons with the Diffusion-Reward method [1] to highlight the advantages of TADPoLe. 

4. The ablation study is also insufficient. It should further explore the design of the 'symlog' component and the selection of the two weights, as these operations lack detailed justification. This analysis is crucial to understand the contribution of each part to the overall performance and to provide a clearer justification for their inclusion.

5. Overclaim. The writing of the paper is dense and lacks clarity in several key areas. The explanations of the methodology and components are not sufficiently detailed, making it challenging for readers to grasp the full picture of how TADPoLe functions and why certain design choices were made. Improving the clarity and coherence of the writing would significantly enhance the paper's readability and accessibility.

[1] Huang, T., Jiang, G., Ze, Y., & Xu, H. (2023). Diffusion Reward: Learning Rewards via Conditional Video Diffusion. arXiv preprint arXiv:2312.14134.
questions: questions: 1. Clarification of Component Integration: Why is the 'symlog' transformation necessary? How does it and 'noise level range' specifically contribute to the performance improvements observed? More detailed explanations and justifications are needed.

2. Source of Ground-Truth: Where do the ground-truth rendered subsequent images come from? Does the method rely on pre-existing well-rendered videos for each environment? This aspect should be clarified to understand the method's applicability.

3. Experimental Expansion: To provide a more comprehensive evaluation, expand the range of tasks and include comparisons with the Diffusion-Reward method. This will help to position TADPoLe's performance in a broader context.

4. Ablation Study: Conduct a more detailed ablation study to isolate the effects of the 'symlog' transformation and the specific weightings used. This will help to understand the contribution of each part to the overall performance and provide a clearer justification for their inclusion.

5. Scalability and Practicality: Address the scalability issue regarding the need for well-rendered videos in environments lacking existing well-performed videos. How does the method handle such cases, and what are the implications for its practicality and scalability?
limitations: limitations: The author has addressed a limitation. It's also recommended to add the unsolvable questions to the limitations.
paper_summary: paper_summary: The paper proposed a reward generating pipeline leveraging a text-conditioned diffusion model with a text prompt to master RL tasks described by the prompt.
The pipeline compares the difference of the generated image with/without the prompt and the original image to calculate a dense reward, which also makes sense intuitively.
strength_weakness: strength_weakness: - The proposed method requires more hyperparameter-tuning of the noise level and the overall reward function. Thus, more investigation is needed before training is conducted, such as Appendix A.
- The proposed method heavily relies on the capability of the external diffusion model to obtain world knowledge. It is a natural sin due to the complexity of the tasks the paper is solving.
- Another minor weakness is that some details of the user study are not presented, such as the variance of ✔ and ✘ and scores (instead of giving the majority and avg).
questions: questions: - Is $\hat{\epsilon}_\phi$ a neural network (Sec 3.1) or a noise (Fig.3)?
- What is the cons of replacing the diffusion model in your method with a CLIP-like model to access the alignment level of image and text? One I can think of is the naturalness cannot be calculated in such case.
- Why is VLM-RM outperforming the proposed method in the standing Humanoid environment? Is it due to the training method of VLM-RM or because it is easier?
- Is it possible to give a concrete example of the failure mode mentioned in the limitation section?
- It would be interesting if the authors can try difficult tasks that even the diffusion model does not know how to generate and see the corresponding results of the RL training and evaluation.
limitations: limitations: See weakness 1. The paper concluded most of its limitation in the main text and Appendix.

Comments: title: Response to Reviewer [2]
Response to Reviewer [1]
Response to Reviewer
Response to rebuttal
comment: On additional robotic experiments: In our experimental sweeps, we have been able to get TADPoLe working on three Adroit tasks (door, pen, hammer), in a similar setup with what is reported in MetaWorld (all hyperparameters are kept the same, e.g. noise level, $w_1$, $w_2$, etc.).  For the base RL backbone algorithm, we find the most success on Adroit from applying the text-conditioned TADPoLe reward to DrM [1].  The associated task prompts are “opening a door” and “spinning a pen”, and “hammering a nail” respectively, and we include the success rate and video demonstration on the bottom of our updated [submission website](https://sites.google.com/view/tadpols/home).  We hope these positive results on complex dexterous manipulation tasks provide a more convincing demonstration of TADPoLe’s capabilities to the reviewer.

We also voluntarily report, for complete transparency, that of the FrankaKitchen suite, in which we select five tasks for evaluation following R3M [2], our approach is only able to solve one of the five tasks (turning the light on) when using the same hyperparameter settings as the MetaWorld experiments.  In fact, LIV [3], in Appendix G4, explicitly states that **fine-tuning is needed for language-grounding** for FrankaKitchen tasks (compare Figure 20 against Figure 21), to form a coherent language-goal reward signal for text-conditioned policy learning. In light of this, TADPoLe performance on FrankaKitchen is expected to be difficult, as TADPoLe uses no fine-tuning at all, but still tries to provide a text-conditioned reward.  The performance TADPoLe can achieve on robotic environments (such as Adroit, MetaWorld, and the FrankaKitchen example) while utilizing *no in-domain demonstrations, or fine-tuning whatsoever* is promising, particularly as such robotic experiments generally have a large visual domain gap from the pretraining data observed by the general-purpose diffusion model (and collecting in-domain examples to alleviate the domain gap can be expensive).

We believe our framework is general enough that future improvements to the underlying diffusion model, for example finetuning on in-domain demonstrations, would help - but our focus in this work is on the *inherent generalization capabilities* from *leveraging large-scale pretrained diffusion models*, which we demonstrate is surprisingly powerful.  As our framework still works with arbitrary image and video diffusion models, we leave exploring the utilization of diffusion models that have explicitly integrated in-domain information (such as through explicit finetuning) for exciting future work.

The main focus of our work is finding a general-purpose, scalable solution for text-conditioned policy learning - rather than solving the entire realm of text-conditioned policy learning for robotics at once, or outperforming even models that leverage in-domain demonstrations.  What we seek to show is that text prompts can be converted into dense rewards without in-domain finetuning, just from text-conditioned visual diffusion models pretrained on general data, to enable policy learning.  Our model formulation, as well as our reward computation are novel, and we demonstrate this generalizes across a variety of robotic environments (Humanoid, Dog, MetaWorld), and across a variety of text prompts and novel behaviors without *finetuning or modification*, and with large reuse in hyperparameters.  Our approach seeks to be general, and scalable - motivations that the reviewer has voiced appreciation for, and accomplishes this by avoiding any reliance on in-domain information - but this generality often comes at the cost of achieving state-of-the-art performance on every single domain, which we hope is understandable without detracting from the surprising insights TADPoLe already provides.

We appreciate the reviewer’s suggestions for strengthening our work, and plan to incorporate these updates into the final camera-ready version of the paper.

[1] Xu et al. DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization. ICLR, 2024.

[2] Nair et al. R3M: A Universal Visual Representation for Robot Manipulation. arxiv preprint, 2022.

[3] Ma et al. LIV: Language-Image Representations and Rewards for Robotic Control. ICML, 2023.
We thank the reviewer for the response; we are happy to have addressed many of the prior concerns, and appreciate the increase in score. We would like to supply additional experimental results towards the interest of the reviewer:

**On alternative normalization:** Indeed, we have previously tried alternative normalization techniques for our reward computation. Apart from symlog and using simple scaling factors directly, we have also tried using symexp, min-max rescaling (subtract an empirical min, divide by an empirical min-max range, and rescaling to [-1, 1]), as well as standardization (subtracting an empirical mean and dividing by an empirical standard deviation). What we have found is that the symlog operation overall is the best reward normalization strategy, in terms of transferring hyperparameter configurations across robotic configurations, visual environments, and desired tasks.  We further note that min-max rescaling and standardization require empirically estimated values, whereas symlog does not.  We provide an updated table:

|Task|Prompt|TADPoLe|TADPoLe-simple-scale|TADPoLe-symexp|TADPoLe-min-max|TADPoLe-standardization|Video-TADPoLe|Video-TADPoLe-simple-scale|Video-TADPoLe-min-max|Video-TADPoLe-symexp|Video-TADPoLe-standardization|
|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|-:|
|Humanoid-Stand|“a person standing”|267.23|**276.67**|241.49|256.81|236.59|-|-|-|-|-|
|Humanoid-Walk|“a person walking”|-|-|-|-|-|**226.29**|4.58|3.68|61.31|134.66|
|Dog-Walk|“a dog walking”|-|-|-|-|-|**81.22**|35.30|9.46|6.15|5.05|

**On Figure 3:** we appreciate the reviewer’s presentation suggestions, and agree that moving Figure 3 to the same row as the ones in Figure 2 could allow additional space.  We have already provided experimental outcomes of goal-achieving policies for Figure 1 and Figure 4, and have left continuous locomotion demonstrations for our attached videos on our website.  However, the additional space from moving Figure 3 could be used to explicitly show static frame rollouts corresponding to the videos on the website within the PDF.
We appreciate Reviewer z3di’s prompt response and engagement; we offer further clarification and results on the provided comments:

**On Question 1:** To the best of our knowledge, prior work on MetaWorld mainly use dense ground-truth rewards, in-domain expert demonstrations, or both (e.g. Diffusion-Reward, referenced by the reviewer, uses a combination of in-domain expert demonstrations and sparse rewards). A more comparable setup is their reported performance on “unseen” tasks (Table 7) where they still use in-domain expert demonstrations (which TADPoLe does not need) to train their video model but the tasks are novel. Among the two overlapping tasks (Drawer-Open and Window-Open), both TADPoLe and Diffusion-Reward achieve near perfect performance, outperforming recently published baselines, such as VIPER (which we note also relies on in-domain expert demonstrations). We are unaware of published work that achieves "state-of-the-art" performance under a comparable setup as TADPoLe (using neither in-domain expert demonstrations nor ground-truth dense rewards) - we welcome clarification on the work the reviewer is referring to.

**Beyond TD-MPC:** As explained in our initial rebuttal, our main motivation is comparing different text-to-reward methods on a consistent, powerful, backbone for an apples-to-apples comparison (with hyperparameters, optimization, architectures, etc. fixed). We select TD-MPC for its competitive performance on the complex Dog and Humanoid tasks, which serve as ideal environments to demonstrate **further** flexible novel behavior synthesis conditioned on text and naturalness. For example, for traditional model-free approaches that do not even have basic strong performance on Humanoid-Walk or Dog-Walk, any failure of our method to learn text-conditioned behavior in such environments could potentially be attributed to the underlying model rather than our proposed reward computation scheme. We selected TD-MPC because it has strong base performance on complex benchmarks (e.g. it is the first approach to even solve the default Dog task), and can therefore clearly showcase the difference between distinct text-to-reward computation methods **on top of it**.

Nevertheless, we have discovered in early iterations of our method that TADPoLe is general across RL backbones. We have implemented RPO with TADPoLe (using the CleanRL implementation, where we replace the reward with the text-conditioned TADPoLe reward, and keep all default hyperparameters) and have achieved competitive performance on OpenAI Gym tasks:
|Task|Prompt|TADPoLe|Ground-Truth|
|:-|:-:|:-:|-:|
|Hopper-v3|“a one-legged stick figure jumping forward over a grid floor”|2925.56|3852.78|
|Walker2d-v4|“a stick figure walking across a grid floor”|2038.06|3872.54|

Note that TADPoLe is not trained on ground-truth environmental signals, but still manages to achieve high ground-truth returns despite being optimized purely to align with a provided text prompt. We agree that more explorations on other RL backbones would further support the generalizability of TADPoLe, beyond the generalization we have already shown over different tasks (specified by flexible text prompts), as well as robotic states and environments (Humanoid, Dog, MetaWorld).

**On real-world experiments:** We would like to clarify a potential misunderstanding on the "real-world experiments" presented in Diffusion-Reward: Diffusion-Reward *does not train any policies for real-world robots*; the only policies they train are in simulation, which we also do in this work. Instead, they use real-world videos to visualize their computed rewards in an *offline manner*. They compare reward curves between an expert and a random policy (as Diffusion-Reward does not have text-conditioning capabilities) to “indicat[e] the **potential** of [their] method for real-world robot manipulation tasks.”

Analogously, for the reviewer's interest, we also visualize TADPoLe rewards for natural videos. As we are now interested in text-conditioned rewards, we compare between different text prompts for a given video demonstration, and show that the more-aligned text prompt has higher predicted reward. This is akin to what we have previously reported in Appendix Figure A1, but now for natural videos. Similar to what is reported in Diffusion-Reward, we visualize our computed dense reward for some real-world robotic arm demos from the Bridge Dataset (Diffusion-Reward did not release their robot-arm videos publicly, nor are they text-annotated). We also visualize some human actions. We provide the updated graph comparisons, along with the videos, on the [submission website](https://sites.google.com/view/tadpols/home), and verify that TADPoLe can determine if a natural video is more aligned with a provided text prompt or not, thus further supporting its use as a text-conditioned reward signal for policy learning. With this, we hope the reviewer increases their perception of our contribution to the community.
The reviewer did not respond to my Question 1. 
The explanation of TD-MPC is not sufficient, and no other algorithm backbone is used to prove the effectiveness of the method during the rebuttal period. 
As for the reason for real-world experiments, diffusion-reward actually added a lot of real-world experiments. Without real-world experiments, I think the contribution to the community is very limited. So I decide to maintain my score.
Thanks for the detailed rebuttal.
Dear Authors,

Thank you for your detailed rebuttal and the clarifications provided. I appreciate your effort in addressing my concerns, especially the clarification that there is no need for expert data or pre-existing well-rendered videos. This has resolved a significant part of my concerns, and I am inclined to adjust my rating to 4 in recognition of it.

However, I still have some remaining concerns that I believe are important to address:

Symlog Regularization: The additional results in Table A6 from your rebuttal indeed highlight the significance of the symlog normalization. This brings up a question regarding the flexibility and robustness of your method—have you considered or experimented with alternative regularization methods? It would be insightful to see if other techniques could provide comparable or even improved performance, which would further strengthen the generalizability of your approach.

Experimental Rigor: While I appreciate the addition of Table A8, which compares your method with an adapted version of Diffusion-Reward, I agree with the general sentiment among reviewers that the experimental evaluation remains somewhat weak. The results from MetaWorld, where most tasks either achieve 100% or 0% success rates, suggest a potential limitation in the diversity and complexity of the tasks. Incorporating more challenging environments, such as those involving interactive tasks with Franka-Kitchen or the dexterous manipulation tasks with Adroit, would provide a more convincing demonstration of your method's capabilities. This would be crucial for further validating the practicality and robustness of TADPoLe.

Presentation: Finally, I recommend revisiting the presentation of your paper. Specifically, Figure 3 could be redesigned to locate in a line with Figure 2, allowing for space to provide a more detailed and visually appealing presentation of your experimental outcomes.

Best regards.