Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion
Authors: Yongyuan Liang
Tingqiang Xu
Kaizhe Hu
Guangqi Jiang
Furong Huang
Huazhe Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Policy Learning
Parameter Generation
Reinforcement Learning
Primary area: diffusion_based_models
TL;DR: We introduce a policy network generator that utilizes conditional diffusion models for behavior-to-policy generation.

Abstract: Can we generate a control policy for an agent using just one demonstration of desired behaviors as a prompt, as effortlessly as creating an image from a textual description?
In this paper, we present **Make-An-Agent**, a novel policy parameter generator that leverages the power of conditional diffusion models for behavior-to-policy generation. Guided by behavior embeddings that encode trajectory information, our policy generator synthesizes latent parameter representations, which can then be decoded into policy networks. 
Trained on policy network checkpoints and their corresponding trajectories, our generation model demonstrates remarkable versatility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs. We showcase its efficacy and efficiency on various domains and tasks, including varying objectives, behaviors, and even across different robot manipulators. Beyond simulation, we directly deploy policies generated by **Make-An-Agent** onto real-world robots on locomotion tasks. Project page: https://cheryyunl.github.io/make-an-agent/.

Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion
Yongyuan Liang!?
Tingqiang Xu
Kaizhe Hu
Guangqi Jiang"
Furong Huang"
Huazhe Xu
1 Shanghai Qi Zhi Institute Tsinghua University
2 University of Maryland, College Park Unniversity of California, San Diego
Abstract
Can we generate a control policy for an agent using just one demonstration of desired behaviors as prompt, as effortlessly creating image from textual description? In this paper, we present Make-An-Agent, novel policy parameter generator that leverages the power of conditional diffusion models for behavior-to to- policy generation. Guided by behavior embeddings that encode trajectory informa- tion, our policy generator synthesizes latent parameter representations, which can then be decoded into policy networks. Trained on policy network checkpoints and their corresponding trajectories, our generation model demonstrates remarkable ver- satility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs. We showcase its efficacy and efficiency on various domains and tasks, in- cluding varying objectives, behaviors, and even across different robot manipulators. Beyond simulation, we directly deploy policies generated by Make-An Make-An-Agent onto real-world robots on locomotion tasks.
Introduction
Policy learning traditionally involves using sampled trajectories from replay buffer or behavior demonstrations to learn policies or trajectory models mapping from state to action modeling narrow behavior distribution. In this paper, we consider a shift in paradigm: moving beyond training policy, can reversely predict optimal policy network parameters using suboptimal trajectories from offline data? This approach would obviate the need to explicitly model behavior distributions, allowing to learn the underlying parameter distributions in the parameter space, thus revealing the implicit relationship between agent behaviors for specific tasks and policy parameters.
Using now-dimensional demonstrations (such as agent behavior) to guide the generation of high- dimensional outputs (policy parameters) a challenging problem. When diffusion models [12 20] have demonstrated highly competitive performance on various tasks including text-to-image synthesis, we are inspired to approach policy network generation as a conditional denoising diffusion process. By progressively refining noise into structured parameters, the diffusion-based based generator can discover various policies that are not only superior performance but also more robust and efficient than the demonstration in the policy parameter space.
[8 explore the concept of training hypernetwork to generate weights for another neural network, they primarily use hypernetworks as an initialization network of meta-learning [7] and then adapt to specific task settings. Our approach diverges from this paradigm by leveraging agent behaviors as direct prompts or to generate optimal policies within the
While prior works on hypernetworks I0
'Code, datasct and vidco released in htttss://cheryyulll htw.bheryyynlggggh....o.e... anaanttt/ Conrespondngg to: cheryunl@use @umd. edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

parameter space, without the need for any downstream policy fine-tuning or adaptation with gradient updates. Since behaviors as the observable manifestation of deployed policies from different tasks often share underlying skills environmental information, our policy generator can exploit these potential correlations in the parameter space, such as shared parameters for similar motion patterns, which leads to enhanced cross-task one-shot generalizability. What we need is an end-to-end behavior-to-poliey generator, not a shared base policy.
To achieve this, we introduce Make-An-Agent featuring three key technical contributions: (I) We propose an autoencoder that encodes policy networks into compact latent representations based on their network layers, which can also effectively reconstruct the original policy from its latent representation. (2) We leverage contrastive learning capture the mutual information between long-term trajectories and their success or future states. This approach yields novel and efficient behavior embedding. (3) We utilize a simple yet effective diffusion model conditioned on the learned behavior embeddings, to generate policy parameter representations, which are then decoded into deployable policies using the pretrained decoder. (4) We construct pretrained dataset of policy network parameters and corresponding deployed trajectories to train our proposed methodology.
To investigate the generation performance of Make-An-Agent, we evaluate our approach in three continuous control domains including diverse tabletop manipulation and real-world locomotion tasks. During test time, we generate policies using trajectories from the replay buffer of partially-trail trriined RL agents. The policies generated by our method demonstrate superior performance compared to policies produced by multi-task (28 23 or meta learning 7 [2 and other hypernetwork based generation methods [1.
Our generator offers several key advantages:
Versatility: Make-An-Agent excels in generating effective policies for wide range of tasks by conditioning on agent behavior embeddings. Since we train the parameter generator for latent parameter representations, it can generate policy networks of varying sizes within the latent space, demonstrating scalability.
Generalizability: Our diffusion-basse generator demonstrates robust generalization, yield- ing proficient policies even for unseen behaviors or unseen embodiments in unfamiliar tasks.
Robustness: Our method can generate diverse policy parameters, exhibiting resilient per- formance under environmental randomness from simulators and real-world environments. Notably, Make-An-Agent can synthesize high-performing policies when fed with noisy trajectories, highlighting the robustness of our model.
2 Backgrounds
Policy Learning. Reinforcement Learning (RL) is structured within the formation of Markov Decision Processes (MDPs) [2], which is defined by the tuple M = (S.AA, A, P, R) Here, S signifies the state space, AA the action space, P the transition probabilities, R the reward function and y the discount factor. RL aims to optimize an agent's policy π S + AA, which outputs action a, based on state s, at each timestep, to maximize cumulative rewards. The optimal policy can be expressed as:
π arg max E
(I)
where represents a trajectory generated by following policy v. In deep RI., policies и are represented using neural network function approximations (22). parameterized by 0x. facilitating the learning of intricate behaviors across high-dimensional state and action spaces.
Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) 12 are generative models that frame data generation through a structured diffusion process, which involves iteratively adding noise to the data and then denoising it to recover the original signal. Given sample æp, the forward diffusion process obtain 1 22. II of increasing noise intensity is typically denoted by:
g x Na,,, VI 3,x,-1, B:1),
2
where q is the forward process, M is Gaussian noise, and Br (0, 1) is is the noise variance.

---

The denoising process, which is the reverse of the forward diffusion, can be formulated as:
Pe (It-1 xx) N (#1-1 μg (Ir. Ee),
where po denotes the reverse process, [Ag and Σg are the mean and variance of the Gaussian distribution respectively, which can be approximated by noise prediction neural network parameterized by 0.
Diffusion models aim to learn reverse transitions that maximize the likelihood of the forward transitions at each time step t. The noise prediction network 0 is optimized using the following objective, as the function mapping from eg (z( 1) to po( 1) is closed-form expression:
DDM (0)
+ VI āte, f)]l]
Here, cm (0, I), is the target Gaussian noise, O₁ := IT 1 B, and vā+zo + vI dge the estimated distribution of z; from the closed-form relation.
Although diffusion models are typically used for image generation through the reverse process, variable x can be generalized to represent diverse entities for generation. In this paper, we adapt to represent the parameters Oz of the policy network policy learning.
Methodology
Inference process
Behavior embelddings
Generated deployable policy
Agent's trajectory
Conditional inputs
denoising
decode
Autoencoder
Latent Diffusion Model
Gaussian noise
noising
encode
vectorize
Latent parameter representations
Forward process
from training dataset
In the inference process policy parameter generation, conditioning on behavior embeddings from the agent's trajectory, the latent diffusion model denoises random noise into a latent parameter representation, which can then be reconstructed as a deployable policy using the autoencoder. The forward process for progressively noising the data is also conducted on the latent space after encoding policy parameters as latent representations.
Figure 1: Overview:
Autoencoder
Long trajectory steps
slep
eacoding
vectorired
mooks
Kem
Projection
Projection e
Policy network parometers
Contrastive Loss
Latent Representations
decodting
Behuvior embeddiings
Figure 2: Autoencoder: Encoding policy param- Figure 3: Contrastive behavior embeddings: eters into the latent space and decoding latent pa- Learning informative behavior embeddings from rameter representations into policy networks. long trajectories with contrastive loss.
Overview. An overview of our proposed methodology is illustrated in Figure To achieve this, we address several key challenges: (1) Developing latent representations of high-dimensiona policy parameters that be effectively reconstructed into foll-functioned policies. (2) Learning embedding of behavior demonstrations that serves as an effective diffusion condition. (3) Training a conditional diffusion model specifically for policy parameter generation.
Parameter representation. We use an MI. with m layers as the common policy approximator. Consequently, when the full parameters of policy are flattened, they form high-dimensional vector.

---

To enable generation with limited computational resources while retaining efficacy, and to support the generation of policies for different domains with varying state and action dimensions, we compress the policy network parameters into a latent space.
Based the policy network architecture, we unfold the parameters following the architecture of the policy network, represented as I [2o, 21,..... -m11), where I; denotes the flattened parameters from each layer. The encoder & encodes each æ; as 24, resulting a parameter latent representation denoted as 2 = [zo, .1..... 2m- where each 2; in the latent space has the same dimension, while the decoder DD can decode into r. To improve the robustness of this procedure, we introduce random noise augmentation in both encoding and decoding during training. Given each vectorized parameter as z, we minimize the objective as,
(5) where = E(x + ED), and &s and ED represent the augmented noise. The architecture of the autoencoder is shown in Figure 2
L= MSE(2, D(E + ED) + +x)),,
For each domain, the autoencoder for parameter representation only needs to be trained once before parameter generation, which can handle policy parameters from different tasks. To facilitate the generalizability of the policy generator across domains, we design the latent parameter representations to have the same dimensions for different domains.
Behavior embedding. Since our goal in learning behavior embeddings is not to model the distri- bution of states and actions, but to provide conditional information for policy parameter generation, we aim for them to encapsulate both crucial environmental dynamics and the key information of the task goal. The principle behind our behavior embeddings is to learn the mutual information between preceding n step trajectories and subsequent states with success signals.
I [ success: {st, az (===))
We propose a novel contrastive method to train behavior embeddings. In Figure we present design demonstration of our contrastive loss. For long trajectory т, we decouple it as the n initial state-action pairs n" = (so. ag. $1, 01 0l...... 8x, an) and the m states after the first success time K as f (sK. 8K+1..... +m Given a batch of trajectory sequences {T:)_=1 which can be presented
as {T f.) we optimize the contrastive objective [16 31 as:
WW
L(os. to. W) = N
log
E h Wo,
where h; (T(""") and U2 (tt are embeddings from different parts of the long trajectory T and W is a learnable metric that measures the similarity between embeddings h; and i..
For each trajectory т, we obtain set of embeddings T. {ha v.). In practice, the choice of specific embeddings can be tailored to the characteristics of different tasks and trajectories. We use (h,, vs) as the conditional input in our experiments.
Flexibility. With the consideration that in many scenarios, rewards are often sparse or non-existent, whereas success signals serve more direct indicator of whether policy has achieved its objective. We therefore use original trajectories that exclude reward information but include success information.
For tasks without explicit success signals, such as locomotion, we segment long trajectories into multiple shorter trajectories. For each segment, we use the last m states as 7 and the 0- n state-action pairs as r". The informative behavior embeddings of long trajectory are concatenated from the embeddings of all the trajectory segments.
This embedding approach strives to capture the essential information for generating behavior-specific policy parameters, including environmental dynamics and task goals, using the most concise repre- sentation possible from long trajectories and prioritizing flexibility and efficiency.
Conditional policy generator. After training the parameter autoencoder and behavior embeddings, for policy parameter x and the corresponding trajectory g deployed by policy I. we can transfer x as latent parameter representation with the autoencoder E and trajectory as behavior embedding Tel The conditional diffusion generator is trained on latent representation 2, conditioning on Te. We optimize the conditional latent diffusion model via the following loss function:
(!) == (0,,)) [l|e -ep(ze, Te( +)
(8)

---

where the neural backbone eø (2,, Te. is implemented as ID convolutional UNet [9 parameterized by and uniformly sampled from {1..... Т. The outputs of our parameter generator can be encoded by DD as deployable policies. During training the diffusion model, both the parameter autoencoder and behavior embedding layers are frozen, which ensures the training stability and efficiency.
Dataset. We build dataset containing tens of thousands of policy parameters and trajectories from deploying these policies. The dataset is obtained from multiple RL training across range of tasks. We utilized the dataset train both the autoencoder and behavior embedding models. Then we use the encoded parameter representations and behavior embeddings derived from the collected trajectory train the conditional diffusion model for policy parameter generation.
4 Experiments
MetaWorld
Robosuite
Quadrupedal Lecomotion
Figure 4: Visualization of MetaWorld, Robosuite, and real quadrupedal locomotion.
We conduct extensive experiments to evaluate Make-An-Agent, answering the following problems:
How does our method compare with other multi-task or learning-to-lee to-learn approaches for policy learning, in terms of performance seen tasks and generalization to unseen tasks?
How scalable our method, and can it be fine-tuned across different domains?
Does our method merely memorize policy parameters and trajectories of each task, or can it generate diverse and new behaviors?
Benchmarks. We include two manipulation benchmarks for simulated experiments and real- world robot tasks to show the performance and capabilities of our method as visualized in Figure
benchmark suite for robotic tabletop manipulation, consisting of a diverse set of motion patterns for the Sawyer robotic arm and interactions with different objects. We selected 10 tasks for training as tasks and for evaluation unseen downstream tasks. Detailed descriptions of these tasks can be found in Appendix C.I The state space of MetaWorld consists of 39 dimensions and the action space has 4 dimensions. The policy network architecture used for Meta World is 4-layer MLP with 128 hidden units, containing a total of 22,664 parameters.
MetaWorld. MetaWorld 29
Robosuite. Robosuite (33, simulation benchmark designed for robotic manipulation, supports various robots such as the Sawyer and Panda arms. We train models on three manipulation tasks: Block Lifting, Door Opening and Nut Assembly, using the single-arm Panda robot. Evaluations conducted on the same tasks using the Sawyer robot. This experimental design aims to validate the practicality of our approach by assessing whether the generated policy be effectively utilized different robots. In the Robosuite environment, the state space comprises 41 dimensions, and the action space consists of 8 dimensions. The policy network employed for this domain contains 23,952 parameters.
Quadrupedal locomotion. To evaluate the policies generated by Make-An-Agent in the real world, we utilize walk-these-wee [14] to train policies on IsaaeGym and use our method to generate actor networks conditioning trajectories from IsaacGym simulation with the pretrained adaptation modules. Then, we deploy the generated policy on real robots environments differ from simulations. The policies generated for real-world locomotion deployment comprise 50,956 parameters.
Dataset. We collect 1,500 policy networks for each task in Mora and Robosuite. These networks sourced from policy checkpoints during SAC [|] training. The checkpoints are saved every
5

---

5000 training steps once the test success rate reaches 1. During the training stage, we fix the initial locations of objects and goals and train the policies using different random seeds. For each task, we require an average of 8 SAC training runs, approximately 30 GPU hours.
For evaluation, the trajectories used as generation conditions are sampled from the SAC training buffer within the first 0.5 million timesteps, which can be highly sub-optimal, under the same environmental initialization. During testing, the generated policies are evaluated in random initial configurations, thoroughly assessing the robustness of policies generated using trajectories from the fixed environment settings.
In RoboSuite experiments, due to the inconsistency in policy networks, we retrain the autoencoder and finetune the diffusion generator trained MetaWorld data. The experimental setup for RoboSuite is almost identical to that of MetaWorld, with the only difference being the robot used during testing.
For real-world locomotion tasks, we save 10,000 policy network checkpoints using walk-these-wee ways (WTW) 14 trained on IsaacGym, requiring a total of 200 GPU hours. The 100 trajectories used as generation conditions are sourced from the first 10,000 training iterations of WTW.
Baselines. We compare Make-An-Agent with four baselines, including multi-task imitation learn- ing (IL), multi-task reinforcement learning (RL), meta-RL with hypernetworks, and meta-IL with transformers. These represent state-of-the-art methods for multi-task policy learning and adaptation. For a fair comparison, each baseline uses the same testing trajectory data for downstream adaptation.
Multi-task BC [24]:
We train multi-task behavior cloning policy using trajectories in training dataset, and then finetune it with test trajectories to adapt in specific tasks.
We train mixture of encoders for 2 million steps (for each task). For RL training, we train the algorithm in dense reward environments and finetune the model using test trajectories with sparse rewards, where feedback is only provided at the end of a trajectory.
Multi-task RL, CARE [23]:
Meta-RL with hypernetworks [ We train a hypernetorr with our training data with dense rewards, which can adapt to different task-specific policies during testing with test trajectories.
Meta Imitation Learning with decision transformer(DT) 4 We train the pre-trained DT model using the training trajectories in our dataset, then use the test trajectories from replay to adapt it to test tasks.
4.1 Performance Analysis
MetaWorld (10 seen tasks)
Robosuite (3 seen tasks)
Multi-task IL
Multi-task RL
Success Rate
Meta RL
80%
80%
Generated top
Generated Best
60%
60%
40%
10
50
100
5
10
50
100
Trajectories
Trajectories
Figure 5: Evaluation of seen tasks with 5 random initializations on Meta World and Robosuite. Our method generate policies using 5/10/50/100 test trajectories. Baselines are finetuned/adapted by the same test trajectories. Results are averaged over training with 4 seeds.
By using test trajectories as conditions, our policy generator can produce an equivalent number of policy parameters. Compared with baselines, we report both the best result among the generated policies and the average performance of the top 5 policies. All algorithms the same task-specific replay trajectories. The difference is that we use them as generation conditions, whereas other methods use them for adaptation.
We define policies achieving a 100% success rate during evaluation as qualified policies. The analysis of qualification rates for policies generated by model is presented in Appendix C22

---

MetaWorld tasks)
Robosuite (unseen robots)
100%
Success Rate
60%
40%
Trajectories
Trajectories
Trejectories
Trejectories
Multi-task IL Multi-task
Meta RL
Generated top
Generated Best
Meta IL
Figure 6: Evaluation of 8 unseen tasks with 5 random initializations on Meta World and Robo- suite. Our method generates policies using 50/100 test trajectories without any finetuning. Baselines adapted using the same test trajectories. Average results are from training with 4 seeds.
Adaptability to environmental randomness on seen tasks. Figure FigureeSdemonstraatee demonstrates the significant advantage of algorithm over other methods on seen tasks. This attributed to the fact that, despite test trajectories originating from the same environment initialization, the generated policy parameters more diverse, thus possessing strong ability adapt environmental randomness. In contrast, other algorithms, when adapted using such singular trajectories, exhibit more limited adaptability in these scenarios. Our experimental design aligns with practical requirements, as real-world randomness is inherently more complex.
Generalizability unseen tasks. Figure (ggure/osshore shoocaesss the superior performance of our algorithm on unseen tasks. Test trajectories originate from the same environment setting for each task, while evaluation occurs in randomly initialized environments. Our policy generator, without fine-tuning, directly utilizes test trajectories as input, demonstrating a remarkable ability to generate parameters tasks exhibits similarities to
that work unseen tasks. The agent's behavior in task behaviors, such as arm dynamics and the path to goals. By effectively combining parameter representations related these features, the generative model successfully generates effective policies. In contrast, baseline methods struggle to adapt in environmental randomness.
These results strongly suggest that algorithm, compared other policy adaptation methods, may offer a superior solution for unseen scenarios. To further investigate robustness in generalization, we added Gaussian noise with standard deviation of 0.1 to actions in test trajectories used for policy generation or adaptation on unseen tasks. Figure ( demonstrates that our method remains resilient noisy inputs, while the performance of the baselines is significantly impacted. We believe this is because our behavior embeddings only need to capture key dynamic information as conditions to generate policies, without directly learning state-action relationships from trajectories, resulting in better robustness.
Trajectory difference. To compare the difference between using test trajectories conditions and the trajectories obtained by deploying the generated policies, we visualize the trajectories during unseen task evaluations. As shown in Figure our diffusion generator which is significantly different from policy learning methods that learn to predict actions or states from trajectory data. We believe that this phenomenon fully illustrates the value of our proposed policy parameter generation paradigm.
synthesize various policies,
Parameter distinction. Beyond trajectory differences, we also investigate the distinction between synthesized parameters and RL policy parameters. We calculate the cosine similarity between the RL policies used to obtain the test trajectories and the parameters generated from these trajectories. As a benchmark, we include the RL policies after 100 steps of finetuning with the test data. For tasks seen during training, the parameters generated by our approach demonstrate significantly greater diversity compared to the RL parameters after fine-tuning, indicating that our generator does not simply memorize training data. On unseen tasks, the similarity between generated parameters and those learned by RL is almost negligible, with most similarities falling below 0.2. This further highlights the diversity and novelty of the policy parameters generated by our method.
Real-world Evaluation We further deploy policies generated from simulation trajectories onto quadruped robot, instructing it to complete tasks as illustrated in Figure I0 Our synthesized policies

---

MetaWerid 18 Unseen Taiks)
Embeddings on
100%s
100%
Success Rate
Success Rate
80%
60%
60%
40%
20%
20%
Trajectories
Trajectories
50 Trajectories
100 Trajectories Emocddinng Layer
Dest from beselines
Generated Top 5
Generated Best
Emocddiing
Embedding
Ours
Figure Evaluation of unseen tasks on Meta- World using noised trajectories.
Figure 8: Ablation studies about using different embeddings conditions in policy generation MetaWorld unseen tasks. (Top models)
Weddow does
Hardl
lock
Figure 9: Trajectory difference: trajectories as conditional inputs V.s. trajectories from synthesized policies outputs MetaWorld 4 unseen tasks.
exhibit smooth and effective responses when faced with these challenging tasks, which highlights the stability of the generated policies under the dynamics randomness of weal-world environments
Seen Taeks
Uneen Tanks
Malding turn aooid stogalig as hide
Maripolag clouunvent the ball settihy tecinard.
Cosire Simlerty
2.8
Costre Smileity
422
Fiseured
Finatuned
8.4
Figure 10: Real-world locomotion tasks, including turning, fast backward movement, and obstacle avoidance on a mat.
Figure 11: Parameter Similarity: Parameter cosine similarity between RL-trained policies and our generated policies or fine-tuned policies.
4.2 Ablation Studies
To better investigate the impact of each design choice in our method on the final results, conduct a series of comprehensive ablation studies. All ablation studies report average results of the Top generation models MetaWorld.
Choice of behavior embeddings. Regarding the choice of conditional embeddings, as illustrated in Figure 3 we concatenate h. and v as generation conditions to maximally preserve trajectory information. Figure shows that utilizing either embedding individually also achieves comparable performance due to our contrastive loss, ensuring efficient capture of dynamics information. Our contrastive behavior embeddings significantly outperform a baseline that adds an embedding layer in the diffusion model to encode trajectories input. These ablation results underscore the effectiveness of our behavior embeddings.
Choice of trajectory length. The trajectory length n used in behavior embeddings can also impact experimental results. Figure 12a demonstrates that overly short trajectories lead performance degradation, probably due to the absence of crucial behavior information. However, beyond 40 steps, trajectory length minimally impacts policy generation, indicating that our method is not sensitive to the length of trajectories.
We thank Kun Lei and Qingwei Ben for their help and support in real-robot applications.

---

Impact of policy network size. The impact of policy network size on generated parameters is also worth discussing, as the network's hidden size influences the dimensionality of parameters to be generated. Figure 12b suggests that hidden size of 128 is suitable choice. Smaller networks may hinder policy performance, while larger ones increase parameter reconstruction complexity.
Impact of parameter number used in training. We study the impact of the number of policy checkpoints included per task the training dataset, as shown in Figure 12c Insufficient training data (<=1000) leads to a significant performance decline across all tasks. With more than 1000 parameters, there is no notable improvement in performance.
Impact of latent representation size. Additionally, Figure 12d illustrates the impact of varying negatively affect the size of the latent parameter representation. Larger latent representations representations is the performance of the generative model. Conversely, when the size of parameter too small, it may hinder the autoencoder's capacity to decode representations deployable policies. This underscores the influence of the parameter autoencoder on the overall effectiveness of the policy network generator.
1025
Success Plate
Success Pair
Succuss Pullaa
45%
80M
99%
42%
605
0%
120
24
2030 Paramaters Truriing
.03 2003 1589
50
1204
Tajuedory Length
Latent Rlgreswiiation
(a) Trajectory Length
(b) Policy Model Size
(c) Parameter Number
(d) Representation Size
Figure 12: Ablation studies of our technical designs on MetaWorld with 50 test trajectories (Top models).
Related Works
Parameter Generation. Learning to generate neural networks has long been a compelling area of research. Since the introduction of Hypernetworks [10] and the subsequent extensions [3, several studies have explored neural network weight prediction. Hypertransformer [32 utilizes Transformers to generate weights for each layer of convolutional neural networks (CNN) using task samples for supervised and semi-supervised learning. Additionally, previous work (2 employs self-supervised learning to learn hyper representations of neural network weights. In the context of using diffusion models for parameter generation, G.pt [17] trains diffusion transformer to generate parameters conditioned on learning metrics such test losses and prediction errors, enabling the optimization method of unseen parameters with a single update. Similarly, p-diff (25] propose a diffusion-
to generate the last two normalization layers without any conditions for classification tasks. In contrast to these prior works, our focus is on policy learning problems. We develop a latent diffusion parameter generator that is more generalizable and scalable, based on agents' behaviors as prompts.
Learning to Learn for Policy Learning. When discussing learning to learn in policy learning, the concept of meta-learning (7) has been widely explored. The goal of meta-RL. 7 6 [5] is to learn a policy that can adapt to any new task from a given task distribution. During the meta-training or meta-testing process, prior meta-RL methods require rewards as supervision for policy adaptation.
Meta-imitation learning 8 5 277 addresses a similar problem but assumes the availability of expert demonstrations. Diffusion models have also been used in meta-learning. Metadiff [30] models the
gradient descent process for task-specific adaptation as a diffusion process to propose diffusion- based meta- meta-learning method. Our work departs from these learning-to-les le-remmn works. Instead, we shift
the focus away from data distributions across tasks and simply leverage behavior embeddings as conditional inputs for policy synthesis in the parameter space.
Conclusion
In this paper, we introduced novel policy generation method based on conditional diffusion models. Targeting the generation of policies in high-dimensional parameter spaces, we employ an autoencoder to encode and reconstruct parameters, incorporating contrastive loss to learn efficient behavior

---

embeddings. By prompting with these behavior embeddings, our policy generator can effectively produce diverse and well-perronning policies. Extensive empirical results across various domains demonstrate the versatility of our approach in multi-task settings, the generalization ability on unseen tasks, and the resilience to environmental randomness. Our work not only introduces fresh perspective on policy learning, but also establishes a new paradigm that delves into the latent connections between agent behaviors and policy parameters.
Limitation. Due to the vast number of parameters involved, we have not yet explored larger and more diverse policy networks. Additionally, the capabilities of the parameter diffusion generator are limited by the parameter autoencoder. We believe there is substantial room for future research explore more flexible parameter generation methods. It would also be interesting apply our proposed generation framework to generate other structures, further facilitating exploration in policy learning within the parameter space.
10

---

References
[1] Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernet- works in meta-reinforcement learning. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1478-1487, PMLR, 14-18 Dec 2023.
[2] Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
[3] Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David Clifton. brief review of hypernetworks in deep learning, arXiv preprint arXiv:2306.06955 2023.
[4] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas. and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-150% 2021.
[5] Yan Duan, Marcin Andrychowicz. Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
[6] Yan Duan, John Schulman, Xi Chen, Peter L.. Bartlett, Ilya Sutskever, and Pieter Abbeel. RIS-2$: Fast reinforcement learning via slow reinforcement learning. CoRR, 0t///611.027777 2016.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[8] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pages 357-368. PMLR, 2017.
[9] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv:1806.0464) 2018.
[10] David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv: 16090091089 2016.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 6156840-6851, 2020.
[13] Tianying Ji, Yongyuan I. Liang, Yan Zeng, Yu Luo, Guowei Xu, Jiawei Guo, Ruijie Zheng, Furong Huang, Fuchun Sun, and Huazhe Xu. Ace: Off-po actor-critic with causality-aware entropy regularization. arXiv preprint arXiv:2402.14522 2024.
[14] Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generaliza- tion with multiplicity of behavior. Conference on Robot Learning. 2022.
(15) Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Advances in Neural Information Processing Systems, 32, 2019.
[16] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 180718700333 03748, 2018
[17] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209.1282 12892, 2022.
11

---

[18] Sahand
Charlotte Morissette, François Robert Hogan, Gregory Dudek, and David Meger. Hypernetworks for zero-shot transfer in reinforcement learning. In AAAI, pages 9579-9587. AAAI Press, 2023.
[19] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1 1286. PMLR, 2014.
[20] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEER/CVF conference on computer vision and pattern recognition, pages (0684-10695, 2022.
[21] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 25:27906-27920, 2022.
[22] David Silver, Aja Huang. Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587)48444889 2016.
[23] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. International Conference on Machine Learning, pages 9767-9779. PMLR, 2021.
[24] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 1805:0190 2018.
[25] Kai Wang. Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion, 2024.
[26] Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint MtXiv:2310.19666 2023.
[27] Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper- decision transformer for efficient online policy adaptation. arXiv preprint arXiv:2304.08887, 2023.
[28] Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777 2020.
[29] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019,
[30] Baoquan Zhang. Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yumming Ye, and Bowen Zhang. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16687-16695, 2024.
[31] Ruijie Zheng, Yongyuan Liang, Xiyao Wang. Shuang Ma, Hal Daumé III, Huazhe Xu, John Langford, Praveen Palanisamy, Kalyan Shankar Basu, and Furong Huang. Premier- TACO few-shot policy learner: Pretraining multitask representation temporal action-driven contrastive loss. In Forty-first International Conference on Machine Learning, 2024.
[32] Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. Hypertranfformer: Model generation for supervised and semi-supervised few-shot learning. In ICML,, volume 162 of Proceedings of Machine Learning Research, pages 27075-27098, PMLR, 2022.
[33] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín- Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12299, 2020.
12

---

A Broader Impact
The broader impact of our research on generating policy parameters using diffusion models is complex interplay of potential benefits and risks. On the one hand, by enhancing the sample efficiency of policy learning in robotics, our method could accelerate the development and deployment robotic systems across various sectors, from manufacturing and healthcare to agriculture. This could lead increased automation, potentially improving efficiency, safety, and accessibility in these fields, ultimately enhancing the quality of life and productivity for society. Additionally, our work introduces robotics community to the potential of policy learning in a new and promising paradigm--wiit could inspire further research, leading to more advanced and efficient algorithms for robot learning and control.
However, it's important to acknowledge the potential negative consequences of increased automation. Job displacement due automation is significant concern, as robots could replace human workers in various industries, leading to economic disruption and social unrest. Furthermore, robots become more autonomous and capable, concerns about safety and ethical implications arise. Ensuring the safe and responsible use of robots will be paramount as our research progresses.
B Implementation Details
All model training are conducted on NVIDIA A40 GPUs.
Autoencoder.
The autoencoder implementation consists of a three-layer MLP encoder and a decoder. Prior to training, each layer of the policy network is flattened and encoded separately. The final mean and std layers are concatenated with the middle layer for encoding.
The hyperparameters used for the autoencoder detailed in Table autoencoder requires 5 GPU hours.
On average, training
Behavior embedding.
The behavior embedding model consists of three-layer MLP embeddings. During training, we concatenate the state and action sequences from the first = 60 steps (each sequence having a length of 3) to form the input for the embedding layer. Subsequently, we concatenate the m states after success as inputs for the v embedding layer. Both embedding layers output 128- 128-dimensional vectors. When utilizing these embeddings as conditional inputs, we concatenate the and embeddings as 256-dimenional conditions.
All hyperparameters about the training of the behavior embeddings be found in Table 2 A single training for the embeddings requires less than I GPU hour.
Conditional diffusion generator. Our diffusion model employs ID convolutional U-Net architec- ture as its backbone, utilizing behavior embeddings as global conditions. It outputs latent parameter representations with the same dimensionality the autoencoder's output.
Training a single diffusion generator requires only GPU hours. All relevant hyperparameters are detailed in Table 3
Hyperparameters
We conduct all experiments with this single set of hyperparameters.
Data collection.
Our dataset collected with [3 26 using RTX 3090 Ti.
C Experiments
C.1 Task Description
MetaWorld Descriptions of tasks and random initialization:
Seen tasks (Training):
window open: Push and open window, Randomize window positions
door open: Open door with revolving joint. Randomize door positions
drawer open: Open drawer. Randomize drawer positions
13

---

Table I: Hyperparameteers for Autoencoder
Hyper-parameter
Value
Table 2: Hyperparameters for Behavior Embedding
backbone
MLP
Hyper-parameter
Value
input dim
22664
backhone
MLP
hidden size
1024
trajectory dim
1020
output
[2, 1024]
success state dim
117
encoder depth
hidden size
1024
decoder depth
output dim
128
input noise factor
0.0001
batch
oulput noise factor
0.001
optimizer
AdamW
batch size
32
leaming rate
le-4
optimizer
AdamW
weight decay
le-4
learming rate
le-3
training epoch
300
weight decay
Se-3
Ir scheduler
CosineAnneeling Warmmestrrss
training epoch
3000
scheduler
CosineAnnealing WarmRestarts
Table 3: Hyperparameters for Diffusion Model
Hyper-parameter
Value
Hyyer-parameter
Value
behavior embedding shape
[256]
kereel size noise scheduler
parameter shape
[2, 1024]
DDIM
inference steps
10
batch size optimizer
128
embedding dim in diffusion steps
128
Adam W
lcarning rate eps training epoch Ir warmup steps
2.0c-4
beta weight decay
[0.95, 0.999]
1.0e-8
5.0e-4
1000
Ir s.scheuulerrr
cosine
500
dial turn: Rotate dial 180 degrees. Randomize dial positions
faucet close: Rotate the faucet elockwise. Randomize faucet positions
button press: Press button. Randomize button positions
door unlock: Unlock the door by rotating the lock clockwise. Randomize door positions
handle press: Press handle down. Randomize the handle positions
plate slide: Slide a plate into cabinet. Randomize the plate and cabinet positions
reach: reach goal position. Randomize the goal positions
Unseen tasks (Downstream):
window close: Push and close a window. Randomize window positions
door close: Close a door with revolving joint. Randomize door positions
drawer close: Open drawer. Randomize drawer positions
faucet open: Rotate the faucet counter-clockwise. Randomize faucet positions
button press wall: Bypass wall and press button. Randomize the button positions
door lock: Lock the door by rotating the lock clockwise. Randomize door positions
handle press side: Press handle down sideways. Randomize the handle positions
coffee-button Push button on the coffee machine. Randomize the position of the button
reach wall: Bypass wall and reach goal. Randomize goal positions
14

---

Robosuite
Descriptions of tasks, robots, and random initialization:
Tasks:
Door: A door with handle is mounted in free space in front of a single robot arm. The robot arm must learn turn the handle and open the door. The door location randomized at the beginning of each episode.
Lift: A cube is placed on the tabletop in front of single robot arm. The robot arm must lift the cube above a certain height. The cube location is randomized the beginning of each episode.
Nut Assembly Single: Two colored pegs (one square and one round) are mounted on the tabletop, and two colored nuts (one square and one round) are placed on the table in front of a single robot arm. The goal is to place either one round nut or one square nut into its peg.
Robots:
Panda: Panda is 7-DoF and relatively new robot model produced by Franka Emika, and boasts high positional accuracy and repeatability. The default gripper for this robot is the PandaGripper, a parallel-jaw gripper equipped with two small finger pads, that comes shipped with the robot arm.
Sawyer: Sawyer is Rethink Robotic's 7-DoF single-arm robot. Sawyer's default RethinkGrip- per model is a parallel-jaw gripper with long fingers and useful for grasping a variety of objects.
C.2 More Results
In addition to reporting the average performance of the top 5 generated results in the main paper, rigorously define "qualified policies" as those achieving a 100% success rate in the test environment. the proportion of qualified policies among 100 policy parameters generated from 100 trajectories. Notably, we maintain an average qualification rate of over 30% on seen tasks.
Furthermore, even unseen tasks, we can generate high- nigh-performing policies using average of only 20 trajectories. Considering that our method does not rely on expert demonstrations, the quality and success rate of our generated policies significantly enhance the sample efficiency of policy learning.
Table 4: Qualified rate and success rate of Top 5/10 models from the generated polices with 100 trajectories on MetaWorld
Seen Tasks
window upen
duor open
drawer open
dial turn
plate slide
button press
handle press
Gaucet close
Qualifted Rote
0.33 0.02
0.27 = 0.04
0.42 + 0.03
0.23 = 0.02
0.45#+0.04
0.32=40.14 14
0.5 + 0.08
0,45+00113
Generated Top 5
1,0 0.0
1.0 1
1,0 0.0
0.82 00.09
1.0 0.0
0.87 0.10
1.0 1 0.0
0,98 0.01
Genernted Top
1.0 0.0
0.87 0.06
1.0 D.0
0.73 0.06
0.94 0.05
0.05
0.96 0.02
0.97 0.01
Unseen Tacks
drower close
Fmncet open
button press
colToe button
handle press
reach wall
door lock
window close
wall
side
Qualifted Rate
0.55 0.04
0.16 + 0.06
0.11 + 0.01
0.08 + 0.013
0:04 + D.03
0.13 0.05
0.13 + 0.04
0.10 + 0.01
Generated Top 5
1.0
1.0
0.97 0.02
0.74 =: 0.21
0.45 0.17
0.94 0.03
1.0 :: 0.0
0.92 0.05
Genernted Top 10
10
0.85 + 0.13
0.95 0.02
0.64 0.23
0.30 d: 0.12
0.72 0.11
0.55 0.05
0.85 0.08
C.3 Details of Real-world Robots
In this section, we detail the real-world robot applications of our method. We deploy synthesized policies on the Unitree Go2 quadruped, designing diverse real-world testing environments to evaluate two key aspects of agent performance: (1) stability during high-speed turning and backward locomo- tion, and (2) robustness of movements on uneven terrain (mats). Our deployment process consists of four key steps:
Obtain actor network parameters and corresponding test trajectories from IsaacGym simula- [14].
tions, where the actors are trained using
Train Make-An-Agen using the acquired training data.
15

---

Generate actor networks from randomly sampled IsaacGym trajectories, covering a variety of training periods.
Equip the Unitree Go2 quadruped with the generated actors and pretrained adaptor, enabling it to complete designed, challenging locomotion tasks.
16

---

Seen Tasks (Average)
Unseen Tasks (Average)
Unseen Tasks with Noise Taaectories
Teppectoyy Lerghh Conerotee Faittes
Hull--takk
Mullitask Multi tack RL Meta RL Meta
Mallikiwk Murti tase RL Meta RL Pizta
Nult: task
Neta RL
Mota
Pajectory Length Conditions
Trajectory Length Conditions
Trsjectory of Consitions
Figure 13: Correlation between condition trajectories and generated policies. Trajectory length accurately reflects the effectiveness of the policies compared the success rate. The maximum episode length in all the tasks is 500 (represents failure).
36.1
31.4
30
GPU Heurs
20.4
7.8
4.1
Vee Methods
Figure 14: Computational budgets of ours and baselines
1.5x
Real-world locomotion: tasks
Multi-task training (seen): 13 tasks
Avoid stapping bouquet while moving
Unseen task/oooot generalization: tasks
155x
Navigating circumvent the and ball switty
Figure 15: Visualization of experimental tasks
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We conduct comprehensive experiments, and the results included in our paper support the main claims made in the abstract and introduction well.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We add paragraph in the Conclusion section, addressing the scope and performance limits our work.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper empirically proposes one effective method to generate policy networks from demonstrations, and doesn't contain theoretical proof.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include detailed descriptions of our methods in the main paper and the Appendix, the information provided should be adequate to reproduce the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all the code used to produce the main results in an anonymous GitHub repo in the supplementary material.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The training and test details included in the Experiment section and the Appendix to facilitate comprehensive understanding our design choices and results.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: An error bar or the variance value is included in all the results, either in table or a figure.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Question:
18

---

Answer: [Yes]
Justification: We provide training GPU hours and the type of computing resources in the Experiments section.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: After reading through the NeurIPS code ethics carefully, we are confident that our research conform well with the spirit of the Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide broader impact section in Appendix A to discuss the positive and negative impacts of research.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not fall into any category of pretrained language models, image generators, or scraped datasets. While we do provide a dataset for trained agents, these are no different than providing model checkpoints and does not contain any sensitive data that may relate to real person or entity.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we've cited any paper code we use in the paper, and follow their license and terms of use correctly.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The open-sourced code and dataset we provided are licensed appropriately, please refer to the anonymous repo for details.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
19


Reviews:
[{"review_id": "I3TyAayXoS", "reviewer": "Reviewer_U6yw", "summary": "This paper presents Make-An-Agent, a conditional diffusion model that generates policy parameters based on demonstration of target behaviors. The authors propose an autoencoder to encode policy network parameters into compact latent representations. The behavior embeddings are learned using contrastive learning between long-term trajectories and their success or future states. Then, a conditional diffusion model is used to generate policy latent code conditioned on the learned behavior embedding, which can then be decoded to policy parameters via the pretrained decoder. Extensive results in simulated environment including MetaWorld and Robosuite demonstrate the effectiveness of the proposed method. Also, there is real-world deployment on quadrupedal locomotion.", "questions": "- Is there any way to roughly predict the performance of the generateed parameters without actually evaluating them? For example, assuming the learned data distribution nicely reflects the given behavior, the generation distribution in the policy parameter space should somewhat follow the mode that gives good performance; perhaps something similar to the \"Parameter distinction\" experiments but among the generation from the proposed model only can be interesting.\n- Is any of the statistics of the generated policy parameters useful? For example, does the mean still a valid policy parameter? Is the variance of the generated parameters correlated with how \"out-of-distribution\" the behavior is?\n- It would be interesting to briefly look at the geometry in the behavior embedding space. For example, how does small perturbation in the behavior embedding affect the performance of the generated policy parameters. Or even more interesting, is it possible to compose behaviors by messing around the conditioning of the diffusion model, which then gives a compositional policy?\n- Could the authors report the performance distribution for all generations (as opposed to the best or top 5)? This can probably provide some insight into the generation behavior and what distribution (w.r.t. the data distribution in generative modeling not performance distribution) is actually learned by the model. Also, it would be interesting to see the performance distribution in seen and unseen tasks.\n- For ablation of policy network, could the authors discuss how architectural difference (beyond hidden size; e.g., constructing policy as very small recurrent network or transformer) can potentially affect the results?\n- In real-world evaluation, could the authors provide more details about the behaviors and policies in trainnig dataset and testing? \n- Given the proposed method being a generator of the policy (which can easily produce many models at a time), it would be also interesting to check the mixture of expert setup. For example, will using something simple like majority voting in generated policies improves robustness in \"Adaptability to environmental randomness on seen tasks\"?\n- Could the authors report some failure cases? Or is there any failure mode like certain type behaviors/demonstrations always lead to bad-performing generated policies?", "limitations": "Limitations are discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The experimental analysis is pretty thorough.", "weaknesses": "- Despite the experiments are overall complete, there are many more interesting aspects to look at, which may further strengthen the paper. Please check questions for more details.\n- The real-world experiment has very limited results presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer U6yw for the positive comments to our writing and experiment thoroughness. Your questions are instrumental in helping us improve the quality of our paper. We have provided detailed responses point by point below.\n\n> Is there any way to roughly predict the performance of the generateed parameters without actually evaluating them? \n\n**Feasibility and efficiency of evaluation:** We generate policies and evaluate them with only one episode to determine their effectiveness. The final results are then reported after evaluation over 10 episodes, consistent with the baselines. The GPU hours consumed by the inference process and evaluation for 100 policies are listed below. We ensure a very low cost for evaluation.\n\n| Task/Average (100 Trajectories)                                             | GPU Hours (100 Trajectories) |\n|---------------------------------------------------|------------------------------|\n| Finetuning (Baselines/Average)                   | 0.40 +- 0.12                          |\n| Inference + Evaluation (100 Policies)| 0.11 +- 0.0                         |\n\n> Is any of the statistics of the generated policy parameters useful? Is the variance of the generated parameters correlated with how \"out-of-distribution\" the behavior is? Is there any failure mode like certain type behaviors/demonstrations always lead to bad-performing generated policies?\n\nFigure 13 in our supplementary material illustrates the relationship between condition trajectories and the performance of generated policies. It can be observed that when the condition trajectories are effective, the generated policies tend to perform better. When the behaviors used for synthesized conditions deviate significantly from the correct actions, the generated policies tend to fail. That is reasonable because failed trajectories may not provide sufficient information as prompts for generation. While condition trajectories do show some correlation with generated policies, it is difficult to use them as a metric to directly judge the performance of generated policies.\n\n> How does small perturbation in the behavior embedding affect the performance of the generated policy parameters. Is it possible to compose behaviors by messing around the conditioning of the diffusion model, which then gives a compositional policy?\n\nWe also show the impact of noisy trajectories on the generated results for unseen tasks in Figure 13, demonstrating a significant advantage over the baselines. For combined trajectories, our method can not generate compositional policies, due to the absence of multi-task compositional policies in our training dataset. However, this presents an interesting possibility that could offer new insights into exploring the parameter space for multi-task learning.\n\n> Could the authors report the performance distribution for all generations (as opposed to the best or top 5)? \n\nAs shown in Figure 13. Note that we choose average trajectory length as a metric because it more accurately reflects the effectiveness of the policies compared to the success rate. For example, policies with the same 80% success rate can exhibit significant differences in task efficiency.\n\n> For ablation of policy network, could the authors discuss how architectural difference (beyond hidden size; e.g., constructing policy as very small recurrent network or transformer) can potentially affect the results?\n\nThe choice of policy architectures determines the type of network parameter data collected for the training dataset. In our experiments, we used MLP policy structures with varying layer numbers, resulting in parameter counts ranging from 2e4 to 16e4 across three domains. The architecture variation did not affect the overall effectiveness in different domains. We believe that our method could also be applied to generate RNN or Transformer policies.\n\n> In real-world evaluation, could the authors provide more details about the behaviors and policies in training dataset and testing?\n\nIn our locomotion tasks, we collect the training dataset using the IsaacGym simulator. We train policies on flat ground to move forward stable without any obstacles and changes in terrain geometry, while randomizing dynamics parameters such as payload mass, motor strength, and gravity offset, to obtain policies under different environment settings.\nIn the real-world deployment, we apply the generated policies to the Unitree Go2 Edu robot, issuing commands to complete behaviors such as sharp turns, obstacle avoidance on mats, and rapid backward movements. We will update the details of these real-world experiments in the appendix.\n\n> Given the proposed method being a generator of the policy (which can easily produce many models at a time), it would be also interesting to check the mixture of expert setups. \n\nIt is a valuable idea that can help us better utilize the large number of generated policies. We employed a straightforward approach by dividing the action space into over 20 discrete intervals. Then we performed majority voting to select the most frequently voted interval. The actions output by policies voting for this interval were averaged to determine the final action.\nWe conduct experiments on three unseen tasks and evaluate using adversarial RL settings with random noise in the range of 0.1 added to actions on 4 random initializations. The results show that **multiple synthesized expert policies can significantly improve robustness**. \n| Robustness to random noise (sr) | door close | coffee button | reach wall| \n|-----|----|----|----|\n| Single RL | 0.32 | 0.44 | 0.52 |\n| Ours (Generated Policies)  | 0.43 | 0.56 | 0.55 |\n| Ours (mixture of experts) | 0.89 | 0.82 | 0.79 |\n\nWe sincerely appreciate your thorough and detailed feedback and hope our replies address all your concerns. All the discussions and experimental results will be included in the final version to further enhance our paper's quality. We look forward to further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ViU51lnr1m", "reviewer": "Reviewer_957J", "summary": "In this work, the authors present Make-An-Agent, which is a method to generate policy parameters given a few intended trajectories from a task. The proposed method is straightforward, which makes it better that it seems to work in the experiments in the work. The method first generates a large dataset of policies as well as their rollout traces, all parametrized in the same way, using standard RL algorithms. Then, the authors learn an autoencoder on top of the policy parameters. Finally, the method learns a trajectory conditional generator using diffusion for creating policy latents. Putting them together, on a new environments the authors generate some trajectories (here, by running SAC for 0.5M steps), and then generate a new policy using the trajectory latent which gets translated to a policy latent by the diffusion model.\n\nThe experiments are done on a real robot locomotion task as well as two manipulation task suites in simulation. In all of these cases, the best generated policies perform well in the experiments compared with meta-learning approaches.", "questions": "- Please detail the compute budget for each of the baselines as well as your method.\n- If possible, please release the dataset as well as the pretrained networks with your code.", "limitations": "- Generating larger networks will be more difficult, so I wish the authors took advantage of any special properties of network weights while training.\n- The comparison metric is fuzzy and can seem unfair.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The approach is novel, and the algorithm and the released code are both quite simple.\n+ The experiments have a wide breadth, and showing the applicability in multiple different domains show the potential of this method.\n+ While the dataset is large (1000+ policies per task) it is not excessively large to be prohibitive.", "weaknesses": "- The reported metric (best generated policy performance, avg top 5 generated policy performance) both seem _fudged_. It seems like the authors are generating a large number of policies, and then picking the best generated policy out of them to get their numbers. This seems dishonest, as for a large number N of generated policies, every method will get a very high score.\n- While the authors report the \"qualification rate\" i.e the % of policies getting 100% success rate in test, I am not sure how this works. As they report the \"best policy score\", and since qualification rate is > 0 for all tasks, why is figure 5 (ours) not all 100%? What am I missing?\n- The drop between top-1 and top-5 is concerning, since I cannot tell whether the authors got lucky finding a perfect policy from the long tail.\n- The authors don't mention the overall compute overhead and how it compares with the baselines, which I believe is a strong consideration for such a method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer 957J for your acknowledgment of our idea's novelty and method's applicability. Thank you for your valuable comments and suggestions, which are of great help to improve the quality of our work. We carefully answer each of your concerns as below.\n> The reported metric (best generated policy performance, avg top 5 generated policy performance) both seem fudged. It seems like the authors are generating a large number of policies, and then picking the best generated policy out of them to get their numbers. \n\nRegarding the reported metrics, we primarily consider the following perspectives :\n1. **Generate vs. Finetune**: Our method can generate one policy per trajectory, while previous methods can only train one policy through fine-tuning with multiple trajectories. Even with multiple finetuning attempts to get the best result, no method can achieve a very high score. (The baseline results we report are the best models obtained during finetuning)\n\n2. **Fairness and Efficiency**: The test trajectories we use are exactly the same as those used for the baselines. The difference is that we do not need to use trajectories for finetuning but generate policies and evaluate them with one episode to determine if the generated policies are effective. The GPU hours consumed by the inference process and evaluation for 100 policies are listed below.\n\n| Task/Average (100 Trajectories)                                             | GPU Hours (100 Trajectories) |\n|---------------------------------------------------|------------------------------|\n| Finetuning (Baselines/Average)                   | 0.40 +- 0.12                          |\n| Inference + Evaluation (100 Policies)| 0.11 +- 0.0                         |\n\n3. **Usage**: Our goal is to ultimately generate an optimal policy with few trajectories, similar to all previous policy learning methods, but our method achieves better performance. Although we obtained many policies, this paper does not further discuss what can be done with the large number of generated policies. They could potentially be used for learning the mixture of experts, which we believe is a promising future direction.\n\n> While the authors report the \"qualification rate\", i.e, the % of policies getting 100% success rate in test, I am not sure how this works. As they report the \"best policy score\", and since qualification rate is > 0 for all tasks, why is figure 5 (ours) not all 100%? What am I missing?\n\nTo efficiently evaluate policies and save on evaluation costs, we initially validate all generated policies with a single episode under 4 random seeds (consuming very little GPU time). Subsequently, the qualified policies, sorted by episode length from shortest to longest (indicating the time spent on task completion), are tested over 10 episodes to report the final results. Therefore, some policies may occasionally fail in certain episodes, resulting in a success rate that does not reach 100%.\nIt should be noted that evaluating 10 episodes follows the standard metrics of all common baselines, ensuring fairness with other papers. We will add these detailed explanations of the evaluation process in the appendix.\n\n> The drop between top-1 and top-5 is concerning, since I cannot tell whether the authors got lucky finding a perfect policy from the long tail.\n\nBased on the evaluation details provided above, this explains why the performance of the top 5 policies may differ from that of the best policy. The performance can fluctuate across multiple episodes due to the random initialization of the environment, which is a normal phenomenon also observed in single RL policy learning.\n\n> The authors don't mention the overall compute overhead and how it compares with the baselines, which I believe is a strong consideration for such a method. Please detail the compute budget for each of the baselines as well as your method. If possible, please release the dataset as well as the pretrained networks with your code.\n\nIn **Line 186**, we state the GPU hours for collecting policy data for training. In **Line 425, 434, and 438**, we mention the GPU hours for training each model. **Our training time requires relatively few GPU hours compared to other methods (much fewer than single RL training)**. All of our datasets and pretrained models will be released to the community for further valuable research, which we believe contributes to researchers who may have limited resources.\n\n| Method                        | Total Computational Budget (GPU Hours) | \n|-------------------------------|----------------------------------------| \n| Single-task RL                | 7.8                                    | \n| Multi-task Learning                 | 20.4                                  |\n| Meta Learning   | 31.4                                   | \n| **Ours (Including data collection)** | 34.1                          |\n| **Ours (Training/inference/evaluation)** | 4.1                         | \n\n> Generating larger networks will be more difficult, so I wish the authors took advantage of any special properties of network weights while training.\n\nThis is a very insightful topic for discussion. In our method, we choose to encode the parameters of each layer of the network, leveraging the properties of the policy network. To handle larger networks, we believe several approaches can be considered: generating parameters for each layer individually, generating parameters for only parts of networks, or utilizing pretrained task representations to reduce the dimension of inputs to the network, allowing smaller actor networks to tackle complex tasks. These are all promising future directions worth exploring, and we will include this discussion in the appendix.\n\nThanks again for reviewing our paper carefully and providing very constructive suggestions. We hope the above resolves your concerns, and we are glad to have any further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fR6a9xgiVl", "reviewer": "Reviewer_Pxd2", "summary": "This paper proposes the Make-An-Agent architecture, which synthesizes a policy neural network from an input trajectory. Make-An-Agent utilizes a parameter and behavior embedding. The behavior embedding is trained with the mutual information between the trajectory and the successful part of the trajectory. The behavior embedding is then used in a diffusion model to generate the policy parameters. This methodology is validated in MetaWorld, Robosuite, and quadrupedal locomotion environments. In MetaWorld and RoboSuite, Make-An-Agent outperforms baselines RL and IL baselines. Further experiments analyze how the trajectories and policies produced by Make-An-Agent differ from those directly trained through RL.", "questions": "1. What fraction of generated policies outperform the baselines in Table 1? And what is the average policy performance between all generated policies, not just the top 5? See my point (1) under weaknesses above.\n\n1. Given that the results in Figure 11b already show performance suffering with increased policy size, how can Make-An-Actor scale to more complex tasks that require higher parameter count policies?", "limitations": "Yes, limitations are discussed in Section 6.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper represents a novel take on learning multi-task policies by directly generating the parameters of a policy network from a demonstration. This is different from prior Meta-RL approaches that adapt existing networks. \n\n1. The paper has extensive empirical comparisons to a variety of baselines from RL and IL. The proposed method outperforms baselines in MetaWorld and Robosuite by a large margin. \n\n1. Make-An-Agent is capable of generating policies that are more diverse (Figure 8) and more robust (Figure 6) than baselines.\n\n1. The behavior embedding is an important aspect of the Make-An-Agent architecture as demonstrated by Figure 7.\n\n1. The paper analyzes the impact of important settings on the system's performance with the demonstration length, policy size and number of policy parameter sets used for training.\n\n1. Implementation details are thoroughly described throughout the paper and in supplementary sections B and C.", "weaknesses": "1. Table 1 only reports the best and top 5 generated policies. However, all policies had to be evaluated in the environment to decide what these top policies are. This is an unrealistic assumption that weakens the significance of the result. The paper must analyze in more detail the distribution of performance between generated policies and compare the full statistics of all generated policies to baselines for a fair comparison.\n\n1. A weakness of directly generating policy parameters is the challenge of scaling up to larger policy networks. For example, policies operating from egocentric visual inputs in 3D spaces typically have tens of millions of parameters. The core idea of directly generating policy parameters is likely incapable of scaling to this setting of 1000x more policy parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the positive feedback from Reviewer Pxd2 on the originality, overall quality, and significance of our work. The valuable comments and suggestions from Reviewer Pxd2 are of great help to improve the quality of our work. Detailed responses regarding each problem are listed below.\n\n> All policies had to be evaluated in the environment to decide what these top policies are. This is an unrealistic assumption that weakens the significance of the result. The paper must analyze in more detail the distribution of performance between generated policies and compare the full statistics of all generated policies to baselines for a fair comparison.\n\nThe evaluation metrics we reported are chosen primarily for the following reasons:\n1. **Feasibility and efficiency of evaluation:** Our method can generate one policy per trajectory, whereas previous methods can only train one policy through fine-tuning with multiple trajectories. As mentioned in our paper, the key difference is that we do not need to use trajectories for finetuning. Instead, we generate policies and evaluate them with one episode to determine their effectiveness. The final results are then reported after evaluation over 10 episodes, consistent with the baselines. The GPU hours consumed by the inference process and evaluation for 100 policies are listed below. We also utilize IsaacGym simulator to evaluate policies before deploying them on the real robot, ensuring evaluation at a relatively low cost.\n\n\n| Task/Average (100 Trajectories)                                             | GPU Hours (100 Trajectories) |\n|---------------------------------------------------|------------------------------|\n| Finetuning (Baselines/Average)                   | 0.40 +- 0.12                          |\n| Inference + Evaluation (100 Policies)| 0.11 +- 0.0                         |\n\n2. **Effectiveness of all generated policies:** We add a set of results in Figure 13 in our supplementary material to present the performance of all the generated policies compared with baselines. Since we do not actually need to use all the generated policies (just as we do not need to use all generated results), but only need to obtain one optimal policy to consistently complete the task, our method provides a new pathway for addressing unseen tasks. This is in contrast to baselines, which cannot achieve good unseen generalization even with finetuning. We also believe that using the generated multiple optimal policies to learn a mixture of experts for decision-making is a potential future direction.\n\n\n> A weakness of directly generating policy parameters is the challenge of scaling up to larger policy networks. For example, policies operating from egocentric visual inputs in 3D spaces typically have tens of millions of parameters. The core idea of directly generating policy parameters is likely incapable of scaling to this setting of 1000x more policy parameters.\n\nThis is indeed a very worthwhile discussion. Using visual encoders to handle image or 3D point cloud inputs is already a common approach. We can apply our method to generate actor or critic networks but with inputs processed through existing pre-trained visual encoders to manage different environmental inputs. Our paper covers a broad range of policy network sizes, from 2e4 to 16e4 parameters, which is among the widest ranges addressed by existing network generation methods. Additionally, we propose other potential solutions for addressing complex networks in our responses below.\n\n> Given that the results in Figure 11b already show performance suffering with increased policy size, how can Make-An-Actor scale to more complex tasks that require higher parameter count policies?\n\nIn Figure 11b, the decline in performance with larger policy sizes is likely due to the autoencoder's inability to provide effective parameter representations for large networks, which significantly impacts the effectiveness of the generated policies (for fairness, we did not modify any model parameters for large networks). When we scale up the hidden size of the autoencoder from 1024 to 2048, policies with 256 hidden sizes achieve very similar results to those with hidden sizes of 128. Therefore, an effective strategy for handling larger networks might be to use multiple autoencoders to encode different layers of the network separately. Alternatively, generating different layers or portions of larger policy networks tailored to specific task features could facilitate faster adaptation to various complex downstream tasks. We believe that exploring how to generate larger networks is a highly promising potential direction.\n\nThank you again for carefully reviewing our paper and providing very constructive suggestions. We will incorporate the above discussions into our final version. We hope that this addresses your concerns, and we are happy to engage in further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "32CjgivXPY", "reviewer": "Reviewer_8Q7F", "summary": "This paper proposes a novel approach to generate policy parameters based on the behavior through diffusion models. The paper leverages the autoencoder to map the parameters of the policy to latent representations. The model demonstrates remarkable generalization abilities on unseen tasks with few-shot demonstrations.", "questions": "* See weakness.\n* What are the states or what would happen after the first success time? Does the env stop if the task is succeeded?", "limitations": "*  The paper has provided several limitations. The paper only generates parameters for simple linear layers. And the learned behavior embeddings are also important.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The method is novel. The paper learns behavior embeddings from agent behaviors to capture environmental dynamics and information about tasks. And it leverages such behavior embeddings as conditions for the parameter generation.\n* The empirical results are rich and strong. The paper evaluates the model across three environments (two simulations and one real world) and shows strong generalization performance on unseen tasks and for unseen robots.\n* This paper also presents well including figures and written structures.", "weaknesses": "* The generated policies seem unstable. The generation by diffusion models are diverse but not stable. And the paper doesn't make sure the generated policies are good enough. And the experiments compare the best or top 5 generated policies. How to evaluate which policy is best? Does it require testing all policies and finding which one is best? In C.2, the paper shows the qualification rate of generated policies. The qualification rates are very weak (mostly less than 0.1) on unseen tasks except \"drawer close\". It's not practical. What if using expert demonstrations?\n* Some settings of experiments seem unreasonable. Why fix the initial locations during the training stage and finetuning/adaptation? This might make the baselines or generated policies only work on fixed initial locations. Similarly, why choose highly sub-optimal trajectories? If so, it may use offline RL approaches to finetune the baselines.\n* Lack of details of baselines. How to finetune the multi-task RL with fixed trajectories (10/50/100)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer 8Q7F's acknowledgment of the novelty, empirical results, and presentation of our paper. Your feedback is very helpful in improving the quality of our work. Below are our detailed responses to each of your questions:\n\n> The generated policies seem unstable. The generation by diffusion models are diverse but not stable. \n\nThe instability in the generated policies primarily arises from two factors:\n\n1. We use trajectories from the replay buffer for generation, which introduces instability in the results. Figure 13 in our supplementary material illustrates the correlation between the effectiveness of condition trajectories and the performance of generated policies. That is reasonable because failed trajectories may not provide sufficient information as prompts for generation.\n\n2. For unseen tasks, the unfamiliarity and sensitivity in exploring the parameter space lead to greater variability in results. However, compared to other generalization methods that struggle to achieve optimal policies on unseen tasks, our approach is capable of discovering optimal policies in such scenarios.\n\n> How to evaluate which policy is best? Does it require testing all policies and finding which one is best? \n\nTo efficiently evaluate policies, we initially validate all generated policies with a single episode under 4 random seeds (**only need 2 minutes**). Subsequently, the qualified policies, sorted by trajectory length from shortest to longest (indicating the used steps for task completion), are tested over 10 episodes to report the final results following all the baselines. Although we need to evaluate all the policies, the GPU hours consumed are significantly less than finetuning required by baselines.\n\n| Task/Average (100 Trajectories)                                             | GPU Hours (100 Trajectories) |\n|---------------------------------------------------|------------------------------|\n| Finetuning (Baselines/Average)                   | 0.40 +- 0.12                          |\n| Inference + Evaluation (100 Policies)| 0.11 +- 0.0                         |\n\n- The qualification rates are very weak (mostly less than 0.1) on unseen tasks except \"drawer close\". It's not practical. What if using expert demonstrations?\n\nFigure 13 illustrates that the performance of generated policies on unseen tasks is strongly influenced by unfamiliar suboptimal condition trajectories.  Using expert demonstrations can largely improve the qualification rate of generated policies on these \"very weak\" unseen tasks.\n\n|         Qualification rate  (%)                           |door lock| button press wall |handle press side| faucet open |reach wall |  coffee button\n|---------------------------------------------------|----------|--------------|---------------|---------------|-------------|-------------|\n|**Ours**|0.43|0.35|0.60|0.39| 0.58 | 0.62|\n\nWe further compared our method with the best baseline (Meta DT) fine-tuned using expert demonstrations. Our approach still outperforms baselines. We will include the experimental results using expert demonstrations in our paper.\n\n|         Success rate  (%)                           | Unseen 8 tasks (Average)|\n|---------------------------------------------------|----------|\n|Baseline |0.79 +- 0.06|\n|**Ours (Top 5)**|1.0 +- 0.0|\n\n- Why fix the initial locations during the training stage and finetuning/adaptation? \n\nDuring training, we fix the initialization because we collect multiple policies from a single RL training, which can effectively reduce the computational cost of data collection (instead of changing the initialization each time to collect only one policy). The test trajectories used for finetuning/generating are derived from a single training buffer, thus having the same initialization. For our evaluation results, we conduct experiments in 4 randomly initialized environments over 10 episodes to obtain results. The results indicate that our generated policies do not only work on fixed initial locations.\n\n- Similarly, why choose highly sub-optimal trajectories? If so, it may use offline RL approaches to finetune the baselines.\n\nAs mentioned in **Lines 187-188**, we select test data from the SAC training replay buffer within the first 0.5 million timesteps considering that expert demonstrations are not always available and using similar expert trajectories might not provide sufficient diversity.\nWe compare our approach with multi-task offline RL methods: skills regularized task decomposition (SRTD). We train the policy using offline training data and then update the policy using test trajectories on unseen tasks. Offline multi-task RL performs worse than imitation learning methods and our approach demonstrates a significant advantage.\n\n|         Success rate  (%)                           | Unseen 8 tasks (Average)|\n|---------------------------------------------------|----------|\n| SRTD |0.61 +- 0.02|\n|**Ours (Top 5)**|0.86 +- 0.07|\n\n[1]Yoo, Minjong, Sangwoo Cho, and Honguk Woo. \"Skills regularized task decomposition for multi-task offline reinforcement learning.\" Advances in Neural Information Processing Systems 2022.\n\n- How to finetune the multi-task RL with fixed trajectories (10/50/100)?\n\nPrevious multi-task RL methods have poor performance on unseen tasks. We update the mixture of encoders and critic networks to improve its generalizability. We will include offline multi-task RL as baselines in our paper. \n\n- What are the states or what would happen after the first success time? \n\nIn all simulator settings, the environment resets after reaching the maximum episode length. The agent's actions continue selected by the policy network after success. We maintain this environment setting consistent with all baselines.\n\nWe thank Reviewer 8Q7F for the detailed and valuable feedback. All the discussions and additional results will be included in the final version. We look forward to further discussions to ensure that our answers address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xiytO1XpqN", "reviewer": "Reviewer_YLrf", "summary": "Make-An-Agent is a novel policy network generator that uses conditional diffusion models to create control policies based on a single demonstration of desired behaviors. By encoding behavior trajectories into embeddings, Make-An-Agent generates latent policy parameter representations that are decoded into functional policy networks. This method, trained on a dataset of policy parameters and their corresponding trajectories, excels in generating versatile and scalable policies across various tasks, including unseen ones, using few-shot demonstrations. Its robustness and efficiency are demonstrated in both simulation and real-world environments, highlighting its ability to produce high-performing policies even from noisy inputs. This approach bypasses traditional behavior modeling, instead leveraging the inherent correlations in agent behaviors to optimize policy generation without the need for further fine-tuning.", "questions": "How does Make-An-Agent perform when faced with tasks that are vastly different from those it was trained on? Are there any specific limitations in task diversity that the model struggles with?\n\n What are the computational requirements for training and deploying Make-An-Agent? How does its efficiency compare to traditional reinforcement learning and other policy generation methods?\n\n How sensitive is the method to the quality of the behavior demonstrations? What happens if the demonstrations contain suboptimal or noisy behaviors?\n\n Can the approach be scaled to more complex tasks and larger environments? What modifications, if any, would be needed to handle such scalability?\n\n What are the contributions of the individual components of the method, such as the autoencoder, contrastive learning for behavior embedding, and the diffusion model? Are there ablation studies that show the impact of each component on the overall performance?\n\n What challenges were encountered during the deployment of the generated policies onto real-world robots? How were these challenges addressed?\n\n The paper mentions robustness to noisy trajectories, but to what extent can the model handle extreme levels of noise or inaccuracies in the demonstrations?", "limitations": "The training process for the autoencoder and diffusion model is likely to be resource-intensive, potentially limiting accessibility for researchers with limited computational resources.\n\nThe effectiveness of the generated policies is heavily reliant on the quality of the input demonstrations. Poor or suboptimal demonstrations could adversely affect the performance of the generated policies.\n\n The evaluation might be limited to specific tasks and environments. A broader range of experiments would be needed to fully assess the model's robustness and versatility across various domains and conditions.\n\n While the method shows promise in the tested domains, its scalability to more complex tasks and larger environments is not fully explored. Handling such scalability might require further modifications and optimizations.\n\n There is a risk that the model might overfit to the types of tasks and behaviors seen during training, which could reduce its effectiveness in truly novel and diverse environments.\n\n While the method has shown success in real-world robotics, the transition from simulation to real-world applications can introduce unforeseen challenges and complexities that need to be addressed comprehensively.\n\n The paper may lack thorough ablation studies to understand the contributions of individual components of the method, such as the autoencoder, behavior embeddings, and diffusion model.\n\nAlthough the method claims robustness to noisy trajectories, the extent to which it can handle extreme levels of noise or inaccuracies in the input demonstrations is not fully detailed.\n\n The sensitivity of the model to various hyperparameters and the process of tuning these parameters for optimal performance is not extensively covered, which can be crucial for practical implementations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a novel method that uses conditional diffusion models for policy generation, a significant shift from traditional policy learning methods.\n2. Make-An-Agent demonstrates the ability to generate effective policies for a wide range of tasks by conditioning on behavior embeddings, showcasing scalability across different domains.\n3. The diffusion-based generator exhibits strong generalization capabilities, producing proficient policies even for unseen behaviors and unfamiliar tasks.\n4. The method can generate diverse and resilient policy parameters, maintaining high performance under environmental variability and with noisy input data.\n5. The effectiveness of Make-An-Agent is validated not only in simulations but also in real-world robotics, highlighting its practical applicability and efficiency.", "weaknesses": "While the paper demonstrates the method's effectiveness in various tasks, a more extensive evaluation across a broader range of environments and conditions could provide a more comprehensive assessment of its robustness and versatility.\n\nThe performance of the generated policies heavily relies on the quality of the input demonstrations, which might not always be optimal or available in all scenarios.\n\nThe model might overfit to the specific types of tasks and behaviors seen during training, potentially reducing its effectiveness in truly novel and diverse environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer YLrf for the insightful feedback and the acknowledgment of our work's novelty and empirical effectiveness. Below, we provide a detailed response to address your concerns:\n\n- **W1 and L3: Limited evaluation**: Our experiments cover **3 domains and 23 tasks**, including key control tasks like tabletop manipulation and locomotion control. As acknowledged by Reviewers 8Q7f, 957J, and U6yw, our experiments are more extensive than all the baselines. We also evaluated our approach in real dynamic tasks, including sharp turns, obstacle avoidance on mats, and rapid backward movements. We present more task visualization in Figure 15 in our supplementary material. If you have any further detailed feedback about our experiments, we would be happy to discuss it further.\n\n- **W2, Q3 and L2: Reliance on input demonstrations**: There may be some misunderstanding here. As stated in **Line 187-188**, we use trajectories from RL replay buffer to generate policies. We do not use optimal trajectories or select specific ones, ensuring availability and diversity of conditions. Experiments with noisy trajectories also show that our method can synthesize effective policies under perturbed conditions.\n\n- **W3 and L5: Overfitting to specific tasks and behaviors**: In **Figure 8**, we compare the behaviors used as generation conditions with the behaviors of generated policies, highlighting the differences between them. **Figure 10** demonstrates that our method explores the parameter space more extensively, not overfitting to specific tasks. Additionally, finetuning the model on significantly different environments (Metaworld to Robosuite) proves our generator's effectiveness across domains.\n\n- **Q1: Performance on vastly different tasks**: We present additional generalization experiments on significantly different tasks like soccer, box close, and sweep into in Metaworld. Our method consistently finds optimal policies, outperforming baselines. As tasks with large differences may have significantly different parameter distributions, this issue exists in both prior policy learning methods and our method. Our work aims to minimize adaptation costs as much as possible.\n\n| Method / Success Rate          | Soccer | Box Close | Sweep Into |\n|--------------------------------|--------|-----------|------------|\n| Meta RL with Hypernetwork      | 42.4%  | 55.2%     | 64.3%      |\n| Meta Decision Transformer      | 55.4%  | 52.1%     | 61.5%      | \n| **Ours** (Top 5)               | 69.2%  | 75.3%     | 87.5%      |\n| **Ours** (Best)                | 92.7%  | 87.4%     | 93.8%      |\n\n- **Q2 and L1: Computational requirements**: In **Line 186**, we specify the GPU hours required for collecting policy data for training. In **Lines 425, 434, and 438**, we mention the GPU hours required for training each model. We compare the computational budget in Figure 14 and the below table. Our method requires relatively few GPU hours compared to baselines (much fewer than single RL training). Our training dataset will be released to the community for further valuable research, which we believe contributes to researchers who may have limited resources.\n\n| Method                        | Total Computational Budget (GPU Hours) | \n|-------------------------------|----------------------------------------| \n| Single-task RL                | 7.8                                    | \n| Multi-task Learning                 | 20.4                                  |\n| Meta Learning   | 31.4                                   | \n| **Ours (Including data collection)** | 34.1                          |\n| **Ours (Training/inference/evaluation)** | 4.1                         | \n\n- **Q4 and L4: Scalability**: Our method handles policy networks with parameters ranging from 2e4 to 1.6e5, suitable for common continuous control scenarios. This is both more diverse and larger than the policy networks used in other policy generation methods. For more complex tasks, we can integrate pre-trained visual foundation models as encoders to process image features while using our method to generate actor networks.\n\n- **Q5, L7 and L9: Component contributions and ablation studies**: **Figure 7** shows results without behavior embedding. Since we use a latent diffusion model as the backbone, we can't remove the autoencoder to directly generate the policy parameters, as this would consume a lot of computational cost. \nThe three parts of our method are not independent and separable but work together through the autoencoder and behavior embedding to reduce the difficulty of generating in the parameter space and to improve efficiency and effectiveness. Our ablation experiments include detailed studies on behavior embedding and parameter representations. The hyperparameters for our model are based on stable diffusion, with minimal impact on results. All models were trained using a single set of hyperparameters without tuning. \n\n- **Q6 and L6: Real-world deployment challenges**: Deploying policies trained in simulators to real-world environments posed challenges due to the changing terrain and stability. We addressed this by adjusting environment randomness during policy dataset collection, making generated policies more adaptable to real-world dynamics.\n\n- **Q7 and L8: Robustness to noisy trajectories**: As mentioned in **Line 237**, we mention that we use the common setting in adversarial RL by adding Gaussian noise with a standard deviation of 0.1 to actions in test trajectories, which can demonstrate the effect of perturbed trajectories on the generated results within a reasonable range. \n\nIn summary, we are grateful to Reviewer YLrf for the detailed feedback. Besides the questions that can be answered within our paper, we will include new discussions from the rebuttal in the appendix to enrich our paper. We hope our response sufficiently addresses your concerns. We eagerly anticipate further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response
### **Summary of Review and Highlights**
We sincerely thank all reviewers for their insightful comments, valuable questions, and helpful suggestions.

We appreciate the positive feedback from all reviewers regarding our paper's **presentation (Reviewers U6yw, 8Q7F)**, **idea novelty (Reviewers 957J, Pxd2, 8Q7F, YLrf)**, **strong empirical performance (Reviewers 957J, Pxd2, 8Q7F, YLrf)**, and **generalization ability of our method (Reviewers Pxd2, 8Q7F, YLrf)**. We are particularly grateful to **Reviewers 8Q7F, 957J, U6yw, and Pxd2 for highlighting the thoroughness and completeness of our experiments**. Our method introduces a novel generative framework for policy learning that not only offers new insights for downstream generalization but also demonstrates impressive effectiveness across various scenarios.

### **Reviewer Concerns**
We carefully consider the reviewers' feedback and responded to each question with detailed explanations and additional experimental results. Here are our responses to the main concerns:

**Concerns about Computational Cost (Reviewers YLrf, 8Q7F, 957J):** As stated in our paper, the computational cost of training our method is even lower than that of single RL training, with the main GPU hours allocated to collecting data for policy networks.  The overall computational time of our method is comparable to baselines as shown in Figure 14. As one of our contributions, we will open-source all models and datasets to support further valuable research by the community.

**Concerns about Scalability (Reviewers YLrf, 957J, Pxd2):** We demonstrate the scalability of our method by showing the parameter number used across different domains, which is more scalable than existing network generation methods. We also discuss various approaches to address more complex networks, which will be included in the appendix.

**Concerns about Evaluation Metrics (Reviewers 8Q7F, 957J, U6yw, Pxd2):** We explain the differences between our approach and prior policy generalization methods (specifically, our approach does not require finetuning) and highlight the efficiency of our evaluation phase. We clarify the choice of evaluation metrics from multiple perspectives and analyze the performance of all generated policies in Figure 13 in our supplementary material.

### **Additional Results**
* We present the relationship between the effectiveness of generated policies and condition trajectories, as shown in Figure 13.
* The computational budgets compared with baselines are presented in Figure 14.
* In response to Reviewer YLrf, we show generalization results on more diverse unseen tasks.
* In response to Reviewer 8Q7F, we include results of policy generation using expert demonstrations and compare them with offline RL baselines.
* In response to Reviewer U6yw, we demonstrate the improvement in robustness of our generated policies using a majority vote approach.

We sincerely thank the reviewers and the AC for their time and thoughtful feedback on our paper. We hope that our responses have effectively addressed all questions and concerns, and we eagerly await further discussion and feedback.

Meta-review: The paper "Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion" introduces an approach for generating control policies using conditional diffusion models based on demonstrations of desired behaviors. The authors propose encoding behavior trajectories into embeddings, which are then used to generate latent policy parameter representations. These representations are subsequently decoded into functional policy networks. The reviewers acknowledge the originality of the idea, the good empirical performance, and the generalization ability of the proposed method.

The strengths of the paper lie in its use of conditional diffusion models for policy generation, which represents a departure from traditional policy learning methods. Make-An-Agent demonstrates the ability to generate effective policies for a wide range of tasks by conditioning on behavior embeddings, which shows its scalability across different domains. The diffusion-based generator exhibits strong generalization capabilities, producing proficient policies even for unseen behaviors and unfamiliar tasks. Moreover, the method can generate diverse and resilient policy parameters, maintaining high performance under environmental variability and with noisy input data. The effectiveness of Make-An-Agent is validated not only in simulations but also in real-world robotics, highlighting its practical applicability and efficiency.

The reviewers raised several important questions and concerns, which the authors addressed in their rebuttal. They provided additional experiments and analyses to support their claims, including the relationship between the effectiveness of generated policies and condition trajectories, the computational budgets compared with baselines, and generalization results on more diverse unseen tasks. The authors also included results of policy generation using expert demonstrations and compared them with offline RL baselines, demonstrating the superiority of their approach.

While one reviewer (Reviewer **YLrf**) recommended rejection, the concerns were not as in-depth as those raised by the other reviewers. The authors have earnestly responded to all the reviews, providing detailed explanations and additional experimental results to address the concerns. As the AC, I do not place significant weight on the reasons provided by Reviewer **YLrf** for rejection, as the authors have adequately responded to the concerns raised, and Reviewer **YLrf** did not respond further.

The reviewers' concerns have mostly been adequately responded to in the rebuttal, and the authors have committed to improving the paper based on the feedback received.  Based on the above reasons, the AC recommends accepting this paper for publication.
Decision: Accept (poster)