Title: Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Model

Policy Learning

Parameter Generation

Reinforcement Learning
Abstract: Can we generate a control policy for an agent using just one demonstration of desired behaviors as a prompt, as effortlessly as creating an image from a textual description?
In this paper, we present **Make-An-Agent**, a novel policy parameter generator that leverages the power of conditional diffusion models for behavior-to-policy generation. Guided by behavior embeddings that encode trajectory information, our policy generator synthesizes latent parameter representations, which can then be decoded into policy networks. 
Trained on policy network checkpoints and their corresponding trajectories, our generation model demonstrates remarkable versatility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs. We showcase its efficacy and efficiency on various domains and tasks, including varying objectives, behaviors, and even across different robot manipulators. Beyond simulation, we directly deploy policies generated by **Make-An-Agent** onto real-world robots on locomotion tasks. Project page: https://cheryyunl.github.io/make-an-agent/.

Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion

Abstract

Can we generate a control policy for an agent using just one demonstration of desired behaviors as a prompt, as effortlessly as creating an image from a textual description? In this paper, we present Make-An-Agent , a novel policy parameter generator that leverages the power of conditional diffusion models for behavior-topolicy generation. Guided by behavior embeddings that encode trajectory information, our policy generator synthesizes latent parameter representations, which can then be decoded into policy networks. Trained on policy network checkpoints and their corresponding trajectories, our generation model demonstrates remarkable versatility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs. We showcase its efficacy and efficiency on various domains and tasks, including varying objectives, behaviors, and even across different robot manipulators. Beyond simulation, we directly deploy policies generated by Make-An-Agent onto real-world robots on locomotion tasks.

1 Introduction

Policy learning traditionally involves using sampled trajectories from a replay buffer or behavior demonstrations to learn policies or trajectory models mapping from state s to action a , modeling a narrow behavior distribution. In this paper, we consider a shift in paradigm: moving beyond training a policy, can we reversely predict optimal policy network parameters using suboptimal trajectories from offline data? This approach would obviate the need to explicitly model behavior distributions, allowing us to learn the underlying parameter distributions in the parameter space, thus revealing the implicit relationship between agent behaviors for specific tasks and policy parameters.

Using low-dimensional demonstrations (such as agent behavior) to guide the generation of highdimensional outputs (policy parameters) is a challenging problem. When diffusion models [12, 20] have demonstrated highly competitive performance on various tasks including text-to-image synthesis, we are inspired to approach policy network generation as a conditional denoising diffusion process. By progressively refining noise into structured parameters, the diffusion-based generator can discover various policies that are not only superior in performance but also more robust and efficient than the demonstration in the policy parameter space.

parameter space, without the need for any downstream policy fine-tuning or adaptation with gradient updates. Since behaviors - as the observable manifestation of deployed policies - from different tasks often share underlying skills or environmental information, our policy generator can exploit these potential correlations in the parameter space, such as shared parameters for similar motion patterns, which leads to enhanced cross-task one-shot generalizability. What we need is an end-to-end behavior-to-policy generator, not a shared base policy.

To achieve this, we introduce Make-An-Agent , featuring three key technical contributions: (1) We propose an autoencoder that encodes policy networks into compact latent representations based on their network layers, which can also effectively reconstruct the original policy from its latent representation. (2) We leverage contrastive learning to capture the mutual information between long-term trajectories and their success or future states. This approach yields a novel and efficient behavior embedding. (3) We utilize a simple yet effective diffusion model conditioned on the learned behavior embeddings, to generate policy parameter representations, which are then decoded into deployable policies using the pretrained decoder. (4) We construct a pretrained dataset of policy network parameters and corresponding deployed trajectories to train our proposed methodology.

To investigate the generation performance of Make-An-Agent , we evaluate our approach in three continuous control domains including diverse tabletop manipulation and real-world locomotion tasks. During test time, we generate policies using trajectories from the replay buffer of partially-trained RL agents. The policies generated by our method demonstrate superior performance compared to policies produced by multi-task [28, 23] or meta learning [7, 27] and other hypernetwork-based generation methods [1]. Our generator offers several key advantages:

Versatility : Make-An-Agent excels in generating effective policies for a wide range of tasks by conditioning on agent behavior embeddings. Since we train the parameter generator for latent parameter representations, it can generate policy networks of varying sizes within the latent space, demonstrating scalability.

Generalizability: Our diffusion-based generator demonstrates robust generalization, yielding proficient policies even for unseen behaviors or unseen embodiments in unfamiliar tasks.

Robustness : Our method can generate diverse policy parameters, exhibiting resilient performance under environmental randomness from simulators and real-world environments. Notably, Make-An-Agent can synthesize high-performing policies when fed with noisy trajectories, highlighting the robustness of our model.

2 Backgrounds

Policy Learning. Reinforcement Learning (RL) is structured within the formation of Markov Decision Processes (MDPs) [2], which is defined by the tuple M = ⟨S , A , P, R , γ ⟩ . Here, S signifies the state space, A the action space, P the transition probabilities, R the reward function and γ the discount factor. RL aims to optimize an agent's policy π : S → A , which outputs action a t based on state s t at each timestep, to maximize cumulative rewards. The optimal policy can be expressed as:

where z represents a trajectory generated by following policy π . In deep RL, policies π are represented using neural network function approximations [22], parameterized by θ π , facilitating the learning of intricate behaviors across high-dimensional state and action spaces.

Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) [12] are generative models that frame data generation through a structured diffusion process, which involves iteratively adding noise to the data and then denoising it to recover the original signal. Given a sample x 0 , the forward diffusion process to obtain x 1 , x 2 , ..., x T of increasing noise intensity is typically denoted by:

where q is the forward process, N is Gaussian noise, and β t ∈ (0 , 1) is is the noise variance.

The denoising process, which is the reverse of the forward diffusion, can be formulated as:

where p θ denotes the reverse process, µ θ and Σ θ are the mean and variance of the Gaussian distribution respectively, which can be approximated by a noise prediction neural network parameterized by θ .

Diffusion models aim to learn reverse transitions that maximize the likelihood of the forward transitions at each time step t . The noise prediction network θ is optimized using the following objective, as the function mapping from ϵ θ ( x t , t ) to µ θ ( x t , t ) is a closed-form expression:

Here, ϵ ∼N ( 0 , I ) , is the target Gaussian noise, ¯ α t := ∏ t s =1 1 -β s , and √ ¯ α t x 0 + √ 1 -¯ α t ϵ is the estimated distribution of x t from the closed-form relation.

Although diffusion models are typically used for image generation through the reverse process, the variable x can be generalized to represent diverse entities for generation. In this paper, we adapt x to represent the parameters θ π of the policy network in policy learning.

3 Methodology

Latent Diffusion

Model

Behavior

embeddings

Conditional

inputs

Latent parameter

representations

Generated

deployable

policy

denoising

Inference process

Agent's trajectory

…

Autoencoder

Forward process

decode

…

encode

noising

from training

dataset

vectorize

vectorized

Policy

network

parameters

Encoder

Decoder

Latent

Representations

…

Autoencoder

decoding

encoding

s

Contrastive

Loss

0

a

0

s

n-1

…

a

n-1

s

n

a

n

Projection

𝝓

s

K

s

K+m

…

Projection

𝝋

h

v

Long trajectory with

𝑛

steps

After the first

success step

𝐾

Behavior embeddings

Overview. An overview of our proposed methodology is illustrated in Figure 1. To achieve this, we address several key challenges: (1) Developing latent representations of high-dimensional policy parameters that can be effectively reconstructed into well-functioned policies. (2) Learning an embedding of behavior demonstrations that serves as an effective diffusion condition. (3) Training a conditional diffusion model specifically for policy parameter generation.

Parameter representation. We use an MLP with m layers as the common policy approximator. Consequently, when the full parameters of a policy are flattened, they form a high-dimensional vector.

To enable generation with limited computational resources while retaining efficacy, and to support the generation of policies for different domains with varying state and action dimensions, we compress the policy network parameters into a latent space.

Based on the policy network architecture, we unfold the parameters following the architecture of the policy network, represented as x = [ x 0 , x 1 , . . . , x m -1 ] , where x i denotes the flattened parameters from each layer. The encoder E encodes each x i as z i , resulting in a parameter latent representation denoted as z = [ z 0 , z 1 , . . . , z m -1 ] , where each z i in the latent space has the same dimension, while the decoder D can decode z into x . To improve the robustness of this procedure, we introduce random noise augmentation in both encoding and decoding during training. Given each vectorized parameter as x , we minimize the objective as,

where z = E ( x + ξ D ) , and ξ E and ξ D represent the augmented noise. The architecture of the autoencoder is shown in Figure 2.

For each domain, the autoencoder for parameter representation only needs to be trained once before parameter generation, which can handle policy parameters from different tasks. To facilitate the generalizability of the policy generator across domains, we design the latent parameter representations to have the same dimensions for different domains.

Behavior embedding. Since our goal in learning behavior embeddings is not to model the distribution of states and actions, but to provide conditional information for policy parameter generation, we aim for them to encapsulate both crucial environmental dynamics and the key information of the task goal. The principle behind our behavior embeddings is to learn the mutual information between preceding n step trajectories and subsequent states with success signals.

We propose a novel contrastive method to train behavior embeddings. In Figure 3, we present a design demonstration of our contrastive loss. For a long trajectory τ , we decouple it as the n initial state-action pairs τ n = ( s 0 , a 0 , s 1 , a 1 , . . . , s n , a n ) and the m states after the first success time K as ˆ τ = ( s K , s K +1 , . . . , s K + m ) . Given a batch of trajectory sequences { τ i } N i =1 which can be presented as { τ n i , ˆ τ i } N i =1 , we optimize the contrastive objective [16, 31] as:

where h i = ϕ θ ( τ n i ) and v i = ψ θ (ˆ τ i ) are embeddings from different parts of the long trajectory τ i and W is a learnable metric that measures the similarity between embeddings h i and v i .

For each trajectory τ , we obtain a set of embeddings τ e = { h i , v i } . In practice, the choice of specific embeddings can be tailored to the characteristics of different tasks and trajectories. We use ( h i , v i ) as the conditional input in our experiments.

Flexibility. With the consideration that in many scenarios, rewards are often sparse or non-existent, whereas success signals serve as a more direct indicator of whether a policy has achieved its objective. We therefore use original trajectories that exclude reward information but include success information.

For tasks without explicit success signals, such as locomotion, we segment long trajectories into multiple shorter trajectories. For each segment, we use the last m states as ˆ τ and the 0 -n state-action pairs as τ n . The informative behavior embeddings of a long trajectory are concatenated from the embeddings of all the trajectory segments.

This embedding approach strives to capture the essential information for generating behavior-specific policy parameters, including environmental dynamics and task goals, using the most concise representation possible from long trajectories and prioritizing flexibility and efficiency.

Conditional policy generator. After training the parameter autoencoder and behavior embeddings, for policy parameter x and the corresponding trajectory g deployed by policy x , we can transfer x as latent parameter representation z with the autoencoder E and trajectory τ as behavior embedding τ e . The conditional diffusion generator is trained on latent representation z , conditioning on τ e . We optimize the conditional latent diffusion model via the following loss function:

where the neural backbone ϵ θ ( z t , τ e , t ) is implemented as a 1D convolutional UNet [19] parameterized by θ and t is uniformly sampled from { 1 , . . . , T } . The outputs of our parameter generator can be encoded by D as deployable policies. During training the diffusion model, both the parameter autoencoder and behavior embedding layers are frozen, which ensures the training stability and efficiency.

Dataset. We build a dataset containing tens of thousands of policy parameters and trajectories from deploying these policies. The dataset is obtained from multiple RL training across a range of tasks. We utilized the dataset to train both the autoencoder and behavior embedding models. Then we use the encoded parameter representations and behavior embeddings derived from the collected trajectory to train the conditional diffusion model for policy parameter generation.

4 Experiments

MetaWorld

Robosuite

We conduct extensive experiments to evaluate Make-An-Agent , answering the following problems:

How does our method compare with other multi-task or learning-to-learn approaches for policy learning, in terms of performance on seen tasks and generalization to unseen tasks?

How scalable is our method, and can it be fine-tuned across different domains?

Does our method merely memorize policy parameters and trajectories of each task, or can it generate diverse and new behaviors?

Benchmarks. We include two manipulation benchmarks for simulated experiments and real-world robot tasks to show the performance and capabilities of our method as visualized in Figure 4.

MetaWorld. MetaWorld [29] is a benchmark suite for robotic tabletop manipulation, consisting of a diverse set of motion patterns for the Sawyer robotic arm and interactions with different objects. We selected 10 tasks for training as seen tasks and 8 for evaluation as unseen downstream tasks. Detailed descriptions of these tasks can be found in Appendix C.1. The state space of MetaWorld consists of 39 dimensions and the action space has 4 dimensions. The policy network architecture used for MetaWorld is a 4-layer MLP with 128 hidden units, containing a total of 22,664 parameters.

Robosuite. Robosuite [33], a simulation benchmark designed for robotic manipulation, supports various robots such as the Sawyer and Panda arms. We train models on three manipulation tasks: Block Lifting, Door Opening and Nut Assembly, using the single-arm Panda robot. Evaluations are conducted on the same tasks using the Sawyer robot. This experimental design aims to validate the practicality of our approach by assessing whether the generated policy can be effectively utilized on different robots. In the Robosuite environment, the state space comprises 41 dimensions, and the action space consists of 8 dimensions. The policy network employed for this domain contains 23,952 parameters.

Quadrupedal locomotion. To evaluate the policies generated by Make-An-Agent in the real world, we utilize walk-these-ways [14] to train policies on IsaacGym and use our method to generate actor networks conditioning on trajectories from IsaacGym simulation with the pretrained adaptation modules. Then, we deploy the generated policy on real robots in environments differ from simulations. The policies generated for real-world locomotion deployment comprise 50,956 parameters.

Dataset. We collect 1500 policy networks for each task in MetaWorld and Robosuite. These networks are sourced from policy checkpoints during SAC [11] training. The checkpoints are saved every

5000 training steps once the test success rate reaches 1. During the training stage, we fix the initial locations of objects and goals and train the policies using different random seeds. For each task, we require an average of 8 SAC training runs, approximately 30 GPU hours.

For evaluation, the trajectories used as generation conditions are sampled from the SAC training buffer within the first 0.5 million timesteps, which can be highly sub-optimal, under the same environmental initialization. During testing, the generated policies are evaluated in 5 random initial configurations, thoroughly assessing the robustness of policies generated using trajectories from the fixed environment settings.

In RoboSuite experiments, due to the inconsistency in policy networks, we retrain the autoencoder and finetune the diffusion generator trained on MetaWorld data. The experimental setup for RoboSuite is almost identical to that of MetaWorld, with the only difference being the robot used during testing.

For real-world locomotion tasks, we save 10,000 policy network checkpoints using walk-these-ways (WTW) [14] trained on IsaacGym, requiring a total of 200 GPU hours. The 100 trajectories used as generation conditions are sourced from the first 10,000 training iterations of WTW.

Baselines. We compare Make-An-Agent with four baselines, including multi-task imitation learning (IL), multi-task reinforcement learning (RL), meta-RL with hypernetworks, and meta-IL with transformers. These represent state-of-the-art methods for multi-task policy learning and adaptation. For a fair comparison, each baseline uses the same testing trajectory data for downstream adaptation.

Multi-task BC [24]: We train a multi-task behavior cloning policy using trajectories in our training dataset, and then finetune it with test trajectories to adapt in specific tasks.

Multi-task RL, CARE [23]: We train an mixture of encoders for 2 million steps (for each task). For RL training, we train the algorithm in dense reward environments and finetune the model using test trajectories with sparse rewards, where feedback is only provided at the end of a trajectory.

Meta-RL with hypernetworks [1]: We train a hypernetwork with our training data with dense rewards, which can adapt to different task-specific policies during testing with test trajectories.

Meta Imitation Learning with decision transformer(DT) [4] We train the pre-trained DT model using the training trajectories in our dataset, then use the test trajectories from replay to adapt it to test tasks.

4.1 Performance Analysis

MetaWorld (10seentasks)

Robosuite (3 seen tasks)

Multi-task IL

100%

100%

Multi-task RL

Rate

Meta IL

80%

80%

Success

Generatedtop5

GeneratedBest

60%

60%

40%

40%

5

10

50

100

5

10

50

100

Trajectories

Trajectories

By using test trajectories as conditions, our policy generator can produce an equivalent number of policy parameters. Compared with baselines, we report both the best result among the generated policies and the average performance of the top 5 policies. All algorithms use the same task-specific replay trajectories. The difference is that we use them as generation conditions, whereas other methods use them for adaptation.

We define policies achieving a 100% success rate during evaluation as qualified policies. The analysis of qualification rates for policies generated by our model is presented in Appendix C.2.

MetaWorld(8unseentasks)

Robosuite(unseenrobots)

100%

Rate

80%

Success

60%

40%

50 Trajectories

100 Trajectories

50 Trajectories

100Trajectories

Multi-task IL

Generated top5

GeneratedBest

Multi-task RL

Meta IL

Adaptability to environmental randomness on seen tasks. Figure 5 demonstrates the significant advantage of our algorithm over other methods on seen tasks. This is attributed to the fact that, despite test trajectories originating from the same environment initialization, the generated policy parameters are more diverse, thus possessing a strong ability to adapt to environmental randomness. In contrast, other algorithms, when adapted using such singular trajectories, exhibit more limited adaptability in these scenarios. Our experimental design aligns with practical requirements, as real-world randomness is inherently more complex.

Generalizability to unseen tasks. Figure 6 showcases the superior performance of our algorithm on unseen tasks. Test trajectories originate from the same environment setting for each task, while evaluation occurs in randomly initialized environments. Our policy generator, without fine-tuning, directly utilizes test trajectories as input, demonstrating a remarkable ability to generate parameters that work on unseen tasks. The agent's behavior in unseen tasks exhibits similarities to seen task behaviors, such as arm dynamics and the path to goals. By effectively combining parameter representations related to these features, the generative model successfully generates effective policies. In contrast, baseline methods struggle to adapt in environmental randomness.

These results strongly suggest that our algorithm, compared to other policy adaptation methods, may offer a superior solution for unseen scenarios. To further investigate robustness in generalization, we added Gaussian noise with a standard deviation of 0.1 to actions in test trajectories used for policy generation or adaptation on unseen tasks. Figure 7 demonstrates that our method remains resilient to noisy inputs, while the performance of the baselines is significantly impacted. We believe this is because our behavior embeddings only need to capture key dynamic information as conditions to generate policies, without directly learning state-action relationships from trajectories, resulting in better robustness.

Trajectory difference. To compare the difference between using test trajectories as conditions and the trajectories obtained by deploying the generated policies, we visualize the trajectories during unseen task evaluations. As shown in Figure 9, our diffusion generator can synthesize various policies, which is significantly different from policy learning methods that learn to predict actions or states from trajectory data. We believe that this phenomenon fully illustrates the value of our proposed policy parameter generation paradigm.

Parameter distinction. Beyond trajectory differences, we also investigate the distinction between synthesized parameters and RL policy parameters. We calculate the cosine similarity between the RL policies used to obtain the test trajectories and the parameters generated from these trajectories. As a benchmark, we include the RL policies after 100 steps of finetuning with the test data. For tasks seen during training, the parameters generated by our approach demonstrate significantly greater diversity compared to the RL parameters after fine-tuning, indicating that our generator does not simply memorize training data. On unseen tasks, the similarity between our generated parameters and those learned by RL is almost negligible, with most similarities falling below 0.2. This further highlights the diversity and novelty of the policy parameters generated by our method.

Real-world Evaluation We further deploy policies generated from simulation trajectories onto a quadruped robot, instructing it to complete tasks as illustrated in Figure 10. Our synthesized policies

MetaWorld(8UnseenTasks)

100%

Rate

80%

Success

60%

40%

20%

0%

50NoisedTrajectories

100NoisedTrajectories

Bestfrombaselines

GeneratedTop5

GeneratedBest

AblationforBehaviorEmbeddingsonMetaWorld(unseen)

100%

80%

Success

60%

40%

20%

0%

50Trajectories

100 Trajectories

Embedding h

Embedding v

EmbeddingLayer

Ours

Handle press side

Door lock

Faucet open

Window close

Original

Generated

0.20

e-2

817

9.34

415y

8.20

0.22

01.14

Original

Original

Generated

Generated

Original

Generated

0.00

3D Visualization of Trajectories

X-dxis

8.95

0.19

0.15

9.4

3D Visualization of Trajectories

x-BKis

9.50

9

9.20

0.4

3D Visualization of Trajectories

x-axis

-0.08

-0.805

=0.84

0.62

0.70

A.69

0.04

3D Visualization of Trajectories

9.06

x-axis

0.08

0.38

9.12

exhibit smooth and effective responses when faced with these challenging tasks, which highlights the stability of the generated policies under the dynamics randomness of real-world environments. ii

Navigating to circumvent the goal and

ball while swiftly moving backward.

Making agile turns to avoid stepping on a

bouquet while moving across a mat

Go2

4.2 Ablation Studies

To better investigate the impact of each design choice in our method on the final results, we conduct a series of comprehensive ablation studies. All ablation studies report average results of the Top 5 generation models on MetaWorld.

Choice of behavior embeddings. Regarding the choice of conditional embeddings, as illustrated in Figure 3, we concatenate h and v as generation conditions to maximally preserve trajectory information. Figure 8 shows that utilizing either embedding individually also achieves comparable performance due to our contrastive loss, ensuring efficient capture of dynamics information. Our contrastive behavior embeddings significantly outperform a baseline that adds an embedding layer in the diffusion model to encode trajectories as input. These ablation results underscore the effectiveness of our behavior embeddings.

Choice of trajectory length. The trajectory length n used in behavior embeddings can also impact experimental results. Figure 12a demonstrates that overly short trajectories lead to performance degradation, probably due to the absence of crucial behavior information. However, beyond 40 steps, trajectory length minimally impacts policy generation, indicating that our method is not sensitive to the length of trajectories.

SeenTasks

UneenTasks

1.0

1.0

Similarity

0.8

Similarity

0.8

0.6

0.6

Cosine

Finetuned

CosineS

Finetuned

0.4

Generated

0.4

Generated

0.2

0.2

Parameter

Parameters

Impact of policy network size. The impact of policy network size on generated parameters is also worth discussing, as the network's hidden size influences the dimensionality of parameters to be generated. Figure 12b suggests that a hidden size of 128 is a suitable choice. Smaller networks may hinder policy performance, while larger ones increase parameter reconstruction complexity.

Impact of parameter number used in training. We study the impact of the number of policy checkpoints included per task in the training dataset, as shown in Figure 12c. Insufficient training data (<=1000) leads to a significant performance decline across all tasks. With more than 1000 parameters, there is no notable improvement in performance.

Impact of latent representation size. Additionally, Figure 12d illustrates the impact of varying the size of the latent parameter representation. Larger latent representations can negatively affect the performance of the generative model. Conversely, when the size of parameter representations is too small, it may hinder the autoencoder's capacity to decode representations to deployable policies. This underscores the influence of the parameter autoencoder on the overall effectiveness of the policy network generator.

(a) Trajectory Length

(c) Parameter Number

seen

seen

seen

seen

100%

unseen

100%

unseen

100%

unseen

100%

unseen

Success Rate

Rate

Rate

Rate

80%

Success

80%

Success

80%

Success

80%

60%

60%

60%

60%

20

40

60

80

40%

64

128

256

500

1000

1500

2000

256

512

1024

2048

TrajectoryLength

ParametersUsed for Training

Latent RepresentationSize

5 Related Works

Parameter Generation. Learning to generate neural networks has long been a compelling area of research. Since the introduction of Hypernetworks [10] and the subsequent extensions [3], several studies have explored neural network weight prediction. Hypertransformer [32] utilizes Transformers to generate weights for each layer of convolutional neural networks (CNN) using task samples for supervised and semi-supervised learning. Additionally, previous work [21] employs self-supervised learning to learn hyper representations of neural network weights. In the context of using diffusion models for parameter generation, G.pt [17] trains a diffusion transformer to generate parameters conditioned on learning metrics such as test losses and prediction errors, enabling the optimization of unseen parameters with a single update. Similarly, p-diff [25] propose a diffusion-based method to generate the last two normalization layers without any conditions for classification tasks. In contrast to these prior works, our focus is on policy learning problems. We develop a latent diffusion parameter generator that is more generalizable and scalable, based on agents' behaviors as prompts.

Learning to Learn for Policy Learning. When discussing learning to learn in policy learning, the concept of meta-learning [7] has been widely explored. The goal of meta-RL [7, 6, 9, 15] is to learn a policy that can adapt to any new task from a given task distribution. During the meta-training or meta-testing process, prior meta-RL methods require rewards as supervision for policy adaptation. Meta-imitation learning [8, 5, 27] addresses a similar problem but assumes the availability of expert demonstrations. Diffusion models have also been used in meta-learning. Metadiff [30] models the gradient descent process for task-specific adaptation as a diffusion process to propose a diffusionbased meta-learning method. Our work departs from these learning-to-learn works. Instead, we shift the focus away from data distributions across tasks and simply leverage behavior embeddings as conditional inputs for policy synthesis in the parameter space.

6 Conclusion

In this paper, we introduced a novel policy generation method based on conditional diffusion models. Targeting the generation of policies in high-dimensional parameter spaces, we employ an autoencoder to encode and reconstruct parameters, incorporating a contrastive loss to learn efficient behavior

embeddings. By prompting with these behavior embeddings, our policy generator can effectively produce diverse and well-performing policies. Extensive empirical results across various domains demonstrate the versatility of our approach in multi-task settings, the generalization ability on unseen tasks, and the resilience to environmental randomness. Our work not only introduces a fresh perspective on policy learning, but also establishes a new paradigm that delves into the latent connections between agent behaviors and policy parameters.

Limitation. Due to the vast number of parameters involved, we have not yet explored larger and more diverse policy networks. Additionally, the capabilities of the parameter diffusion generator are limited by the parameter autoencoder. We believe there is substantial room for future research to explore more flexible parameter generation methods. It would also be interesting to apply our proposed generation framework to generate other structures, further facilitating exploration in policy learning within the parameter space.

References

Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforcement learning. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1478-1487. PMLR, 14-18 Dec 2023.

Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.

Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning. arXiv preprint arXiv:2306.06955, 2023.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.

Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611.02779, 2016.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.

Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pages 357-368. PMLR, 2017.

Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv:1806.04640, 2018.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

Tianying Ji, Yongyuan Liang, Yan Zeng, Yu Luo, Guowei Xu, Jiawei Guo, Ruijie Zheng, Furong Huang, Fuchun Sun, and Huazhe Xu. Ace: Off-policy actor-critic with causality-aware entropy regularization. arXiv preprint arXiv:2402.14528, 2024.

Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. Conference on Robot Learning, 2022.

Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Advances in Neural Information Processing Systems, 32, 2019.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

William Peebles, Ilija Radosavovic, Tim Brooks, Alexei Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209.12892, 2022.

Sahand Rezaei-Shoshtari, Charlotte Morissette, François Robert Hogan, Gregory Dudek, and David Meger. Hypernetworks for zero-shot transfer in reinforcement learning. In AAAI, pages 9579-9587. AAAI Press, 2023.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.

Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyperrepresentations as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 35:27906-27920, 2022.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.

Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with contextbased representations. In International Conference on Machine Learning, pages 9767-9779. PMLR, 2021.

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.

Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion, 2024.

Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668, 2023.

Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyperdecision transformer for efficient online policy adaptation. arXiv preprint arXiv:2304.08487, 2023.

Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019.

Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16687-16695, 2024.

Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Shuang Ma, Hal Daumé III, Huazhe Xu, John Langford, Praveen Palanisamy, Kalyan Shankar Basu, and Furong Huang. Premier-TACO is a few-shot policy learner: Pretraining multitask representation via temporal action-driven contrastive loss. In Forty-first International Conference on Machine Learning, 2024.

Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. Hypertransformer: Model generation for supervised and semi-supervised few-shot learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27075-27098. PMLR, 2022.

Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020.

A Broader Impact

The broader impact of our research on generating policy parameters using diffusion models is a complex interplay of potential benefits and risks. On the one hand, by enhancing the sample efficiency of policy learning in robotics, our method could accelerate the development and deployment of robotic systems across various sectors, from manufacturing and healthcare to agriculture. This could lead to increased automation, potentially improving efficiency, safety, and accessibility in these fields, ultimately enhancing the quality of life and productivity for society. Additionally, our work introduces the robotics community to the potential of policy learning in a new and promising paradigm-which could inspire further research, leading to more advanced and efficient algorithms for robot learning and control.

However, it's important to acknowledge the potential negative consequences of increased automation. Job displacement due to automation is a significant concern, as robots could replace human workers in various industries, leading to economic disruption and social unrest. Furthermore, as robots become more autonomous and capable, concerns about safety and ethical implications arise. Ensuring the safe and responsible use of robots will be paramount as our research progresses.

B Implementation Details

All model training are conducted on NVIDIA A40 GPUs.

Autoencoder. The autoencoder implementation consists of a three-layer MLP encoder and a decoder. Prior to training, each layer of the policy network is flattened and encoded separately. The final mean and std layers are concatenated with the middle layer for encoding.

The hyperparameters used for the autoencoder are detailed in Table 1. On average, training an autoencoder requires 5 GPU hours.

Behavior embedding.

The behavior embedding model consists of two three-layer MLP embeddings. During training, we concatenate the state and action sequences from the first n = 60 steps (each sequence having a length of 3) to form the input for the h embedding layer. Subsequently, we concatenate the m = 3 states after success as inputs for the v embedding layer. Both embedding layers output 128-dimensional vectors. When utilizing these embeddings as conditional inputs, we concatenate the h and v embeddings as 256-dimenional conditions.

All hyperparameters about the training of the behavior embeddings can be found in Table 2. A single training for the embeddings requires less than 1 GPU hour.

Conditional diffusion generator. Our diffusion model employs a 1D convolutional U-Net architecture as its backbone, utilizing behavior embeddings as global conditions. It outputs latent parameter representations with the same dimensionality as the autoencoder's output.

Training a single diffusion generator requires only 4 GPU hours. All relevant hyperparameters are detailed in Table 3.

Hyperparameters We conduct all experiments with this single set of hyperparameters.

Data collection. Our dataset is collected with [13, 26] using RTX 3090 Ti.

C Experiments

C.1 Task Description

MetaWorld Descriptions of tasks and random initialization:

window open: Push and open a window. Randomize window positions

door open: Open a door with a revolving joint. Randomize door positions

drawer open: Open a drawer. Randomize drawer positions

dial turn: Rotate a dial 180 degrees. Randomize dial positions

faucet close: Rotate the faucet clockwise. Randomize faucet positions

button press: Press a button. Randomize button positions

door unlock: Unlock the door by rotating the lock clockwise. Randomize door positions

handle press: Press a handle down. Randomize the handle positions

plate slide: Slide a plate into a cabinet. Randomize the plate and cabinet positions

reach: reach a goal position. Randomize the goal positions

Unseen tasks (Downstream):

window close: Push and close a window. Randomize window positions

door close: Close a door with a revolving joint. Randomize door positions

drawer close: Open a drawer. Randomize drawer positions

faucet open: Rotate the faucet counter-clockwise. Randomize faucet positions

button press wall: Bypass a wall and press a button. Randomize the button positions

door lock: Lock the door by rotating the lock clockwise. Randomize door positions

handle press side: Press a handle down sideways. Randomize the handle positions

coffee-button: Push a button on the coffee machine. Randomize the position of the button

reach wall: Bypass a wall and reach a goal. Randomize goal positions

Robosuite Descriptions of tasks, robots, and random initialization:

Tasks:

Door: A door with a handle is mounted in free space in front of a single robot arm. The robot arm must learn to turn the handle and open the door. The door location is randomized at the beginning of each episode.

Lift: A cube is placed on the tabletop in front of a single robot arm. The robot arm must lift the cube above a certain height. The cube location is randomized at the beginning of each episode.

Nut Assembly - Single: Two colored pegs (one square and one round) are mounted on the tabletop, and two colored nuts (one square and one round) are placed on the table in front of a single robot arm. The goal is to place either one round nut or one square nut into its peg.

Robots:

Panda: Panda is a 7-DoF and relatively new robot model produced by Franka Emika, and boasts high positional accuracy and repeatability. The default gripper for this robot is the PandaGripper, a parallel-jaw gripper equipped with two small finger pads, that comes shipped with the robot arm.

Sawyer: Sawyer is Rethink Robotic's 7-DoF single-arm robot. Sawyer's default RethinkGripper model is a parallel-jaw gripper with long fingers and useful for grasping a variety of objects.

C.2 More Results

In addition to reporting the average performance of the top 5 generated results in the main paper, we rigorously define "qualified policies" as those achieving a 100% success rate in the test environment. Table 4 presents the proportion of qualified policies among 100 policy parameters generated from 100 trajectories. Notably, we maintain an average qualification rate of over 30% on seen tasks.

Furthermore, even on unseen tasks, we can generate high-performing policies using an average of only 20 trajectories. Considering that our method does not rely on expert demonstrations, the quality and success rate of our generated policies significantly enhance the sample efficiency of policy learning.

C.3 Details of Real-world Robots

In this section, we detail the real-world robot applications of our method. We deploy synthesized policies on the Unitree Go2 quadruped, designing diverse real-world testing environments to evaluate two key aspects of agent performance: (1) stability during high-speed turning and backward locomotion, and (2) robustness of movements on uneven terrain (mats). Our deployment process consists of four key steps:

Obtain actor network parameters and corresponding test trajectories from IsaacGym simulations, where the actors are trained using walk-these-ways [14].

Train Make-An-Agent using the acquired training data.

Generate actor networks from randomly sampled IsaacGym trajectories, covering a variety of training periods.

Equip the Unitree Go2 quadruped with the generated actors and a pretrained adaptor, enabling it to complete designed, challenging locomotion tasks.

100

200

300

400

500

Trajectory Length of Conditions

100

200

300

400

500

Trajectory Length of Generated Policies

Seen Tasks (Average)

Ours

Multi-task IL

Multi-task RL

Meta IL

100

200

300

400

500

Trajectory Length of Conditions

100

200

300

400

500

Unseen Tasks (Average)

Ours

Multi-task IL

Multi-task RL

Meta IL

100

200

300

400

500

Trajectory Length of Conditions

100

200

300

400

500

Unseen Tasks with Noise Trajectories

Ours

Multi-task IL

Multi-task RL

Meta IL

Multi-task RL

OursOurs (Training+Evaluation)

Methods

10

20

30

40

GPU Hours

7.8

20.4

31.4

34.1

4.1

Computational Budget

Multi-task training (seen): 13 tasks

Real-world locomotion: 2 tasks

1.5x

Avoid stepping on a bouquet while moving across a mat

1.5x

Navigating to circumvent the goal and ball swiftly

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We conduct comprehensive experiments, and the results included in our paper support the main claims made in the abstract and introduction well.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We add a paragraph in the Conclusion section, addressing the scope and performance limits of our work.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper empirically proposes one effective method to generate policy networks from demonstrations, and it doesn't contain theoretical proof.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include detailed descriptions of our methods in the main paper and the Appendix, the information provided should be adequate to reproduce the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all the code used to produce the main results in an anonymous GitHub repo in the supplementary material.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are included in the Experiment section and the Appendix to facilitate a comprehensive understanding of our design choices and results.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: An error bar or the variance value is included in all the results, either in a table or a figure.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide training GPU hours and the type of computing resources in the Experiments section.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: After reading through the NeurIPS code of ethics carefully, we are confident that our research conform well with the spirit of the Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide a broader impact section in Appendix A to discuss the positive and negative impacts of our research.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method does not fall into any category of pretrained language models, image generators, or scraped datasets. While we do provide a dataset for trained agents, these are no different than providing model checkpoints and does not contain any sensitive data that may relate to a real person or entity.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we've cited any paper or code we use in the paper, and follow their license and terms of use correctly.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The open-sourced code and dataset we provided are licensed appropriately, please refer to the anonymous repo for details.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMake-An-Agent is a novel policy network generator that uses conditional diffusion models to create control policies based on a single demonstration of desired behaviors. By encoding behavior trajectories into embeddings, Make-An-Agent generates latent policy parameter representations that are decoded into functional policy networks. This method, trained on a dataset of policy parameters and their corresponding trajectories, excels in generating versatile and scalable policies across various tasks, including unseen ones, using few-shot demonstrations. Its robustness and efficiency are demonstrated in both simulation and real-world environments, highlighting its ability to produce high-performing policies even from noisy inputs. This approach bypasses traditional behavior modeling, instead leveraging the inherent correlations in agent behaviors to optimize policy generation without the need for further fine-tuning.\n\nSTRENGTHS:\n1. The paper introduces a novel method that uses conditional diffusion models for policy generation, a significant shift from traditional policy learning methods.\n2. Make-An-Agent demonstrates the ability to generate effective policies for a wide range of tasks by conditioning on behavior embeddings, showcasing scalability across different domains.\n3. The diffusion-based generator exhibits strong generalization capabilities, producing proficient policies even for unseen behaviors and unfamiliar tasks.\n4. The method can generate diverse and resilient policy parameters, maintaining high performance under environmental variability and with noisy input data.\n5. The effectiveness of Make-An-Agent is validated not only in simulations but also in real-world robotics, highlighting its practical applicability and efficiency.\n\nWEAKNESSES:\nWhile the paper demonstrates the method's effectiveness in various tasks, a more extensive evaluation across a broader range of environments and conditions could provide a more comprehensive assessment of its robustness and versatility.\n\nThe performance of the generated policies heavily relies on the quality of the input demonstrations, which might not always be optimal or available in all scenarios.\n\nThe model might overfit to the specific types of tasks and behaviors seen during training, potentially reducing its effectiveness in truly novel and diverse environments.\n\nQUESTIONS:\nHow does Make-An-Agent perform when faced with tasks that are vastly different from those it was trained on? Are there any specific limitations in task diversity that the model struggles with?\n\n What are the computational requirements for training and deploying Make-An-Agent? How does its efficiency compare to traditional reinforcement learning and other policy generation methods?\n\n How sensitive is the method to the quality of the behavior demonstrations? What happens if the demonstrations contain suboptimal or noisy behaviors?\n\n Can the approach be scaled to more complex tasks and larger environments? What modifications, if any, would be needed to handle such scalability?\n\n What are the contributions of the individual components of the method, such as the autoencoder, contrastive learning for behavior embedding, and the diffusion model? Are there ablation studies that show the impact of each component on the overall performance?\n\n What challenges were encountered during the deployment of the generated policies onto real-world robots? How were these challenges addressed?\n\n The paper mentions robustness to noisy trajectories, but to what extent can the model handle extreme levels of noise or inaccuracies in the demonstrations?\n\nLIMITATIONS:\nThe training process for the autoencoder and diffusion model is likely to be resource-intensive, potentially limiting accessibility for researchers with limited computational resources.\n\nThe effectiveness of the generated policies is heavily reliant on the quality of the input demonstrations. Poor or suboptimal demonstrations could adversely affect the performance of the generated policies.\n\n The evaluation might be limited to specific tasks and environments. A broader range of experiments would be needed to fully assess the model's robustness and versatility across various domains and conditions.\n\n While the method shows promise in the tested domains, its scalability to more complex tasks and larger environments is not fully explored. Handling such scalability might require further modifications and optimizations.\n\n There is a risk that the model might overfit to the types of tasks and behaviors seen during training, which could reduce its effectiveness in truly novel and diverse environments.\n\n While the method has shown success in real-world robotics, the transition from simulation to real-world applications can introduce unforeseen challenges and complexities that need to be addressed comprehensively.\n\n The paper may lack thorough ablation studies to understand the contributions of individual components of the method, such as the autoencoder, behavior embeddings, and diffusion model.\n\nAlthough the method claims robustness to noisy trajectories, the extent to which it can handle extreme levels of noise or inaccuracies in the input demonstrations is not fully detailed.\n\n The sensitivity of the model to various hyperparameters and the process of tuning these parameters for optimal performance is not extensively covered, which can be crucial for practical implementations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to generate policy parameters based on the behavior through diffusion models. The paper leverages the autoencoder to map the parameters of the policy to latent representations. The model demonstrates remarkable generalization abilities on unseen tasks with few-shot demonstrations.\n\nSTRENGTHS:\n* The method is novel. The paper learns behavior embeddings from agent behaviors to capture environmental dynamics and information about tasks. And it leverages such behavior embeddings as conditions for the parameter generation.\n* The empirical results are rich and strong. The paper evaluates the model across three environments (two simulations and one real world) and shows strong generalization performance on unseen tasks and for unseen robots.\n* This paper also presents well including figures and written structures.\n\nWEAKNESSES:\n* The generated policies seem unstable. The generation by diffusion models are diverse but not stable. And the paper doesn't make sure the generated policies are good enough. And the experiments compare the best or top 5 generated policies. How to evaluate which policy is best? Does it require testing all policies and finding which one is best? In C.2, the paper shows the qualification rate of generated policies. The qualification rates are very weak (mostly less than 0.1) on unseen tasks except \"drawer close\". It's not practical. What if using expert demonstrations?\n* Some settings of experiments seem unreasonable. Why fix the initial locations during the training stage and finetuning/adaptation? This might make the baselines or generated policies only work on fixed initial locations. Similarly, why choose highly sub-optimal trajectories? If so, it may use offline RL approaches to finetune the baselines.\n* Lack of details of baselines. How to finetune the multi-task RL with fixed trajectories (10/50/100)?\n\nQUESTIONS:\n* See weakness.\n* What are the states or what would happen after the first success time? Does the env stop if the task is succeeded?\n\nLIMITATIONS:\n*  The paper has provided several limitations. The paper only generates parameters for simple linear layers. And the learned behavior embeddings are also important.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Make-An-Agent architecture, which synthesizes a policy neural network from an input trajectory. Make-An-Agent utilizes a parameter and behavior embedding. The behavior embedding is trained with the mutual information between the trajectory and the successful part of the trajectory. The behavior embedding is then used in a diffusion model to generate the policy parameters. This methodology is validated in MetaWorld, Robosuite, and quadrupedal locomotion environments. In MetaWorld and RoboSuite, Make-An-Agent outperforms baselines RL and IL baselines. Further experiments analyze how the trajectories and policies produced by Make-An-Agent differ from those directly trained through RL.\n\nSTRENGTHS:\n1. The paper represents a novel take on learning multi-task policies by directly generating the parameters of a policy network from a demonstration. This is different from prior Meta-RL approaches that adapt existing networks. \n\n1. The paper has extensive empirical comparisons to a variety of baselines from RL and IL. The proposed method outperforms baselines in MetaWorld and Robosuite by a large margin. \n\n1. Make-An-Agent is capable of generating policies that are more diverse (Figure 8) and more robust (Figure 6) than baselines.\n\n1. The behavior embedding is an important aspect of the Make-An-Agent architecture as demonstrated by Figure 7.\n\n1. The paper analyzes the impact of important settings on the system's performance with the demonstration length, policy size and number of policy parameter sets used for training.\n\n1. Implementation details are thoroughly described throughout the paper and in supplementary sections B and C.\n\nWEAKNESSES:\n1. Table 1 only reports the best and top 5 generated policies. However, all policies had to be evaluated in the environment to decide what these top policies are. This is an unrealistic assumption that weakens the significance of the result. The paper must analyze in more detail the distribution of performance between generated policies and compare the full statistics of all generated policies to baselines for a fair comparison.\n\n1. A weakness of directly generating policy parameters is the challenge of scaling up to larger policy networks. For example, policies operating from egocentric visual inputs in 3D spaces typically have tens of millions of parameters. The core idea of directly generating policy parameters is likely incapable of scaling to this setting of 1000x more policy parameters.\n\nQUESTIONS:\n1. What fraction of generated policies outperform the baselines in Table 1? And what is the average policy performance between all generated policies, not just the top 5? See my point (1) under weaknesses above.\n\n1. Given that the results in Figure 11b already show performance suffering with increased policy size, how can Make-An-Actor scale to more complex tasks that require higher parameter count policies?\n\nLIMITATIONS:\nYes, limitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present Make-An-Agent, which is a method to generate policy parameters given a few intended trajectories from a task. The proposed method is straightforward, which makes it better that it seems to work in the experiments in the work. The method first generates a large dataset of policies as well as their rollout traces, all parametrized in the same way, using standard RL algorithms. Then, the authors learn an autoencoder on top of the policy parameters. Finally, the method learns a trajectory conditional generator using diffusion for creating policy latents. Putting them together, on a new environments the authors generate some trajectories (here, by running SAC for 0.5M steps), and then generate a new policy using the trajectory latent which gets translated to a policy latent by the diffusion model.\n\nThe experiments are done on a real robot locomotion task as well as two manipulation task suites in simulation. In all of these cases, the best generated policies perform well in the experiments compared with meta-learning approaches.\n\nSTRENGTHS:\n+ The approach is novel, and the algorithm and the released code are both quite simple.\n+ The experiments have a wide breadth, and showing the applicability in multiple different domains show the potential of this method.\n+ While the dataset is large (1000+ policies per task) it is not excessively large to be prohibitive.\n\nWEAKNESSES:\n- The reported metric (best generated policy performance, avg top 5 generated policy performance) both seem _fudged_. It seems like the authors are generating a large number of policies, and then picking the best generated policy out of them to get their numbers. This seems dishonest, as for a large number N of generated policies, every method will get a very high score.\n- While the authors report the \"qualification rate\" i.e the % of policies getting 100% success rate in test, I am not sure how this works. As they report the \"best policy score\", and since qualification rate is > 0 for all tasks, why is figure 5 (ours) not all 100%? What am I missing?\n- The drop between top-1 and top-5 is concerning, since I cannot tell whether the authors got lucky finding a perfect policy from the long tail.\n- The authors don't mention the overall compute overhead and how it compares with the baselines, which I believe is a strong consideration for such a method.\n\nQUESTIONS:\n- Please detail the compute budget for each of the baselines as well as your method.\n- If possible, please release the dataset as well as the pretrained networks with your code.\n\nLIMITATIONS:\n- Generating larger networks will be more difficult, so I wish the authors took advantage of any special properties of network weights while training.\n- The comparison metric is fuzzy and can seem unfair.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Make-An-Agent, a conditional diffusion model that generates policy parameters based on demonstration of target behaviors. The authors propose an autoencoder to encode policy network parameters into compact latent representations. The behavior embeddings are learned using contrastive learning between long-term trajectories and their success or future states. Then, a conditional diffusion model is used to generate policy latent code conditioned on the learned behavior embedding, which can then be decoded to policy parameters via the pretrained decoder. Extensive results in simulated environment including MetaWorld and Robosuite demonstrate the effectiveness of the proposed method. Also, there is real-world deployment on quadrupedal locomotion.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow\n- The experimental analysis is pretty thorough.\n\nWEAKNESSES:\n- Despite the experiments are overall complete, there are many more interesting aspects to look at, which may further strengthen the paper. Please check questions for more details.\n- The real-world experiment has very limited results presented.\n\nQUESTIONS:\n- Is there any way to roughly predict the performance of the generateed parameters without actually evaluating them? For example, assuming the learned data distribution nicely reflects the given behavior, the generation distribution in the policy parameter space should somewhat follow the mode that gives good performance; perhaps something similar to the \"Parameter distinction\" experiments but among the generation from the proposed model only can be interesting.\n- Is any of the statistics of the generated policy parameters useful? For example, does the mean still a valid policy parameter? Is the variance of the generated parameters correlated with how \"out-of-distribution\" the behavior is?\n- It would be interesting to briefly look at the geometry in the behavior embedding space. For example, how does small perturbation in the behavior embedding affect the performance of the generated policy parameters. Or even more interesting, is it possible to compose behaviors by messing around the conditioning of the diffusion model, which then gives a compositional policy?\n- Could the authors report the performance distribution for all generations (as opposed to the best or top 5)? This can probably provide some insight into the generation behavior and what distribution (w.r.t. the data distribution in generative modeling not performance distribution) is actually learned by the model. Also, it would be interesting to see the performance distribution in seen and unseen tasks.\n- For ablation of policy network, could the authors discuss how architectural difference (beyond hidden size; e.g., constructing policy as very small recurrent network or transformer) can potentially affect the results?\n- In real-world evaluation, could the authors provide more details about the behaviors and policies in trainnig dataset and testing? \n- Given the proposed method being a generator of the policy (which can easily produce many models at a time), it would be also interesting to check the mixture of expert setup. For example, will using something simple like majority voting in generated policies improves robustness in \"Adaptability to environmental randomness on seen tasks\"?\n- Could the authors report some failure cases? Or is there any failure mode like certain type behaviors/demonstrations always lead to bad-performing generated policies?\n\nLIMITATIONS:\nLimitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}