Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion

Authors: yongyuan  liang
tingqiang  xu
kaizhe  hu
guangqi  jiang
furong  huang
huazhe  xu
shanghai  qi
zhi  institute
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:35:59

Abstract: Can we generate a control policy for an agent using just one demonstration of desired behaviors as a prompt, as effortlessly as creating an image from a textual description?
In this paper, we present **Make-An-Agent**, a novel policy parameter generator that leverages the power of conditional diffusion models for behavior-to-policy generation. Guided by behavior embeddings that encode trajectory information, our policy generator synthesizes latent parameter representations, which can then be decoded into policy networks. 
Trained on policy network checkpoints and their corresponding trajectories, our generation model demonstrates remarkable versatility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs. We showcase its efficacy and efficiency on various domains and tasks, including varying objectives, behaviors, and even across different robot manipulators. Beyond simulation, we directly deploy policies generated by **Make-An-Agent** onto real-world robots on locomotion tasks. Project page: https://cheryyunl.github.io/make-an-agent/.

Main paper: 
 introduction: 
 Policy learning traditionally involves using sampled trajectories from a replay buffer or behavior demonstrations to learn policies or trajectory models mapping from state s to action a, modeling a narrow behavior distribution. In this paper, we consider a shift in paradigm: moving beyond training a policy, can we reversely predict optimal policy network parameters using suboptimal trajectories from offline data? This approach would obviate the need to explicitly model behavior distributions, allowing us to learn the underlying parameter distributions in the parameter space, thus revealing the implicit relationship between agent behaviors for specific tasks and policy parameters.Using low-dimensional demonstrations (such as agent behavior) to guide the generation of highdimensional outputs (policy parameters) is a challenging problem. When diffusion models [12, 20] have demonstrated highly competitive performance on various tasks including text-to-image synthesis, we are inspired to approach policy network generation as a conditional denoising diffusion process. By progressively refining noise into structured parameters, the diffusion-based generator can discover various policies that are not only superior in performance but also more robust and efficient than the demonstration in the policy parameter space.While prior works on hypernetworks [10, 1, 18] explore the concept of training a hypernetwork to generate weights for another neural network, they primarily use hypernetworks as an initialization network of meta-learning [7] and then adapt to specific task settings. Our approach diverges from this paradigm by leveraging agent behaviors as direct prompts or to generate optimal policies within the parameter space, without the need for any downstream policy fine-tuning or adaptation with gradient updates. Since behaviors -as the observable manifestation of deployed policies -from different tasks often share underlying skills or environmental information, our policy generator can exploit these potential correlations in the parameter space, such as shared parameters for similar motion patterns, which leads to enhanced cross-task one-shot generalizability. What we need is an end-to-end behavior-to-policy generator, not a shared base policy.To achieve this, we introduce Make-An-Agent , featuring three key technical contributions: (1) We propose an autoencoder that encodes policy networks into compact latent representations based on their network layers, which can also effectively reconstruct the original policy from its latent representation. (2) We leverage contrastive learning to capture the mutual information between long-term trajectories and their success or future states. This approach yields a novel and efficient behavior embedding. (3) We utilize a simple yet effective diffusion model conditioned on the learned behavior embeddings, to generate policy parameter representations, which are then decoded into deployable policies using the pretrained decoder. (4) We construct a pretrained dataset of policy network parameters and corresponding deployed trajectories to train our proposed methodology.To investigate the generation performance of Make-An-Agent, we evaluate our approach in three continuous control domains including diverse tabletop manipulation and real-world locomotion tasks. During test time, we generate policies using trajectories from the replay buffer of partially-trained RL agents. The policies generated by our method demonstrate superior performance compared to policies produced by multi-task [28, 23] or meta learning [7, 27] and other hypernetwork-based generation methods [1] . Our generator offers several key advantages:• Versatility: Make-An-Agent excels in generating effective policies for a wide range of tasks by conditioning on agent behavior embeddings. Since we train the parameter generator for latent parameter representations, it can generate policy networks of varying sizes within the latent space, demonstrating scalability.• Generalizability: Our diffusion-based generator demonstrates robust generalization, yielding proficient policies even for unseen behaviors or unseen embodiments in unfamiliar tasks.• Robustness: Our method can generate diverse policy parameters, exhibiting resilient performance under environmental randomness from simulators and real-world environments. Notably, Make-An-Agent can synthesize high-performing policies when fed with noisy trajectories, highlighting the robustness of our model. 
backgrounds: 
 Policy Learning. Reinforcement Learning (RL) is structured within the formation of Markov Decision Processes (MDPs) [2] , which is defined by the tuple M = ⟨S, A, P, R, γ⟩. Here, S signifies the state space, A the action space, P the transition probabilities, R the reward function and γ the discount factor. RL aims to optimize an agent's policy π : S → A, which outputs action a t based on state s t at each timestep, to maximize cumulative rewards. The optimal policy can be expressed as:EQUATIONwhere z represents a trajectory generated by following policy π. In deep RL, policies π are represented using neural network function approximations [22] , parameterized by θ π , facilitating the learning of intricate behaviors across high-dimensional state and action spaces.Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) [12] are generative models that frame data generation through a structured diffusion process, which involves iteratively adding noise to the data and then denoising it to recover the original signal. Given a sample x 0 , the forward diffusion process to obtain x 1 , x 2 , ..., x T of increasing noise intensity is typically denoted by:EQUATIONwhere q is the forward process, N is Gaussian noise, and β t ∈ (0, 1) is is the noise variance.The denoising process, which is the reverse of the forward diffusion, can be formulated as:EQUATIONwhere p θ denotes the reverse process, µ θ and Σ θ are the mean and variance of the Gaussian distribution respectively, which can be approximated by a noise prediction neural network parameterized by θ.Diffusion models aim to learn reverse transitions that maximize the likelihood of the forward transitions at each time step t. The noise prediction network θ is optimized using the following objective, as the function mapping from ϵ θ (x t , t) to µ θ (x t , t) is a closed-form expression:EQUATIONHere, ϵ ∼ N (0, I), is the target Gaussian noise, ᾱt := t s=1 1 -β s , and√ ᾱt x 0 + √ 1 -ᾱt ϵ is the estimated distribution of x t from the closed-form relation.Although diffusion models are typically used for image generation through the reverse process, the variable x can be generalized to represent diverse entities for generation. In this paper, we adapt x to represent the parameters θ π of the policy network in policy learning.Behavior embeddings In the inference process of policy parameter generation, conditioning on behavior embeddings from the agent's trajectory, the latent diffusion model denoises random noise into a latent parameter representation, which can then be reconstructed as a deployable policy using the autoencoder. The forward process for progressively noising the data is also conducted on the latent space after encoding policy parameters as latent representations. Behavior embeddings0 a 0 s n-1 … a n-1 s n a n Projection 𝝓 s K s K+m … Projection 𝝋 h vLong trajectory with 𝑛 steps After the first success step 𝐾 Overview. An overview of our proposed methodology is illustrated in Figure 1 . To achieve this, we address several key challenges: (1) Developing latent representations of high-dimensional policy parameters that can be effectively reconstructed into well-functioned policies. (2) Learning an embedding of behavior demonstrations that serves as an effective diffusion condition. (3) Training a conditional diffusion model specifically for policy parameter generation.Parameter representation. We use an MLP with m layers as the common policy approximator. Consequently, when the full parameters of a policy are flattened, they form a high-dimensional vector.To enable generation with limited computational resources while retaining efficacy, and to support the generation of policies for different domains with varying state and action dimensions, we compress the policy network parameters into a latent space.Based on the policy network architecture, we unfold the parameters following the architecture of the policy network, represented as x = [x 0 , x 1 , . . . , x m-1 ], where x i denotes the flattened parameters from each layer. The encoder E encodes each x i as z i , resulting in a parameter latent representation denoted as z = [z 0 , z 1 , . . . , z m-1 ], where each z i in the latent space has the same dimension, while the decoder D can decode z into x. To improve the robustness of this procedure, we introduce random noise augmentation in both encoding and decoding during training. Given each vectorized parameter as x, we minimize the objective as,EQUATIONwhere z = E(x + ξ D ), and ξ E and ξ D represent the augmented noise. The architecture of the autoencoder is shown in Figure 2 .For each domain, the autoencoder for parameter representation only needs to be trained once before parameter generation, which can handle policy parameters from different tasks. To facilitate the generalizability of the policy generator across domains, we design the latent parameter representations to have the same dimensions for different domains.Behavior embedding. Since our goal in learning behavior embeddings is not to model the distribution of states and actions, but to provide conditional information for policy parameter generation, we aim for them to encapsulate both crucial environmental dynamics and the key information of the task goal. The principle behind our behavior embeddings is to learn the mutual information between preceding n step trajectories and subsequent states with success signals.EQUATIONWe propose a novel contrastive method to train behavior embeddings. In Figure 3 , we present a design demonstration of our contrastive loss. For a long trajectory τ , we decouple it as the n initial state-action pairs τ n = (s 0 , a 0 , s 1 , a 1 , . . . , s n , a n ) and the m states after the first success time K as τ = (s K , s K+1 , . . . , s K+m ). Given a batch of trajectory sequences {τ i } N i=1 which can be presented as {τ n i , τi } N i=1 , we optimize the contrastive objective [16, 31] as:EQUATIONwhere h i = ϕ θ (τ n i ) and v i = ψ θ (τ i ) are embeddings from different parts of the long trajectory τ i and W is a learnable metric that measures the similarity between embeddings h i and v i .For each trajectory τ , we obtain a set of embeddings τ e = {h i , v i }. In practice, the choice of specific embeddings can be tailored to the characteristics of different tasks and trajectories. We use (h i , v i ) as the conditional input in our experiments.Flexibility. With the consideration that in many scenarios, rewards are often sparse or non-existent, whereas success signals serve as a more direct indicator of whether a policy has achieved its objective. We therefore use original trajectories that exclude reward information but include success information.For tasks without explicit success signals, such as locomotion, we segment long trajectories into multiple shorter trajectories. For each segment, we use the last m states as τ and the 0 -n state-action pairs as τ n . The informative behavior embeddings of a long trajectory are concatenated from the embeddings of all the trajectory segments. This embedding approach strives to capture the essential information for generating behavior-specific policy parameters, including environmental dynamics and task goals, using the most concise representation possible from long trajectories and prioritizing flexibility and efficiency.Conditional policy generator. After training the parameter autoencoder and behavior embeddings, for policy parameter x and the corresponding trajectory g deployed by policy x, we can transfer x as latent parameter representation z with the autoencoder E and trajectory τ as behavior embedding τ e . The conditional diffusion generator is trained on latent representation z, conditioning on τ e . We optimize the conditional latent diffusion model via the following loss function:EQUATIONwhere the neural backbone ϵ θ (z t , τ e , t) is implemented as a 1D convolutional UNet [19] parameterized by θ and t is uniformly sampled from {1, . . . , T }. The outputs of our parameter generator can be encoded by D as deployable policies. During training the diffusion model, both the parameter autoencoder and behavior embedding layers are frozen, which ensures the training stability and efficiency.Dataset. We build a dataset containing tens of thousands of policy parameters and trajectories from deploying these policies. The dataset is obtained from multiple RL training across a range of tasks. We utilized the dataset to train both the autoencoder and behavior embedding models. Then we use the encoded parameter representations and behavior embeddings derived from the collected trajectory to train the conditional diffusion model for policy parameter generation. We conduct extensive experiments to evaluate Make-An-Agent , answering the following problems: 
experiments: 
 MetaWorld Robosuite Quadrupedal Locomotion• How does our method compare with other multi-task or learning-to-learn approaches for policy learning, in terms of performance on seen tasks and generalization to unseen tasks? • How scalable is our method, and can it be fine-tuned across different domains? • Does our method merely memorize policy parameters and trajectories of each task, or can it generate diverse and new behaviors?Benchmarks. We include two manipulation benchmarks for simulated experiments and real-world robot tasks to show the performance and capabilities of our method as visualized in Figure 4 .MetaWorld. MetaWorld [29] is a benchmark suite for robotic tabletop manipulation, consisting of a diverse set of motion patterns for the Sawyer robotic arm and interactions with different objects. We selected 10 tasks for training as seen tasks and 8 for evaluation as unseen downstream tasks. Detailed descriptions of these tasks can be found in Appendix C.1. The state space of MetaWorld consists of 39 dimensions and the action space has 4 dimensions. The policy network architecture used for MetaWorld is a 4-layer MLP with 128 hidden units, containing a total of 22,664 parameters.Robosuite. Robosuite [33] , a simulation benchmark designed for robotic manipulation, supports various robots such as the Sawyer and Panda arms. We train models on three manipulation tasks: Block Lifting, Door Opening and Nut Assembly, using the single-arm Panda robot. Evaluations are conducted on the same tasks using the Sawyer robot. This experimental design aims to validate the practicality of our approach by assessing whether the generated policy can be effectively utilized on different robots. In the Robosuite environment, the state space comprises 41 dimensions, and the action space consists of 8 dimensions. The policy network employed for this domain contains 23,952 parameters.Quadrupedal locomotion. To evaluate the policies generated by Make-An-Agent in the real world, we utilize walk-these-ways [14] to train policies on IsaacGym and use our method to generate actor networks conditioning on trajectories from IsaacGym simulation with the pretrained adaptation modules. Then, we deploy the generated policy on real robots in environments differ from simulations.The policies generated for real-world locomotion deployment comprise 50,956 parameters.Dataset. We collect 1500 policy networks for each task in MetaWorld and Robosuite. These networks are sourced from policy checkpoints during SAC [11] training. The checkpoints are saved every 5000 training steps once the test success rate reaches 1. During the training stage, we fix the initial locations of objects and goals and train the policies using different random seeds. For each task, we require an average of 8 SAC training runs, approximately 30 GPU hours.For evaluation, the trajectories used as generation conditions are sampled from the SAC training buffer within the first 0.5 million timesteps, which can be highly sub-optimal, under the same environmental initialization. During testing, the generated policies are evaluated in 5 random initial configurations, thoroughly assessing the robustness of policies generated using trajectories from the fixed environment settings.In RoboSuite experiments, due to the inconsistency in policy networks, we retrain the autoencoder and finetune the diffusion generator trained on MetaWorld data. The experimental setup for RoboSuite is almost identical to that of MetaWorld, with the only difference being the robot used during testing.For real-world locomotion tasks, we save 10,000 policy network checkpoints using walk-these-ways (WTW) [14] trained on IsaacGym, requiring a total of 200 GPU hours. The 100 trajectories used as generation conditions are sourced from the first 10,000 training iterations of WTW.Baselines. We compare Make-An-Agent with four baselines, including multi-task imitation learning (IL), multi-task reinforcement learning (RL), meta-RL with hypernetworks, and meta-IL with transformers. These represent state-of-the-art methods for multi-task policy learning and adaptation. For a fair comparison, each baseline uses the same testing trajectory data for downstream adaptation.Multi-task BC [24] : We train a multi-task behavior cloning policy using trajectories in our training dataset, and then finetune it with test trajectories to adapt in specific tasks.Multi-task RL, CARE [23] : We train an mixture of encoders for 2 million steps (for each task). For RL training, we train the algorithm in dense reward environments and finetune the model using test trajectories with sparse rewards, where feedback is only provided at the end of a trajectory.We train a hypernetwork with our training data with dense rewards, which can adapt to different task-specific policies during testing with test trajectories.Meta Imitation Learning with decision transformer(DT) [4] We train the pre-trained DT model using the training trajectories in our dataset, then use the test trajectories from replay to adapt it to test tasks. By using test trajectories as conditions, our policy generator can produce an equivalent number of policy parameters. Compared with baselines, we report both the best result among the generated policies and the average performance of the top 5 policies. All algorithms use the same task-specific replay trajectories. The difference is that we use them as generation conditions, whereas other methods use them for adaptation.We define policies achieving a 100% success rate during evaluation as qualified policies. The analysis of qualification rates for policies generated by our model is presented in Appendix C.2. Adaptability to environmental randomness on seen tasks. Figure 5 demonstrates the significant advantage of our algorithm over other methods on seen tasks. This is attributed to the fact that, despite test trajectories originating from the same environment initialization, the generated policy parameters are more diverse, thus possessing a strong ability to adapt to environmental randomness. In contrast, other algorithms, when adapted using such singular trajectories, exhibit more limited adaptability in these scenarios. Our experimental design aligns with practical requirements, as real-world randomness is inherently more complex.Generalizability to unseen tasks. Figure 6 showcases the superior performance of our algorithm on unseen tasks. Test trajectories originate from the same environment setting for each task, while evaluation occurs in randomly initialized environments. Our policy generator, without fine-tuning, directly utilizes test trajectories as input, demonstrating a remarkable ability to generate parameters that work on unseen tasks. The agent's behavior in unseen tasks exhibits similarities to seen task behaviors, such as arm dynamics and the path to goals. By effectively combining parameter representations related to these features, the generative model successfully generates effective policies. In contrast, baseline methods struggle to adapt in environmental randomness.These results strongly suggest that our algorithm, compared to other policy adaptation methods, may offer a superior solution for unseen scenarios. To further investigate robustness in generalization, we added Gaussian noise with a standard deviation of 0.1 to actions in test trajectories used for policy generation or adaptation on unseen tasks. Figure 7 demonstrates that our method remains resilient to noisy inputs, while the performance of the baselines is significantly impacted. We believe this is because our behavior embeddings only need to capture key dynamic information as conditions to generate policies, without directly learning state-action relationships from trajectories, resulting in better robustness.Trajectory difference. To compare the difference between using test trajectories as conditions and the trajectories obtained by deploying the generated policies, we visualize the trajectories during unseen task evaluations. As shown in Figure 9 , our diffusion generator can synthesize various policies, which is significantly different from policy learning methods that learn to predict actions or states from trajectory data. We believe that this phenomenon fully illustrates the value of our proposed policy parameter generation paradigm.Parameter distinction. Beyond trajectory differences, we also investigate the distinction between synthesized parameters and RL policy parameters. We calculate the cosine similarity between the RL policies used to obtain the test trajectories and the parameters generated from these trajectories. As a benchmark, we include the RL policies after 100 steps of finetuning with the test data. For tasks seen during training, the parameters generated by our approach demonstrate significantly greater diversity compared to the RL parameters after fine-tuning, indicating that our generator does not simply memorize training data. On unseen tasks, the similarity between our generated parameters and those learned by RL is almost negligible, with most similarities falling below 0.2. This further highlights the diversity and novelty of the policy parameters generated by our method.Real-world Evaluation We further deploy policies generated from simulation trajectories onto a quadruped robot, instructing it to complete tasks as illustrated in Figure 10 . Our synthesized policies exhibit smooth and effective responses when faced with these challenging tasks, which highlights the stability of the generated policies under the dynamics randomness of real-world environments. ii Navigating to circumvent the goal and ball while swiftly moving backward.Making agile turns to avoid stepping on a bouquet while moving across a mat To better investigate the impact of each design choice in our method on the final results, we conduct a series of comprehensive ablation studies. All ablation studies report average results of the Top 5 generation models on MetaWorld.Choice of behavior embeddings. Regarding the choice of conditional embeddings, as illustrated in Figure 3 , we concatenate h and v as generation conditions to maximally preserve trajectory information. Figure 8 shows that utilizing either embedding individually also achieves comparable performance due to our contrastive loss, ensuring efficient capture of dynamics information. Our contrastive behavior embeddings significantly outperform a baseline that adds an embedding layer in the diffusion model to encode trajectories as input. These ablation results underscore the effectiveness of our behavior embeddings.Choice of trajectory length. The trajectory length n used in behavior embeddings can also impact experimental results. Figure 12a demonstrates that overly short trajectories lead to performance degradation, probably due to the absence of crucial behavior information. However, beyond 40 steps, trajectory length minimally impacts policy generation, indicating that our method is not sensitive to the length of trajectories.ii We thank Kun Lei and Qingwei Ben for their help and support in real-robot applications.Impact of policy network size. The of policy network size on generated parameters is also worth discussing, as the network's hidden size influences the dimensionality of parameters to be generated. Figure 12b suggests that a hidden size of 128 is a suitable choice. Smaller networks may hinder policy performance, while larger ones increase parameter reconstruction complexity.Impact of parameter number used in training. We study the impact of the number of policy checkpoints included per task in the training dataset, as shown in Figure 12c . Insufficient training data (<=1000) leads to a significant performance decline across all tasks. With more than 1000 parameters, there is no notable improvement in performance.Impact of latent representation size. Additionally, Figure 12d illustrates the impact of varying the size of the latent parameter representation. Larger latent representations can negatively affect the performance of the generative model. Conversely, when the size of parameter representations is too small, it may hinder the autoencoder's capacity to decode representations to deployable policies. This underscores the influence of the parameter autoencoder on the overall effectiveness of the policy network generator.  
related works: 
 Parameter Generation. Learning to generate neural networks has long been a compelling area of research. Since the introduction of Hypernetworks [10] and the subsequent extensions [3] , several studies have explored neural network weight prediction. Hypertransformer [32] utilizes Transformers to generate weights for each layer of convolutional neural networks (CNN) using task samples for supervised and semi-supervised learning. Additionally, previous work [21] employs self-supervised learning to learn hyper representations of neural network weights. In the context of using diffusion models for parameter generation, G.pt [17] trains a diffusion transformer to generate parameters conditioned on learning metrics such as test losses and prediction errors, enabling the optimization of unseen parameters with a single update. Similarly, p-diff [25] propose a diffusion-based method to generate the last two normalization layers without any conditions for classification tasks. In contrast to these prior works, our focus is on policy learning problems. We develop a latent diffusion parameter generator that is more generalizable and scalable, based on agents' behaviors as prompts.Learning to Learn for Policy Learning. When discussing learning to learn in policy learning, the concept of meta-learning [7] has been widely explored. The goal of meta-RL [7, 6, 9, 15] is to learn a policy that can adapt to any new task from a given task distribution. During the meta-training or meta-testing process, prior meta-RL methods require rewards as supervision for policy adaptation. Meta-imitation learning [8, 5, 27] addresses a similar problem but assumes the availability of expert demonstrations. Diffusion models have also been used in meta-learning. Metadiff [30] models the gradient descent process for task-specific adaptation as a diffusion process to propose a diffusionbased meta-learning method. Our work departs from these learning-to-learn works. Instead, we shift the focus away from data distributions across tasks and simply leverage behavior embeddings as conditional inputs for policy synthesis in the parameter space. 
conclusion: 
 In this paper, we introduced a novel policy generation method based on conditional diffusion models.Targeting the generation of policies in high-dimensional parameter spaces, we employ an autoencoder to encode and reconstruct parameters, incorporating a contrastive loss to learn efficient behavior By prompting with these behavior embeddings, our policy generator can effectively produce diverse and well-performing policies. Extensive empirical results across various domains demonstrate the versatility of our approach in multi-task settings, the generalization ability on unseen tasks, and the resilience to environmental randomness. Our work not only introduces a fresh perspective on policy learning, but also establishes a new paradigm that delves into the latent connections between agent behaviors and policy parameters.Limitation. Due to the vast number of parameters involved, we have not yet explored larger and more diverse policy networks. Additionally, the capabilities of the parameter diffusion generator are limited by the parameter autoencoder. We believe there is substantial room for future research to explore more flexible parameter generation methods. It would also be interesting to apply our proposed generation framework to generate other structures, further facilitating exploration in policy learning within the parameter space.  
 
 Appendix: i Code, dataset and video are released in https://cheryyunl.github.io/make-an-agent/. Corresponding to: cheryunl@umd.edu 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper "Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion" introduces an approach for generating control policies using conditional diffusion models based on demonstrations of desired behaviors. The authors propose encoding behavior trajectories into embeddings, which are then used to generate latent policy parameter representations. These representations are subsequently decoded into functional policy networks. The reviewers acknowledge the originality of the idea, the good empirical performance, and the generalization ability of the proposed method.

The strengths of the paper lie in its use of conditional diffusion models for policy generation, which represents a departure from traditional policy learning methods. Make-An-Agent demonstrates the ability to generate effective policies for a wide range of tasks by conditioning on behavior embeddings, which shows its scalability across different domains. The diffusion-based generator exhibits strong generalization capabilities, producing proficient policies even for unseen behaviors and unfamiliar tasks. Moreover, the method can generate diverse and resilient policy parameters, maintaining high performance under environmental variability and with noisy input data. The effectiveness of Make-An-Agent is validated not only in simulations but also in real-world robotics, highlighting its practical applicability and efficiency.

The reviewers raised several important questions and concerns, which the authors addressed in their rebuttal. They provided additional experiments and analyses to support their claims, including the relationship between the effectiveness of generated policies and condition trajectories, the computational budgets compared with baselines, and generalization results on more diverse unseen tasks. The authors also included results of policy generation using expert demonstrations and compared them with offline RL baselines, demonstrating the superiority of their approach.

While one reviewer (Reviewer **YLrf**) recommended rejection, the concerns were not as in-depth as those raised by the other reviewers. The authors have earnestly responded to all the reviews, providing detailed explanations and additional experimental results to address the concerns. As the AC, I do not place significant weight on the reasons provided by Reviewer **YLrf** for rejection, as the authors have adequately responded to the concerns raised, and Reviewer **YLrf** did not respond further.

The reviewers' concerns have mostly been adequately responded to in the rebuttal, and the authors have committed to improving the paper based on the feedback received.  Based on the above reasons, the AC recommends accepting this paper for publication.

Reviews: review: paper_summary: paper_summary: This paper presents Make-An-Agent, a conditional diffusion model that generates policy parameters based on demonstration of target behaviors. The authors propose an autoencoder to encode policy network parameters into compact latent representations. The behavior embeddings are learned using contrastive learning between long-term trajectories and their success or future states. Then, a conditional diffusion model is used to generate policy latent code conditioned on the learned behavior embedding, which can then be decoded to policy parameters via the pretrained decoder. Extensive results in simulated environment including MetaWorld and Robosuite demonstrate the effectiveness of the proposed method. Also, there is real-world deployment on quadrupedal locomotion.
strength_weakness: strength_weakness: - Despite the experiments are overall complete, there are many more interesting aspects to look at, which may further strengthen the paper. Please check questions for more details.
- The real-world experiment has very limited results presented.
questions: questions: - Is there any way to roughly predict the performance of the generateed parameters without actually evaluating them? For example, assuming the learned data distribution nicely reflects the given behavior, the generation distribution in the policy parameter space should somewhat follow the mode that gives good performance; perhaps something similar to the "Parameter distinction" experiments but among the generation from the proposed model only can be interesting.
- Is any of the statistics of the generated policy parameters useful? For example, does the mean still a valid policy parameter? Is the variance of the generated parameters correlated with how "out-of-distribution" the behavior is?
- It would be interesting to briefly look at the geometry in the behavior embedding space. For example, how does small perturbation in the behavior embedding affect the performance of the generated policy parameters. Or even more interesting, is it possible to compose behaviors by messing around the conditioning of the diffusion model, which then gives a compositional policy?
- Could the authors report the performance distribution for all generations (as opposed to the best or top 5)? This can probably provide some insight into the generation behavior and what distribution (w.r.t. the data distribution in generative modeling not performance distribution) is actually learned by the model. Also, it would be interesting to see the performance distribution in seen and unseen tasks.
- For ablation of policy network, could the authors discuss how architectural difference (beyond hidden size; e.g., constructing policy as very small recurrent network or transformer) can potentially affect the results?
- In real-world evaluation, could the authors provide more details about the behaviors and policies in trainnig dataset and testing? 
- Given the proposed method being a generator of the policy (which can easily produce many models at a time), it would be also interesting to check the mixture of expert setup. For example, will using something simple like majority voting in generated policies improves robustness in "Adaptability to environmental randomness on seen tasks"?
- Could the authors report some failure cases? Or is there any failure mode like certain type behaviors/demonstrations always lead to bad-performing generated policies?
limitations: limitations: Limitations are discussed in the paper.
paper_summary: paper_summary: In this work, the authors present Make-An-Agent, which is a method to generate policy parameters given a few intended trajectories from a task. The proposed method is straightforward, which makes it better that it seems to work in the experiments in the work. The method first generates a large dataset of policies as well as their rollout traces, all parametrized in the same way, using standard RL algorithms. Then, the authors learn an autoencoder on top of the policy parameters. Finally, the method learns a trajectory conditional generator using diffusion for creating policy latents. Putting them together, on a new environments the authors generate some trajectories (here, by running SAC for 0.5M steps), and then generate a new policy using the trajectory latent which gets translated to a policy latent by the diffusion model.

The experiments are done on a real robot locomotion task as well as two manipulation task suites in simulation. In all of these cases, the best generated policies perform well in the experiments compared with meta-learning approaches.
strength_weakness: strength_weakness: - The reported metric (best generated policy performance, avg top 5 generated policy performance) both seem _fudged_. It seems like the authors are generating a large number of policies, and then picking the best generated policy out of them to get their numbers. This seems dishonest, as for a large number N of generated policies, every method will get a very high score.
- While the authors report the "qualification rate" i.e the % of policies getting 100% success rate in test, I am not sure how this works. As they report the "best policy score", and since qualification rate is > 0 for all tasks, why is figure 5 (ours) not all 100%? What am I missing?
- The drop between top-1 and top-5 is concerning, since I cannot tell whether the authors got lucky finding a perfect policy from the long tail.
- The authors don't mention the overall compute overhead and how it compares with the baselines, which I believe is a strong consideration for such a method.
questions: questions: - Please detail the compute budget for each of the baselines as well as your method.
- If possible, please release the dataset as well as the pretrained networks with your code.
limitations: limitations: - Generating larger networks will be more difficult, so I wish the authors took advantage of any special properties of network weights while training.
- The comparison metric is fuzzy and can seem unfair.
paper_summary: paper_summary: This paper proposes the Make-An-Agent architecture, which synthesizes a policy neural network from an input trajectory. Make-An-Agent utilizes a parameter and behavior embedding. The behavior embedding is trained with the mutual information between the trajectory and the successful part of the trajectory. The behavior embedding is then used in a diffusion model to generate the policy parameters. This methodology is validated in MetaWorld, Robosuite, and quadrupedal locomotion environments. In MetaWorld and RoboSuite, Make-An-Agent outperforms baselines RL and IL baselines. Further experiments analyze how the trajectories and policies produced by Make-An-Agent differ from those directly trained through RL.
strength_weakness: strength_weakness: 1. Table 1 only reports the best and top 5 generated policies. However, all policies had to be evaluated in the environment to decide what these top policies are. This is an unrealistic assumption that weakens the significance of the result. The paper must analyze in more detail the distribution of performance between generated policies and compare the full statistics of all generated policies to baselines for a fair comparison.

1. A weakness of directly generating policy parameters is the challenge of scaling up to larger policy networks. For example, policies operating from egocentric visual inputs in 3D spaces typically have tens of millions of parameters. The core idea of directly generating policy parameters is likely incapable of scaling to this setting of 1000x more policy parameters.
questions: questions: 1. What fraction of generated policies outperform the baselines in Table 1? And what is the average policy performance between all generated policies, not just the top 5? See my point (1) under weaknesses above.

1. Given that the results in Figure 11b already show performance suffering with increased policy size, how can Make-An-Actor scale to more complex tasks that require higher parameter count policies?
limitations: limitations: Yes, limitations are discussed in Section 6.
paper_summary: paper_summary: This paper proposes a novel approach to generate policy parameters based on the behavior through diffusion models. The paper leverages the autoencoder to map the parameters of the policy to latent representations. The model demonstrates remarkable generalization abilities on unseen tasks with few-shot demonstrations.
strength_weakness: strength_weakness: * The generated policies seem unstable. The generation by diffusion models are diverse but not stable. And the paper doesn't make sure the generated policies are good enough. And the experiments compare the best or top 5 generated policies. How to evaluate which policy is best? Does it require testing all policies and finding which one is best? In C.2, the paper shows the qualification rate of generated policies. The qualification rates are very weak (mostly less than 0.1) on unseen tasks except "drawer close". It's not practical. What if using expert demonstrations?
* Some settings of experiments seem unreasonable. Why fix the initial locations during the training stage and finetuning/adaptation? This might make the baselines or generated policies only work on fixed initial locations. Similarly, why choose highly sub-optimal trajectories? If so, it may use offline RL approaches to finetune the baselines.
* Lack of details of baselines. How to finetune the multi-task RL with fixed trajectories (10/50/100)?
questions: questions: * See weakness.
* What are the states or what would happen after the first success time? Does the env stop if the task is succeeded?
limitations: limitations: *  The paper has provided several limitations. The paper only generates parameters for simple linear layers. And the learned behavior embeddings are also important.
paper_summary: paper_summary: Make-An-Agent is a novel policy network generator that uses conditional diffusion models to create control policies based on a single demonstration of desired behaviors. By encoding behavior trajectories into embeddings, Make-An-Agent generates latent policy parameter representations that are decoded into functional policy networks. This method, trained on a dataset of policy parameters and their corresponding trajectories, excels in generating versatile and scalable policies across various tasks, including unseen ones, using few-shot demonstrations. Its robustness and efficiency are demonstrated in both simulation and real-world environments, highlighting its ability to produce high-performing policies even from noisy inputs. This approach bypasses traditional behavior modeling, instead leveraging the inherent correlations in agent behaviors to optimize policy generation without the need for further fine-tuning.
strength_weakness: strength_weakness: While the paper demonstrates the method's effectiveness in various tasks, a more extensive evaluation across a broader range of environments and conditions could provide a more comprehensive assessment of its robustness and versatility.

The performance of the generated policies heavily relies on the quality of the input demonstrations, which might not always be optimal or available in all scenarios.

The model might overfit to the specific types of tasks and behaviors seen during training, potentially reducing its effectiveness in truly novel and diverse environments.
questions: questions: How does Make-An-Agent perform when faced with tasks that are vastly different from those it was trained on? Are there any specific limitations in task diversity that the model struggles with?

 What are the computational requirements for training and deploying Make-An-Agent? How does its efficiency compare to traditional reinforcement learning and other policy generation methods?

 How sensitive is the method to the quality of the behavior demonstrations? What happens if the demonstrations contain suboptimal or noisy behaviors?

 Can the approach be scaled to more complex tasks and larger environments? What modifications, if any, would be needed to handle such scalability?

 What are the contributions of the individual components of the method, such as the autoencoder, contrastive learning for behavior embedding, and the diffusion model? Are there ablation studies that show the impact of each component on the overall performance?

 What challenges were encountered during the deployment of the generated policies onto real-world robots? How were these challenges addressed?

 The paper mentions robustness to noisy trajectories, but to what extent can the model handle extreme levels of noise or inaccuracies in the demonstrations?
limitations: limitations: The training process for the autoencoder and diffusion model is likely to be resource-intensive, potentially limiting accessibility for researchers with limited computational resources.

The effectiveness of the generated policies is heavily reliant on the quality of the input demonstrations. Poor or suboptimal demonstrations could adversely affect the performance of the generated policies.

 The evaluation might be limited to specific tasks and environments. A broader range of experiments would be needed to fully assess the model's robustness and versatility across various domains and conditions.

 While the method shows promise in the tested domains, its scalability to more complex tasks and larger environments is not fully explored. Handling such scalability might require further modifications and optimizations.

 There is a risk that the model might overfit to the types of tasks and behaviors seen during training, which could reduce its effectiveness in truly novel and diverse environments.

 While the method has shown success in real-world robotics, the transition from simulation to real-world applications can introduce unforeseen challenges and complexities that need to be addressed comprehensively.

 The paper may lack thorough ablation studies to understand the contributions of individual components of the method, such as the autoencoder, behavior embeddings, and diffusion model.

Although the method claims robustness to noisy trajectories, the extent to which it can handle extreme levels of noise or inaccuracies in the input demonstrations is not fully detailed.

 The sensitivity of the model to various hyperparameters and the process of tuning these parameters for optimal performance is not extensively covered, which can be crucial for practical implementations.

Comments: title: Eagerly awaiting your valuable feedback Reviewer YLrf (for the final 12 hours)
Thank Reviewer 957J for your replay and suggestions!
Thank you for your inspiring reply!
Thanks for the rebuttal
Thank Reviewer 8Q7F for your inspiring reply!
Required Action: Please Respond to the Author Rebuttal
Required Action: Please Respond to the Author Rebuttal
Required Action: Please Respond to the Author Rebuttal
Thank Reviewer Pxd2 for your supportive feedback!
comment: Dear Reviewer YLrf,

As the discussion period draws to a close in 12 hours, we are delighted to have received positive feedback from the other four reviewers and are very eager to ensure that our response has adequately addressed your concerns as well.

We believe that the clarification in our paper would solve your concerns on input demonstrations/overfitting/computational cost/ablation/robustness, along with the additional results and discussions in rebuttal could solve your questions regarding experiments/scalability/real-world deployment.

We deeply appreciate your contribution to the community through your review. Your insights are valuable to us. We eagerly await your response.

Warm Regards,

Paper 8467 Authors
Dear Reviewer 957J,

Thank you for your constructive suggestions and understanding. We would like to provide further clarification on the concerns you raised:

> In the real world, the majority of time and energy is spent on evaluating the robot in a real environment, which is a lot more expensive than simple GPU hours.

We completely agree with this point. To ensure a relatively low cost on evaluations, in our real-robot experiments, we utilize the IsaacGym simulator to collect policies and trajectories and evaluate the generated policies before deploying them on the real robot. Our experiments aim to demonstrate that policies generated and evaluated in simulation can still achieve excellent results in real-world scenarios.

For tasks where it is not feasible to evaluate in simulations, we try to discuss alternative methods inspired by other reviewers for utilizing multiple policies in decision-making, such as majority voting and using the mixture of experts, to improve the robustness and stability of real-robot decisions. We believe these are very promising directions and hope our work will inspire the community to explore more.

> Release of full code and pretrained model library

As we have promised, we will release all datasets, pretrained models, and training code in our final version. We greatly appreciate your support.

> Disambiguating between GPU hours needed for training (done offline) and rollouts/evaluation needed for model selection (done online).

Based on your suggestion, we will provide a detailed report on the computational costs associated with each part of our method. Additionally, we will include a separate discussion of the evaluation costs and alternative approaches in the limitations section.

Thank you again for your positive comments and valuable suggestions regarding real-world evaluation, which have greatly helped us further discuss the weaknesses and improve the quality of our paper. We appreciate the time and effort you’ve put into reviewing and the rebuttal process. We will supplement our paper based on all of these discussions.

Best wishes,

Paper 8467 Authors
Thank you for the response and the discussion. After reading this rebuttal, the only point of contention that remains for me is how efficiency is reported. In the real world (which is where I study robotics) the majority of time and energy is spent on evaluating the robot in a real environment, which is a lot more expensive than simple GPU hours. Since the number of runs required is not disambiguated from GPU hours needed to train/generate/fine-tune, I am keeping my current score.

I still recommend acceptance of the paper, which would be a strong accept with:
1. Release of full code and pretrained model library, and
2. Disambiguating between GPU hours needed for training (done offline) and rollouts/evaluation needed for model selection (done online.)
Dear Reviewer U6yw,

Thank you for your thoughtful feedback and understanding! We further discuss the points you raised.

> mixture-of-experts computational overhead

When using the mixture-of-experts approach for decision making, while it involves using multiple policies to output actions, the inference cost for these policies is very low. As a result, the computational cost is less than 1/10 of the baseline fine-tuning cost. We will include detailed statistics on this in the final version of our paper.

> the statistics of the generated policy parameters in parameter space 

Given the high dimensionality of the parameter space, we primarily compare the trajectories obtained from deployed policies. Directly averaging the generated parameters does not yield ideal results, mainly because the parameter distribution in the latent parameter space and the parameter space are not smooth.

> the use of other architectures like RNN or transformer

If we use RNN as the policy backbone, and the parameter count of the RNN is not larger by an order of magnitude, it is feasible to use our model to generate RNNs. Our method directly flattens the network parameters into a vector before encoding and generating, without imposing special requirements on the network structure during the generation process.

We sincerely appreciate your positive comments and insightful discussions. The constructive suggestions for practical applications are crucial for improving the quality of our paper, and we will ensure that all discussions and new results are included in the final version.

Best wishes!

Paper 8467 Authors
Thanks for the rebuttals. The mixture-of-experts result is really nice. It would be nice to include this result with the computational overhead of doing so (which I assume is pretty marginal) in the final version. For Fig.13 (re "the statistics of the generated policy parameters"), I was previously referring to experimenting in the parameter space but this result also looks nice. Lastly, for architecture, I don't think the use of other architectures like RNN or transformer (on non-reactive or partially-observable tasks) is trivially extendable; but I am not 100% positive. 
Overall, the new results look nice. I will keep my rating.
Dear Reviewer 8Q7F,

Thank you for your reply and understanding. We will update our paper accordingly based on your and other reviewers' comments. 

We sincerely appreciate your positive comments and the discussions during rebuttal, which are of great help of improving our paper and also contribute to the community.

Best wishes!

Paper 8467 Authors
Thanks for the detailed response. I maintain my positive rating.
Dear Reviewer YLrf,


As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.


Best regards,

Area Chair, NeurIPS 2024
Dear Reviewer 957J,


As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.


Best regards,

Area Chair, NeurIPS 2024
Dear Reviewer U6yw,


As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.


Best regards,

Area Chair, NeurIPS 2024
Dear Reviewer Pxd2,

Thank you for your prompt reply and constructive suggestions.

* In our GPU hours comparison, we have included the evaluation and inference costs. For very slower simulators, we strongly agree that filtering policies by selecting condition trajectories that are close to optimal may reduce evaluation time costs. Even if more interaction time is required, the evaluation cost in most environments is still significantly lower than the cost of fine-tuning.

* In our real-robot experiments, as discussed in our paper, we utilize the IsaacGym simulator to collect policies and trajectories and also evaluate the generated policies before deploying them on the real robot, which ensures that we conduct evaluations at a relatively low cost. Our experiments aim to demonstrate that policies generated and evaluated in simulation can still achieve excellent results in real-world scenarios. 

* Figure 13 presents results from the door unlock (seen) and coffee button (unseen) environments, both of which exhibit medium performance across all seen/unseen tasks. 

* We chose to use trajectory length as it more accurately reflects the performance of the condition trajectories compared to the success condition. For example, two trajectories might both achieve success, but their lengths could differ, with shorter trajectories indicating closer to optimal and more efficient performance. Using trajectory length to evaluate policies also makes it easier to compare with condition trajectories.

* We fully acknowledge your suggestion, and we provide a table showing the generated policies' success rates across all condition trajectories for all seen/unseen tasks. We can see that on seen tasks, our method can generate more than 55% of policies that perform better than the best baselines. On unseen tasks, on average at least 35% of the policies perform better than the best baselines. This fully demonstrates the superiority of our method. 

* Additionally, if we select trajectories with success signals for policy generation, in seen tasks, more than 82% of the policies outperform the baselines, and in unseen tasks, more than 50% do so. For environments where evaluation is challenging, using success trajectories can also more efficiently yield optimal policies. We will include these results as histograms in the final version of our paper.

| Average success rate (seen) | Generated policies across all trajectories |
 | ------ | ----- | 
|   0%       |    21.2%     |  
|   0-60%       |   5.2%      |         
|   60-80%       |  17.7%       |          
|   80-100%       |    26.5%     |                 
|   100%       |    29.4%     |         

| Average success rate (unseen) | Generated policies across all trajectories |
 | ------ | ----- | 
|   0%       |    27.8%     |  
|   0-60%       |   12.0%      |         
|   60-80%       |   28.2%      |          
|   80-100%       |    19.6%     |         
|   100%       |    12.4%     |   

| Average success rate (seen) | Generated policies across success trajectories |
 | ------ | ----- | 
|   0%       |    3.2%     |  
|   0-60%       |   6.9%      |         
|   60-80%       |  11.2%       |          
|   80-100%       |    8.4%     |                 
|   100%       |    70.3%     |         

| Average success rate (unseen) | Generated policies across success trajectories |
 | ------ | ----- | 
|   0%       |    10.7%     |  
|   0-60%       |   17.8%      |         
|   60-80%       |   21.4%      |          
|   80-100%       |    18.9%     |         
|   100%       |    31.2%     |   

**In summary**, we agree that the evaluation cost could become a weakness of our method when generating multiple policies. However, it is undeniable that the broader exploration of parameter spaces offers more effective solutions for policy generalization compared to prior works. We also believe that using these multiple policies for a mixture of experts could be a promising direction.

Thank you for acknowledging the novelty of our work and for providing such insightful feedback. We welcome any further discussions and greatly value the opportunity to continue improving our research.

Best Regards,

Paper 8467 Authors
Thank you for the response. While I still appreciate the novelty of the approach, like other reviewers, I am also concerned about the evaluation criteria for selecting the best policies. The GPU hours comparison in the rebuttal is not a representative comparison because for Make-An-Agent, this also includes evaluation costs, which could be prohibitively expensive depending on the environment, as in the real world or a slower simulator. Figure 13 in the rebuttal is also unclear. Which environment are these results for? Why compare episode length in the tasks? A clearer comparison would be to generate a histogram of success rates, not trajectory lengths, for the Make-An-Agent generated policies across all condition trajectories.
Dear Reviewer 8Q7F,

Thank you for your time and effort in reviewing our work and providing insightful feedback. 

> Collecting data from fix locations and highly sub-optimal would hugely hurt the baselines' performance.

In our rebuttal results, we have demonstrated that our method not only excels in suboptimal data but also significantly outperforms all baselines when using expert demonstrations with random initialization. It's important to note that our method is not an imitation learning approach. Offline replay datasets are widely used in the evaluation of offline RL and meta RL, which is why we selected this evaluation metric.

We fully agree with your suggestion and will include experiments using expert demonstrations in the appendix to fairly compare our method with meta IL baselines.

> It's expensive and unsafe to evaluate the policy (like real-world) while it's simpler to collect expert demonstrations with teleoperation systems.

In real-robot experiments as mentioned in our paper, we utilize the IsaacGym simulator to collect policies and trajectories and also evaluate generated policies before deploying them on the real robot, which ensures that we conduct evaluations at a relatively low cost.

There is no contradiction between using simulation for initial evaluation and using expert demonstrations for generation. If optimal real-world data is available, using it to generate policies would certainly be preferable. Our experiments aim to demonstrate that policies generated and evaluated in simulation can still yield excellent results in real-world scenarios.

> About real-world experiments, the paper seems only to provide some visualizations. What about the success rate and other baselines.

Thank you for your suggestions. Real-world visualization is the most straightforward way to observe the performance of policies, especially since designing success rate metrics for real-robot locomotion tasks is challenging as shown in prior works. Additionally, none of the baselines we listed include real-world experiments, and the network sizes used in those baselines are not applicable to real-robot scenarios. 

We evaluate our generated policies against offline RL methods in low-speed testing and high-speed testing following [1]. The results demonstrate that the policies generated by our method exhibit better stability and performance on real robots.

| Low-Speed Testing | IQL | BC | Ours | 
|---|---|---|---|
|Reward |  11.9 |  14.4 | 24.8 |

| High-Speed Testing | IQL | BC | Ours | 
|---|---|---|---|
|Reward |  10.4 |  10.7 | 20.9 |

[1] Margolis, Gabriel B., and Pulkit Agrawal. "Walk these ways: Tuning robot control for generalization with multiplicity of behavior." Conference on Robot Learning. PMLR, 2023.

We hope our responses address your concerns, and we are open to further discussion. Thank you once again for your valuable feedback.

Warm Regards, 

Paper 8467 Authors
Thanks for the response.
* collecting data from **fix locations** and **highly sub-optimal** would hugely hurt the baselines' performance. I understand the work has achieved better results with such settings. I'm curious about which factor results in the weak performance of baselines.
* In some settings, it's expensive and unsafe to evaluate the policy (like real-world) while it's simpler to collect expert demonstrations with teleoperation systems. 
* About real-world experiments, the paper seems only to provide some visualizations. What about the success rate and other baselines.