HyPoGen: Optimization-Biased Hypernetworks for Generalizable Policy Generation
Authors: Hanxiang Ren
Li Sun
Xulong Wang
Pei Zhou
Zewen Wu
Siyan Dong
Difan Zou
Youyi Zheng
Yanchao Yang
Venue: ICLR 2025 Poster (2025)
Keywords: hypernetwork
policy generation
behavior cloning
Primary area: applications to robotics, autonomy, planning

Abstract: Policy learning through behavior cloning poses significant challenges, particularly when demonstration data is limited. In this work, we present HyPoGen, a novel optimization-biased hypernetwork for policy generation. The proposed hypernetwork learns to synthesize optimal policy parameters solely from task specifications --  without accessing training data -- by modeling policy generation as an approximation of the optimization process executed over a finite number of steps and assuming these specifications serve as a sufficient representation of the demonstration data. By incorporating structural designs that bias the hypernetwork towards optimization, we can improve its generalization capability while only training on source task demonstrations. During the feed-forward prediction pass, the hypernetwork effectively performs an optimization in the latent (compressed) policy space, which is then decoded into policy parameters for action prediction. Experimental results on locomotion and manipulation benchmarks show that HyPoGen significantly outperforms state-of-the-art methods in generating policies for unseen target tasks without any demonstrations, achieving higher success rates and underscoring the potential of optimization-biased hypernetworks in advancing generalizable policy generation. Our code and data are available at: https://github.com/ReNginx/HyPoGen.

Published as a conference paper at ICLR 2025
HoPoGen: OPTIMIZATION- HYPERNETWORKS FOR GENERALIZABLE POLICY GENERATION
Hanxiang Ren" ZJU & HKU
Li Sun' HKU
Xulong Wang
Pei Zhou
Zewen Wu
ZJU
HKU
HKU & Transcengram
Siyan Dong HKU
Difan Zou HKU
Youyi Zheng ZJU
Yanchao Yang* HKU
ABSTRACT
Policy learning through behavior cloning poses significant challenges, particu- larly when demonstration data is limited. In this work, we present HyPoGen, a novel optimization-biased hypernetwork for policy generation. The proposed hypernetwork learns to synthesize optimal policy parameters solely from task specifications without accessing training data by modeling policy generation as an approximation the optimization process executed over a finite number of steps and assuming these specifications serve sufficient representation of the demonstration data. By incorporating structural designs that bias the hyper- network towards optimization, we can improve its generalization capability while only training on source task demonstrations. During the feed-forward prediction pass, the hypernetwork effectively performs an optimization in the latent (com- pressed) policy space, which is then decoded into policy parameters for action prediction. Experimental results on locomotion and manipulation benchnarks show that HyPoGen significantly outperforms state-of-the the-art methods in generat- ing policies for unseen target tasks without any demonstrations, achieving higher success rates and underscoring the potential of optimization-biant -iased hypernetworss in advancing generalizable policy generation. Our code and data are available at: com/ RRNggiinnn Hawp///yPPGen
INTRODUCTION
Behavior cloning (BC) (Pomerleau 1991 is highly promising in real-world applications, trains agents directly from expert demonstrations, which can be more easily acquired in certain scenarios than designing a proper reward function for Reinforcement Learning (RL). Moreover, BC alleviates the complexity in training for long-horizon tasks that may exist in infinitedimeensinall spaces.
While BC can bypass the challenges of reward function design and RL training difficulties by learning directly from expert demonstrations, it comes with its own limitations: (i) Substantial data requirement. Gathering sufficient high-quality expert data is lime-consuming and labor-intensive, as it requires days, if not months, of manual recording of numerous instances of the desired behavior (spatial-temporal trajectories). This process can be particularly burdensome in complex or hazardous tasks. (ii) Insuficient generalization ability. Policies trained in one environment often fail to perform well in different ones due to overfitting to specific scene variations and task dynamics encountered during training (Zhang et al. 2018 Song al. 2020). Such lack of generalization limits the policy's applicability in complex, real-world scenarios with varying conditions.
In response to these challenges, recent work attempts to enhance the generalization capability through Hypernetworks (Ha et al. 2017 Rezaci-Shoshhieeeeeeee et al. (202)). These networks learn a mapping from the task embedding (specification) space to the parameter space of target (policy) networks, enabling capturing the similarity between different tasks, which is crucial for transferability, and promoting better utilization of limited demonstrations.
First Authors (hanxiang. rew www.:/zu.uuu..nn. edu.cn, sunlids@ coa.iie@cccnnethh..kkk
1 Work done as Research Assistant at the InfoBodied AI Lab HKK Corresponding Authors
$ Senior Author

---

Published as a conference paper at ICLR 2025
Seen:
Unseen:
Policy Generation Networks
HyPoCen Hyyer/em Cond Policy
Figure 1: Policy generation networks are trained on source (seen) task specifications with corre- sponding demonstrations using behavior cloning, and then directly applied to target (unseen) task specifications to generate policy network parameters for action prediction. By introducing optimiza- tion inductive biases into the proposed hypernetoork HyPoGen,
achieve better performance on out-of-domain specifications without resorting to additional demonstration collection and training.
Despite their success, previous works usually employ hypernetworks in target-network-ageee manner, typically by structuring them as multi-layer perceptrons (MLPs), and are subject to several limitations: (i) Overfitting risk: Training hypernetwork to map task embedding to the set of parameters of a policy network faces the same problem of overfitting to the tasks being learned, similar to the generalization difficulty encountered in supervised learning. (ii) Disregard for the optimization process: These hypernetworks do not account for the fact that the optimal weights of a target network the result of optimization through (stochastic) gradient descent. (iii) Ignorance of target network structure: The hyperettwors predict different parts of the policy network simultaneously and do not utilize the information flow induced by the structure of the target network.
These issues hinder the deployment of hypernetworss at a large scale for generalizable policy synthe- sis, for which we attempt to investigate and propose a potentially in-principle solution. Generalization from completely unrelated tasks, like piano playing to pizza cooking, is exceptionally challenging. A more achievable type of generalization occurs tasks with similar characteristics, such as an agent learning to handle objects of various shapes or operate under unseen dynamics. In such cases, tasks are defined by environmental and embodiment parameters (specification) that help the agent leverage and generalize through sharable characteristics across different tasks.
To address these issues, we present HyPoGen, method that biases Hypernetworks for Policy Generation from task specifications. HyPoGen employs inductive biases in both the optimization process and the interdependencces within the target network structure to facilitate generalizable policy synthesis. Our approach imitates the iterative optimization in the latent parameter space, updating policy network parameters solely based on task specifications rather than relying on manu- ally collected demonstration data. To further bias the hypernetwork training towards optimization with structural information of the policy network, HyPoGen mimics the interdependent gradient computation (involved in the backward propagation) during the (forward) synthesis procedure in the latent parameter space. This is achieved by predicting the neural "gradients" of each conceptual network block according the chain rule relationship. The predicted latent parameters can then be decoded back to the original parameter space, enabling action prediction when loaded into the target policy network.
We conduct an extensive evaluation within both locomotion and manipulation environments. Our proposed method, trained on the DeepMind Control Suite (Tunyasuvunakool et al.. 2020) and ManiSkill (Mu et al.) (021). outperforms state-of-the-are (SOTA) methods in both settings. HyPoGen achieves a significant enhancement in success rate for various stiffness and arm lengths ManiSkill environments. Furthermore, our experiments validate that HyPoGen performs optimization based solely on task specifications without actual demonstration data. In summary, our main contributions include:
We propose novel hypernetwork architecture termed HyPoGen that integrates the inductive bias of the optimization process of the target network under behavior cloning.
We introduce effective approach for learning (latent) neural gradient updates, explicitly modeling the interdependencies of gradients across different target network blocks.
We comprehensively benchmark our approach against previous methods both locomotion and manipulation tasks, demonstrating its superiority in generalizable policy generation.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Behavior cloning and policy generation BC (Bain & Sammut, 1995) enables efficient learning and replication of policies through expert demonstration data. Due to its ability to directly imitate demonstrated behavior without requiring aine-graind reward functions or extensive interactions with
the environment, BC has shown remarkable effectiveness when combined with RL tasks (Lee &
Zhang, 2021 Goecks et al., 2019; Kumar et al. 2022 Rajeswaran et al. 2017 Nair et al. 2018). For instance, it has been used to pre-train Q-functions by minimizing the temporal difference (TD) error between the demonstrated expert policy and the optimal policy (Hester et al., (018),01), or to guide exploration through reward and policy shaping (Subramanian et al. 2016 Brys et al. 2015). Current
behavior guidance methods strive to explore strategies beyond the original demonstrations (Gao et al.
2018 Brown et al. 2020), but generalization remains fundamental challenge.
A primary issue of BC is the lack of generalization to unseen scenarios due to distributional mismatch (Ross et al. 2 Given the high cost of collecting extensive demonstration data, efforts made to learn generalizable policies with a limited number of demonstrations (Kim et al., 2013 Mandlekar et al. 2020] Prados al., 2024). Several methods achieve better generalization by exploring perturbations in the policy parameter space (Such et al., (017). known as evolutionary algorithms (Salimans et al., 2017). Further approaches add noise to the parameters and optimize through policy gradient descent to facilitate exploration (Plappert et al. 2017 Cao et al.) 2020). Other works focus on generalizing to long-horizon, multi-stage tasks by leveraging the compositionality of small set of given demonstrations (Mandlekar et al.) 2020). They learn goal-directed policies from stochastic rollouts during the behavior cloning phase to achieve controllable behaviors. However, existing works still struggle with contextual variations and perform poorly in zero-shot settings.
Hypernetworks (Ha et al. 2017 generally speaking, use a neural network to generate the weights of a target network. During training, only the learnable hypernetwork weights are optimized. One of the noticeable merits of hypernetworks is the parameter efficiency. It achieves this by generating weights of multiple target networks on related tasks, performing well in soft-weight sharing (Chauban cet al., 2023b; Von Oswald et al. 2019 Navon et al. 2021 Zhao et al. 2020) and weight compression (Ha et al.] 2017). Hypernetworks hold a good connection with meta-learning, primarily due to their intrinsic capability to facilitate the learning-to-lle leam process. Research such as (Sendera et al., 2023] Przewięžlikowski al.] (202)) employs hypernetworks encapsulate the inner-update loop within the MAML framework (Mishra et al., 2017). These approaches help position hypernetworks as a meta-optimizer, concept that has also inspired the methodology adopted in our study.
In the field of multi-task RL and meta-RI,, which focus on policy generation and generalization, hypernetworks have achieved successful applications in areas such as morphology control (Xiong let al. [2023] (024), episodic memory (BG et al. (2024), context-aware response (Beukman et al. 2024
and dynamics model generation (Xian et al. 2021). For instance, using an MLP-based hypernetwork
to learn a universal policy for different robotic morphologies can maintain computational e efficiency while achieving inference performance comparable to that of Transformers (Xiong et al., 2024).
Other approaches demonstrate that when incorporating contextual information into policy adaptation, generating the weights of decision adapters with a hypernetwork can effectively circumvent the computational constraints caused by the expansion of learnable parameters (Beukman et al. (0024). As far as we observe, these methods (Xian et al. 2021 Beck et al. (023) focus on the combination of the hypernetworks and zero-shot contextual RL. However, our method focuses on the hypernetoork structure and explicit modeling of the optimization procedure of the target network for achieving generalizable policy generation.
Learned Optimizers In this work, we design the hypernetwork to work like an optimizer. This is in concept similar to learned optimizers (LO) (Harrison et al. 2022 Bengio et al.) [2013] Runarsson & Jonsson| 2000 (Andrychowicz et al., 2016 Wichrowska et al. (017). LO are models designed to enhance the efficiency and effectiveness of optimization processes. Unlike traditional hand-crafted algorithms such as SGD or Adam, LO leverages meta-learning to learn optimization strategies directly. Their primary goal is to generalize across tasks and outperform standard optimizers by adapting to specific problem structures. A key distinction is that LO is meta-learned and requires tuning on novel tasks. In contrast, our proposed structure predicts the update scheme directly from task specifications, eliminating the need for any data at test time.
3

---

Published as a conference paper at ICLR 2025
M
SGD
0"
M
H
H
(b)
(d)
Figure 2: (a) Given expert demonstrations D (MM. one can always achieve the optimal policy l* via BC with stochastic gradient descent (SGD), which also defines the generalization upper bound in the context of BC for task M. (b) Due to the lack of demonstrations, conventional hypernetworks generate parameters & for the task M, by supervised learning the mapping from task to parameters, subject to overfitting given infinite solution space. (c) Even without actual demonstrations, we propose that the task M is informative of the data, and can thus be used to guide the iterative update for predicting the optimal parameters 0+ (as in (a), Sec. 4.1 which in turn constrains the solution space and promotes prediction as optimization (from the task instead of data, Sec. 4.2). (d) With
the proposed architectural inductive bias, the hypernetwork in (c) is indeed performing iterative optimization (action prediction loss is decreasing) as the iteration (block) number increases.
POLICY GENERATION WITH HYPERNETWORKS
Given a Markov Decision Process (MDP) M (S, R. where S is the state space, A the action space, T SS x A P(S) is the transition function, R: S x A R is the reward function, and Y E [0, 1] is the discount factor, an optimal control policy π : S + (.A) is supposed to maximize the expected (discounted) return in the context of reinforcement learning (RL). Typically, the policy
network π is a mulli-layee perceptron (MLP) (Rakelly et al./ 2019 Sarafian et al.) 2021 Rezaei]
Shoshtari al. 2023 Beck et al. 2023). However, training a policy network with RL is challenging (lbarz et al. (021). e.g., due to large exploration space and optimization difficulties.
To resolve the training difficulties in RI., behavior cloning (BC) trains policy network in supervised manner with a dataset of expert demonstrations D {(s),, ((st,o()) a with N₂ being the number of demonstrations and si, aj being the state and action at time respectively. The optimal policy can then be obtained by minimizing the action prediction loss L(0, D) = Σ., (( (st: af). with é as discrepancy metric and e θ the parameters of policy network æ. Despite the stability and efficiency of the training procedure, BC relies heavily on expert data and may not generalize to unseen task specifications. To generalize, one has to collect demonstrations covering all different situations, which is time-consuming and costly, especially for real-world applications.
Problem formulation. We aim to maximize the utilization of the limited demonstrations for BC by developing a mechanism that can automatically synthesize the "optimal" policy network (parameters) from a novel specification of a given task. Specifically, for source task set S and target task set T. we denote the source tasks characterized by the corresponding MDP as M. (MylM, A,, Ty. 7).j j 5), and similarly the target tasks are M, = (Mylj T), with M nM, = Ø. e.g.. differing environmental settings or dynamics. Moreover, for source task M, Ms, there exists a collection of demonstrations D,, whereas for the target tasks in My, no demonstrations are available. Our goal is to train network with Ω as parameters, 7 -M - O to produce the weights 0 of the policy network æ after consuming the task specification M, such that ( (+H(MM minimizes the BC loss derived with the demonstrations D(M).
Formally, the neural network H functions as a hypernetwork (Ha et al. 2017). and can be trained by minimizing the following:
C(2, (MM, = ())
(1)
3,t
where j is the task (specification) index and i, go through different demonstrations and time steps within a task. After training, the hypernetwork 7 should be able to synthesize the optimal parameters for each source task (specification) in Ms, i.e., L(2, (M,, D))) jes) is minimal. Note that without demonstrations for the target tasks Mr. the performance of the synthesized policy networks fully
Note that the loss computed with demonstrations, which is only used here for validation purposes.
4

---

Published as a conference paper at ICLR 2025
Overview of HyPoGen
Policy Network
X
Policy Updiete
@
By pormet Black
Byperact Black
500
gx
II Policy Gradients Estimation
Black
Bleck
Block N
Layer
8
0%
.b
0g
Vyby
Task Spreification
Activation Estimatioo
20 8
21
N
Veliy
Valta
8
Vohy
Sum
Ox
8m
8x
Network Forwarding
Gradicat
80
000
89
Figure 3: The overall pipeline of the proposed HyPoGen. Top: HyPoGen uses an iterative update scheme (Eq. 66 to generate the policy network parameters 0 from the task specification M. Bottom: inside each hypernet block, we calculate the neural "gradients" for each block of the policy network way resembling the backpropagation process (Eq.
depends on the generalization capability of H. while only being trained on the source tasks Mg and the demonstrations {Dyli S). Next, elaborate on the generalization gap of as mapping from the task specification to the optimal parameters, and how we can improve the generalization by injecting an inductive bias into structure that encourages optimization instead of memorization.
HYPERNETWORKS AS OPTIMIZATION WITHOUT DATA
Motivation. As pointed out by (Heckel Yilmaz 2021 Nakkiran et al. 2021 I) an optimization process with proper number of steps tends to yield parameters that generaliza lize better than model directly memorizing the training set. Currently, the architecture of hypernetwors is dominated by MLPs, especially those focusing on policy generation (Chauhan et al.) 2023a). While an MLP is capable of mapping a task specification to the parameters of a policy network, remains a very naive approach, offering limited generalization ability due to its lack of inductive bias. Building on this insight, we propose a novel hypernetwork to function as an iterative optimization process, rather than simply memorizing the best parameters for each training task. This operation is similar to the task-specific training with SGD and thus could potentially induce better generalization performance due 1) this inductive bias constrains the search space for the hypernetwork, thus, less prone to overfitting to the training tasks; and 2) optimization is low-degree-offfreed of-freedom operation and, learned, will approach the upper bound by task-specific training using demonstrations. Next, will first introduce the designing principle in Sec. 4.1 and then we detail the proposed hypernetwork with the introduced bias in Sec. 4.2) An overview the motivation presented in Fig. []] which also shows evidence that the proposed hypernetwork is performing optimization when checking the action prediction loss.
4.1 BIASING HYPERNETWORKS TOWARDS OPTIMIZATION
The overall pipeline of the proposed HyPoGen is illustrated in Fig. In this section, start by deriving the gradient formula for target networks on a given datapoint (Eq. 4). and then we argue that the dependency specific data can be resolved (Eq. 5.. Finally, we present the model structure that incorporates the optimization inductive bias targeting MLP-based policy networks (Eq. 6 7 g
Since most policy networks are MLPs due to their training efficiency and scalability, and without loss of generality, we can put policy into the following format:
π (s; 0)
On h N-1
o 1 ... hes o h (s)
where 0 = (9, N and N is the number of layers (or blocks) in the policy network म. Please note that each layer h does not need be fully connected layer, e.g., h can be block of layers
5

---

Published as a conference paper at ICLR 2025
sub-network. The grouping of the parameters of a policy network into the format presented in Eq. 2ddpeedddds on its actual topology, and we mainly experiment with the MLP structure, given its popularity in the policy learning literature.
Furthermore, to fit single data point (s.a) under the & distance, gradient step updates the policy network parameters
gh+l 6 A. (=(; -a) (x;
00
where, A > (T(ss; a) denotes the effective stepsize and OT (s; /00 can be computed using the chain rule:
On (m{
N- IhN-m+1 ( (ZN-m; 0 (0m II
Th 1;
00,. n
02N-m
00,
with Zn h(( 2n 0n) the activation of the n-th layer (block). Please note that even though Eq.
and Eq. Arre instantiated with a single state-action pair, the counterparts computed with dataset of demonstrations share a similar format. Therefore, we proceed describe the proposed hypemeeworr architecture without further laboring.
4.2 NEURAL ESTIMATION OF POLICY UPDATES WITHOUT DATA
Accurate computation of the updates in Eq 3 for deriving the optimal policy is difficult due to the lack of demonstration data for the target tasks, for example, both X- (=; (=T(s=0)-a)) (a) and ( 8*) /00 in Eq. 3 are data-dependent. However, it possible to approximate these updates (at least theoretically), if we treat the specification M of a target task as sufficient representation of its demonstrations.
More specifically, the update of the policy parameters over all demonstrations is
(T: 0)
A = (((.a)))(((M))) A (T((s:))- -
=(0,,
(5)
00
Note that the expectation is taken over the distribution of training demonstrations D(M). which is defined by the task specification M. The update of the policy parameters can be seen as a function F that only depends on the policy network parameters and the distribution of the demonstrations (D(M)).
With the above reasoning, is sufficient to use an encoded representation of the task specification
6(M) to represent (M(M) and estimate the policy updates A = =(0, o ((M)) ). We use the same task encoder as in previous works (Rezaci-Shoshtar et al. 2023 Sarafian al. 2021). though these works directly map the task specification to policy parameters. Taking this ground, we propose the following hypernetwork that leverages the structures presented in Eq. optimization generalization, as well as tackling the lack of demonstration at test time.
for promoting the
Iterative policy update with "neural gradients". Inspired by Eq. 3 and the above reasoning, propose network modules and vb. such that the output of the proposed hypernetwork is as follows:
H(M)= = 0k and 0² = =001 (0(00=), & 10000100
6(M)),
(6)
where A and w estimate the "stepsize" (i.e., A (T(::) a)) and "gradient" of the k-th policy update from the task embedding (specification) ((M). The total number of updates K, and 6 is set of learnable parameters. To improve the training efficiency, we also compress/recover the policy network parameters via an encoder and a decoder so that the hypernetwork can perform the policy generation in a latent space. (details can be found in Sec. A.2)
We further incorporate the optimization inductive bias from Eq. into the structure of & to esti- mate the corresponding terms. Let V (, Ōh (,))) 0.)/ represent the gradients with respect to the activations and parameters, respec- tively. Then, we divide gradient estimation function &b into two sub-networks vi and wff im- plemented by MLPs, to estimate Vahn, Veh, respectively. Moreover, we instantiate another series of MLPs o; n = L...N that compute the pseudo estimation of the activation 2n 's, i.e. di (e(M)). One could also make the estimation of 2, 's dependent
/8 and (2,
2, oi of the current estimate of the latent parameters, but we omit the dependence for clarity and leave the necessity experimental justifications.

---

Published as a conference paper at ICLR 2025
With these, the neural estimate of V V,,, and Veh, for each iteration & computed
h.hn = ( 0n) and Voh, (En-1,
with the superscript omitted for clarity, and now we have the output of a/s as:
(0,0
Voln
(8)
m=11
n=1
which is the neural counterpart of the gradient computation illustrated in Eq. Given these structural designs, we can denote the proposed volicy-generation hypernetwork as H. {(*) Jk-1 o, (o% V v 1 ( 0"), which are the learnable parameters.
Training loss. Finally, we can train the proposed hypernetwork H for policy generation (HyPoGen) using the demonstrations of source tasks (M,. D)) by minimizing the BC loss presented in Eq. ii an end-to-end manner. Next, we study the effectiveness and generalization of the proposed hypernetwork in synthesizing policies for target tasks (from only the task specifications) without accessing any demonstrations.
EXPERIMENTS
The experiments are designed to evaluate the generaliza- tion capabilities of existing policy generation methods and
HopeiZoro
ByPoGen
Roward
our proposed method, to demonstrate the effectiveness which biases the hypernetwork towards optimization. We leverage two sets of tasks for evaluation. The first derived from MuJoCo environments, which provide ro- bust platform for concept verification and are widely used by existing methods. The second set of tasks is sourced from the ManiSkill environment, which includes replicas of real robot arms and is more closely aligned with real-world scenarios. By conducting experiments both MuJoCo and ManiSkill environments, we aim comprehensively evalu- ate our methods, demonstrating their versatility in handling various types of task.
100
Speed
Figure 4: Our model shows better gen- eralization than the baseline Hyper- Zero (details in Sec. 5.2)
Experiments conducted individually on five selected representative tasks. Each task varied by a group of specifications. It is important to emphasize that we focus on the generalization among different specifications within each task, rather than developing generic policy generation for drastically different tasks. For each task, we split the specifications into training set, denoted as Mgs and a test set, denoted Mr, and Mg Mr Ø. The methods are trained on the expert trajectories from Mss and during test time, we roll out policy for each specification in My and report the average of their performance (on the designated specification). Experimentally, instead of using the full MDP M input, omit the common component in M and use the task specification (length/target speed, etc.) as the input.
5.1 ENVIRONMENT SETUP
MuJoCo. Following HyperZero Coperrero Rezeeii Rezaei-Shoshtari Shoshtari al. (2023), we use three environments from DeepMind Control Suite [Tunasuvunakool et (2.20) (a wrapper of the MuloCo environment Todorov et al. (2012) for evaluation: Cheetah, Walker, and Finger. For each task, we examine three types of specifications: (1) the desired speed of the object, (2) the torso length of the object, and (3) the combinations of speed and length.
ManiSkill. We utilize the ManiSkill2 environment Mu et al. (2021): Gu et al. (2023) to validate
the real-world applicability. We select two tasks, LiftCube and PlaceCuee and conduct experiments using the Franka Panda robotic arm. We examine four types of specifications: (1) the size the target cube, (2) the stiffness the controller, (3) the damping of the controller, and (4) the arm length of the robot. The detailed settings of the task specifications can be found in Sec. B.1
7

---

Published as a conference paper at ICLR 2025
Data collection and evaluation protocols. For MuJoCo Environments, we pre-train TD3 Fuji moto et al. (2018) agent on each specification separately for x 10 steps as the expert. For each specification, 10 trajectories of length 1000 are collected as the demonstration. We randomly split the specifications into training and test sets using a 20% to 80% ratio. At test time, we evaluate the performance by caleulating the average reward of roll-out policies. The training/test splitting process is repeated five times, and the average is calculated to reduce the effects of randomness.
For Maniskill Environments, we pre-train PPO agent Schulman al. (2017) for 4 10 steps and collect 1000 successful trajectories for each specification. We uniformly sample 30 % of the data for training, and use the remaining 70% for testing. At test time, we calculate the average success rate and the average episode length for evaluation.
Baselines. For a fair comparison, all methods (including ours) use the same policy network archi- tecture and specification encoder. The baselines summarized follows:
1. Specification conditioned policy (Cond Policy): This method aims to learn a general policy for all specifications. First, the network encodes the task specification using an encoder. It then takes the concatenation of the encoded specification and the state as input to predict the action.
2. Specification conditioned policy with UVFA Schaul al. 2015): : This method predicts actions and values under the UVFA framework. It first predicts the action in a similar way to specification- conditioned policy. Besides, it adds an extra branch to predict the action-value functions Q(s, à) where is the current state and à is the predicted action, effectively utilizing the action-value in the expert rollouts as an extra supervision signal.
3. Specification conditioned Meta Policy: This method trains the policy network with the MAML Finn et al. (017)) framework. It aims to learn a set of weights that could adapt to different specifications. It predicts action similarly to apecification-conditonn differently through an inner-outer loop process, where the inner loop adapts the model to the support set, and the outer loop optimizes the model's initial parameters for rapid adaptation. Note that this method requires few-shot fine-tuning at test time.
policy but trains
4. PEARL Rakelly et al. (2019): Unlike other baselines, this method does not require access to contextual information. instead keeps a replay buffer and uses it to generate an environment de- scriptor via variational inference approach. This way, it can effectively leverage past experience and enable efficient off-policy learning. Note that this method also requires few-shot fine-tuning at test time.
5. HyperZero Rezaci-Shoshtar et al. (2023): It uses MLP-based hypernetwork to predict the weight of policy networks. The hypernet takes the encoded specification as input and outputs the policy network parameters.
We conduct zero-shot policy generation evaluations for Cond Policy, UVFA, HyperZero, and method. In contrast, the evaluations for Meta Policy and PEARL are carried out in few-shot transfer setting. More specifically, these two methods are fine-tuned using expert trajectories from the test specifications Mr, thus breaking the no test-time demonstration assumption.
Implementation details of HyPoGen. We apply the same task encoder design as in Hyper- Zero (Rezaei-Shoshtari al. 2023). which consists of 6 Res-Blocks. Following -kezaai-Shoshtari et al. 2023), the policy networks are implemented as two-layer MLPs with hidden dimension 256 for MuloCo and three-layer MLPs with hidden dimension 256 for ManiSkills. The input and output dimensions are specified by the task.
We K = hypernet blocks and apply Adam (Kingma & Ba 2015) optimizer with learning rate 1 x 10 We train our model on an NVIDIA 4090 GPU with a batch size of 512 for 2000 epochs (approximately 11 hours); we use the same hyperparameters in ManiSkill except for training 1000 epochs (approximately 10 hours). We detail our hyperparameters in Sec. B.2
5.2 QUALITATIVE RESULTS
To intuitively illustrate the generalization ability of our method, we compare its performance with HyperZero (Rezaci-Shoshtari et al. 2023) on the Cheetah environment in MuloCo. Both methods
8

---

Published as a conference paper at ICLR 2025
Table Comparison of the average reward achieved by different methods on the MuJoCo tasks, The bold and underline numbers indicate the best and second-best results.
Reward
Cheetah
Finger
Walker
Method
speed
length
&lenght
speed
length
speed&length
speed
length
speed&&length
Cond Policy
433.24
$74.70
356.68
379.66
409.30
289.58
151.95
336.71
186.15
UVFA
396.86
588.56
340.78
383.65
422.62
287.43
115.60
294.87
177.81
Meta Policy
337.35
579.41
199.63
125.89
217.58
114.78
44.54
46.22
44.44
PEARL
177.16
214.65
160.42
$2.06
54.13
53,03
HyperZoro
695.73
895.46
602.39
596.80
536.56
353.68
328.48
642.22
393.93
HyPoGen(Ours)
819.23
926.90
623.76
835.21
657.12
365.85
436.26
706.16
409.88
Expert Rollout
869.59
963.07
927.12
975.71
959.42
913.40
722.68
897.11
814.43
Table 2: Comparison of the success rate (%) and episode length (#steps) on the ManiSkill tasks. For each specification, we roll out 100 trajectories and compute the average success rate and episode length. Again, bold and underline numbers indicate the best and second-bess results.
% Rate 1. ASteps
Luf
Piic&&PlaceCube
Method
cube size
stiffiess
damping
arm length
cube size
stiffiess
damping
arm length
Cond Policy
89.93, 32,09
0.00, 200.00
56,28, 94.27
69.36, 71.40
69.81,
2.20, 196.56
39.26. 131.41
43.77. 123.02
UVFA Policy
90.27. 32.40
0.00,
200.00
78.67,
53.53
72.18, 66.11
71.75. 71.91
0.00, 200.00
29.53.
146.68
36.85. 134.42
41.14
0.00, 200.00
0.06, 199.90
76.45. 56.95
177.89
PEARL
84.47, 41.60
59.87,
86.70
35.67. 132.23
78.73, 52.86
21.06, 163.28
24.44, 156.22
12.58,
177.54
16.08, 171.93
HyperZero
86,60, 38.24
0.00,
200.00
31.33,
140.90
63.64, 81.59
25.87,
155.27
0.00, 200.00
0.00,
200.00
13.92, 174.91
Hy (urr))
35.87.39.37
97.20,
16.72
93,28.
24.78
85.73,39.44
39.44
72.87.
68.97
78.33. 58.76
41.26. 125.99
52.54, 106.75
Expert Rollout
94.13,
96.73, 17.88
97.28, 16.30
93.64, 24.74
69.87, 73.26
75.92, 62.65
73.72, 66.19
54.77, 101.14
are trained on target speeds and 10 and tested on speeds from to 9, with reward curves shown in Fig. 4 These experiments show that even though the different tasks are varied by single parameter (i.e., speed), transferring between them is challenging. While both methods perform similarly on training tasks, HyperZero's 's performance degrades significantly on unseen tasks with larger train-test gaps, whereas HyPoGen maintains stable performance, showing superior generalization.
5.3 COMPARISONS
MuJoCo. The test-time average rewards obtained from different methods are reported in Tab. [ We can observe that the rewards obtained from conditioned policies (Cond Policy and UVFA) are considerably lower than those from the expert. This suggests that the effectiveness of using specifications as network input (condition) limited. The two few-shot tnne-tuning methods, Meta Policy and PEARL, also perform much worse than the expert rollout. HyperZero outperforms all other baselines, illustrating that hypernetwork is a more powerful policy generation paradigm than the conditioning methods. Our method yields the best results across all tasks and specifications, demonstrating the advantage of incorporating optimization inductive bias over direct policy generation. Please refer to Sec. C. for detailed results in each specification and Tab. 13 in the appendix for the standard deviation of the results.
ManiSkill. We report the average success rate and episode length in Tab. 2 and the standard deviation of episode length in Tab. 14 in the appendix. The max step of a trial is set to 200. All methods can achieve high success rates across different cube size specifications. We contend that this task is relatively simple, as it primarily involves controlling the clamp's 's opening width, which is nearly linear to the cube size. In more challenging scenarios where controller stiffness, damping. and arm length are varied, a linear relationship is no longer valid. As a result, the performances of different methods vary and degenerate. In the most challenging stiffness specifications, the conditional policy methods, the meta-learning baseline, and HyperZero fail to learn reasonable policy. In contrast, the proposed method HyPoGen is able to generate policies that achieve high success rates and low episode length, demonstrating its superior generalization ability. Additionally, in-depth experiments in Sec. C.33 verified that the poor performance of the basclines is attributed not to a lack of capacity, but their inductive bias, further highlighting HyPoGen's superior generalization.

---

Published as a conference paper at ICLR 2025
5.4 ANALYSIS
In this section, we perform analysis and ablation studies to validate our model from different aspects. By default, use the task Cheetah from MuJoCo, with different speed specifications.
Does HyPoGen actually perform optimization? Since we aim at biasing the hypernetwork towards optimization, one may be interested in knowing: (I) if HyPoGen simply remem- bers a fixed set of parameters for each specification, and (2) will the "neural gradients" work in similar way to ordinary gradients? i.c., if they are really optimizing the BC loss.
Table 3: Statistics of the generated parameters by HyPoGen. We report the average magnitude of the param- eters and their standard deviation.
Magnitude
Std.
For the first question, We fix the input specification M and examine the response of HyPoGen with various input weights 80. Specifically, we replace the values of 8 with uniformly random values within the range of [-0.1,0.1] and then com-
fel. weight bii.
43.92
9.32
6.17
2.20
fc2. weight
9.31
5.53
fc2.bias
0.44
0.31
pare the outputs of g for different initial values of go. The statistics of weights and biases in g
reported in Tab. S It is clear that the differences significant, indicating that HyPoGen is not simply memorizing a certain set of fixed parameters for given task specification.
As for the second question, since HyPoGen requires significantly fewer update steps than conven- tional gradient descent (8 versus thousands), the "neural gradient" encapsulates much more informa- tion than the local gradient. Therefore, it is not appropriate directly compare the val- ues of neural gradients with conventional ones. Instead, we report the BC loss after each optimization step (i.e. BCLoss(0*)). The results in Tab. show that BC loss is con- stantly decreasing after each update, verifying the effectiveness of HyPoGen in predicting neu- ral "gradients" that optimize the weights of the network. Thus, HyPoGen not only gener- ates different updates for different input weights, but also "optimizes" in the right direction.
Table 4: BC loss after each "gradient" update.
#Updates
2
3
5
6
7 8
Loss@Cheetall 71.49 61.13 46.30 40.64 21.95 11,21 3.19 1.61
Loss@LifiCube 9.041 5.202 2.972 2.68 1.418 0.948 0.644 0.123
Does the optimization-based hypernetwork improve training convergence? We compare our method with HyperZero, examining how much reward (test-time) they can achieve when trained with the same epochs, and how many epochs each requires to achieve the same reward. The results are reported in Tab. 5 Our method achieves higher reward with the same number of epochs. Besides, our method requires much fewer epochs of training to reach a given reward value. These results demonstrate the superior training efficiency of our method. Due to limited space, please refer to Sec. for more experimental analysis.
Table 5: Convergence property at certain epochs reward.
250 epochs
600 reward
HyperZero HyPoGen
571.3
394
732.9
74
6 DISCUSSION
We propose a novel hypernetwork, HyPoGen, which explicitly models the gradient-based optimiza- tion process in the architecture for policy synthesis, capitalizing on the generalization capability of test-time optimization. Additionally, we introduce an effective building block to mimic the interde- pendencies of the gradient flow while training the policies. This structural design further enhances the interpretability and generalizability of the proposed hypernetwork. Through comprehensive benchmarking, we demonstrate the superiority of our method terms of generalization under limited learning budgets (both demonstration and computation) and the effectiveness of each proposed component. Our work not only contributes to the development of more efficient and generalizable policy learning algorithms but also provides insights for future research in the area of hypernet- works. However, generating policies without seeing demonstration data for completely different tasks remains challenging and is not fully resolved by method. Future work may explore different optimization techniques and more sophisticated building blocks that can better generalize across a more diverse range of tasks.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our work. The code, datasets, and instructions necessary replicate our experiments publicly available at https:/ //tp:////htttb/u/ com Rete//eNNgiii htt.//eennnninnneeeeeiee We hope these resources will assist future re- search in both reproducing and building upon our work.
ACKNOWLEDGMENT
This work is supported by the Early Career Scheme of the Research Grants Council (grant 27207224), the HKU-100 Award, a donation from the Musketeers Foundation, the HKU Seed Fund for PI Research, and in part by the JC STEM Lab of Autonomous Intelligent Systems funded by The Hong Kong Jockey Club Charities Trust. Difan Zou and Youyi Zheng would like to thank the support by the NSFC Grant (No. 62172363, No. 62306252) and Guangdong NSF Grant (No. 2024A15 501444)). Siyan Dong would like to thank the Institute of Data Science at HKU for postdoctoral scholarship. The authors would also like thank the HKU-Shanghai ICRC for providing valuable computing resources.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta- reinforcement learning. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.). Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 1478-1487. PMLR, 14-18 Dec 2023.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks?, pp. 265 (65--87. Routledge, 2013.
Michael Beukman, Devon Jarvis, Richard Klein, Steven James, and Benjamin Rosman. Dynamics generalisation reinforcement learning via adaptive context-aware policies. Advances Neural Information Processing Systems, 36, 2024.
Prasanna Kumar BG, Yash Upadhyay, Nunna Vishnu Teja, Penta Aravinda Swamy, et al. Advanced multi-task reinforcement learning utilising task-adeptive episodic memory with hypernetoork inte- gration. In 2024 2nd International Conference Intelligent Data Communication Technologies and Internet of Things (IDClo'), pp. 635-640. IEFF, 2024.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. santer-thhn-demosstraatte imitation learning via aano demonstrations. In Conference robot learning, pp. 330-359, PMLR, 2020.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Twenty-found international joint conference on artificial intelligence, 2015.
Junjie Cao, Weiwei Liu, Yong Liu, and Jian Yang. Generalize robot learning from demonstration to variant scenarios with evolutionary policy gradient. Frontiers Neurorobotics, 14:21, 2020.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Sohcila Molaci, and David A Clifton. A brief review of hypernetworks in deep learning. arXiv preprint arXiv:2306.066 06955, 2023a.
Vinod Kumar Chauhan, Jiandong Zhou, Soheila Molaci, Ghadeer Ghosheh, and David A Clifton. Dynamic inter treatment information sharing for heterogeneous treatment effects estimation. arXiv preprint ar Xiv::2300. 15984, 2023b.

---

Published as a conference paper at ICLR 2025
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Yang Gao, Huazhe Xu, Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv: 1802.0531,, 2018.
Vinicius G Goecks, Gregory M Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint arXiv:1910.0428: 2019.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Man- iskill2: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenRe- view.net, 2023. URL [tllittts/s/ppeeeeiieeee
net wwwpp/dddpd.bbbb//po.p.p.pireepp ?id=b cateb___DDyyvD__1
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworss. In International Conference Learning Representations, 2017. URL
James Harrison, Luke Metz, and Jascha Sohl-Dickstein.
closer look at learned op- timization: Stability, robustness, and inductive biases. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, http:/appprrrr_e wttp:/fiwesspaee//eeepeaaaaeeeee /
2022, 2022. URL http: http:://apppps.app 184c1e1 [BCCCA44B.000772222222222222--2-
Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how to eliminate it. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. UtL//tttss//ppeeeeeeee....
Forum?id=t.. jobbw
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings the AAAI conference on artificial intelligence, volume 32, 2018.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Roboties Research, 40(4-5):698-72) 2021.
Beomjoon Kim, Amir-massoud Farahmand. Joelle Pineau, and Doina Precup. Learning from limited demonstrations. Advances in Neural Information Processing Systems, 26, 2013.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ICLR (Poster), 2015.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618, 2022.
Zachary E Lee and K Max Zhang. Generalized reinforcement learning for building control using behavioral cloning. Applied Energy, 304: 17602, 2021.
Ajay Mandlekar, Danfei Xu, Roberto Martin-Mattin, Silvio Savarese, and Li Fei-Fei. Learning to gen- eralize across long-horizon tasks from human demonstrations, arXiv preprint arXiv:2003.06085, 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and P. Abbeel. A simple neural attentive meta- meta-learner. In International Conference Learning Representations, 2017.
12

---

Published as a conference paper at ICLR 2025
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill bench- mark with large-scale demonstrations. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL h www.:eeeeeeeeeeccceaeeeeeeeeeeeo
In Joaquin Vanschoren and Sai-Kit Yeung
neurips wwtpppppp////22// wow.eroounddd...
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6292-6299, IEEE, 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12): 2021.
Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypemetworks. In International Conference on Learning Representations, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, higg-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 700.00195, 2017.
Dean A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. mavinoonnsnniiggaioonn.. Neural Computation, 3:88-97, 1991. URL httptttt///aappt
7813587]
Adrian Prados, Santiago Garrido, and Ramon Barber. Learning and generalization of task- parameterized skills through few human demonstrations. Engineering Applications of Artificial Intelligence, 133:108310 10, 2024.
Marcin PrzewiçZlikowkii Przemystaw Przybysz, Jacek Tabor, M Zięba, and Przemyslaw Spurek. Hypermaml: Few-shot adaptation of deep models with hypernetworks. arXiv preprint arXiv:2205.1574: 2022.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arvi:: 1 1009.700889 10087, 2017.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforrement learning via probabilistic context variables. International conference on machine learning, pp. 5331-5340. PMLR. 2019.
Sahand Rezaei- Shosharri,,,,,ri, Charlotte Morissette, Francois R Hogan, Gregory Dudek, and David Meger. Hypernetworks for zero-shot transfer in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9579-9587, 2023.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Thomas Philip Runarsson and Magnus Thor Jonsson. Evolution and design of distributed learn- ing rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00, pp. 59-63. IEEE, 2000.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategics as a scalable alternative reinforcement learning. arXiv preprint aiviv: 17o:: 0703033844 2017.
13

---

Published as a conference paper at ICLR 2025
Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In International Conference on Machine Learning, pp. 9301-9312. PMLR. 2021.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi-
mal policy optimization algorithms. ArXiv, htss/1707,063444 2017. URL /ttps://app
semanticscho wtt.ccoo.lr..org/oooouttt:D::::. 286950522266222
Marcin Sendera, Marcin Przewięzlikowski, Konrad Karanowski, Maciej Zięba, Jacek Tabor, and Przemystaw Spurek. Hypershot: Few-shot learning by kernel hypernetworks. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2469-2478, 2023.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Kaushik Subramanian, Charles I. Isbell Jr, and Andrea Thomaz. Exploration from demonstration for interactive reinforcement learning. In Proceedings of the 2016 international conference autonomous agents multiagent systems, pp. 447-456, 2016.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv: 1712.06667, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033.
IEEE, 2012. doi: 10.1109/ 0.01...3.66099
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6: 10002, 2020. ISSN 2665-9638.
Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. Continual learning with hypernetworks. arXiv preprint 190.19990000655 2019.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo. Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International conference machine learning, pp. 3751-3760. PMLR, 2017.
Zhou Xian, Shamit Lal, Hsiao- Yu Tung, Emmanouil Antonios Platanios, and Katerina Fragkiadaki. Hyperdynamics: Meta-learning object and agent dynamics with hypernetwrkss. arXiv preprint arXiv:2103.09453 2021.
Zheng Xiong, Jacob Beck, and Shimon Whiteson. Universal morphology control via contextual modulation. In International Conference on Machine Learning, pp. 38286-38300. PMLR, 2023.
Zheng Xiong, Risto Vuorio, Jacob Beck, Matthieu Zimmer, Kun Shao, and Shimon Whiteson. Distilling morphology--conditione hypernetworks for efficient universal morphology control. arXiv preprint arXivv2402.06570, 2024.
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. CoRR, abs/1 005/06004889, 2018.
Dominic Zhao, Seijin Kobayashi. João Sacramento, and Johannes von Oswald. Meta- Meta-learning via hypernetworks. In 4th Workshop on Meta-Learning NeurIPS 2020 (MetaLeam 2020). NeurIPS, 2020.
14

---

Published as a conference paper at ICLR 2025
A MORE DETAILS OF THE METHOD
A. WHY AN MLP CAN BE APPLIED AS A GRADIENT APPROXIMATOR FOR MLPS
We show that &c BW:
where W,6 are the weight and bias of MLP layer, are very close the form of MLP in this section.
The forward process of MLP layer is:
2x = o -
(9)
where in is the hidden variable, and the activation function. By taking the derivatives, we have
Ozn An(
(10)
aw,
Ozn
= An (2n-1)
(11)
Öbn
where
(m(=n-1) 1) = diag( (Wn Citaglo(WWnz,, + bn))
(12)
To get from dc__
to OW. we apply the chain rules.
BC
aW.
dc Oc dzN Ozn+1 Ozn
(13)
awn
daN dzN-1 Ozn OWn
Oc
N Oz dzn
(14)
daN i-nt1
02i-1 OWn
And
85n can also be calculated from the forward process 05n-1
Omn (2n1)))
(15)
0z,-1
Note that (Eq. [2] is just the output another MLP with a different final activation. And other operations in (Eq. 10 14 15) are fundamentally linear in nature. This linearity implies that the forms of these gradient expressions bear a strong resemblance to those typically encountered in MLP. As mentioned above, it is reasonable to use an MLP gradient approximator for MLPs.
A.2 OPTIMIZATION IN THE LATENT PARAMETER SPACE
To ease training and memory consumption, the proposed hypemnetork predicts the policy updates in latent space. Specifically, employ a layer-level encoder Es., to compress each layer of the target network into a latent space before feeding them into the hypernet blocks. During the optimization process, parameters consistently updated within the latent space. After the optimization process, a layer-level decoder Do,, is utilized to reconstruct the latent representation back into the parameter space of This approach ensures that the size of our optimization network parameters does not expand significantly with an increase in the target network parameters, thereby enhancing the model's applicability. For ease of understanding, we have illustrated these less critical details in Fig. 3
B ADDITIONAL EXPERIMENTAL DETAILS
B.
DETAILED SPECIFICATIONS EACH TASK
B.1.1 MUJoCo
Desired speed. Tab. showws the desired speed specifications used for each task. The default value is utilized in the experiments that involve changing torso length. Conversely, during the torso length experiments, the default value of the desired speed is applied.
15

---

Published as a conference paper at ICLR 2025
Table 6: Details of desired speed specifications.
Environment
Range
No. of Samples
Default Value
Cheetah
[-10, 0), (0, 10], increment 0.5
Walker
[-5, 0), (0, 5]. increment 0.25
40
+1
Finger
[115, 0), (0, 15), increment 1.0
30
+15
Table 7: Details torso length specifications.
Environment
Range
No. of Samples
Default Value
Cheetah
[0.3, 0.7], increment 0.01
41
0.5
Walker
[0.1, 0.5), increment 0.01
41
0.3
Finger
[0.1. 0.4], increment 0.01
31
0.16
Table 8: Details of speed and length specifications.
Environment
Range of Speed Parameter
Range of Length Parameter
No. Samples
Chectah
[+1, +10), increment 1.0
[0.3, 0.7), inerement 0.05
10 x 9 grid
Walker
[+1,+5], +5], increment 0.5
[0.1, 0.5), increment 0.05
9 grid
Finger
[+1, +15], increment 1.0
[0.1, 0.4), increment 0.05
15 x grid
Torso length. In all environments, the torso length parameter refers to the geometric size of the torso of the controlled object. Different sizes change the physical properties such as mass, center of gravity, etc. A complete list of torso lengths we used is shown in Tab. 7
Desired speed and torso length. Now, list all the parameters in the experiments of jointly changing speed and length parameters in Tab.
B.1.2 MANIŠKILL.
The specifications for ManiSkill are shown Tab. Note that the term "arm length" represents ratio of the arm length to its original length.
Table 9: Details of task specifications in ManiSkill Environment.
Specification
Range of Parameters
No. of Samples
Default Value
Cube Size
[0.01, 0.03], increment 0.001
21
0.02
Controller Stiffiess
[500, 1500]. increment 50
21
1000
Controller Damping
[50, 150], increment 10
21
100
Arm Length
[0.5, 2.0], increment 0.1
16
1.0
B.2 ADDITIONAL IMPLEMENTATION DETAILS
We implemented our method in Py Torch Paszee Paszke et al. 2019) and the hyperparameters are reported in
Tab. TO Tab. 11
and Tab. T2
16

---

Published as a conference paper at ICLR 2025
Table 10: Hyperparameters of TD3.
Hyperparameter
Setting
Learning rate
1 10
Optimizer
Adam
Mini-batch size
256
Actor update frequency
Target networks update frequency
Target networks soft-update
0.01
Target policy smoothing stddev. clip
0.3
Hidden dim.
256
Replay buffer capacity
10
Discount
0.99
Seed frames
4000
Exploration steps
2000
Exploration stddev. schedule
linear(1.0, 0.1, 1c6)
Table 11: Hyperparameters of PPO.
Hyperparameter
Setting
Learning rate
3 x 10
Optimizer
Adam
Mini-batch size
400
Gamma
0.85
Target KI.
0.09
Value function coefficient
0.5
Clip Range
0.2
Lambda Gae
0.95
Max Grad Norm
0.5
Table 12: Hyperparameters of HyPoGen.
Hyperparameter
Setting
Learning rate
10
Optimizer
Adam
Mini-batch size
512
Hidden dim. of target policy networks
256
Task embedding dim.
256
Weight embedding dim.
256
No. Layers K
No. MLP layers Hypernet Blocks
MLP hidden dim. in Hypernet Blocks
128
C MORE COMPARISON AND ANALYSIS
C.1 DETAILED COMPARISON ON RACH SPECIFICATION
We showcase the rewards and their standard deviations of the rollout policies in MuloCo in Fig
Fig. 6 and Fig. 7
We also show the success curves of different specifications in ManiSkill in Fig.
17

---

Published as a conference paper at ICLR 2025
NyfoGwnyk.n tyeeram
UM
UM
Het0Gmn0knn Crrd
Card
Steed
Tageet Steed
Taggst Steed
(a) Cheetah
(b) Finger
(c) Walker
Figure 5: Average reward and std curves generalization different desired speed specifications. The solid lines present the mean, and the shaded region presents the standard deviation. Our method achieves the best results in most cases.
VMA
VMA
hear
follcy
Oue Pelry
Card
Trroo Length
Longth
Longth
(a) Cheetah
(b) Finger
(c) Walker
Figure 6: Average reward and std curves generalization to different torso length specifications. Our method achieves the best results most cases.
USA
IESOA OUTE Pare
USA
PART
USA
Bu Langle
Doger Langs
B Langie
"Jmn Spord
(a) Cheetah
(b) Finger
(c) Walker
Figure 7: 3D reward surfaces generalization different speed and length specifications. Our method achieves the best results in most cases.
(a) Cube size
(b) Stiffness
(c) Damping
(d) Length
Figure 8: The success rate curve of different specifications LiftCube task of Maniskill Environment.
18

---

Published as a conference paper at ICLR 2025
(a) Cube size
(b) Stiffness
(c) Damping
(d) Length
Figure 9: The success rate curve of different specifications on PickCube task of Maniskill Environ- ment.
C.2 STANDARD DEVIATION OF MAIN EXPERIMENTS
Below, we report the standard deviation of the rewards MuJoCo environments in Tab. 13 and the standard deviation of episode length in ManiSkill in Tab. T4
Table 13: Standard deviations on MuloCo. The mean values are reported in Tab. I in the paper.
Cheetah lengths
Finger
Walker
Method
speed
spead&lleggth
spood
lengths
speed&length
speed
lengths
speed&llongth
Cond Policy
92.76
146.06
88.43
116.61
104.42
67.42
101.47
181
136.32
UVFA
86.59
165.48 48
84.68
100.03
111.45
63.04
79.65
125.10
Meta Policy
51.64
128.34
29.29
23.82
50.87
16.58
21.26
19.85
17.03
19.39
33.16
HyperZero
84.08
103.30
137.36
107.90
115.00
80.67
140.51
195.85
191.87
HyPoGen
81.62
100.69
80.12
Expert Rollout
2.27
3.16
9,16
4.51
10.12
13.56
9.63
17.21
18.64
Table 14: Standard deviations of episode length on Maniskill. The mean values reported in Tab. the paper.
Lift Cabe
Pick& Cube
Method
cube size
stiffness
damping
length
cube size
stiffness
damping
Policy
UVFA Policy
49.57
0.00
52.15
68.64
74,70
0.00
64.10
65.76
69.12
40.19
PEARL
65.42
38.89
18.40
70.52
64.23
71.22
30.36
$2.70
HyperZero
58.62
0.00
47.44
68.76
51.38
0.00
0.00
35.00
PoCnnnn
76.25
Expert Rollout
35.01
37.33
47.28
25.29
60.89
66.02
61.77
66.11
C.3 ANALYSIS OF POOR PERFORMANCE OF BASELINES IN MANISKILL
To further analyze whether the poor performance baselines is due to the lack of generalization ability simply the model's capacity. First, we report the model size of each method in Tab. 15 Then, we report the original baselines' training performance in Tab, [6] and the training/test performance From these tables, along with
of the baselines with equal capacity to our models in Tab. Tab we can see that even without capacity increase, some methods can already fit well on the training data. For example, HyperZero on the Pick&Place cube size task achieves 78.20% success rate compared to 25.87% at test time, and its performance drops from 92% to 31 % on the LiftCube damping task. Additionally, the performance of Cond Policy drops from 63% to 39%.
In theory, increasing the model capacity should make it easier to overfit. However, due to learning difficulty, the training performance actually decreased. For example, from Tab. [6 and Tab. [7] HyperZero dropped from 86% to 69% the LiftCube arm-length task. When learning is not an issue, we observe that training performance improves, but generalization does not improve or even worsens. This is evidenced by Cond Policy's 's performance where training performance increased from 47% to 61 %, but test performance dropped from 43% to 26%.
the Pick&Place Cube arm-length task,
19

---

Published as a conference paper at ICLR 2025
These results and observations confirm that the proposed architecture design ensures the robustness of the learning process and eliminates the difficulty of training when capacity is increased. Moreover, it ensures that increased capacity does not incur overfitting and generalization issues, while this property is not observed with other models.
Table 15: Comparison of parameter sizes for different methods (Original vs Augmented), in Millions.
Cond Policy
UVEA
Meta Policy
PEARL
HyperZero
HyPoGen'8 blocks)
Original Augmented
0.7
0.7
0.7
0.7
25.1
70.0
64.1
63.8
64.1
64.1
72.0
70.0
Table 16: Training- teiming--imme success rate of baselines with original model size in Maniskill environments.
Lift Cabe
Plak& Cube
Method
cube size
stitlness
damping
length
cube size
stiflness
damping
length
Cond Policy
94.67
0.00
98.33
86.60
2.56
63.00
46.67
UVFA
94,17
0.00
97.33
89.00
87.40
0.00
68.50
49.00
Meta Policy
83.83
0.00
0.00
77.20
17.00
0.56
0.00
11.33
PEARL
64.33
27.60
22.67
HyperZero
92.67
0.00
92.00
86.00
78.20
0.00
0.00
3,00
HyPonGen
Expert Rollout
96.00
95.33
90.75
98.50
87.00
82.33
58.25
84.00
Table 17: Training/test-time success rate of baselines with equal model size to ours in Maniskill environments.
Lift Cube
Plak&Place Cuhe
Method
cube size
stiffiess
damping
arm length
cube size
stiffiness
damping
arm length
Cond Policy
95.56/ 28.67
0.00/0.00
63.33/ 14.63
89.17/5633
82.22/ 26.67
4.44/1.11
32.22
60.83 26.67
UVFA
95.56/ 28.89
0.00/0.00
64.44/ 16.48
83.33/58.18
81.11 /2.996
4.44/
15.56/7.02
38.33 18.21
Meta Policy
0.00/0.22
0.00/0.00
0.00/0.00
1.11/0.42
0.00/0.18
PEARL
0.00/0.00
0.00/0.22
0,00/0,74
5.00/1.21
0.00/0.62
2.22/0.00
0.00/
0.83/0.00
HyperZero
94.44/28.00
0.00/0.00
63.33/22.22 22.22
69.17/32.73
71.78/25.05 /5.62
0.00/0.00
0.00/ 0.00
21.67 8.46
HyPoGen
89.33/885.88
96.50/97.20
/63.2/
35,60/85,75
22.40/72.7
81.33/78.33
68.67 52,54
Expert Rollout
96.00/ 94.13
95.33 /9.77
90.75 99.22
98.50/93.64
87.00/ 69.87
82.33/75.92
58.25 73.72
84.00/54.77 54.77
C.4 ADDITIONAL ANALYSIS
Scalibility: HyPoGen generate deeper target networks? We conducted experiments deeper policy networks, and the results show that our method consistently outperforms the SOTA method HyperZero. as reported in Tab. 18
In the comparisons section, we use two-layer and three-layer MIPs for MuJoCo and ManiSkill, re- spectively. Here, we gradually increase the num-
Table 18:
Comparison of HyPoGen and Hyper-
Zero with different layers in target networks.
ber of layers in the policy networks. As the num- ber of layers (i.e., the number of parameters to be generated by the hypernet) increases, the learn- ing difficulty of the hypernet also increases. As observed from the table, our method consistently surpasses HyperZero terms of obtained rewards at test time.
#Layers
7
HyperZem
695.7
704.5
434.4
528.6
509.3
HyPoGen
819.2
821.9
819.
665.6
622.5
Generalization ability w.r.t the amount of training data. We compare our method with HyperZero to emphasize our generalization ability. In the experiment, we split 50% of the task specifications for testing and vary the fractions of the remaining specifications for training. The fractions range from 5% to 50%. The results are reported in Tab. 19 Our method achieves constantly better results than HyperZero. Note that as the fraction decreases, our method exhibits more significant improvements. It further demonstrates the strong generaliza- tion ability of our model.
Table 19: Zero-shot transfer with dif- ferent amounts of training data.
HyperZero
HyPoGon (ours)
385.90
644.85
10%
557.98
756.13 13
20%
668.13
754.88
30%
813.14
841.12
50%
854.27
884.44
20

---

Published as a conference paper at ICLR 2025
Analysis hyperparameters. We show results the influence of hyperparameters in model. We first validate the proper number of hypernet blocks (i.c., the optimization steps) in Tab, 20 We can observe that the performance reaches the peak when K 8. In Tab. 21 we present the results of different MLP layers inside each hypernet block. The performance achieves the highest when the number of layers is set to 2. We also examine the feature dimensions of hidden layers in the hypernet block. The results are reported in Tab. 22 When setting the hidden dimension as 128, the network produces the best result.
Table 20: Performance with different optimization steps (No. of hypernet blocks).
K
Reward and Std.
1
746.06 83.54
775.55 68.26
5
825.76 76.61
8
856.88 61.73
10
832.76 78.50
20
817.9 60.92
Table 21: Performance with different number of layers in each hypernet block.
No. MLP Layers
Reward and Std.
2
856.88 a 61.73
3
795 70.39
5
834.99 = 78.62
8
752.74 + 119.31
10
772.82 + 117.64
Table 22: Performance with different hidden dimensions in hypernet block.
Hidden Dimension
Reward and Std.
16
847.83 = 59.95
32
817.12 = 67.35
64
840.02 61.24
128
856.88 61.73
256
834.5 80.36
C.5 STANDARD DEVIATION OF THE SUCCESS RATE ON MANISKILL
The standard deviations success rate in Manskill environment are shown Tab. 23 The proposed method has the same deviation level as other baseline methods but is higher in the mean value.
Table 23: Average and standard deviations of success rate on Maniskill,
Lift damping
Plac& Cube
Method
cube
stiffness
arm length
cube size
stiffness
damping
amm longth
Cond Policy
89.93 + 8.00
4.00
56.28 + 10.00
69.36
14.00
69.81
18.00
2.20 2.00
39.26 17.00
43.77 17.00
UVFA
90.27
0.00
78.67
10.00
72.18
15.00
71.75 17.00
0.00
29:53 14.00
36.85 14.006
Meta Policy
84.67 = 12.00
0.00 0.00
0.06
0.00
76.45 15.00
12.62
10.00
0.22 2
0.21 0.00
8.15 7.00
HyperZero
86.60 + 11.00
0.00 + 0.00
31.33 10.00
63.64 + 14.00
25.87 10.00
0.00 + 0.00
0.00
13.92 + 6600
HyPoGen
85.87 2 11.00
97.20 4 3.00
93.28 a 6.00
85.73 a 10.00
72.87
17.00
78.33 15.00
41.26 18.00
52.54 = 20.00
21

---

Published as a conference paper at ICLR 2025
C.6 STANDARD DEVIATIONS SUCCESS EPISODES
To align with established work Rezaei-Shoshtarn et al. (2023), the result is shown in Tab. count in the failure episodes, which increases the variance reported in Tab T3 14
In Tab. 24 25
we report the average of only success episodes As shown in the table, the standard
deviations are of reasonable scale relative the averages.
Table 24: Average and standard deviations of rewards on MuloCo on success episodes.
Cheetalt
Finger
Walker
Melhod
speed
lengths
speed&length
speed
lengths
speesl&lenghh
speed
lengths
speed&klength
Cond Policy
510.50
609.70
370.11
378.38
441.97
301.73
604.32
315.07
384.91
83.8
130.1
78.2
118.2
91.2
61.6
50,9
108.0
80,00
UVFA
452.58
657.83
358 22
385.83
462.36
292.53
504.84
244.98 98
170.07
89.7
161.0
100.7
93.5
54.5
52.7
97.3
73.3
Meta Policy
355.62
697.48
232.19
122.97
174.37
123.83
84.56
92.13
81,74
+93.1
+21.9
+20.6
13.3
+.2
9,6
PEARL
296.29
816.72
217.93
121.06
132.60
135.19
79.39
79.08
74.83
22.6
1 69.1
25.1
1. 11.2
2: 30.6
1: 26.9
1: 6.8
1 10.8
2
HyperZero
772.16
917.14
626.06
618.75
565.97
384.97
764.24
508.74
537.99
72.9
+ 84.2
+ 129.2
109.9
+ 105.7
# 76.9
6..7.
# 80.5
+ 112.6
HyPoGen
845.81
949.86
653.53
854,99
697.78
396.76
839.94
636.54
601.73
67.7
* 49.7
91.4
111.8
* 103.2
* 79.2
92.9
* 66.1
92.0
Table 25: Average and standard deviations of episode length on Maniskill on success episodes.
Lift Cube
Pick&PPaace Cube
Method
cabe size
stiffness
damping
arm&leegtt
cuhe size
stiffness
damping
arm&lengih
Cond Policy
13.46
7.09
0,00 a 0.00
0.00 1 0.00
0.00 0.00
21.03 10.13
0.00 0.00
32.57 a 14.27
26.22 18.96
UVFA
14.56 9.89
0.00 0.00
14
15.99 6.73
21.80 + 9.80
0.00 + 0.00
25.33 12.54
23.28 11.84
Meta Policy
12.45
0.00
PEARL
12.60 = 5,91
0.00 a 0.00
126.11 0.79
13.28 4:222
46.72 13.28
33.36 8.42
134,20 + 4.71
65.95 # 11.32
HyperZero
13.31
0.00
0.00
HyPoGen
7.66
11.41 1.56
12.18 2.14
12.68 + 3,67
20.43 + 9.94
19.55 9.35
23.98 + 11.78
23.46 14.02
C.7 IN AND OUT-OF--RRAINNGG-DISTRIIIS
PERFORMANCE
To better evaluate the generalization ability of the proposed methods, we evaluate all methods within one standard deviation of training distribution fin-distribution) and outside that -uttof-ddistibbution) Our proposed method exhibits superior performance over baseline methods in both ii-distriuttion and out-distribution settings. This further highlights the strong generalization ability of the proposed method.
With the result shown in Tab. 26 27 28 29
Table 26: The in-distribution average and standard deviations of rewards on MuloCo.
Cheetah
Finger
Walker
Method
lengths
loeod&longth
speed
lengths
spood&Eleggh
speed
lengths
speod&llnggh
Cond Policy
521.81
627.72
330.69
399.36
433.41
318.38
324.24
654.04
400.58
2 85.4
# 131.5
70.6
+ 129.4
# 102.2
2 59.1
# 57.6
+ 110.1
2843
466.93
85.3
157.6
+ 788.2
* 109.8
+ 96.6
+ 59.1
48.3
104.5
+ 72.7
Meta Policy
308.15 15
702.66
188.45
126.04
165.49
122.81
88.71
86.05
82.95
39.3
+ $9.4
25.88
24.4
34.2
19.7
10.4
6.4
11.0
PEARL
251.10
819.34
210.16
118.19
137.03
130.73
78.37
79.31
77.15
18.0
70.22
+1.77
+.35.7
222.8
6.25
10.8
8.6
HyperZero 796.92
941.41
640.15
691.46
585.75
446.38
$55.23
792.82
596.58
+ 72.1
$2222
108.7
HyPuGen 837.94
956.45
653.97
880.00
706.10
459.51
621.51
306.10
645.18
63.5
2 44.8
90.8
+ 103.1
2 106.3
+ 93.5
95.5
106.3
a 97.5
"MuloCo does not provide success indicator, thus when the reward of each frame is less than 0.05 for 50 consecutive frames, it considered a failure.
22

---

Published as a conference paper at ICLR 2025
Table 27: The out-of-distribution average and standard deviations of rewards on MuJoCo.
Cheetah
Finger
Walker
Method
speed
lengths
speed&&length
speed
lengths
speed&&length
speed
lengths
spead&length
Cond Policy
488.88 88
574.32
427.34
343,66
460.60
236.09
294,89
486,90
321.78
80.8
127.2
89.3
99.8
* 67.3
44.6
36.2
103.
# 62.6
UVEA
440.42
676.61
394.64
308.00
453.66
236.09
247.72
477.46
340.54
4 97.8
= 167.2
2.87
8377
# 87.6
44.66
2 =6922
+ 83.5
77.66
Meta Policy
429.27
687.59
297.92
117.87
190.76
78.68
52.0
100.2
38.3
17.8
23.7
# 22.3
-18.3
5.8
2.9
PEARL
374.23
811.10
229.90
125.88
124.18
144.47
80.16
79.56
70.50
30.5
66.7
40.33
11.9
#111
29.1
#75.5
10.8
7.00
HyperZero
721.07
866.78
165.16
486.52
530.86
253.91
380.68
702.61
400.35
+83.,6
109.4
11799
8.8
88.3
1.4.1
1.47.2
82.1
121.8
HyPoGen
857.70
936.57
652.87
812.36
682,02
260.78
663.83
682,02
487,32
# 59.5
+ 92.3
* 126.6
* 97.4
48.3
88.7
97.4
77.44
Table 28: The in-distribution success rate and espisode length on Maniskill.
Lift Cube
Plak& Cube
Method
cube size
stiffness
damping
lrn&length
cube
stiffness
domping
arm&klength
Policy
93.42, 12.77
0.00, 0.00
95.00,
66.30,
72.20,
2.19.
61.00,
UVFA
92.75. 13.83
0.00, 0.00
92.80, 10.39
69.90, 16.11
80.20, 21.87
0.00, 0.00
65.00. 16.50
52.00, 20.21
Meta Policy
85.58, 11.65
0.00, 0.00
0.00, 0.00
78.20, 13.13
T5.10, 22.86
0.28, 8.00
0.25, 5.75
11.50,
85.00,
59.56,
93.80,
80.00,
25.70,
23.31
25.15, 21.50
32.00,
25.60,
HyperZero
88.58, 12.83
0.00, 0.00
79.60, 10.90
61.10, 14.55
40.20,
32.40
0.00, 0.00
0.00, 0.00
21.33. 14.13
HyPoGen
89.25, 12.64
97.00, 11.46
95.60, 10.53
84.30, 12.57
81.90, 19.75
79.75, 19.81
69.00, 17.67
65.17, 21.15
Table 29; The out-of-distribution success rate and episode length Maniskill.
Cabe
Pick&Place
Method
cube size
stilTness
damping
armnklength
cube sizze
stillines
dampinn
arm&length
Cond Policy
76.00, 16.23
0.00, 0.00,0.00
0.00
41.38, 14.92
100.00. 14.06
65.83, 20.68
2.22, 24.33
33.53, 36.51
29.00, 28.28
80.33, 17.49
0.00, 0.00
73.23, 15.52
00, 14.74
57.67. 21.67
20.07, 27.68
23.86. 25.92
Meta Policy
81.00, 15.63
0.00, 0.00
0.08. 1.85
59.00, 12.92
8.50. 16.80
0.00, 0.00
0.20, 4.40
5.29, 19.12
PEARL
82.33, 14.95
61.11. 6.73
86.50, 9.54
66.00, 12.35
20.00, 28.62
30.00, 21.28
35.75, 21.49
16,20, 28.61
HyperZero
78.67. 15.25
0.00, 0.00
12.77. 10.79
89.00
13.82
2.00, 9:40
0.00, 0.00
0.00, 0.00
7.57, 15.36
HyPoGen
72.33, 14.71
98.00, 11.21
92.38, 12.81
100.00, 13.75
57.83, 21.57
72.67, 18.52
33.87, 25.66
41.71, 25.45
D VISUALIZATION
Due to the page limit, we only present quantitative evaluations in the experiment section. Here, we visualize the results the simulation environments and qualitatively compare our method with the baselines. The visual results of MuJoCo tasks are shown in Fig. 10 II
14, and [5]
As seen in Fig. 10 our method can obtain more stable results with different speeds. When the target speed is -5.0 or -8.0 (the first and second rows), our method has faster actual speed (better performance). Meanwhile, HyperZero is unable to maintain balance when the speed is 5.0 (the third row). In Fig. [1 HyperZero struggles maintain balance, while method generates reasonable results. In the Finger Spin environment in Fig 12 HyperZero stagnates for a long time when the speed equals -5.0 and 5.0. On the contrary, our method works well for all settings, demonstrating its generalization ability and stability. The same as the results of different speeds, in different torso lengths, the results (in Fig. [3 14 and [15) demonstrate generalization and stability of our method. Our method encounters fewer failures. and also guides the agents to act at faster speed. The visual results on ManiSkill tasks are shown in Fig 16 page see the visual results.
20
and Please turn the next
23

---

Published as a conference paper at ICLR 2025
1 50
t= 100
t= 150
t= 200
t= 250
t= 300
t= 350
t= 400
Spd -3.5
K<
HyPoGen
Spd 3.5
7
(
Spd -3.5
<
R
T
HyperZero
Spd 3.5
Spd -3.5
7
1
1
7
M
Cond Policy
1
Spd 3.5
K
Spd -3.5
Meta Policy
Spd 3.5
N
X
Spd -3.5
T
7
UVFA
M
1
5pd 3.5
Spd -3.5
Pearl
Spd 3.5
Figure 10: Comparison Cheetah environment with different speeds.
24

---

Published as a conference paper at ICLR 2025
1 50
t= 100
t= 150
t= 200
t= 250
t= 300
t= 350
t= 400
1
Spd -0.5
HyPoGen
Spd 3.0
Spd -0.5
HyperZero
Spd 3.0
4
Spd -0.5
Cond Policy
5pd 3.0
Spd -0.5
Meta Policy
Spd 3.0
Spd -0.5
UVFA
5pd 3.0
Spd -0.5
Pearl
Spd 3.0
Figure 11: Comparison in Walker environment with different speeds.
25

---

Published as a conference paper at ICLR 2025
1 50
t= 60
t= 70
t= 80
t= 90
t= 100
t= 110
t= 120
Spd -5.0
HyPoGen
11
Spd
Spd -5.0
HyperZero
7
Spd
CI
Spd -5.0
Cond Policy
Spd 5.0
Spd -5.0
-
Meta Policy
Spd 5.0
Spd -5.0
UVFA
5pd 5.0
7 - -
7
Spd -5.0
Pearl
Spd 5.0
Figure 12: Comparison Finger environment with different speeds.
26

---

Published as a conference paper at ICLR 2025
1 50
== 100
t= 150
t= 200
= 250
t= 300
t= 350
t= 400
L.e 0.3
HyPoGen
Len 0.4
E
R
R
Len 0.3
L
F
Y
HyperZero
Len 0.4
Len 0.3
A
Cond Policy
x
Len 0.4
Y
Lee 0.3
Meta Policy
Len 0.4
Leen 0.3
2~
2~
ZA
UVFA
Len 0.4
NN
2~
1
Lon 0.3
2~
2~
Pearl
Len 0.4
Figure 13: Comparison Cheetah environment with different torso lengths.
27

---

Published as a conference paper at ICLR 2025
1 50
== 100
t= 150
t= 200
t 250
t= 300
t= 350
t= 400
I.en
HyPoGen
17
Len 0.4
Lee 0.2
0.2
HyperZero
Len 0.4
Tenn 0.2
Cond Policy
Len 0.4
Leen 0.2
Meta Policy
Len 0.4
I.en 0.2
UVFA
Len 0.4
Loo 0.2
Pearl
Len 0.4
Figure 14: Comparison in Walker environment with different torso lengths.
28

---

Published as a conference paper at ICLR 2025
1 50
t= 60
t= 70
t= 80
t= 90
t= 100
t= 110
t= 120
L.en
HyPoGen
Len 0.2
7
T.en 0.1
HyperZero
J
7
1
Len 0.2
Len 0.1
Cond Policy
Len 0.2
J
L.en 0.1
Meta Policy
1
Len 0.2
Ieen 0.1
UVFA
Len 0.2
Len 0.1
L
L
L
L
Pearl
1
I
Len 0.2
L
L
Figure 15: Comparison Finger environment with different torso lengths.
29

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t 60
Size 0.010
HyPoGen
Size 0.025
Size 0.010
HyperZero
Size 0.025
Size 0.010
Cond Policy
Size 0.025
Size 0.010
Meta Policy
Size 0.025
Size 0.010
UVFA
Size 0.025
Size 0,010
Pcarl
Size 0.025
Figure 16: Comparison in LiftCube with different cube sizes.
30

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Stif 750
HyPoGen
Stiff 1250
Stiff 7.5
HyperZero
Stiff 1250
Stiff 750
Cond Policy
Stiff 250
Stiff 750
Meta Policy
Stiff 2250
Stiff 750
UVFA
Stiff 2250
750
Stiff 750
Pcarl
Stiff 250
Figure 17: Comparison LiftCube with different controller stiffness.
31

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Damp
HyPoGen
Damp 95
Damp
HyperZero
Damp 95
Damp
Cond Policy
Damp 95
Damp
Meta Policy
Damp 95
Damp
UVFA
Damp 95
Damp
Pcarl
Damp 95
Figure 18: Comparison LiftCube with different controller damping.
32

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Len 1.1
HyPoGGn
Len 1.6
Len 1.1
ZyperZero
Len 1,6
Cond Policy
Len
Len 1.6
Len 1.1
Meta Policy
Len 1.6
Len
UVFA
Len 1.6
Len 1.1
Pcarl
Len 1.6
Figure 19: Comparison in LiftCube with different arm lengths.
33

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Size 0.010
HyPoGen
Size 0.025
Size 0.010
HyperZero
Size 0.025
Size 0.010
Cond Policy
Size 0.025
Size 0.010
Meta Policy
Size 0.025
Size 0.010
UVFA
Size 0.025
Size 0.010
Pcarl
Size 0.025
Figure 20: Comparison PickCube with different cube sizes.
34

---

Published as a conference paper at ICLR 2025
t=1 1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Stiff 750
HyPoGen
Stiff 1250
Stiff 750
HyperZero
Stiff 1250
Stiff 750
Cond Policy
Stiff 1250
Stiff 750
Meta Policy
Stiff 2250
Stiff 750
UVFA
Stiff 1250
750
Stiff 750
Pcarl
Stiff 250
Figure 21: Comparison in PickCube with different controller stiffness.
35

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Damp
HyPoGen
Damp 95
Damp
HyperZero
Damp 95
Damp
Cond Policy
Damp 95
Damp
Meta Policy
Damp 95
Damp
UVFA
Damp 95
Damp
Pcarl
Damp 95
Figure 22: Comparison in PickCube with different controller damping.
36

---

Published as a conference paper at ICLR 2025
t=1
t= 5
t= 10
t= 20
t= 30
t= 40
t= 50
t= 60
Len 1.1
HyPoGen
Len 1.6
Len 1.1
HyperZero
Len 1,6
Len 1.1
Cond Policy
Len 1.6
Len 1.1
Meta Policy
Len 1.6
Len
UVFA
Len 1.6
Len 1.1
Pcarl
Len 1.6
Figure 23: Comparison PickCube with different arm lengths.
37


Reviews:
[{"review_id": "1cQAnNA9HA", "reviewer": "Reviewer_49Ju", "summary": "The paper introduces HyPoGen, a hypernetwork that generates policies for unseen tasks without demonstrations by learning policy parameters directly from task specifications. By structuring the network to mimic optimization, HyPoGen generalizes effectively despite limited training data. Experiments show it outperforms state-of-the-art methods on locomotion and manipulation tasks, achieving higher success rates in unseen scenarios. The authors will release the code and models.", "questions": "- Could the proposed approach work in discrete action space?\n    - Is there any potential difficulties or benefits of applying this approach for discrete actions?\n- In the experiments section, it seems that the generated policy was adapted in the scenario in which task is slightly modified, such as the desired speed of the agent changed. Can authors provide more experiments that policy is generated for more distinct tasks, such as meta-world?\n- Though the authors provide average success rate, it would be also important to show the variance of the success rate.\n\n[1] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning, Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, Sergey Levine", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper was well-written.\n- The experimental results show that performance improvement is significant.", "weaknesses": "- In general, I am curious about the limitation of the proposed method. Please refer to the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OiHlUF54Iv", "reviewer": "Reviewer_JcCr", "summary": "This paper proposes a novel hypernetwork architecture for policy generation in behavior cloning, with a focus on generalization to unseen tasks without requiring extra expert demonstrations. Unlike existing methods that apply hypernetworks to generate policy network parameters directly from a task specification, the proposed hypernetwork architecture, HyPoGen, iteratively generates a policy network by simulating gradient-based optimization, where the pseudo \"gradients\" are conditioned on the task specification. Additionally, the generation of these pseudo \"gradients\" across different layers of the policy network is structured to follow the chain rule. The primary idea is that the inductive biases introduced by simulating gradient-based optimization can help improve policy generation quality and improve generalization to unseen tasks. Experimental results show that HyPoGen outperforms baseline methods in generating policies for unseen tasks without any demonstration and that the policy performance improves during the simulated optimization guided by the trained hypernetwork.", "questions": "1. Is my understanding of the training procedure of the proposed hypernetwork correct, i.e. are all trainable parameters of the hypernetwork, including $\\theta_0$, trained simultaneously in an end-to-end fashion?\n\n2. Are any parts of the hypernetwork parameters retrained for each initial value of $\\theta_0$ when performing the analysis of Table 3? See Weakness 2 above for further context of this question.\n\n3. Appendix B.1 provides the range, granularity, and number of samples of tasks specifications in different environments used in the experiments. Is task specification sampling done uniformly? Is the source-test task partition also uniform? If so, for the cases where only one task parameter varies, could there be very similar specifications among the source tasks for most test tasks? For example, Table 6 shows that the Cheetah environment has about 40 possible specifications, and there are 40 samples, so nearly every specification appears in either source or target tasks (assuming sampling without replacement).\n\n4. It is mentioned that the optimization is performed in a compressed latent space. What is the dimension of this latent space?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses the challenge of generalizing policy generation to unseen tasks without any demonstration, a realistic and difficult problem in imitation learning.\n\n2. The main contribution of the paper is the novel hypernetwork architecture, HyPoGen, which introduces inductive biases to generate a policy network by simulating gradient-based optimization. The architecture performs iterative pseudo \"gradient\"-descent where the pesudo \"gradients\" are enforced to follow chain rule across different layers of the policy netowrk, an interesting and sensible approach.\n\n3. It is demonstrated through extensive experiments on locomotion and manipulation benchmarks that HyPoGen outperforms baseline methods in generalizing to unseen tasks without any demonstration. Additionally, several empirical cases studies (notably in Table 4) provide evidence that the proposed hypernetwork architecture effectively simulates an iterative optimization process. \n\n4. The paper is well organized and clearly written, with a thorough discussion of related work to place it within existing research. The rationale behind the HyPoGen architecture design is clearly explained, and the construction of the hypernetwork's inner components is presented effectively.", "weaknesses": "1. The training process of the proposed hypernetwork is described somewhat briefly in Section 4.2. Although the set of learnable parameters and the loss function (Eq. 1) are discussed, likely implying end-to-end training over source tasks to minimize the BC loss (Eq. 1) while optimizing all learnable parameters simultaneously, this part could be elaborated more explicitly. Including pseudocode in the appendix might be helpful if the training procedure is not this straightforward. \n\n2. In Section 5.4, there is an analysis of the statistics of the generated policy network parameters with different initial values of $\\theta_0$ (Table 3). Are other hypernetwork parameters, apart from $\\theta_0$, kept fixed across different initial values of $\\theta_0$, or are they retrained separately for each initial value of $\\theta_0$? In either case, the presented statistics in Table 3 might not be able to sufficiently answer the question whether HyPoGen remembers a fixed set of parameters for each specification. Specifically, if the other hypernetwork parameters are not retrained, observing the statistics of generated policy network parameters may be less meaningful as the hypernetwork is not adapting to the $\\theta_0$ value in use; if the hypernetwork is retrained for different initial values of $\\theta_0$, it could still be the case that HyPoGen remembers a fixed set of parameters for each specification for a given $\\theta_0$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jy1ghWsSWJ", "reviewer": "Reviewer_RPpm", "summary": "In low-data settings, using a hypernetwork can be promising. However, classical hypernetworks have critical issues, such as overfitting, which can limit their generalization capability. To address this, the authors modify the role of the hypernetwork. Instead of directly predicting the policy weights, they iteratively compute the weights in a manner similar to multi-step gradient descent. This reformulates the optimization problem for the hypernetwork to predict gradients instead of weights directly, which the authors empirically show leads to better performance and generalization on locomotion and manipulation tasks.", "questions": "I think mentioning learned optimizers : https://arxiv.org/pdf/2209.11208 can be relevant to the overall framing of this work. \n\nI also am wondering how well would directly training the hyper network on the true gradients be, such that instead of it resembling back propagation, it is exactly back propagation (assuming no training error) .", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work strengthens hypernetworks by altering the training optimization be guided to output intermediate gradient steps, a novel approach for hypernetworks, especially in policy settings.\n\nThe writing and figures are clear, and the experimentation is thorough, with a wide range of baselines that use the same policy network architecture and specification encoder. Performance consistently surpasses that of the baselines.\n\nI found the section \"Does HyPoGen actually perform optimization?\" particularly interesting and insightful that showcases how it is actually like an optimizer.", "weaknesses": "Overall, I think this work is strong, and I would be interested in seeing additional analysis on how closely the predicted intermediate gradients align with true gradients that would result from directly optimizing the policy. While the authors have shown that the iterative optimization process effectively reduces the loss over time, it would be valuable to quantify the similarity between these predicted gradients and the actual gradients obtained through standard gradient descent. Such an analysis could fortify the claim that the hypernetwork is indeed guided to predict meaningful gradients, rather than simply memorizing an update path. This comparison would strengthen the case for the hypernetwork's ability to emulate true gradient-based optimization, supporting its effectiveness in generalizing to unseen tasks. While this is not a critical weakness, it would serve as a valuable addition that could make the work even more compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jG4hm65pPK", "reviewer": "Reviewer_2Bw1", "summary": "The authors propose to incorporate inductive biases about optimization into a hyper-network framework that generates policies for different tasks. They do so by parameterizing the hypernetwork updates based on the chain rule and also on a delta to the current weights. They show that these inductive biases help HyPoGen outperform previous methods when generalizing across a set of related tasks.", "questions": "- What happens if HyPoGen is applied for many more steps (Fig. D)? Does the loss keep going down, or does it “overfit” eventually?\n- What is the strategy for choosing the number of hyper-optimization steps? Do harder problems (more out-of-distribution for example), require more optimization steps?\n- Does HyPoGen work for larger policy models / harder environments (the policy networks at the moment seem to be only 2 layer MLPs)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The method is presented and motivated well.\n- The experiments provide compelling evidence that HyPoGen outperforms previous methods (at least based on the average).\n- The idea of neural gradients for policy generation is interesting. The methods presented in this paper for incorporating optimization inductive biases into hypernetworks could lead to future interesting work.", "weaknesses": "- The variances for Table 1 / 2 (reported in Table 13/14) are quite high (on the order of 100-200). This makes it harder to argue that HypoGen is better than the previous methods when the gap is often on the order of a few tens of units. It would be fairer to report exact confidence intervals in the main text to make the comparison more clear.\n- The computational costs of HyPoGen aren’t directly addressed. Since it requires multiple steps to keep decreasing the loss, how does the computational cost compare to other methods (especially something like HyperZero)? This makes me think that other baselines could be relevant to assess the tradeoff between computational cost and performance. For example, instead of parametrizing the full chain rule (equation 6/7), can the majority of the benefit be obtained by framing the problem just as a delta from the previous step (equation 5). \n- While the paper claims that HypoGen leads to better generalization, Fig. 4,5,6,7,8,9 show generalization somewhat qualitatively and it is not super clear (other than Fig. 4) that HyPoGen generalizes better than the other methods. Are their more precise metrics that could show this generalization more clearly (i.e. report performance within one std of training distribution and outside of that for the relevant methods).\n\nThe authors present an interesting method, but I think clearer evaluations and details on the computational requirements would make the paper stronger and help isolate the benefit of the optimization inductive bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)