$\textit{Hyper-GoalNet}$: Goal-Conditioned Manipulation Policy Learning with HyperNetworks
Authors: Pei Zhou
Wanting Yao
Qian Luo
Xunzhe Zhou
Yanchao Yang
Venue: NeurIPS 2025 poster ()
Keywords: Visuomotor Hypernetwork
Goal-Conditioned Policy
Primary area: applications

Abstract: Goal-conditioned policy learning for robotic manipulation presents significant challenges in maintaining performance across diverse objectives and environments. We introduce *Hyper-GoalNet*, a framework that generates task-specific policy network parameters from goal specifications using hypernetworks. Unlike conventional methods that simply condition fixed networks on goal-state pairs, our approach separates goal interpretation from state processing -- the former determines network parameters while the latter applies these parameters to current observations. To enhance representation quality for effective policy generation, we implement two complementary constraints on the latent space: (1) a forward dynamics model that promotes state transition predictability, and (2) a distance-based constraint ensuring monotonic progression toward goal states. We evaluate our method on a comprehensive suite of manipulation tasks with varying environmental randomization. 
Results demonstrate significant performance improvements over state-of-the-art methods, particularly in high-variability conditions. Real-world robotic experiments further validate our method's robustness to sensor noise and physical uncertainties. Our code and trained models will be made publicly available.

Hyper-GoalNerr Goal-Conditioned Manipulation Policy Learning with HyperNetworks
Pei Zhou'
Wanting Yao 1.2
Qian Luo'
Xunzhe Zhou'
Yanchao Yang'
Infooodied Lab, The University Hong Kong University of Pennsylvania {pezhou, , qianluo1 xunzhe,__ 2contenooooooeeeetttttte hku. hk wtyao8seas upenn. edu, yanchaoy@hku. hk
Abstract
Goal-conditioned policy learning for robotic manipulation presents significant chal- lenges in maintaining performance across diverse objectives and environments. We introduce Hyper- GoalNet, a framework that generates task-specific policy network parameters from goal specifications using hypernetworks. Unlike conventional methods that simply condition fixed networks on goal-state pairs, our approach separates goal interpretation from state processing- the former determines network parameters while the latter applies these parameters to current observations. To enhance representation quality for effective policy generation, we implement complementary constraints on the latent space: (1) a forward dynamics model that promotes state transition predictability, and (2) a distance-based constraint ensuring monotonic progression toward goal states. We evaluate our method on a comprehen- sive suite of manipulation tasks with varying environmental randomization. Results demonstrate significant performance improvements over state-of-the-ar methods, particularly in high-variability conditions. Real-wor Real-world robotic experiments further validate our method's 's robustness to sensor noise and physical uncertainties. Code is available at: https: /ttp://gtttt...... thub. com/ com//antiiggao//ypprrr goalnet
Introduction
Goal-conditioned policy learning enables embodied agents to adjust their actions based on current state observations and specified goals 88 27 46). By integrating goal information into decision
making, agents leverage knowledge across various tasks, enhancing adaptability 7 12 35 in
hierarchical reinforcement learning and complex imitation learning 17
Conventional approaches typically concatenate goal observations with current states as input to a
fxee-parameter network 9 [56 (53) This design creates fundamental limitation: the network must process all possible goal-current state combinations using the same fixed weights, conflating "what" process (current state) with "how" to process it (goal-dependent strategy). Consequently, these architectures often struggle with generalization to novel goals and complex manipulation tasks that require different processing strategies depending on the goal specification.
We aim to rethink this relationship by treating goals not as additional input features but as specifica- tions that determine how current observations should be processed. Hypemetworks neural networks that generate weights for another network offer a natural implementation of this perspective. By explicitly modeling goals determinants of policy parameters rather than inputs, hypernetworks effectively disentangle task-dependent processing (defined by goals) from state-dependent processing (applied to current observations) [19 45). This approach better aligns with biological goal-directed
Work done intern HKU.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Nyper-GoalNet
Specified gool
Current ebs.
Specified gool
Current abs.
Obe. Update
HyperNiet
Policy
Enw
Action
Looding
Adaptive
Weights
Action
Target
Non-adaptive Goal-cond, Policy
Figure 1: The proposed Goal-Conditioned Policy Generation framework (Hyper-GoalNel and conventional goal-conditioned policies. Existing methods typically employ a eeeeep policy network that processes concatenated current observations and goal states, treating goals mostly as additional inputs. In contrast, our approach formulates policy learning as an adaptive generation task, where the goal image determines the parameters of the policy network itself transforming goals from inputs into specifications that define how current observations should be processed. This allows for more effective handling of diverse goals and complex manipulation tasks.
behavior, where prefrontal regions interpret task goals and dynamically modulate processing in sensorimotor circuits accordingly B2 [5]]
To address these challenges, we present Hyper-GoalNet, hypernetork based framework for robotic manipulation illustrated in Fig. Our approach employs a hypernetwork to dynamically generate target policy parameters conditioned on specified goals. When loaded into the policy network, these parameters enable the processing of current observations without requiring further access to goal information. This architecture creates a clear separation of concerns: the hypernetwor interprets what the goal means for processing strategy, while the generated policy focuses exclusively on transforming current observations into appropriate actions. key advantage of this goal-aware design is that it enables the system autonomously detect task completion during execution. By training the hypernetwork to model the conditional distribution of effective policy weights given goal specifications, we obtain system that adapt its processing strategy diverse manipulation tasks.
Our technical contributions center on effectively applying hypernetworks for parameter-adaptive goal-conditioned policies learning. First, we adapt optimization-insplee hypernetwork architectures generating policy parameters conditioned on goal specifications, creating a framework that dynamically determines how current observations should be processed. Second, we introduce effective latent space shaping technique that imposes two critical properties: (1) predictability of future states through learned dynamics model, and (2) preservation of physical relationships through distance constraints that ensure monotonic progression toward goals. These properties create an ideal representation space for hypemetwork, providing clear signals about how policy parameters should change as states approach goals.
Our extensive experiments across multiple manipulation tasks show that Hyper-GoalNet signifi- cantly outperforms state-of-the-art methods, achieving higher success rates on complex contact-rich manipulations. Notably, while conventional approaches fail almost completely in high-variabilltyy environments, our method maintains robust performance. Ablation studies confirm the critical importance of our proposed components in the parameter-adaptive goal-conditioned policy lean- ing framework. Finally, real-robot experiments demonstrate that our parameter-adaptive approach succeeds in physical environments where conventional methods struggle with sensor noise and environmental variations. These results confirm that explicitly modeling goals as determinants of processing strategy rather than additional inputs creates more effective and robust framework for goal-conditioned manipulation.
Related Work
Goal-conditioned policy. Goal-conditioned policy learning has attracted significant attention for developing versatile and generalizable agents [44 52 50 57 25). Traditional methods augment

---

state spaces with goal information and train policies that condition on these augmented states [3 exemplifies this approach by allowing agents Hindsight Experience Replay (HER) [T]
44 30 54
to learn from failures by reinterpreting unsuccessful outcomes as alternative goals. However, these methods typically suffer from increased complexity and require extensive tuning to manage the parameters associated with goal conditioning [39
Our approach distinguishes itself by utilizing hypernetworks to dynamically generate policy weights, thereby reducing parameters in the
33 48).
policy network while enhancing scalability without extensive retraining [20 43)
In the realm of imitation learning for goal-conditiones policies, several frameworks have demonstrated
promising results by learning from pre-collected demonstrations [29] 9
53 Recent goal- coal-conditioned
behavior cloning approaches such as C-BeT [9] and MimicPlay [53 have advanced long- ong-horizon
manipulation tasks. However, these methods typically require sequences of achievable goal images, which are challenging to obtaín in practice. Moreover, while performing well in basic pick-and-place scenarios, they often struggle with contact-rich tasks that demand precise environmental aware- ness [3]] Our method overcomes these limitations through effective latent space shaping, requiring only single goal image while maintaining robustness across diverse manipulation scenarios.
Hypernetworks and Cognitive science insights for goal-directed behavior. Our work draws inspiration from cognitive science research on human goal-directed behavior, where meta-cognitive
strategies and higher-level planning mechanisms enable adaptive actions 6 36 37 Studies show
that humans efficiently manage cognitive resources and flexibly adapt to different goals through
higher-level representations 18 [0]. Current policy learning methods incorporating cogmitive principles often focus on imitation learning to mimic human strategies 2 16 14], but can be limited by demonstration quality and diversity [88 42 [2]. Hypernetworks have been explored in robotic control [2155 [3, though primarily within reward- reinforcement learning (RL) settings 53 This fundamental difference in training paradigms, RL versus our reward-free behavior cloning (BC), means their end-to-end algorithms are not directly adaptable. We clarify, however, that their core hypernetwork architectures can be decoupled from the RL framework. By embedding cognitive insights into our hypernetwork architecture, we emulate human-like flexible adaptation 66 36 37).
Hyper-GoalNe GoalNet's ss capacity to generate goal-specific policy parameters without extensive retraining addresses practical challenges of goal-conditioned learning (20) while mirroring key cognitive mechanisms, offering a biologically plausible framework for adaptable policy generation.
3 Method
Let D (m.)aM be a dataset consisting of M robotic manipulation demonstrations, where each trajectory comprises sequence of observation-action pairs, i.e., Ti = {(of, ) with a, denoting continuous-valued action and 0y representing tuple containing high-dimensional state observations. Specificully, 0j includes an RGB image I captured by a single front-view camera, as well as the proprioceptive information s, of the embodied agent. Given this formulation, our objective is to develop a generalizable goal-conditioned policy learning framewor that enables efficient adaptation to diverse manipulation tasks.
We propose a shift from conventional goal-conditioned policies, which typically use fixed parameters while processing both current and goal images. Our key insight is that the goal image inherently specifies how the current image should be processed to generate appropriate actions. Therefore, we argue that the policy parameters themselves which determine the processing mechanism should adapt based on different goal specifications. To realize this insight, we leverage hypernetworks to dynamically generate task-specific policy parameters conditioned on goal images, rather than directly conditioning a fixed policy network on both current and goal observations. This approach creates a more flexible and efficient framework where the processing of current states is explicitly tailored to the specified goals.
The full pipeline is illustrated in Fig. 2. Next, we elaborate on the key components of our approach. In Sec. 3.1] we describe how we adapt hypernetwork architectures to effectively generate varying geal-rea policies. In Sec. 3.2 we introduce an effective latent space shaping techniques that significantly enhance the performance of goal-conditioned policy generation by enforcing meaningful geometric structure in the representation space. Finully, in Sec. 3.3 we present the test-time inference pipeline that integrates our trained model (Hyper-Gnoe Goaee- to accomplish diverse manipulation tasks, highlighting the practical advantages of our parameter-adaptive approach.
3

---

(a) Adaptive Policy Generation
(b) Latent Shaping
Target Palicy
Action
Propri.
Shate Encodirr
Gurrent ebx
Net
Feature
Action
Feature @hime
Images
RGB
Genenoted weights
Waight
Good image
Latent
bet
Peoture @tine
RG8 Encedir
Hyper
Manifoll
Des.
Feoture etime
Network
Sesl Feature
Figure 2: An overview of the proposed Hoper-GoalNet framework. (a) Adaptive Policy Genera- tion: Unlike conventional approaches with fixed parameters, our hypernetwork dynamically generates task-specific policy parameters conditioned on goal images. This creates parameter- adaptive target policy that processes current observations (RGB images and proprioception) through a multimodal encoder to produce actions tailored to specific goals. (b) Latent Shaping: Our approach enhances performance by explicitly structuring the latent space in two ways: predictive network models state transitions to improve temporal dynamics, while geometric constraints ensure distances to goal states monotonically decrease during successful trajectories (detailed in Sec. 3.2)
3.1 Goal-Conditioned Hypernetworks
We formulate coal-conditioned policy learning parameter generation task rather than direct conditioning problem. This formulation reframes the challenge from "what action to take given current and goal observations" to "what processing parameters
given the goal."
More formally, given a current observation og and a desired goal observation Og, we model the condi- tional distribution over target policy weights that will transform the scene into the goal configuration. With the set of robotic manipulation demonstrations D, we learn the distribution Hi( oc: og):
H(O Ocr 0g) = Po (0 o I, 0g = Ir), where I, Ty E T E D and L' I.
For practical implementation, we focus on single goal state rather than a sequence of goals, and use RGB image observations to condition the hypernetwork. Since our primary objective is to investigate the efficacy of colal-ondiiooeed policy generation for manipulation tasks rather than developing a full probabilistic model, we approximate this deterministic mapping:
H O x
(2)
where O denotes the observation space and θ represents the space of target policy parameters. With a current-goal observation pair (o., 0y), the hypemetwork H. produces policy that guides the transition from current state o, to goal state 0g through action execution.
Hypernetwork architecture. To implement our parameter-adaptive approach, we adopt hyper- network architecture that efficiently generates policy parameters for achieving specified goals. The architecture must be capable of capturing the complex relationships between current states, goal states, and the required actions to bridge them.
We leverage an irrmmiiaationinnspiren architecture following 40) which provides beneficial inductive bias for our parameter generation task. This approach mimics iterative optimization by refining policy parameters through multiple feed-forward steps:
0x
H(O,, 0g),
where g represents the final policy parameters after K refinement iterations, with each update computed as:
gh = -k + >* ( a)). a = ((oc,
(Bk-1,
Here, neural modules X and 2 serve as learned analogs to step sizes and gradients in optimization, operating on embeddings @00c: 0g) of the current and goal observations. This mechanism enhances the hypernetwork's 's ability to generate effective task-specific policy parameters and improves general- ization new goal specifications. Once we obtain the goal-conditioned policy weights 0, we can process the current observation through the generated policy to predict appropriate actions.

---

Hypernetwork Training. We train our hypernetwork H. O O + θ to generate parameters for the target visuomotor policy ((= f) using behavior cloning (BC) on demonstration data. The generated policy takes the current observation (comprising image 7, and proprioception 8:) and outputs actions for execution.
To enhance robustness, we utilize a sequence of L consecutive observations as input to the policy, cap- turing temporal dependencies under non-Markovian assumption. The training objective minimizes the BC loss between demonstrated actions aj and predicted actions at:
policy
Σ Σ i=1 ISKKENN
Elab ot), of T H(ob.o4))
(5)
Here, e represents the Mean Squared Error (MSE) loss. The end-to-end training procedure works as follows: for a given current observation oj and goal image oi., the hypernetwork H. generates the weights for the policy network т. This goal-conditions sequence o;_ Lit to predict the action át. The resulting loss is backpropagated through both the policy network and the hypernetwork. We restrict goal representations to image inputs since proprioceptive goal states may not always be available in practical applications. This formulation allows the hypernetwork to learn how to generate goal-specific processing mechanisms (target policy parameters) from visual goals, embodying our key insight that goal images determine how current observations should be processed.
policy then processes the observation
3.2 Latent Space Shaping
A critical insight in our approach is that the effectiveness of parameter-adaatiie policies depends significantly on the quality of the representation space in which observations are embedded. While hypemetwork can directly generate policy parameters from raw observations, find that explicitly shaping the latent representation space substantially enhances performance. Given the high dimensionality and redundant information in RGB images, we employ an image encoder & to extract task-relevant features and compress them into low-dimensional latents E().
We identify two fundamental properties that, when enforced in the latent space, particularly benefit our aurameter-adaptive approach: predictability and physical structure preservation. The first property ensures the latent space facilitates modeling of state transitions, making the hypernetwork' 's task of generating appropriate policy parameters more tractable. The second property ensures that the geometric relationships in the latent space meaningfully reflect physical relationships between states, enabling the generated policies to exploit these structured representations.
Enhancing predictability through dynamic modeling. To improve the predictability of latent representations, we introduce dynamics model that forecasts future states the latent space. By training this model to predict state transitions while simultaneously shaping the encoder & ttrouuhh backpropagation, we create latent space where sequential relationships are explicitly captured. This significantly benefits our hypernetwork, it needs generate policy parameters that leverage these sequential relationships to guide transitions from current to goal states.
Formally, consider discrete-time dynamical system with state representation E 2 and control input 0 E A. The forward dynamic model is:
(6) where = E(4) represents the latent encoding at time t, is the executed action, and is the transition dynamics parameterized by 4. For practical implementation, we approximate this with
ê+1 (2(4++1 (2,,aa). ax),
deterministic model & : Z x A Z. The corresponding learning objective minimizes the prediction
loss:
Speed E,wD [e( (((((2z (2,, at), 2611),
where is distance metric in the latent space. When well-trained, we further finetune & through & to shape representations that capture both current state and potential transition information.
Preserving physical structure through distance constraints. For our parameter-adaptive approach to be effective, the latent space must preserve the physical structure of the task, particularly the progression towards goals. We formalize this as requirement that the distance between the current state and goal state should monotonically decrease along goal-reaching trajectories. This property is
5

---

especially valuable for our hypernetwrkk, as provides a clear signal about how policy parameters should change as states approach goals.
Specifically, for any goal-reaching trajectory To E D, where T; ((o),, aه) we enforce:
dg(o} of.) 2 de(o)+11 ofy), Vj <j<
(8)
where de denotes distance metric in the latent space. To explicitly model this behavior, propose the following loss function:
Ldist EIND Σ max (0, 8 d(zz+1, =g) d(z) =g)),
(9)
where d(31, 22) |||= 22 represents the Euclidean distance between latent features, 2j
E(1))
and 2g = {lg)) denote the image and goal image latents, respectively. The margin parameter 3 2 0 enforces minimum decrease distance between consecutive states and the goal. Empirically, setting 3 = 0 suffices to induce the desired monotonic progression.
This shaped latent space creates ideal foundation for our parameter- adaptiv approach, as it encodes both the predictive dynamics and geometric structure needed for the hypernetwork to effectively generate goal-tailored policy parameters. Figure 3] illustrates how our latent space shaping approach compares to alternative methods, showing the enhanced structure that benefits our goal-conditioned policy generation.
3.3 Hyper-GoalNet for Manipulation
Having established our parameter-adapiii architecture and latent space shaping techniques, we now present our complete framework, Goppee. which synthesizes these components for effective goal-conditioned manipulation. The overall training objective combines our policy generation loss with the latent space shaping terms:
Lityper GoalNer = Spolicy + Xpred Lpred + Adist Laist:
(10)
where Apred and Adia are weight coefficients balancing the contributions of predictability and structural constraints. The framework trained end-to-end using gradient descent, allowing all components to co-adapt for optimal performance.
During inference for task completion, Tlyper-GoalNer generates goal-specific policy parameters by feeding the concatenated latent features [žg. 21] into the hypernetwork H. where 2g E {1,)) and 2, E(4) are the latent representations of the goal and current observations. The generated goal-specific parameters / = H(2,, 2g) are then loaded into the target policy (-; 0), which processes the current observation sequence produce actions that guide the agent toward the goal.
offers two principal advantages over conventional goal goal-conditioned policies:
1) Paraneter-adaptive policy generation: By dynamically synthesizing policy parameters based on goal specifications, our approach effectively changes how goal-conditioned policies operate. Rather than relying fixed network with static parameters to handle all possible goals, Nyper-GoalNet generates compact and efficient processing pathways that are tailored to specific goals.
2) Natural goal completion detection: Our shaped latent space provides an elegant solution to the challenging problem of goal completion detection. The distance metric de (ot, 0g) in the latent space serves as natural criterion for determining when a goal has been achieved, enabling autonomous goal transitions without external supervision.
The test-time task evaluation procedure for Hyper-GoalNet is formalized in Algorithm I Given an initial observation To and goal observation Ig. the algorithm iteratively generates policy parameters, samples actions, and applies them until either the goal is reached (as determined by the latent distance falling below a threshold procedure demonstrates how our parameter-adaptive approach seamlessly integrates into practical robotic control scenarios.
maximum number of steps T is achieved. This simple yet effective
4 Experiments
In this section, we present comprehensive experimental evaluation across suite of simulated and real-robot manipulation tasks designed to address the following questions: 1) How effectively does

---

Algorithm Hyper-GoalNet: Test-Time Task Evaluation
Input:
Initial observation Io. Goal observation I Encoder E, Policy generation hypernetwork 7.
Modules:
Parameters: Max steps T, Goal completion threshold
1: 4 t lo. +- 0, done +- false
2: while T and not done do
3:
&- (((E(()))E(1)
Generate policy weights Sample action
4:
â < [(0-LL4;; 8)
5:
I++1, done < Env (āz)
Apply action and Env. interaction
6:
if (((E(1++)),ELL)) < done then
7:
return SUCCESS 8: end if
9:
1, 5 441. + ++1
end while
11:
return TIMEOUT
Hyper- Goaeesss parameter-adaptive approach generate successful policies for diverse manipulation tasks? 2) To what extent does our latent space shaping enhance the performance of the hypernetwork for goal-conditioned policy generation? 3) How does Hyper-GoalNet compare with conventional goal-conditioned methods and alternative representation learning approaches? Through extensive empirical analysis, we validate the effectiveness of our aarameter-adaptive framework.
4.1 Experiment Setup
We evaluate our approach using Robosuite, a comprehensive robotics benchmark designed for both short and long-horizon manipulation tasks [3]. This framework provides a standardized suite of environments, from which we select multiple contact-rich tabletop manipulation tasks: coffee manipulation, threading. mug cleanup, nut assembly, three-piece assembly, and several long-ho long-horizon tasks including coffee preparation and kitchen manipulation. To assess robustness across varying initial conditions, we use three difficulty levels (do, dj, d2), where higher indices correspond to increased environmental variability, particularly in object pose initialization (position and orientation). Each experimental environment features robotic manipulator positioned adjacent to a workspace containing task-specific manipulable objects.
Simulation Environment.
Training Protocol. Our approach follows the behavior cloning paradigm, utilizing dataset based on MimicGen [3I] For each task, we employ a training dataset of 950 demonstrations, where each timestep comprises front-view RGB images (128 128 resolution), robot proprioceptive states 8, E S, and corresponding ground-truth actions a₂ E A. The training procedure employs the Adam optimizer (26) with cosine learning rate schedule (8 We initialize the learning rate 5 10-4 and maintain uniform loss balancing coefficients (A₃ = 1 for all components). Our model trained for 500 epochs with batch size of 256, by default. Detailed implementation and training protocols
are provided in the Sec.
4.2 Main Results
Baselines. We compare parameter-adaptive approach against state-of-the-ar goal-conditioned methods that use fixed network parameters. All methods are trained on pre-collected demonstrations from MimicGen B1 and modified to operate with single future image the goal specification for
fair comparison:
GCBC (29 [5]: Coal-Conditioned Behavioral Cloning concatenates current and goal observations as input to fixed policy network, learning a direct mapping from this concatenated representation to actions through supervised learning on demonstration data.
Play-supervised Latent Motor Plans learns a latent plan space from demonstration data, then trains a ixed-parameter policy conditioned on both the current state and the inferred latent plan for the specified goal.
Play-LMP 29
7

---

Table 1: Comparison with state-of-the-art goal-conditioned methods. Success rates (higher is better) are computed over 50 rollouts across various manipulation tasks with increasing difficulty levels (d0-d2). The experimental setup uses only two historical observations and a single goal image, representing a practical deployment scenario. Our method consistently outperforms conventional fixed-paametter approaches, demonstrating the effectiveness of dynamically generating policy param- eters based on goal specifications.
Coffee t
Mug-eleanup f dl
Three piece Assemb.
Threading
Nur Assemb,
Method
Avg.
d0
d1
d2
Avg.
d0
Avg.
d0
d1
d2
Avg.
d0
d1
d2
Avg.
d0
GCBC
0.00 0.00 0.00
0.00
0.00
0.00
0.00
0.00 0.00
0.00
0.00
0.00
0.00 0.00
0.00
0.00
0.00
Play-LMP
0.00 0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MimicPlay
0.28
0.28 0.16
0.24
0.26
0.06
0.16
0.06
0.06
0.00
0.04
0.18
0.02
0.00
0.07
0.03
0.12
C-BeT
0.92 0.00
0.74
0.55
0.30
0.50
0.40
0.00
0.02
0.00
0.01
0.62
0.22
0.12
0.32
0.34
0.32
Ours
0.94 0.76 0.62
0.77
0.78
0.46
0.62
0.52
0.20
0.04
0.25
0.82
0.32
0.24
0.46
0.55
0.52
Table 2: Long-horizon task performance. Our parameter-adaptive approach excels in complex sequential tasks, outperforming fixed-parameter methods across difficulty levels.
Table 3:
Component ablation analysis. La-
tent space shaping significantly enhances perfor- mance, while proper distance metrics and train- ing are crucial for robustness.
Coffee Preparation
Kitchenf
Coffee
Method
d0
dl
Avg.
d0
d1
Avg.
Avg.
Method
d2
GCBC
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Ourstuf: opoch ())
Ourstwlo shaping)
0.92
0.92
0.000
0.62
0.00
0.51
0.31
Play-LMP
0.00
0.00
0.00
0.00
Oursdistt-start img.)
0.50
0.52
0.32
0:45
MimicPlay
0.34
0.00
0.17
0.86
0.18
0.52
0.35
Oursfoos dist)
C-BeT
0.82
0.04
0.43
0.78
0.70
0.74
0.59
C-BeTww/ shaping)
0.80
0.61
0.64
0.69
Ours
0.80 0.50
0.65
1.00
0.80
0.90
0.78
Ours
0.94
0176
0.62
0.77
C-BeT [ Conditional Behavior Transformer uses self-attention to compress observation history into latent representation, which is combined with the goal state to condition fixedparammete transformer that predicts actions.
MimicPlay is a self-supervised approach that learns general robotic skills from unstructured telcoperation data, which consists of continuous sequences observations and actions from a human video. For our experiments, this method is adapted into goal-immage-onnditioneed policy, with implementation details provided the Appendix.
MimicPlay 53
Evaluation Metrics. We evaluate performance using task completion success rates over 50 inde- pendent rollouts with randomly initialized, previously unseen environmental configurations. We impose maximum trajectory lengths of = 600 or 800 steps for contact-rich tasks and = 1600 for long-horizon tasks. While our parameter-adaative approach enables autonomous task completion we use aavironment-providee terminal signals for standardized evaluation across all methods. Success is indicated S 1 if rollout i completes within steps, and S; = 0 otherwise.
detection through latent space metrics (Algorithm [
Quantitative Results. Tables D and present success rates across multi-step and long-horizon tasks, respectively. For each task, success is determined by tusk-speciin criteria provided by the environment such as correct object placement, successful insertion, proper assembly configuration, or completion of sequence of subtasks for long-horizon scenarios. Our parameter-adaptic approach outperforms fixed-parameter methods across these diverse evaluation criteria and difficulty levels. This superior performance stems from our hypernetwork's 's ability to dynamically generate task-specific policy parameters tailored to each goal, resulting in more effective coal-directed behavior. Particularly in high- nigh-variability environments (difficulty levels dl-d2), our method demonstrates greater robustness and adaptability compared to conventional approaches highlighting the advantage of having policy parameters explicitly conditioned on goals rather than using fixed parameters for all scenarios.
Analysis of Likelihood- Based Baselines. We diagnose the poor performance of GCBC and Play- LMP as a fundamental issue of their learning objective, not implementation. These methods, adapted from reputable thiry-party code, aim to maximize the log-likelihood of expert actions. We found this leads to severe overfitting on the training data, evidenced by large gap between low training loss and high validation loss. Such memorization-be based learning fails to generalize to the subtle variations
8

---

present in our high precision test scenarios. This limitation of likelihood based models in complex settings is corroborated by prior work (9. In contrast, our hypernetwork's design imposes beneficial structural bias: by emulating an optimization process to generate parameters, it is incentivized to learn a functional mapping from goal to policy, ensuring better generalization and avoiding the overfitting issues that plague the baselines.
4.3 Ablation Study
Hypernetwork Architecture Analysis. We evaluate our optimizationiispiiie hypernet- work design against HyperZero [41]. a promi- nent alternative architecture. Architecturally, HyperZero encodes conditioning information into a meta-embedding that is then transformed to produce the parameters for the target policy network. Because this direct mapping can pro- duce parameters with a numerical range mis- aligned with that of an optimally trained net- work, it often requires special initialization to stabilize training. To ensure stable and robust comparison, we therefore implemented two enhanced initialization schemes for HyperZero. The first, ScalarInit, introduces a learnable scalar to control the initial scale of the hypernetwor''s output. The second, Bias-Ini [5], is designed for nigh-dimensional conditioning inputs and constrains the parameter range by incorporating learnable biases alongside weights that are initialized zero. As shown in Table 4 even with these stabilization techniques, our architecture, which requires no special initialization, vastly outperforms HyperZero. This demonstrates that the iterative refinement mechanism in our design is inherently more effective at capturing the relationship between goals and appropriate policy parameters. These results highlight the critical role of architectural choice in developing robust parameter-adaptive policies. More details can be found in Sec. C
Table 4: Ablating hypernetwork architectures. Our method with standard initialization is com- pared against HyperZero variants stabilized with enhanced initializations.
Coffee
Task (%)
f
Method
d0
d1
d2
Avg.
HyperZero ScalarInit
16
18
14
16
HyperZero Bias-Init
30
18
0
16
Ours (Standard Init.)
94
76
62
77
Latent Space Shaping Analysis. Table 3 shows that proper latent space shaping is critical for our parameter-adaptive framework. Removing shaping ("Ours w/o shaping") severely degrades performance harder difficulty levels, while our specific choices of using goal-relative distances and Euclidean metrics prove superior to alternatives. F visually confirms how our approach creates more consistent monotonic progression toward goals compared to unshaped representations. Notably, applying our shaping techniques to the baseline C-BeT improves its performance, yet its performance lags compared to ours, which in turn signifies the importance of the parameter-adaativve framework as well as the synergy between policy generation and latent shaping. Our training strategy also matters- unfreezing the R3M visual encoder [34] only after 20 epochs ensures stable parameter generation. These results validate our insight that latent spaces should reflect physical progression toward goals to effectively support parameter-ade adaptive policy learning.
Visualization. Figures 3 and illustrate how our latent space shaping creates representations that directly benefit our parameter-adaptinn ap- proach. The plot shows the latent distance the goal (y-axis) over the execution timesteps of a
Dero
Damo
Deria
Dare's
Demo
Highly noldy trend
Monotonic trand
Distence to the gool
robotic task rollout (x-axis), where later steps are progressively closer to the goal. Unlike R3M
embeddings which show significant fluctuations, our method produces consistently monotonic distance reductions toward goal states. This structured latent space offers two key advantages for our hypernetwork: (1) provides clearer sig- nals for generating appropriate policy parame- ters as the agent progresses toward goals, and (2) it enables reliable autonomous detection of task completion based on latent distances. The visualization confirms that our combined predictive modeling and distance constraints create opti-
Time Seeps
Tiin Staae (b) Hoplr-Gelll Embsedding
(6) REM Embedding
Figure 3: Comparison of (a) unshaped R3M em- heddings versus (b) our shaped latent space, show- ing L2 distances to goal states along multiple tra- jectories. Our shaping creates consistent mono- tonic decreases in distance-to-goal, facilitating more effective parameter generation.

---

mal representations for goal coal-condi parameter generation, enhancing both performance and interpretability.
Goal Completion Detection. To substan- tiate our claim that the shaped latent space enables autonomous goal completion de- tection, we supplement the qualitative evi- We eval-
Table 5: Quantitative validation of autonomous goal completion detection. Our method's s latent distance- based success rate (Auto SR) is compared against the environment's ground truth (Env SR) on the Coffee tasks. High Accuracy and Recall validate its reliability.
dence from Figures 3 ? and TO
uate this capability by comparing our au-
tonomous detection, where success is deter- mined by a latent distance threshold (Auto SR), against the environment's 's ground ground-truth signal (Env SR). T the results using two key metrics: Accuracy, which measures the agreement between the two signals, and Recall, which measures method's ability to identify true successes reported by the environment. The strong alignment, evidenced by average accuracy of 86.6% and recall of 94.3%, provides strong empirical evidence that our latent-distance based approach is a reliable autonomous completion detector.
Task
Auto SR
Env SR
Accuracy
Recall
DO
0,96
0.94
94%
98%
DI
0.78
0.76
90%
95%
D2
0.74
0.62
76%
90%
Mean
86.6 %
94.3%
4.4 Real Robot Experiments
We validate our parameter-adaptive approach physical hardware using the Realman Robotics Platform, featuring 7-DoF manipulator with a I-DoF parallel gripper (Figure 7). We eval- uate four diverse manipulation tasks sweep, pick&place, pull, and stack with 15 trials per task. Due to hardware constraints limiting con- trol to joint angles without end-effector pose information, we exclude MimicPlay, which re- quires precise 3D end-effector trajectories.
Table 6: Real-robot experiment results. (suc-
cesses/total trials).
Method
Pickplace
Pull
Stack
Sweep
GCBC
0/15
0/15
0/15
2/15
Play-LMP
0/15
0/15
0/15
5/15
C-BeT
2/15
6/15
5/15
8/15
Ours
14/15
15/15
14/15
15/15
As shown in Table conventional tred-parameter approaches struggle significantly in real-world conditions where environmental noise, perception uncertainties, and imperfect demonstrations create substantial challenges. In contrast, our method maintains high success rates across all tasks, including those requiring precise contact-rich interactions. This real-world performance gap highlights key advantage of our ceruumeter-ddattiv approach: by dynamically generating task-specific policy parameters based on goal images, our method better adapts to real-world variations and demonstration imperfections that weren't 't encountered during training. Detailed experimental protocols are provided in the Sec.
5 Discussion
Conclusion. Our cerameter-adaptivvv approach represents an effective move in goal-conditioned policy learning by dynamically generating policy parameters based on goal information rather than using fixed parameters with conditioning. The consistent performance improvements across tasks demonstrate that "how" observations should be processed is inherently dependent on the goal specification. Our latent space shaping techniques prove critical for this architecture impossng physical structure and predictive capacity provides clearer signals for the hypernetwork to generate effective policy parameters. Overall, our results suggest that explicitly modeling the relationship between goals and processing mechanisms offers a promising direction for more flexible and robust robotic control.
Limitations. Our method's primary limitation is its reliance on well- strl-urtre latent space, which is challenging to form for highly complex tasks and requires demonstration data with clear goal progression. This data dependency creates key failure mode: dusti--distribution goals can cause the hypernetwork to generate erratic policies. Furthermore, as offline-trained method, its dynamic generation of parameters for novel goals lacks explicit safety guarantees against unforeseen states, making the integration of robust safety constraints critical direction for future research.
10

---

Acknowledgments and Disclosure of Funding
This work supported by the Early Career Scheme of the Research Grants Council (RGC) grant 27207224, the HKU-100 Award, a donation from the Musketeers Foundation, and in part by the JC STEM Lab of Autonomous Intelligent Systems funded by The Hong Kong Jockey Club Charities Trust.
References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-45) 2009.
[3] Sayantan Auddy, Jakob Hollenstein, Matteo Saveriano, Antonio Rodríguez-Sánchez, and Justus Piater. Scalable and efficient continual learning from demonstration via hypemetwork- generated stable dynamics model. arXiv preprint arXiv:2311.036600 2023.
[4] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
[5] Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforceemet learning. In Conference on Robot Learning, pages 1478-1487. PMLR, 2023.
[6] Matthew M Botvinick, Yael Niv, and Andew Barto. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. cognition, 113(3):262-22)) 280, 2009,
[7] Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan, Ping Luo, Heming Cui, Ma, and Hongyang Li. Closed-loop visuomotor control with generative expectation for robotic manipulation. In The Thirty-eighth Annual Conference Neural Information Processing Systems.
[8] Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models embodied ai. arXiv preprint arXiv:24/1.00785 2024.
[9] Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play policy: Conditional behavior generation from uncurated robot data. arXiv preprint ar1iv:2210. 10047, 2022.
[10] Nathaniel D Daw, Yacl Niv, and Peter Dayan. Uncertainty- competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature neuroscience, 8(12):1704-1111 2005.
[11] Peter Dayan and Nathaniel D Daw. Decision theory, reinforcement learning, and the brain. Cognitive, Affective, Behavioral Neuroscience, 8(4):429-44 2008.
[12] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
[13] Yilun Du, Sherry Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156-9172, 2023.
[14] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benehmarking deep reinforcement learning for continuous control. In International conference machine learning. pages 1 PMLR, 2016.
[15] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What essential for offline rl via supervised learning? arXiv preprint 1r21v 2021.
11

---

[16] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical in- teraction through video prediction. Advances in neural information processing systems, 29, 2016.
[17] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning, arXiv preprint arXiv:/004.03012, 2017.
[18] Michael Frank, Lauren C Seeberger, and Randall C O''reeil O'reellyy. By carrot by stick: cognitive reinforcement learning in parkinsonism. Science, 306(5703):1940--194, 2004.
[19] Tomer Galanti and Lior Wolf. On the modularity of hypernetworks. Advances in Neural Information Processing Systems, 13:10499-10419, 2020.
[20] David Ha and Jürgen Schmidhuber. World models. arXiv preprint VrXiv::803.10122,,
[21] Shashank Hegde, Zhchui Huang, and Gaurav S Sukhatme. Hyperppo: A scalable method for finding small policies for robotic control. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 10821-10828 IEEE, 2024.
[22] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[23] Yizhou Huang, Kevin Xie, Homanga Bharadhwaj. and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In 202/ IEEE International Conference on Roboties and Automation (ICRA), pages 799-805. IEEE, 2021.
[24] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey, Journal of artificial intelligence research, 4:237-285, 1996.
[25] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, al. Openvla: An open-source vision-language-aati action model. arXiv preprint 0rXiv:2406.09244, 2024.
[26] Diederik Kingma. Adam: A method for stochastic optimization.
arXiv preprint
arXiv:1412.6988 2014.
[27] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08290 2022.
[28] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[29] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[30] Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in Neural Information P Processing Systems, 35:310-323, 2022.
[31] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang. Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17549 2023.
[32] Earl K Miller and Jonathan D Cohen. An integrative theory of prefrontal cortex function. Annual review of neuroscience, 2771)2 2001.
[33] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018.
[34] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022.
12

---

[35] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Advances in neural information processing systems, 32, 2019.
[36] Randall C O'Reilly and Michael J Frank. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural computation, 18(2):283-328, 2006.
[37] Giovanni Pezzulo and Paul Cisek. Navigating the affordance landscape: feedback control as process model of behavior and cognition. Trends cognitive sciences, 20(6):414 424, 2016.
[38] Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[39] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep rl for model-based based control. arXiv preprint arXiv: 1802. 09081, 2018.
[40] Hanxiang Ren, Li Sun, Xulong Wang, Pei Zhou, Zewen Wu, Siyan Dong, Difan Zou, Youyi Zheng, and Yanchao Yang. Hypogen: Optimization- biased hypernetworks for generalizable policy generation. In The Thirteenth International Conference on Learning Representations.
[41] Sahand Meger. Hypernetworks for zero-shot transfer in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence volume 37, pages 9579-9587, 2023.
Charlotte Morissette, Francois R Hogan, Gregory Dudek, and David
[42] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- rional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[43] Cicero Nogueira dos Santos, Youssef Mroueh, Inkit Padhi, and Pierre Dognin. Learning implicit generative models by matching perceptual features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4461-4470, 2019.
[44] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[45] Simon Schug, Seijin Kobayashi, Yassir Akram, João Sacramento, and Razvan Pascanu, Atten- tion hypernetwork. arXiv preprint arXiv:2406.058 05816, 2024.
[46] Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, and Andy Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 202/ IEEE International Conference Roboties and Automation (ICRA), pages 4568-4575. IEEE, 2021.
[47] Nur Muhammad Shafullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning A modes with one stone. Advances in neural information processing systems, 35:22955-22968, 2022.
[48] Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 63-70. IEEE, 2024.
[49] Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
[50] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
[51] I Toni. The cerebellum and parietal cortex play specific role in coordination: pet study. Neuroimage, 14(4):899-911, 2001.
13

---

[52] Quan Vuong, Sergey Levine. Homer Rich Walke, Karl Pertsch, Anikait Singh. Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition CoRI.2023, 2023.
[53] Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizen imitation learning by watching human play. arXiv preprint arXiv:2302. 12422, 2023.
[54] Huilin Xu, Jian Ding, Jiakun Xu, Ruixiang Wang, Jun Chen, Jinjie Mai, Yanwei Fu, Bernard Ghanem, Feng Xu, and Mohamed Elhoseiny. Diffusion-based imaginative coordination for bimanual manipulation. arXiv preprini arXiv:2507. / /296, 2025.
[55] Hongxiang Yu, Anzhe Chen, Kechun Xu, Zhongxiang Zhou, Wei Jing, Yue Wang, and Rong Xiong. hyper-networt based end-to-end visual servoing with arbitrary desired poses. IEEE Robotics and Automation Letters, 8(8):4769-477 2023.
[56] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages (094-1100, PMLR, 2020.
[57] Pei Zhou, Ruizhe Liu, Qian Luo, Fan Wang. Yibing Song, and Yanchao Yang. Autoegp: Closed- loop concept-guides policies from unlabeled demonstrations. In The Thirteenth International Conference Learning Representations.
[58] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martnn Abhishek Joshi. Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
14

---

Appendix
In this section, we present supplementary materials detailing the methodological framework and experimental procedures used in this study.
A Manipulation Task Details
Task Details. Our experimental evaluation conducted within the Robosuite [58] simulation environment, utilizing the benchmark dataset from Mimicgen [31]. We primarily investigated complex tabletop manipulation tasks that encompass diverse robotic skills. The selected tasks are characterized as follows:
Coffee: A multi-step manipulation task requiring precise object handling, where the robot must grasp a coffee capsule, insert it into the designated slot of the coffee machine, and securely close the machine's lid.
Mug cleanup: A sequential task involving both articulated object interaction and object placement. The robot must coordinate drawer manipulation and object transportation, culminating in storage of mug.
Three piece assembly: A multi-step assembly task demanding spatial reasoning and precise manipulation. The robot must stack three components in specific sequence to achieve compact assembly configuration.
Threading: A high-precision manipulation task requiring fine motor control. The robot must accurately orient and manipulate a needle for successful insertion through a minimal aperture.
Nut Assembly: A manipulation task involving precise grip control and spatial alignment for successful mechanical assembly. this task, the success rates for the two nuts are measured separately, and the overall success rate is then calculated.
Coffee Preparation: An extended sequential task combining multiple sub-goals, includ- ing cup positioning, drawer manipulation, capsule retrieval, and coffee maker operation, culminating in fully prepared coffee setup.
Kitchen: complex sequence involving appliance interaction, object manipulation, and spatial reasoning. The task includes stove operation, cookware handling, and precise object placement.
These tasks are specifically selected for their comprehensive representation of challenging robotic manipulation scenarios, featuring contact-rich interactions, precise object manipulation, and complex multi-step sequences. Each task requires a combination of skills including spatial reasoning, and sequential decision- merii..n-mmaing.... The visual representation of these manipulation tasks and their key phases are illustrated in Figure
Data Processing and Observation Space. The demonstration data from Mimicgen is initially preprocessed by segmenting complete demonstrations into trajectory subsequences to facilitate learning. Our framework implements constrained observation context with length of 2, including front-view RGB images and the agent's proprioceptive state information. This design choice means that the agent's 's decision-making process is based solely on the current frame and one historical frame, deliberately limiting the temporal horizon enhance real- world applicability. Given focus goal-conditioned policy learning, the observation space of the hypernetwork is augmented with a single RGB goal image representing feasible target state. The input modalities are structured as follows:
Visual observations: RGB images with dimensions 128x128 pixels for both contextual and goal representations.
Proprioceptive state: compact 9-dimensional vector encoding essential agent state infor- mation.
This deliberately constrained observation space creates a partially observable environment that closely aligns with real- world robotics scenarios, where complete state information is rarely available. While
15

---

(o) Coffee dD
(b) Coffee d1
(c) Coffee d2
(d) Mug clearup dO
clecnup di
assembly
assembly
(i) Threading
Threading dl
(K) Threading
(1) Nut assembly do
(m) Coffeee preparation dD
(n) Coffee: preperation d!
(o) Kitchen d0
Kitchen
Figure 4: Key phases of diverse manipulation tasks experimental evaluation.
this design choice enhances the practical applicability of our approach, it also introduces significant challenges:
Single-goal scenarios necessitate hypernetwork architectures to efficiently handle the de- manding requirements of goal-conditioned policy generation.
Limited temporal context requiring efficient of historical information.
Partial observability demanding robust state estimation and feature extraction.
Complex vision- based reasoning with constrained visual information.
Such challenging conditions
validate our method's effectiveness under realistic constraints, demonstrating its potential for real-world deployment.
Goal Specification Strategy. Our approach implements systematic strategy for goal specification across both training and evaluation phases. During training, we employ a dynamic goal sampling mechanism where the goal image is stochastically selected from future frames within the same demonstration, subsequent to the current timestep. This design offers two key advantages:
Goal Feasibility: By sampling from actual demonstration frames, we inherently guarantee the physical feasibility and reachability of the specified goals.
Goal Diversity: The random sampling mechanism ensures sufficient variation in goal states, promoting the learning of robust and generalizable policy.
During the evaluation phase, the goal specification mechanism leverages the Mimicgen framework to generate feasible goal states, facilitating potential transfer from simulation to physical systems.
B Comparison with Baselines
Given the challenging nature of our experimental setting as shown in Table I existing baseline methods demonstrate limited success in task completion. ensure comprehensive evaluation, we introduce modified versions of existing approaches and additional baseline methods adapted for our scenario. The following section details these enhanced baseline implementations and their comparative performance.
16

---

Table 7: Additional quantitative experiments with more baseline methods across different tasks. i indicates methods using sequence of visual frames as goals rather than a single goal image, with extended observation length of 10 frames versus the standard 2. $ indicates methods with access to extra wrist-mounted camera images. The wrist views, extended observation sequence, and extended goal sequences provide richer observational and task-guidance information for augmented baselines. Hyper-GoalNet leverages only one goal image and two front-view observations, indicating a more easy-to-be-applics ap-apee setup while being more effective.
Coffee
Mug-cleanup d1
Three piece Assemb.
Threading
Nut Assemb,
Method
Avg.
d0
d1
42
Avg.
d0
Arg.
d0
d1
d2
Avg.
d0
d1
d2
Avg.
d0
MimicPlay-Ott
0.80
0.84
0.88
0.84
0.68
0.58
0.63 0.50
0.38
0.02
0.30
0.32
0.06
0.06
0.15
0.10
0.44
MimicPlay-M
0.28 0.28
0.16
0.24
0.26
0.06
0.16
0.06
0.06
0.00
0.04
0.18
0.02
0.00
0.07
0.03
0.12
C-BeT
0.92 0.00 0.74
0.55
0..30
0.50
0.40
0.00
0.02
0.00
0.01
0.62
0.22
0.12
0.32
0.34
0.32
)
0.94 0.60
0.62
0.72
0.72
0.54
0.63
0.22
0.02
0.78
0.20
0.18
0.39
0.67
0.50
Hyper GoalNet
0.94 0.76
0.62
0.77
0.78
0.46
0.62
0.52
0.20
0.04
0.25
0.82
0.32
0.24
0.46
0.55
0.52
We also conducted additional comparative experiments with BeTT 47 (implemented in both its original setting and a modified setting). And a variant of our method, Hyper- GoalNet(G) is also introduced. Experiments were performed across all 16 MimicGen tasks, with results for contact-rich tasks and long- monghhorizon tasks presented in Tables 7 and g respectively. Please note that the success rates reported in Table reflect a modified evaluation criterion in the simulation environment, resulting in slight variations from the kitchen task results presented in Table [ The implementation details are explained below.
C-BeT 9 and MimicPlay (53
Details about the Baselines. We selected four task-specific baselines and reimplemented them under settings generally consistent with Hyper- The reimplementation details are as follows.
GCBC 29 15: Goal-Conditioned Behavioral Cloning (GCBC) is the most general framework for learning goal-conditioned policies. It consists of a perception module, a visual encoder, and a RNNN-based goal-conditioned policy module. GCBC takes in 9-dimensional proprioceptive state, a current front-view RGB image, and goal RGB image as input and predicts the action distribution to transfer the current state the goal state. The model is trained end-to-end with the objective of maximizing the log-likelihood of the ground-truth action in the predicted distribution. The obser- vation sequence length and the predicted action sequence length are both restricted to 5 steps.
Play-LMP [29]: Play-Supervised Latent Motor Plans (Play-LMP) builds upon the foundation of GCBC, aiming learn reusable plan representations and task-agnostic control from play data. Play-LMP consists of three main components: 1) Plan recognition module: maps the input sequence to distribution in the latent plan space. 2) Plan proposal module: generates multiple conditional prior solutions based on the current and goal states. 3) Plan and goal-conditioned policy: predicts actions conditioned on the current state, goal state, and latent plan sampled from the plan proposals. Similar to GCBC, both the observation sequence length and the predicted action sequence length are restricted steps. The model is trained end-to-end.
MimicPlay [33: MimicPlay employs a hierarchical learning framework consisting of two training stages. In the high-level training stage, the model takes the robot's 's end-effector pose, along with the current visual observation and goal observation, as input to predict the future pose trajectory of the robot's end-effector. This component is referred to the high-level planner. In the low-level training stage, the high-level planner with the best validation performance from the previous stage is loaded and its parameters are frozen. The model then continues training using a 9-dimensional robot proprioceptive state and visual observations (both current and goal) input predict the robot's actions.
Please note that in the original MimicPlay low-level training setup, in addition to the current front-view RGB image, mrist-mounted RGB image is also used input, which may contribute to its higher success rate. To ensure a fair comparison with our method, we modify this setup by replacing the wrist-mounted image with duplicate front-view image
17

---

Table 8: Additional evaluation on long-horizon tasks. it indicates methods using a sequence of visual frames as goals rather than single goal image, with an extended observation length of 10 frames versus the standard 2. j indicates methods with access to extra wrist-mounted camera images. The wrist views, extended observation sequence, and extended goal sequences provide richer observational and task -uiddgui information for augmented baselines. Ours achieves similar performance compared to the baseline with access to wrist view images, extended observation sequences and a sequence of goal images, which demonstrates the effectiveness of our method in achieving long-horizon tasks with much less guidance information (effort).
Coffee Prep. Avg.
Kitchen
Method
Avg.
d0
d1
d0
dl
Avg.
MimicPlay-Ol#
0.86
0.68
0.77
1.00
0.70
0.85
0.81
MimicPlay-Mi
0.34
0.00
0.17
0.86
0.18
0.52
0.35
C-BeT
0.82
0.04
0.43
0.78
0.70
0.74
0.59
Heper-GGolNNer(G)
0.80
0.50
0.65
1.00
0.88
0.94
0.80
Hyper-GoalNet
0.80
0.50
0.65
1.00
0.80
0.90
0.78
Table 9: Comparison with augmented BeT. it represents extended context length and additional wrist-view images. Although BeT is not originally designed as a goal-conditioned policy method, its augmented version serves strong baseline. Our method still outperforms the augmented BeT.
Method
Cof. d
Cof. d2 1
Mug. d1 t
Avg.
BeTt
0.66
0.42
0.26 0.46
0.45
Ours
0.94
0.62
0.67
during the low-level training process. Additionally, we adopt the same goal-specified strategy described above, rather than providing the entire prompt video used in MimicPlay's original test-time evaluation setting.
Heaer-Go We introduce another variant of our method, Hlpper-GollNet(C) where the hypernetwork backbone takes only single goal image as input, without requiring the current image during both training and testing phases. All other settings remain unchanged. During inference, Hyper-GoalNet(G) generates the weights for the lightweight target policy only once and maintains them fixed during rollouts, resulting in improved computational efficiency. As shown in Table 7 and T Hyper-GoalNedl still outperforms the baselines while utilizing much smaller lightweight policy network, demonstrating both the effectiveness and efficiency our approach.
Comparison with BeT [47]. We conducted comparative experiments with Behavior Transformer (BeT), a state-of-the-art approach for multi-modal behavioral learning. BeT employs k-means clustering to discretize continuous actions and utilizes transformers to model categorical distributions across action bins, incorporating an action correction head to refine discretized actions into continuous ones. Despite not being explicitly designed as goal-conditioned policy, BeT has emerged as a robust baseline in current robot learning literature. Given that the original BeT implementation for the Franka Kitchen task was limited to state space observations and lacked compatibility with Robosuite tasks, we enhanced its architecture by incorporating a pretrained image encoder [34]. To strengthen the baseline comparison, we augmented BeT with additional wrist camera observations a feature absent in our method-ane extended the context length from 2 (used in our approach) to 4, which typically facilitates more effective policy learning for BeТ. Consequently, BeT acquires more image observations per timestep than our method, while maintaining the same image resolution of 128x 128 pixels.
Table presents experimental results for some randomly selected tasks, where Cof. d0 and Cof. d2 represent Coffee d0 and Coffee d2 tasks, respectively, and Mug. dl denotes the Mug cleanup dl task. Notably, despite utilizing reduced contextual information, our method demonstrates superior performance across all tasks in terms of success rates. These results substantiate our method's 's robust capability in behavior learning, even under more constrained observational conditions.
18

---

Comparison with C-BeT 19]. Conditional Behavior Transformer (C-BeTT) is goal-conditioned version of BeT, a behavior prediction model that compresses an agent's observation history and the goal state into a compact latent representation using self-attention, which is then transformed along with discrete action representations to efficiently predict the agent's future behaviors. To ensure fair comparison, we configured C-Be'T with identical observation settings to our method, maintaining context length 2 and utilizing a single goal image. The comparative results are presented in Table] and Table g
Comparison with Original MimicPlay (MimicPlay-O).. the original MimicPlay experiment setting, retained the wrist image an input during the low-level training stage. For the test-time evaluation process, we provided the pretrained model with prompt videos in HDF5 format generated by MimieGen. In contrast to our approach, MimicPlay-O has access wrist-mounted camera images, uses an extended observation length of 10 frames instead of 2, and utilizes a sequence of visual frames goals, rather than single goal image configuration.
Comparison with another Modified MimicPlay (MimicPlay-M). In this experimental setting, we removed the wrist image an input during the low-level training stage, since the wrist image is not easy to obtain for goal specification. For the test-time evaluation process, we provided the pretrained model with prompt videos in HDF5 format generated by MimicGen. Please note that this experimental setup slightly differs from the one in our baseline setting. In contrast to our approach, MimicPlay-M uses an extended observation length of 10 frames instead of 2, and utilizes a sequence of visual frames as goals, rather than single goal image configuration.
Efficiency Analysis. We evaluate the computational efficiency by measuring the average inference time per action step during deployment. Table 10 presents the average inference latency per step across different methods, measured over 40,000 steps on a single NVIDIA RTX 3090 GPU. Our proposed Ge))e)) demonstrates superior computational efficiency while maintaining state- of-the-art performance. This efficiency stems from our novel approach of dynamically generating weights for lightweight target policy. Specifically, Hyper- Goger-GGalNet(G) generates suitable set of policy weights at the beginning each rollout based on the goal image. These weights remain fixed throughout the execution, eliminating the need for repeated weight generation and thus significantly reducing the computational overhead during deployment.
Table 10: Average Inference Time Per Step
Method
GCBC
Play-LMP
C-BeT
Hyper-GoalNet Hoper-Go)lNer(G)
Time (ms)
15.47
22.78
13.61
6.33
1.46
C
Comparison with Other Hypernetworks
This section provides additional details on the comparison between our proposed hypernetwork architecture and HyperZero, particularly regarding computational requirements.
C.1 Training Time & Memory Requirements
To ensure fair and direct comparison, all experiments were conducted single NVIDIA RTX 4090 GPU. The training hyperparameters, including batch size, learning rate, and optimizer settings, were kept identical for both our method and the HyperZero baseline. The only modification was the hypernetwork architecture itself. This controlled setup ensures that any observed differences in resource consumption directly attributable the design of the hypernetwork module.
As detailed Table [T] our method requires slightly more resources than HyperZero in terms of per-epoch training time and memory usage. However, this modest computational overhead is coupled with the substantial performance gains documented the main paper, highlighting the efficiency and effectiveness of our architectural design.
19

---

Table 11: Computational resource comparison. The table shows per-epoch training time and memory footprint for our method versus HyperZero under identical hyperparameter settings. "Frozen" and "Unfrozen" refer to the state of the visual encoder.
Metric
HyperZero
Ours
Training Time Epoch
~0ss
1104
Memory (Frozen Encoder)
3,038 MB
4,916 MB
Memory (Unfrozen Encoder)
13,452 MB
14,844 MB
D Policy Learning Details
Overview. A conventional sequential docision-makingg problem can be formalized as discrete- finite Markov decision process (MDP) defined by a 7-tuple M = (O,A,, Po: 7: H), where:
denotes the observation space,
A represents the action space,
O x Ax R2 defines the transition probability distribution,
7 E [0, is the discount factor,
H specifies the temporal horizon of the process.
In the context of imitation learning, we define a complete state-action trajectory as 7 (on, ot; 0r), where the initial state is sampled op po(on), actions generated by the policy a2 ~ TT (- (-))), and state transitions follow o1+1 ~ P(-lo),,)).
Traditionally, the objective in goal-conditioned decision- problems is to identify optimal goal-conditioned policy my that maximizes the expected discounted reward:
(TT) Ey
y'r(o,, ae 0c41lo0)
(11)
However, approach diverges from this conventional framework in several key aspects: Reward- Free Learning: Operating within a behavior cloning paradigm, we lack access to explicit reward signals. Instead, we aim learn goal-specific policy that maps states to optimal actions purely from demonstrations. Goal-Specific Policy Generation: Rather than learning a universal goal- conditioned policy, hypernetwork architecture generates specialized policies for specific goals, conditioned on the current RGB observation and a target goal image. Non-Markovian Extension: We relax the Markovian assumption to incorporate temporal dependencies. The resulting policy formulation becomes:
Tel (aclor--1.0 oz)
(12)
This extended formulation enables the policy to leverage information from context window of length 2, enhancing its capacity to handle complex, cemporally-depender manipulation sequences.
Model Architecture. Our architectural design addresses the challenges of visuomotor manipulation tasks, which require processing of high-dimensional visual inputs rather than simple state-based representations. The architecture comprises several key components integrated handle visual and proprioceptive information effectively. Visual Processing Pipeline: To bridge the gap between nigh-dimensional visual inputs and hypernetwrr processing capabilities, we employ a pre-trained visual encoder (34] to compress RGB images into compact latent representations. This encoder's s training follows a two-phase strategy: Initial phase (first 20 epochs): Parameters remain frozen to establish stable feature representations; Fine-tuning phase: Parameters become trainable to optimize task-specific visual features.
Hypernetwork Configuration. Our hypernetwork uses the HyPoGen architecture 44 with 8 optimization blocks. processes encoded current and goal images to generate the parameters for 3-layer MLP target policy, architecture that can be flexibly extended in depth and width. This
20

---

Ours
uf. at epoch
w/o shaping
start img.
COs. dist
Normalized distance to the goal
6.
M M
6.
0.0
50
100
130
200
250
Time Steps
Figure 5: Normalized distance between current and goal states computed with different latent spaces along policy rollouts.
notlort Genersted Policy
Retiout Perkr
Rollout Deneraned Pollcy
Inttal Ooere
Gool Stana
Gast
coffee d2
threading
Figure 6: Distance to the goal in the latent space along policy rollouts across different manipu- lation tasks. The consistent decreasing trend across diverse tasks demonstrates that our learned latent representations effectively capture the physical progress toward task goals, establishing meaningful correspondence between latent-space distances and real-world task completion.
meta-learning approach enables dynamic policy adaptation based on specified goals while main- taining computational efficiency. Beside the image encoder mentioned above, the action generation incorporates several specialized components: Predictive Model: Implemented as an MLP operating in the compressed latent space, leveraging the reduced dimensionality for efficient dynamics modeling, Proprioceptive Encoder: compact MLP processes low-dimensional proprioceptive states, providing essential agent state information, Feature Integration: Temporal image features concatenated with proprioceptive information at each timestep, Target Policy: A lightweight MLP processes the integrated features generate appropriate control actions. This architecture efficiently handles the complexity of visuomotor tasks while maintaining computational tractability through dimensionality reduction and feature integration.
Training Details. Our framework implements an end-to-end training paradigm with controlled experimental conditions for reproducibility. Using fixed random seed, we partition the dataset into 950 training and 50 validation demonstrations across all tasks. Training employs a batch size of 256 and the Adam optimizer with an initial learning rate of 5 x 10 coupled with cosine annealing for learning rate decay. The model trains 500 epochs without weight decay or dropout regularization in the hypernetwork component. The training and evaluation procedures were performed on single NVIDIA GeForce RTX 3090 or RTX 4090 GPU. ensure fair comparison, methods evaluated in our experiments were trained using this identical configuration.
21

---

Figure 7: The real robot workspace.
E
Ablation Studies
We conducted ablation experiments to evaluate the impact of various methodological choices, with quantitative results presented in Table Figure and Jilustatil the normalized distance between current and goal states under different experimental configurations. Our analysis includes several key variations:
uf. at epoch O. Full parameter unfreezing from epoch 0, where all model components trainable from initialization.
w/o shapping. Removal latent shaping technique detailed in Section
dist img. Alternative distance computation between current and start images, rather than current and goal images.
cos. dist. Implementation of cosine distance metric in place of Euclidean distance.
C-Bet(w/ shaping). We implement C-Bet and incorporate our proposed additional shaping method while maintaining the same experimental setup.
These systematic variations enable us to quantify the contribution of each design choice to the overall system performance.
Real Robot Experiment Setting
Real Robot Platform. All real-world experiments were conducted RealMan RMC-DA dual-
arm manipulator, which comprises two 7-degree-of-freedoo robotic arms, each rated for a 5 kg payload and fitted with parallel-jaw gripper. While the platform supports bimanual manipulation, this work focuses specifically on evaluating Tlyper-GoalNet's capabilities in single-arm tabletop manipulation tasks. The extension bimanual manipulation remains as future work. The robot mounted in front of a 1.25 m x 0.75 m tabletop, which serves as the exclusive workspace for all manipulation tasks. A standardized (e.g., cubes) to more complex shapes-in placed on the table according to predefined configurations. And on the tabletop there might be some distractor object. To support perception, we employ an overhead RGB-D sensor (Intel RealSense D435i). The entire workspace is shown in Figure
of test objects-ranging from simple geometric primitives
Task Details. To validate the versatility and robustness of our method across broad spectrum of manipulation skills, selected four representative tabletop tasks. Each task emphasizes a different
22

---

(a) Stacking
(b) Pick-and-Place
(c) Drawer Pulling
(d) Sweeping
Figure 8: Key phases of diverse manipulation tasks in real robot experimental evaluation.
core competency-objeee localization, precision grasping, and surface contact manipulation and is defined as follows:
Pick-and-Place. The robot must perceive and localize specified cubic object within the workspace, plan collision-free trajectory, execute stable grasp, and transport the object to predefined target location (e.g., a plate). Success measured by the accuracy of the final placement and the repeatability across trials.
Stacking. Extending the sock-and-place paradigm, this task requires the robot to grasp a source cube, position it directly above target cube resting the tabletop, lower it until gentle contact is detected via proximity or vision-based cues, and then release to complete the stack. Success is defined by the source cube being neatly aligned atop the target cube.
Drawer Pulling. The robot must detect drawer handle, plan an approach to engage the handle with its gripper. and execute a controlled pulling maneuver extend the drawer along its linear guide. Performance is evaluated by the final extension distance achieved without stalling.
Sweeping. The end-effector is equipped with broom attachment. The robot must locate and gather a target object scattered on the tabletop, then sweep it into a designated collection zone (e.g., dustpan). Success is defined by the target object being fully contained within the collection zone at the end of each trial.
Data Processing and Observation Space. Our real-robot evaluation follows the same protocol as in simulation: at each timestep. the policy receives the two most recent observations and single goal image. We collect approximately 70-100 human teleoperation trajectories per task for training. Vision is acquired with an Intel RealSense D435i depth camera: we concatenate its depth channel with the RGB channels form 4-channel images of resolution 128 128, which are normalized to 10 before input to the encoder. Since explicit end-effector poses are unavailable, we represent the current proprioceptive state by the previous action comprrsii seven joint angles and one gripper command- resulting in an 8-dimensional vector. All modalities are synchronized identically to the simulation setting, ensuring a seamless transfer between simulated and real-world experiments.
G Additional Visualization
provide additional visualization of the shaping. Figure and Figure 10 demonstrate the ap- proximate monotonic trend of the distance between the current state and the goal state is consistent across different task scenarios. This consistent pattern substantiates the robustness of our shaping mechanism and validates its task-agnostic applicability.
23

---

Rollout Generated
Rofout Generated Pollcy
Goal State
State
Pumber 0TSNpA
Numberd
(o) Coffee d0
(b) Coffeee
Rellout Conenated Policy
Rtolout by Gemerated Policy
Indial
Intta State
Oizano wuth
(c)
Mag dleamup
Rellout Generated Policy
Rollout by Generabed Policy
Inbial State
Inttl State
Mug cleanup
(f) Thres piece assemblly
Intial
Ofte Ond
Sicana Wet Goll
Goel Stute
Gool State
Mumber
Three assimblly
(h) Three assembly d2
Figure Additional visualization: Distance to the goal the latent space along policy rollouts across different manipulation tasks. The consistent decreasing trend across diverse tasks demonstrates that our learned latent representations effectively capture the physical progress toward task goals, establishing meaningful correspondence between latent-space distances and real-world task completion.
H Future Work
Building upon our current findings, we identify three primary directions for future research: First, we aim to incorporate multi-goal reasoning to handle complex sequential tasks. Second, we plan to extend our framework by integrating foundation models to develop a more generalized policy generation framework, potentially enabling broader task generalization and enhanced adaptability

---

Rollout
Rollout
Iettl
C
Ciaoao prt Col
batt Ond
Gool State
Geal State
Pumbor 2802
Threading dD
Thrnaddng di
Rollout by Generoted Pollcy
Rollout Generated Polkyy
Gol
(K) Threading d2
(1) hut assembly d0
Rollout Generated Policy
Rellout Generated Policy
5 Col
Wember Stcts
Nambor Stape
(m) Coffiee preparation d0
(r) Coffee properation
Policy
Rollout Policy
intaal State
Intral State
Ooal Mtat
Goel Stute
Kitchan
Additional visualization: Distance to the goal in the latent space along policy rollouts across different manipulation tasks. The consistent decreasing trend across diverse tasks demonstrates that our learned latent representations effectively capture the physical progress toward task goals, establishing meaningful correspondence between latent-space distances and real-world task completion.
Figure 10:
across diverse manipulation scenarios. Third, we plan to combine our parameter-adaptive approach with reinforcement learning reduce dependence demonstration data.
25

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim our contribution in the introduction and method section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss limitations including challenges in long-term planning, further work on scaling up, and multi-goal reasoning design.
Justification:
26

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not does not involve the theoretical part and the proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include implementation details in the experiment section and training details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We don't ton't provide open access to the code now, but the data is publicly available. We will open the source code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c www.ddddeeeooeuummmmccco ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
28

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We show the training and testing details, as well as model architecture in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not show error bars in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention the compute resources in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
29

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In our manipulation tasks, have not encountered any safeguard issues.
Guidelines:
30

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the methods, datasets and models originated from other works are cited in reference.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will release relative assets upon acceptance. We welcome new creation based system.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
31

---

Answer:
[NA]
Justification:
Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The development of our core methods does not incorporate LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy (https://neurr) http:///eeerrpssscceeeeeeeeeeeee /ttp://o//oeeee//////////////// for what should or should not be described.
32


Reviews:
[{"review_id": "BpY2B3fzHc", "reviewer": "Reviewer_bvF4", "summary": "In this work, the authors introduce a novel framework to learn goal-conditioned policies by separating the goal conditioning from the state parameters. Specifically, the approach uses hypernetworks to generate target policy parameters based on the goals. This is achieved by shaping the latent space to maintain the predictability of future states through a learned dynamics model and preserving physical relationships through distance constraints so that there is a monotonic progression towards goals. Through experimental validation, the authors show that their hypernetwork-based framework outperforms baseline goal-conditioned methods that use fixed network parameters. Similarly, the authors conduct ablation studies to compare the hypernetwork architecture and latent space shaping while also testing their approach on a real robot.", "questions": "Why do many of the goal-conditioned methods have a 0.00 success rate across all three difficulty levels? Is there a more simplified benchmark where they have at least some successes? The same idea applies to the ablation study with Hyperzero, as it seems to have achieved 0 successes overall.", "limitations": "Yes, the authors adequately address the limitations, though it is mainly in the appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper discusses a novel concept in detail with both theoretical and experimental justifications for each of the design choices for the Hyper-GoalNet framework, including the choice of the hypernetwork architecture as well as the latent space shaping approach. The experiment setup includes a robotics benchmark, ablation studies and evaluations on a real robot, not to mention the additional benchmarks in the appendix. The paper is also well-written and easy to understand. Lastly, the results show significant improvements over the baselines, highlighting the success of the approach. The paper could discuss the limitations in greater detail for the final discussion section instead of only describing it in the appendix. In addition, the experiment task variety or difficulty should be adjusted (particularly for the hypernetwork architecture ablation study) in order to have a more detailed comparison, as a 0% success rate across all tasks for Hyperzero.", "was_revised": true, "final_justification": "My concerns regarding the experiment results have been resolved in the detailed rebuttal by the authors. I am going to keep my original score of accept.", "rebuttal": "Dear Reviewer bvF4,\n\nWe would like to sincerely thank the reviewer for their insightful feedback and positive assessment of our work. We are particularly grateful for your recognition of several key strengths, including the novelty of our concept with its detailed theoretical and experimental justifications, the comprehensive experiment setup spanning from benchmarks and ablation studies to real-robot evaluation, and the clarity of the paper as being 'well-written and easy to understand.' We are also encouraged that you highlighted the significant improvements our method achieves over baselines. We have carefully considered your comments and address your questions below.\n\n***\n>**W.1 & Q.1** The need for a more detailed comparison for HyperZero and other baselines.\n\n**A. to W.1 & Q.1:**\nThank you for this critical question. This point touches upon the core motivation for both our work and our experimental design.\n\nFirst, as you noted, the manipulation tasks we focus on are intentionally **challenging**. The failure of some baselines on these complex tasks is precisely what motivates the need for a more robust architectural paradigm like our Hyper-GoalNet.\n\nTo provide the more detailed comparison you suggested, we have conducted further analysis:\n\n**1. Analysis of HyperZero:**\n\nIn our original experiments, we directly adopted the official architecture of HyperZero for our tasks to ensure a fair comparison. The difference from our method was the hypernetwork architecture itself; all other settings, including the target policy network (a lightweight 3-layer MLP) and hyperparameters, were kept identical. To maintain fairness, we initially avoided any method-specific tricks like special initializations.\n\nHowever, inspired by your feedback, we revisited the HyperZero ablation with an improved initialization trick. We attribute HyperZero's training instability to an inherent limitation in its design: its approach of directly generating an entire set of network weights makes the training process fragile and highly dependent on tricks like initialization to place the initial weights in a suitable distribution. In contrast, our method is inspired by the process of iterative optimization (akin to gradient descent), which allows for more efficient use of the hypernetwork's capacity and yields better-structured target parameters.\n\nTo stabilize HyperZero, we therefore introduced a learnable scalar that acts on the generated weights, effectively placing them in a suitable initial range. We found this technique stabilizes its training process, allowing the loss to converge to a lower value (around 0.14-0.18, though still not as low as our method). This indeed leads to non-zero success rates. The updated results on the coffee task provide the more detailed comparison you were looking for:\n\n|Task|HyperZero (New)|Ours|\n|-|-|-|\n|coffee (d0)|0.16|0.94|\n|coffee (d1)|0.18|0.76|\n|coffee (d2)|0.14|0.62|\n\nThese new results offer a more nuanced view, confirming that while HyperZero can be improved,our proposed architecture still significantly outperforms it across all difficulty levels. This reinforces our claim that our specific hypernetwork design and latent space shaping are critical for success.\n\n**2. Analysis of GCBC and Play-LMP:**\n\nThe challenges with these baselines are multi-faceted:\n\n* **Implementation:** The original authors for these methods did not provide open-source code. We therefore relied on third-party GitHub implementations, making minimal modifications only to align with our experimental setting (e.g., optimizer, learning rate) while preserving the core algorithm. It is noteworthy that in the experiments of a recent paper (e.g. C-BeT), GCBC and Play-LMP also demonstrated near-zero success on similar complex tasks, which corroborates our findings.\n* **Fundamental Methodological Limitation:** After careful inspection, we identified a key technical reason for their failure on our tasks. Both GCBC and Play-LMP model policy learning as a maximum likelihood estimation problem, optimizing the log-probability (log_prob) of actions from a learned Gaussian distribution. We observed that optimizing via probability is more prone to **overfitting** on the training dataset compared to direct regression with a distance loss (e.g., MSE). This was empirically validated: both methods achieved very low training loss, but their validation loss failed to converge to a low value, indicating poor generalization.\n* **Observed Failure Mode:** In practice, this overfitting prevents them from succeeding in novel test scenes with variations. While they could learn a coarse trajectory, they consistently failed at the fine-grained, precise movements required for successful grasping and placing, which are critical for task completion.\n\nTo further strengthen our baseline validation, we have also added comparisons to additional state-of-the-art baselines, as detailed in our response to Reviewer EGhA (W2). The new results are summarized below:\n\n|Method|Coffee d0|Coffee d1|Coffee d2|Mug d0|Inference Type|\n|-|-|-|-|-|-|\n|Diffusion Policy (DP)|**0.94**|0.74|0.44|0.68|Iterative|\n|Goal-Conditioned DP|0.92|**0.78**|**0.66**|0.74|Iterative|\n|**Hyper-GoalNet (Ours)**|**0.94**|0.76|0.62|**0.78**|**Single Pass**|\n\nAs these results show, our Hyper-GoalNet achieves performance that is highly competitive with, and on some tasks superior to, these powerful diffusion-based models. More importantly, our method accomplishes this in a single inference pass, whereas diffusion-based policies require a much more computationally expensive iterative denoising process. This highlights a significant practical advantage of our approach in terms of efficiency and suitability for real-time applications.\n\nIn summary, we believe the failures of the initial baselines are not superficial but stem from fundamental limitations when faced with complex, long-horizon manipulation tasks. The new results for HyperZero provide the granular comparison you requested, and these additional comparisons against diffusion models further validate the effectiveness and efficiency of our proposed architecture. We will add this detailed discussion and the new results to the final version of the paper."}, {"review_id": "DJNEU1yiKj", "reviewer": "Reviewer_uJn1", "summary": "This paper proposes a novel network architecture for goal-conditioned policy learning, where the goal is specified via an image. The central contribution is the use of a hypernetwork that generates the policy network’s weights conditioned on the goal image. The approach is evaluated through behavioral cloning on a set of manipulation tasks, originally introduced in the MimicGen benchmark. Additionally, the authors report results on a real robot, further demonstrating the feasibility of their method.", "questions": "- Can the authors clarify in which of the presented tasks the goal image provides information not already present in the current state? In the absence of ambiguity, what is the motivation for using a goal-conditioned architecture?\n\n\n- Could the authors provide videos of the real robot experiments to support their results and enable a better qualitative assessment of the learned policies?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper is clearly written and easy to follow. Figures are well-designed and contribute to the reader’s understanding. I love figure 1 & 2, both are very well designed and executed.\n- The proposed architecture, involving a hypernetwork for goal-conditioning, appears to be novel in the context of visual goal-conditioned policies and robot manipulation. While I am not deeply familiar with all related work in this subdomain, the architectural choice seems original and well-motivated.\n- The experimental design strikes a good balance between simplicity and realism. The chosen tasks are tractable but still non-trivial, providing a reasonable testbed for evaluation.\n- The inclusion of real robot experiments is a strong point and adds practical credibility to the method, though more details would be welcome (see suggestions below).\n\n\n## Weaknesses\n\n1. **Necessity of Goal Conditioning**  \n   It is unclear whether the tasks presented in the paper truly require goal conditioning. In many of the simulated scenarios, the current observation already appears to encode sufficient information to determine the correct action. Goal images are typically most useful when the task involves ambiguity or requires disambiguating between multiple possible outcomes. As such, the relevance of the goal-conditioned architecture in these settings is somewhat questionable.\n\n2. **Venue Fit**  \n   While the work is technically sound and includes real-world experiments, its primary contributions lie at the intersection of robotics and representation learning. As such, it might be more naturally aligned with a conference like CoRL. That said, the paper could still be of interest to the NeurIPS community, particularly those working on goal-conditioned policies and representation learning for control.", "was_revised": true, "final_justification": "See my original review. Both of my main weaknesses were not resolved (and honestly also cannot be resolved within a rebuttal). I am going to keep my original score of weak accept", "rebuttal": "Dear Reviewer uJn1,\n\nThank you for your insightful feedback and for recognizing the strengths of our work. We are grateful for your positive comments on the paper's clarity, the novelty of our proposed architecture, and the practical value of our real-world experiments. Your thoughtful assessment is much appreciated. We have carefully considered your questions and concerns, which we address point-by-point in the following.\n\n***\n>**W.1 & Q.1** Necessity of Goal Conditioning\n\n**A. to W.1 & Q.1:**\nWe thank the reviewer for this insightful question regarding the motivation for goal-conditioning in our chosen tasks. Our primary motivation for using a goal-conditioned architecture is not merely to solve a single task, but to learn a general and versatile policy that can be directed to achieve a wide spectrum of goal states, including critical intermediate steps of a task.\n\n1. Motivation Beyond Ambiguity: Learning a General Skill\n\nThe core advantage of our approach lies in its ability to generalize. Instead of learning a reactive policy that simply maps a state to an action for one specific outcome (e.g., \"put the lid on the coffeemaker\"), our goal-conditioned policy learns a much more fundamental and reusable skill: \"given a goal image, manipulate the environment to match that image.\" This allows the policy to:\n\n- **Target any valid goal state**, not just the final one from the training data.\n- **Understand and achieve intermediate goals**, demonstrating a deeper comprehension of state progression rather than just overfitting to successful final trajectories.\n\n2. Autonomous Goal Completion Detection\n\nA direct and powerful capability stemming from this architecture is autonomous goal completion detection. Our policy continuously compares the latent representation of the current observation with that of the goal image. When the latent distance falls below a threshold, the policy intrinsically \"knows\" the task is complete and can terminate. This capacity for self-assessment is impossible for a non-conditioned policy and is a key feature of our method, which we qualitatively illustrated in Figures 3, 10, and 11.\n\nTo quantitatively validate this capability, we performed a new analysis comparing our policy's autonomous success detection (Auto SR) with the environment's ground-truth success signal (Env SR). We introduce two key metrics:\n- **Accuracy**: The agreement rate between our method's decision and the ground truth.\n- **Recall**: Our method's ability to correctly identify all true successes. We emphasize recall as it is crucial for a policy not to miss a completed task and continue acting unnecessarily.\n\nThe results for the Coffee tasks are presented below:\n\nTable: Quantitative Results for Autonomous Goal Completion Detection\n| Task | Auto SR | Env SR | Accuracy | Recall |\n|-| :---: | :---: | :---: | :---: |\n|D0|0.96|0.94|94%|98%|\n|D1|0.78|0.76|90%|95%|\n|D2|0.74|0.62|76%|90%|\n|Mean|-|-|86.6%|94.3%|\n\nThe results show a strong alignment with the ground-truth environment signal, achieving an average accuracy of 86.6% and, most importantly, a high average recall of 94.3%. This confirms that our latent-distance-based mechanism is a reliable method for autonomous completion detection, a critical capability enabled directly by our goal-conditioned design.\n\nRevision Plan: In our revised manuscript, we will:\n\n1. Clarify our motivation in the introduction, explicitly stating that our goal is to learn a generalizable policy capable of achieving a spectrum of goals and performing autonomous completion detection.\n2. Add this new quantitative analysis and table to the experimental section to provide strong empirical evidence for this key capability.\n\nWe believe this clarification and the new results effectively demonstrate that goal-conditioning provides a crucial advantage—generalization and self-awareness—that goes far beyond simply resolving ambiguity in the initial state.\n\n***\n>**W.2** Venue Fit\n\n**A. to W.2:**\n\nWe thank the reviewer for their thoughtful consideration of our paper's positioning. We agree that our work has strong connections to the robotics community, as the real-world validation is a crucial component of our research.\n\nHowever, as the reviewer insightfully points out, the primary contributions of our paper are deeply rooted in machine learning topics that are of broad interest to the NeurIPS community. Our work focuses on:\n\n1. Representation Learning for Control: At its core, our paper investigates how to learn effective representations from high-dimensional visual inputs (goal images) to condition policies.\n\n2. A Novel Generative Architecture: Our central contribution is the use of a hypernetwork to generate the parameters of an entire policy network. This positions our work within the broader study of generative models, exploring how one network can produce the weights for another.\n\nWe believe that robotics provides a challenging and meaningful testbed for these fundamental ML concepts. Proving the effectiveness of our representation learning and generative architecture on complex, real-world manipulation tasks demonstrates their practical viability and robustness.\n\nTherefore, given this focus on goal-conditioned representation learning and generative hypernetwork architectures, we believe that our paper is well-aligned with the interests of the NeurIPS community and will be of significant value to the community.\n\n***\n>**Q.2** Real Robot Experiment Videos\n\n**A. to Q.2:**\nWe thank the reviewer for this suggestion. We agree that videos are essential for qualitative assessment and have prepared them to be shared via a project website, which we will link in the camera-ready version.\n\nDue to NeurIPS policy regarding anonymity, we cannot provide the link during the rebuttal period. In the interim, we would direct the reviewer to our Appendix (specifically the \"Real Robot Experiment Setting\" section), which details our hardware, setup, and procedures. We look forward to sharing the full videos, along with all code and experimental details, upon the paper's acceptance."}, {"review_id": "dpbXKzeLSf", "reviewer": "Reviewer_EGhA", "summary": "This paper introduces Hyper-GoalNet, a novel framework for goal-conditioned manipulation policy learning that uses hypernetworks to dynamically generate task-specific policy parameters based on goal specifications. Unlike conventional approaches that concatenate goal and state information as input to fixed-parameter networks, this method treats goals as specifications that determine how current observations should be processed. The approach is evaluated on multiple manipulation tasks in simulation and real-robot experiments.", "questions": "If I understand it right, the method only predicts the action of the current timestep, so why don’t the authors apply some techniques such as action chunking?", "limitations": "The authors have discussed the limitations of the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The core idea of using hypernetworks to generate goal-specific policy parameters is innovative and well-motivated.\n- The paper includes extensive experiments across multiple manipulation tasks with varying difficulty levels, thorough ablation studies, and real-robot validation.\n- The paper is well-written and easy to follow.\n\n**Weaknesses:**\n\n- The method is evaluated exclusively in single-view camera configurations. The authors do not demonstrate how their approach scales to multi-view camera setups, which are standard in most robot learning systems. Given that the method incorporates a latent shaping module, scaling to multiple cameras may introduce additional complexity and learning challenges.\n- The experimental baselines appear outdated and insufficient for comprehensive evaluation. For instance, the authors use C-BeT [1] but omit its more recent variant VQ-BeT [2]. Additionally, diffusion policies [3, 4], which have gained significant adoption in recent robot learning research, are not included in the comparison. While the authors' method employs behavior cloning loss, making direct comparison with diffusion-based approaches potentially unfair, they should clarify how their method could extend to other policies and justify the exclusion of diffusion-based baselines to strengthen their evaluation.\n- The real robot experimental results raise concerns, as all baseline methods exhibit surprisingly poor performance. Given that these appear to be standard robot learning tasks, the authors should clarify what specific aspects make these tasks challenging and explain the underlying factors contributing to the baselines' poor performance.\n- I am also concerned about the scaling ability of the method. With the number and complexity of tasks increasing, relying on the current observation and specific goals to predict the parameters of the policy could be challenging.\n---\n[1] Cui, Zichen Jeff, et al. \"From play to policy: Conditional behavior generation from uncurated robot data.\" ICLR'23.\n\n[2] Lee, Seungjae, et al. \"Behavior generation with latent actions.\" ICML'24.\n\n[3] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" RSS'23.\n\n[4] Reuss, Moritz, et al. \"Goal-conditioned imitation learning using score-based diffusion policies.\" RSS'23.", "was_revised": true, "final_justification": "Based on the authors' rebuttal, which provided clarifications on experimental protocols and included new results with diffusion-based policies and video goals, I am increasing my assessment to a score of 4.\n\nHowever, I maintain reservations regarding the benchmark comparisons. The baselines used in the paper are relatively old. While the authors have included results from diffusion policies in the rebuttal, these perform similarly to their proposed method. It's unclear how much improvement it will bring in diffusion-based policies. I believe it would be crucial to include a comprehensive comparison between a diffusion-based Hyper-GoalNet and a pure diffusion policy.", "rebuttal": "Dear Reviewer EGhA,\n\nThank you for your constructive feedback and for recognizing the strengths of our paper. We are grateful for your positive comments on our core idea being \"innovative and well-motivated,\" our \"extensive experiments,\" and the paper's clarity as \"well-written and easy to follow.\" We have carefully considered your concerns and address your comments and questions in the following.\n\n***\n>**W.1** Single-View Camera Configuration\n\n**A. to W.1:**\nWe thank the reviewer for this question about camera configurations. Our focus on a single-view setup is a deliberate choice, as we argue it is essential for achieving real-world deployability and generalization. Also, a key to generalization in robotics is the ability to train on massive and diverse datasets. While multi-view systems are information-rich, their setup complexity present significant barriers to this kind of large-scale data acquisition:\n1. **Barrier to Scale**: The need for complex calibration and the high deployment cost make it challenging to gather data from hundreds of different environments, fundamentally limiting data diversity.\n2. **Barrier to Simplicity**: Specifying geometrically consistent goals across multiple views adds significant complexity to the data collection and annotation pipeline, further hindering scalability.\n\nIn contrast, a single-view approach is far more scalable. It lowers the barrier for massive data collection (potentially including heterogeneous web data) and enables deployment in novel, un-instrumented environments. We argue that forcing the model to learn from this constrained but realistic input fosters more robust and generalizable representations, paving a more promising path towards agents that can operate \"in the wild.\"\n\nMoreover, the Hyper-GoalNet framework is flexible. Its core mechanism—generating policy parameters from a goal—is agnostic to the vision encoder. Features from multiple cameras could be readily integrated (e.g., via attention) and processed by our latent shaping module. This sensor fusion is a feasible extension but is orthogonal to our primary contribution.\n\nRevision Plan: In the revised manuscript, we will clarify the motivation for our single-view focus, emphasizing its benefits for generalizability and scalability, and discuss multi-view extensions in the future work section.\n\n***\n>**W.2** Additional Experimental Baselines\n\n**A. to W.2:**\nWe thank the reviewer for this valuable suggestion. To strengthen our evaluation, we have performed new experiments on some tasks against a state-of-the-art **Diffusion Policy (DP)** and a strong **Goal-Conditioned DP** baseline. \nRegarding VQ-BeT, we also worked to include it as a baseline. We endeavored to follow the official implementation, but encountered significant implementation challenges (e.g., the official, (only) image-based model failed to converge, and our efforts to stabilize its VQ-VAE pre-training stage for our high-precision visuomotor data proved non-trivial), preventing us from obtaining stable results within the rebuttal period. We are continuing this effort and aim to include it in the final version.\n\nThe new results are summarized below:\n\n|Method|Coffee d0|Coffee d1|Coffee d2|Mug d0|Inference Type|\n|-|-|-|-|-|-|\n|Diffusion Policy (DP)|**0.94**|0.74|0.44|0.68|Iterative|\n|Goal-Conditioned DP|0.92|**0.78**|**0.66**|0.74|Iterative|\n|**Hyper-GoalNet (Ours)**|**0.94**|0.76|0.62|**0.78**|**Single Pass**|\n\nThese results highlight a crucial trade-off between performance and efficiency. To provide a robust comparison, we benchmarked against a state-of-the-art Diffusion Policy (DP) and an even stronger Goal-Conditioned DP (GC-DP), which leverages the goal image via concatenation. While powerful, both diffusion baselines require a computationally intensive, iterative sampling process at inference time.\n\nIn contrast, our method is a single-pass, feed-forward network, making it significantly more efficient. Despite this key efficiency advantage, the results show our method achieves performance comparable to the strong, iterative GC-DP baseline.  We attribute this compelling result to our core contribution: the hypernetwork's ability to directly generate high-quality, specialized parameters from the goal, bypassing the need for iterative refinement.\n\nIn addition to these policy-level comparisons, we have also added new baselines analyzing the hypernetwork architecture itself, the details of which can be found in our response to Reviewer nbKf's Question 5 (a) and (c).\n\nRevision Plan: We will add this discussion and the new results into our revised manuscript, which we believe substantially strengthens our paper's evaluation.\n***\n>**W.3** Real Robot Performance Gap\n\n**A. to W.3:**\nWe thank the reviewer for this critical question. The significant performance gap on the real robot warrants this deeper explanation. The tasks require high precision, which penalizes policies that are not robust to the noise and observation shifts inherent to the real world. Our analysis points to two distinct failure modes for the baselines:\n\n1. GCBC & Play-LMP: Overfitting from Likelihood-based Training. These methods failed due to their likelihood-based training objective (i.e., maximizing action log_prob), typically under a Gaussian assumption, which is more prone to overfitting than direct regression (e.g., MSE). We observed this overfitting empirically, as their validation loss failed to converge converge to very low values despite low training loss, indicating a poor generalization capability. Similar failure cases also occurred in simulation, where these overfitted policies could only reproduce coarse trajectory motions and consistently failed at the crucial, high-precision steps. This weakness was naturally amplified on the real robot, where noise and observation shifts are unavoidable.\n2. C-BeT: Lack of Robustness to Compounding Errors. For C-BeT, the issue is more nuanced. Its powerful Transformer architecture excels at fitting the in-distribution training data. However, it struggles with robustness. In the real world, a small amount of sensor noise can push the robot into a slightly out-of-distribution (OOD) state. A fixed-parameter model like C-BeT makes a small prediction error, which leads to a subsequent state that is even further from the training distribution. This triggers a cascade of compounding errors throughout the trajectory, ultimately leading to task failure.\n\nWhy Hyper-GoalNet Succeeds:\n\nOur method is inherently more robust to these challenges precisely because it does not rely on a single, fixed-parameter network. Instead, Hyper-GoalNet dynamically generates the parameters of the policy network on-the-fly, conditioned on the current (and potentially noisy) observation and the specific goal. This mechanism acts as a form of rapid, instance-specific adaptation. The ability to generate and refine high-quality parameters makes the resulting policy far more resilient to observation noise and helps prevent the accumulation of minor errors.\n\nRevision Plan: We will add a detailed discussion of these distinct failure modes and the underlying reasons for our method's robustness to the appendix of our revised manuscript. Thank you for prompting this deeper analysis.\n***\n>**W.4** Scaling Ability\n\n**A. to W.4:**\nWe thank the reviewer for raising this forward-looking question about the scalability of our method. We agree that scaling to a larger number and greater complexity of tasks is a crucial long-term goal for any manipulation policy.\n\nOur current work focuses on establishing the core principle of Hyper-GoalNet: dynamically generating policy parameters from a goal specification (i.e., a goal image) for a lightweight MLP policy. We view this as a foundational proof-of-concept. However, the framework itself is inherently designed for scalability. We believe the key to scaling is not to simply increase the capacity of a single, fixed model, but to enhance the components of our modular system. Specifically, our hypernetwork architecture is compatible with:\n\n1. More advanced guidance: Replacing the single goal image with richer goal specifications, such as videos.\n2. More powerful target policies: Generating the parameters for more advanced network architectures beyond a simple MLP, such as a Diffusion Policy.\n\nTo validate this potential, we conducted a experiment on scaling a single Hyper-GoalNet architecture across four distinct manipulation tasks. In this setup, we equipped the hypernetwork with a more powerful diffusion-based target policy and used video demonstrations as goal information. The success rates across the subtasks were as follows:\n\n|Task|Subtask 1|Subtask 2|Subtask 3|Subtask 4|\n|-|-|-|-|-|\n|Scaled HyperNet|~25/25|~25/25|~25/25|~24/25|\n\nPlease note that this was an initial experiment to test the feasibility of scaling. The results are highly promising, demonstrating that when equipped with more advanced components, the HyperNet framework can achieve very strong performance across multiple, diverse tasks. This provides compelling evidence for the scalability of our approach.\n\nRevision Plan: We are very excited by these initial findings and will make scaling a central focus of our future work. We will add a discussion of this to the paper to highlight the future potential of our framework.\n***\n>**Q.1** Action Chunking\n\n**A. to Q.1:**\nThank you for this insightful question. We apologize for the omission. In fact, we did employ action chunking in all of our experiments. Our policy predicts a short sequence of future actions at each timestep, and the agent executes this chunk sequentially before the policy is invoked again.\n\nRevision Plan: We omitted this detail for brevity to maintain focus on our core contribution, the Hyper-GoalNet framework itself. We agree this is an important implementation detail and will add it to the experimental setup section in our revised manuscript. Thank you for pointing this out."}, {"review_id": "BdPbnHKcSp", "reviewer": "Reviewer_nbKf", "summary": "This paper introduces Hyper-GoalNet, a framework that uses hypernetworks to generate\ntask-specific policy network parameters from goal specifications for robotic manipulation.\nThe authors also propose latent space shaping techniques using forward dynamics modeling\nand distance constraints to enhance performance. Experiments are performed on 6\nRobosuite tasks, and comparisons are made with several baselines. Ablation studies on the\nuse of a different hypernetwork (HyperZero) and the effectiveness of the two new latent\nspace objectives are performed. The proposed approach is further tested on a real world\nrobot, leveraging four evaluation tasks.", "questions": "1.​ Why do GCBC, Play-LMP, and HyperZero achieve 0% success across all tasks and\ndifficulty levels? Please provide: (a) implementation details for each baseline, (b)\nverification that the original papers' results can be reproduced, (c) hyperparameter\nsearch procedures used, and (d) any debugging steps taken. If technical errors were\nfound and fixed, I would reconsider the quality score. Without proper baseline\nvalidation, the claimed improvements cannot be trusted.\n\n2.​ How does your approach differ from Beck et al. [1]'s hypernetwork-based meta-RL\nmethod? Your claim that \"hypernetwork-based RL approaches cannot be directly\nadapted to goal-conditioned behavioral cloning\" (lines 88-90) is contradicted by your\nown straightforward adaptation of HyperZero. Please clearly explain: (a) what makes\nyour adaptation novel compared to [1], and (b) why a comparison with [1] is not\nincluded. Adding this comparison or clearly establishing novelty would address my\noriginality concerns.\n\n3.​ Can you provide complete architectural and training details for reproducibility?\nSpecifically: (a) exact policy network architecture used for all methods, (b) complete\nhypernetwork architecture beyond Eq. 4, (c) the loss function ℓ in Eq. 5, (d) training\nprocedure including the R3M unfreezing strategy, (e) hyperparameter selection\nprocess, and (f) training time/memory requirements. Providing these details would\nimprove the clarity score and enable verification of results.\n\n4.​ How do you verify the claimed \"natural goal completion detection\" capability? You\nclaim this as a key advantage (lines 222, 313) but use environment-provided terminal\nsignals in experiments. Please show experiments where: (a) your method\nautonomously detects task completion using latent distances, (b) comparison with\nenvironment ground truth, and (c) success/failure rates using autonomous vs.\nenvironment detection. Demonstrating this capability would strengthen the\nsignificance of your contribution.\n\n5.​ What causes HyperZero's training instability (Figure 5) and how did you attempt to\naddress it? The loss curves suggest HyperZero never properly converges. Did you\ntry: (a) different learning rates or optimization strategies, (b) architectural\nmodifications, (c) different initialization schemes? Understanding whether this is an\ninherent limitation of HyperZero or an implementation issue is crucial for fairly\nevaluating your approach's advantages.\n\nAddressing questions 1 and 3 satisfactorily would improve Quality and Clarity scores.\nResolving question 2 would improve Originality. Demonstrating question 4 would enhance\nSignificance. A thorough response to all questions could potentially change the overall\nrecommendation from Reject to Borderline Accept/Accept.", "limitations": "The authors discuss some of the major limitations of their approach.They might add a\ndiscussion on failure modes when goal-specifications lie outside the training distribution or\nwhen latent distances mispredict completion (if this feature was actually used), which could\nlead to unsafe behaviors.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Originality*\n\nThe proposed latent space shaping technique is simple but principled and it shows benefits\nin the ablation studies. The use of hypernetworks in goal-conditioned behavioral cloning is\nnovel.\nHowever, it is not clear whether the first claimed contribution is actually a novelty. In the\nIntroduction, the authors say “we adapt optimization-inspired hypernetwork architectures for\ngenerating policy parameters conditioned on goal specifications, creating a framework that\ndynamically determines how current observations should be processed”, and again in\nsection 3 “To realize this insight, we leverage hypernetworks to dynamically generate\ntask-specific policy parameters [...] tailored to the specified goals”. While the related work\nsection briefly mentions recent RL approaches leveraging hypernetworks, saying they\ncannot be directly adapted to goal-conditioned behavioral cloning (lines 88-90), the ablation\nstudy in 4.3 compares Hyper-GoalNet to “Hyper-Zero [39], an alternative hypernetwork\narchitecture, under identical training conditions”. The adaptation seems therefore\nstraightforward.\n\n*Significance*\n\nThe use of hypernetworks in goal-conditioned behavioral cloning might be of interest to the\ncommunity, offering a well-motivated novel combination of existing techniques.\nHowever, while the related work section briefly mentions recent RL approaches leveraging\nhypernetworks, important references such as [1, 2] are missing. The relationship with\nprevious hypernetwork-based RL approaches is unclear. The authors should expand the\nrelated work section, explaining why such approaches cannot be directly adapted to\ngoal-conditioned behavioral cloning (lines 88-90), and how they differ from the proposed\nmethod. Otherwise, they should add comparisons to other hypernetwork-based approaches,\nto assess the superiority of the proposed approach. Some concerns regarding the presented\nresults for the chosen baselines raise doubts on the actual impact this work might have on\nthe community (see quality section for detailed comments and suggestions).\n\n*Quality*\nThe paper includes experiments across multiple manipulation tasks (long- vs short-horizon,\n3 difficulty levels) in simulation (Robosuite) and real robot experiments, with ablation studies.\nHowever, the results are not entirely convincing.\n\n●​ In line 222, the authors claim that one of the advantages of Hyper-GoalNet is its\nability to perform “natural goal completion detection”. However, they say they use\n“environment-provided terminal signals” for fair comparisons with the baselines (lines\n278-281). Then, in section 4.3, they claim again that a key advantage of their\napproach is that “it enables reliable autonomous detection of task completion based\non latent distances”. How is this verified?\n\n●​ The discussion of the quantitative results in Tables 1-2-4 is too superficial, simply\nsaying that the proposed approach outperforms the others. However, the baseline\nimplementations seem weakened, with GCBC, Play-LMP, and HyperZero obtaining\n0.0 success rates on all tasks and for all difficulty levels. The authors should say how\nthe baseline methods were re-implemented, how the hyperparameters were chosen,\ncheck for technical errors in their implementation, and carefully justify any disruptive\nresults like the ones that are currently reported. The training curves in Figure 5 seem\nto suggest there are issues in HyperZero’s training, which might be the cause of its\npoor performance.\n\n●​ Success rates are computed over 50 rollouts, but error bars are absent in all\npresented results. This is particularly problematic in Tables 6-8, where some of the\nbaseline approaches have success rates that are closer to the ones obtained by\nHyper-GoalNet.\n\n*Clarity*\n\nOverall, the paper is hard to follow. There are inconsistencies in terminology, missing\ninformation, and the organization can be improved.\n\n●​ The section on “Hypernetwork training” is not very clear. What is ℓ in equation 5?\n\n●​ In section 4.3, the authors compare against HyperZero, which is an alternative\nhypernetwork architecture, never introduced before. The authors should consider\nadding a brief description of such architecture where the other baselines are\ndescribed, in section 4.2.\n\n●​ In all experiments, it is not clear what the architecture of the policy network is and\nwhether it is the same for the baselines. Could performance differences might also\nbe due to the architectural differences?\n\n●​ In lines 313-315, the authors say: “Our training strategy also matters – unfreezing the\nR3M visual encoder [32] only after 20 epochs ensures stable parameter generation.”\nThis aspect was never mentioned before, and the overall training procedure is\nunclear, and the code is unavailable at this time. The authors should consider adding\nthis information in the Methods section.\n\n●​ In line 314, the authors refer to epochs, while in Figure 3 they report time steps on\nthe x axis. Are the two terms referring to the same thing? Try to be consistent with\nthe terminology.\n\n●​ In Table 9, the authors report average inference time per step, and claim the superior\nefficiency of their method. What about training times? No information about the\nmemory and total execution time needed to reproduce the experiments is provided.\n\n●​ Overall, results are not reproducible with the information provided. Details about the\narchitectures of the policy network and the hypernetwork are missing. The training\nprocedure is not clear, and the authors do not specify how hyperparameters were\nchosen.\n\n[1] Beck, J., Jackson, M. T., Vuorio, R., & Whiteson, S. (2023, March). Hypernetworks in\nmeta-reinforcement learning. In Conference on Robot Learning (pp. 1478-1487). PMLR.\n\n[2] Y. Huang, K. Xie, H. Bharadhwaj and F. Shkurti, \"Continual Model-Based Reinforcement\nLearning with Hypernetworks,\" 2021 IEEE International Conference on Robotics and\nAutomation (ICRA), Xi'an, China, 2021, pp. 799-805, doi: 10.1109/ICRA48506.2021.9560793.", "was_revised": true, "final_justification": "I appreciate the authors' detailed rebuttal and additional details on the experiments. However, I remain minorly concerned about the overall clarity and completeness of the paper.", "rebuttal": "We thank the reviewer nbKf for the insightful feedback and for recognizing the novelty of using hypernetworks in goal-conditioned behavioral cloning. Our responses below, supported by new experiments, aim to address your questions (also thanks for organizing the weaknesses into these structured questions) and resolve the primary concerns listed in the 'Weaknesses' section.\nDue to space limitations, we focus on the major points here. All other minor edits and clarifications will be incorporated directly into the revised manuscript.\n***\n>**Q.5** HyperZero Training Instability, Analysis, and Solutions\n\n**A. to Q.5:**\nWe appreciate the question about HyperZero's training instability. We address this first to clarify HyperZero’s issues and highlight our method’s architectural advantages.\n\n(a) Diagnosis and Initial Comparison\nOur initial attempts to stabilize HyperZero with varied learning rates and architectures failed. For the original submission, we avoided specialized initializations to ensure a fair comparison, as our method requires none. We attribute HyperZero’s instability to its architecture, which learns a direct goal-to-parameter mapping that is highly sensitive to initialization. In contrast, our architecture is inherently more robust because it simulates an optimization process, iteratively refining parameters through a series of blocks within a single forward pass.\n\n(b) Enhanced Initialization Schemes for HyperZero\nBased on your suggestion, we tested two initialization schemes:\n- **ScalarInit**: A learnable scalar to scale the hypernetwork’s output, stabilizing training by controlling initial weight scale.\n- **Bias-HyperInit[1]**: Implemented as recommended in [1] for its compatibility with high-dim conditioning inputs.\n\n(c) Comparative Results\nNew results show that these initializations improve HyperZero, but our method remains superior without requiring such techniques.\n\nSuccess Rate (%) on Coffee Tasks with Different Initializations\n|Method|D0|D1|D2|\n|-|-|-|-|\n|ScalarInit|16%|18%|14%|\n|Bias-HyperInit[1]|30%|18%|0% |\n|Ours(no special init)|94%|76%|62%|\n\nThese results confirm the effectiveness of HyperZero’s initialization schemes, yet our Hyper-GoalNet architecture remains superior and more robust.\n\nRevision Plan: We will update Fig. 5 with new training curves and add this analysis to Sec. 4.3.\n\n***\n>**Q.1** Analysis of Baseline Performance\n\n**A. to Q.1:**\nWe thank the reviewer for this insightful question regarding the baselines. Below, we clarify our approach, provide analytical results, and outline revisions.\n\n(a) HyperZero Analysis & Results\n\nFor a fair comparison, we initially used the official HyperZero using the same setup as our method but without special initializations. Following your feedback, we found HyperZero’s training is sensitive to initialization. Applying a specific initialization technique (see Q5 response) significantly improved its success rate. Our method still outperforms, but these results ensure a fairer comparison. \n\nRevision Plan: We will update Tab. 4 with new HyperZero results and add a note detailing the setup.\n\n(b) GCBC and Play-LMP Analysis\n- **Implementation**: Without official code, we used reputable third-party implementations, adapted for our data pipeline with identical core hyperparameters.\n- **Performance Diagnosis**: We verified that the failure stems from design, not implementation errors. We observe that both methods optimize policy via expert action log-probability, leading to overfitting (low training loss, high/stagnant validation loss). This incurs generalization issues to unseen test set variations, especially for high-precision grasping/placing. This finding is consistent with other work (e.g., C-BeT).\nOur hypernetwork design avoids such overfitting by a structural bias that forces the prediction to follow an optimization process instead of just a memorization of the mapping. \n\nRevision Plan: We will add a paragraph to Sec. 4.2 with these details, technical reasons for performance (likelihood-based vs. distance-based loss), and citing corroborating prior work. We have also added stronger baselines (e.g., Diffusion Policy; see response to Reviewer EGhA), which we believe addresses your concerns.\n***\n>**Q.2** Comparison with Hypernetwork-based Meta-RL\n\n**A. to Q.2:**\nWe thank the reviewer for this question, which helps us clarify our work's novelty relative to hypernetwork-based RL methods [1].\n\n(a) Our Novelty and Clarification\n- **Paradigm vs. Architecture**: Our claim that RL methods \"cannot be directly adapted\" refers to the training paradigm. The full RL algorithms of [1, 2] are inapplicable in our reward-less BC setting. We only adapted architectures (e.g., HyperZero), not the learning methods themselves. We will clarify this on lines 88-90.\n- **Architectural Design**: Our method, inspired by iterative optimization, is inherently more stable and robust than methods that directly generate \"sub-optimal\" parameters and rely on initialization fixes like Bias-HyperInit [1].\n- **Task Conditioning**: Our use of high-dim goal images provides superior scalability and generalization over the low-dim one-hot vectors used for discrete tasks in [1], enabling continuous goal specification.\n\n(b) Comparison and Planned Revisions\nWe did not include a comparison due to these fundamental differences(RL vs. BC). However, prompted by your review, we have now benchmarked against a key component of [1]. As shown in our Q5 response, our method significantly outperforms a baseline with Bias-HyperInit[1], confirming our approach's benefits.\n\nRevision Plan: We will:\n1. Distinguish the RL vs. BC paradigms in the introduction\n2. Expand the Related Work to discuss [1] and [2], highlighting their key differences.\n3. Integrate these new results [1] into our main experimental tables.\n\nWe believe these changes will clearly establish our work's novelty and contributions.\n***\n>**Q.3** Architectural and Training Details\n\n**A. to Q.3:**\nWe thank the reviewer for these important questions on reproducibility. We will add to the appendix these details.\n\n(a) Policy Network Architecture\n* GCBC, Play-LMP, & HyperZero: Details are in our response to Q. 1.\n* C-BeT: We adapted its official, state-based implementation for our challenging image-based setting by adding a ResNet-18 encoder. This ensured a fair comparison.\n* MimicPlay-M: We used the official implementation within our identical task setup.\n\n(b) Hypernetwork Architecture\nOur hypernetwork architecture is inspired by [38] and works as follows: First, the goal image is encoded into a goal embedding. This embedding is then fed into a series of hypernetwork blocks. Each block executes a process analogous to a step of Policy Gradient estimation, iteratively optimizing a set of policy parameters as if performing gradient descent. The final, optimized parameters are then applied to a lightweight 3-layer MLP, which serves as the target policy network for predicting actions.\n\nDue to space limitations, we cannot provide a full implementation breakdown. However, we are committed to release our complete codebase upon publication to ensure full reproducibility.\n\n(c) The loss function *ℓ* in Eq. 5 is the Mean Squared Error loss.\n\n(d) Training Procedure\nThe model is trained end-to-end. The process per step is:\n1.  The hypernetwork takes the goal image as input and generates the parameters for the target policy.\n2.  The target policy takes the current observation as input and outputs an action.\n3.  The final loss is a sum of the classic behavioral cloning loss (MSE) and our proposed latent shaping losses.\n\nUnfreezing strategy: To protect the pretrained R3M vision encoder from noisy gradients early in training, we keep the encoder frozen for the first 20 epochs and then finetune it jointly with the model once the policy has stabilized. Crucially, this exact strategy was applied to all baseline methods for a fair comparison.\n\n(e) Hyperparameter Selection\nWe found our iterative, optimization-inspired method to be highly robust to the choice of hyperparameters. To maintain a fair comparison, we performed a light search and set the initial learning rate for all methods to 4e-5, using the Adam optimizer.\n\n(f) Training Time & Memory Requirements\nAll experiments were conducted on a single NVIDIA RTX 4090 GPU and all methods completed training within one day. While our method requires slightly more resources than HyperZero, this is coupled with a substantial performance gain.\n||HyperZero|Ours|\n|-|-|-|\n|Training Time/Epoch|~90s|~104s|\n|Memory(Frozen)|3038MB|4916MB|\n|Memory(Unfrozen)|13452MB|14844MB|\n***\n>**Q.4** Verifying Natural Goal Completion Detection\n\n**A. to Q.4:**\nWhile our original submission qualitatively illustrated this concept of automatic detection(Figs. 3, 10, 11), we used environment signals in our main results to ensure a fair comparison with all baselines.\n\nTo quantitatively validate this capability, we now compare our autonomous detection (Auto SR) with the environment's ground truth (Env SR) using these metrics:\n\n* Auto SR: Success determined by our method's latent distance threshold.\n* Env SR: Success determined by the environment signal.\n* Accuracy: Agreement rate between Auto SR and Env SR.\n* Recall: Auto SR's ability to identify true successes from Env SR, a crucial metric for not missing completed tasks.\n\nResults for Autonomous Goal Completion Detection (Coffee Tasks)\n|Task|Auto SR|Env SR|Accuracy|Recall|\n|-|-|-|-|-|\n|D0|0.96|0.94|94%|98%|\n|D1|0.78|0.76|90%|95%|\n|D2|0.74|0.62|76%|90%|\n|Mean|-|-|86.6%|94.3%|\n\nThis strong alignment with the ground-truth signal, shown by an average accuracy of 86.6% and recall of 94.3%, confirms that our latent-distance-based approach is a reliable autonomous completion detector.\n\n---\nWe thank the reviewer again for these thoughtful questions and hope our clarifications reinforce the value and clarity of our contributions. We would be happy to address any further suggestions or questions you may have."}]

Author rebuttal:

Meta-review: The paper presents Hyper-GoalNet, a framework for goal-conditioned manipulation using hypernetworks to generate task-specific policy parameters from goal specifications instead of concatenating goals with states. The approach is evaluated on multiple manipulation tasks in simulation and real-robot experiments. The proposed architecture, involving a hypernetwork for goal-conditioning seems to be novel in the context of visual goal-conditioned policies and robot manipulation. The paper is well written. The results presented show significant improvement over baselines. One weakness pointed out by one of the reviewers was whether the goal conditioning is needed in the performed experiments. The authors are suggested to consider that seriously and revise their paper accordingly. Overall, a strong contribution.
Decision: Accept (poster)