Hierarchical Value-Decomposed Offline Reinforcement Learning for Whole-Body Control
Authors: Zhilong Zhang
Yunpeng Mei
Xinghao Du
Hongjie Cao
Haonan Wang
Pengyuan Min
Chenyu Wang
Pengfei Chen
Chenbo Xin
Yijie Wang
Wenyu Luo
Yihao Sun
Yidi Wang
Lei Yuan
Gang Wang
Yang Yu
Venue: ICLR 2026 Poster (2026)
Keywords: robot policy learning
offline reinforcement learning
whole-body control
Primary area: applications to robotics, autonomy, planning

Abstract: Scaling imitation learning to high-DoF whole-body robots is fundamentally constrained by the scarcity of expert demonstrations. In contrast, large amounts of suboptimal data are readily available and offer a practical way to alleviate supervision bottlenecks in real-world whole-body control. However, leveraging such data introduces two central challenges: how to extract informative signals from imperfect trajectories, and how to cope with the increased learning complexity induced by high-dimensional control. To overcome this, we propose **HVD** (Hierarchical Value-Decomposed Offline Reinforcement Learning). The offline RL formulation provides principled data selection over suboptimal datasets, enabling the policy to prioritize high-value behaviors while down-weighting harmful ones. Complementarily, hierarchical value decomposition organizes learning along the robot’s kinematic structure, improving credit assignment and reducing learning complexity in high-DoF systems. Built on a Transformer-based architecture, HVD supports *multi-modal* and *multi-task* learning, allowing flexible integration of diverse sensory inputs. To enable realistic evaluation and training, we further introduce **WB-50**, a 50-hour dataset of teleoperated and policy rollout trajectories annotated with rewards and preserving natural imperfections, including partial successes, corrections, and failures. Experiments show HVD significantly outperforms existing baselines in success rate across complex whole-body tasks. Our results suggest effective policy learning for high-DoF systems can emerge not from perfect demonstrations, but from structured learning over realistic, imperfect data. Our code is available at https://github.com/LAMDA-RL/HVD.

Published as a conference paper at ICLR 2026
HIERARCHICAL VALUB-DECOMPOSE OFFLINE REIN- FORCEMENT LEARNING FOR WHOLE-BODY CONTROL
Zhilong Zhang 22 Yumpeng : Xinghao D Hongjie C Haonan Wang', Pengyuan Min', Chenyu Wang', Pengfei Chen Chenbo Xin2, Vijie Wang', Wenyu Luo", Yihao Sun', Yidi Wang" Lei Yuan' Gang Wang Yang Yu
Natiinal Key Laboratory for Novel Software Technology, Nanjing University, China Schooo of Artificial Intelligence, Nanjing University, China "Scoool of Automation, Beijing Institute of Technology, China "Mila-Quebec Al Institute Université de Montréal, Canada
(zhangzl, duxh, wangyd) 0 (00.00000055mmmmm......... edu.cn mmeiynnpeng,hongjiee. (chenpengfei,,xncceenooo wangyijie, gangwang) yihao. sahao.sunemila..uubee.. yyuanl,yuy) @ny.
mpy, wey (bbittbbit.... edu.iteedd.cr
yangwaggg @bit edu. en
ABSTRACT
Scaling imitation learning to high-DoF whole-body robots is fundamentally con- strained by the scarcity of expert demonstrations. In contrast, large amounts of suboptimal data are readily available and offer a practical way to alleviate super- vision bottlenecks in real-world whole-body control. However, leveraging such data introduces two central challenges: how to extract informative signals from imperfect trajectories, and how cope with the increased learning complexity induced by high-dimensional control. To overcome this, we propose HVD (Hi- erarchical Value-Decomposed Offline Reinforcement Learning), The offline RL formulation provides principled data selection over suboptimal datasets, enabling the policy to prioritize high-vale behaviors while down- weighting harmful ones. Complementarily, hierarchical value decomposition organizes learning along the robot's kinematic structure, improving credit assignment and reducing learning complexity in high-DoF systems. Built on a Transformer- architecture, HVD supports multi-modal and multi-task learning, allowing ficxible integration diverse sensory inputs. To enable realistic evaluation and training, we further introduce WB-50, a 50-hour dataset teleoperated and policy rollout trajectories annotated with rewards and preserving natural imperfections, including partial suc- cesses, corrections, and failures. Experiments show HVD significantly outperforms existing baselines in success rate across complex whole-body tasks. Our results suggest effective policy learning for high-DoF systems can emerge not from perfect demonstrations, but from structured learning over realistic, imperfect data. Our code is available at https://github.com/ LAw.:///AAAAAAAAAAAAA
INTRODUCTION
Imitation learning has become central paradigm for robotic policy learning, enabling robots to
acquire complex skills directly from demonstrations (Pomerleau 1991] Ross al., 2011: Brantley
al. 2020 Feng al. 22266) Recent advances such diffusion based action generation (Chi et al.
2023 Ze et al. 2024| Liu ct al. 2024) and vision language action models for instruction conditioned
control (Kim et al. 2024 Black al., 2024| Liu al. 2024 Hu al. 2024 Intelligence al.
2025)
further extend imitation learning toward unified perception, reasoning, and control systems,
positioning it foundation for general purpose roboties.
"Equal contribution. "ooresponniig author.

---

Published as a conference paper at ICLR 2026
However, scaling imitation learning to high DoF whole body robots exposes a fundamental bottleneck: expert data searcity. Whole body control requires coordinated motion across many joints, dramatically enlarging the state and action spaces (Bellman 1966 Kober et al. 2013). Collecting high quality demonstrations through teleoperation is cognitively demanding and physically costly, making large scale expert supervision impractical (Zhou al. 2023 Diang al. 2025). In contrast, large amounts of suboptimal data naturally arise from teleoperation and policy rollouts. These trajectories often contain partial successes, corrections, and failures, and therefore provide a scalable but imperfect source of supervision.
Leveraging such data introduces two central challenges. First, suboptimal trajectories mix useful behaviors with harmful ones, requiring principled mechanisms to extract informative signals. Second, high DoF control increases learning complexity, making credit assignment difficult in long horizon and high dimensional settings. Existing imitation learning methods typically assume expert demon- strations, while standard offline reinforcement learning struggles to scale effectively to structured whole body systems with multi modal observations (Levine et al. 2020; Kalashnikov et al. 2018
Kumar et al. 2022).
To address these challenges, we propose HVD (Hierarchical Value Decomposed Offline Reinforce- ment Learning), a structured offline RL framework for whole body control. We first perform reward labeling over collected trajectories to provide consistent supervision signals across suboptimal data. Built upon these reward annotated datasets, HVD leverages offline RL for value based data selection, enabling the policy prioritize high value behaviors while suppressing harmful ones. Meanwhile, hierarchical value decomposition organizes learning along the robot kinematic structure, improving credit assignment and reducing learning complexity in high DoF systems. Implemented with Transformer backbone, HVD supports multi-modal and multi-task learning, making it more flexible.
To support realistic evaluation and training, we introduce WB 50, 50 hour dataset of telcoperated and policy rollout trajectories annotated with rewards and preserving natural imperfections. Experiments on diverse whole body manipulation tasks show that HVD significantly outperforms existing baselines in success rate. Our results suggest that effective high DoF policy learning can emerge not from perfect demonstrations, but from structured learning over abundant imperfect data.
The primary contributions of this work are as follows:
We propose HVD, offline RL method for whole-body control via hierarchical Q-value decomposition with temporal chunking, enabling precise credit assignment in high-DoF, long-horizon tasks.
We implement HVD using a Transformer-based architecture that supports multi-modal inputs and multi-task learning.
We introduce WB-50, 50-hour whole-body robotics dataset of imperfect, reward-labeled trajectories.
We demonstrate that HVD outperforms baselines across diverse whole-body tasks and policy architectures, and validate the effectiveness of HVD on multi-task settings.
PRELIMINARIES
Markov Decision Process. We model the robot control task Markov decision process (MDP) defined by the tuple M = (S,A, P.r. H), where H is the horizon. The state space S = Sats x Sprop includes egocentric observations (c.g.. images, point clouds) and proprioceptive state. The action space A C Rd consists of joint-level commands for a d-DoF robot. and r(s" a²) E [o, 1] is the reward function and P(sh+I|s* |s" a²) characterizes the non-stationary transition function of this MDP, which is a critical assumption because the whole-body movement and the constraints of the camera's 's field of view (FoV) result in partial observability, making the observation-base transitions appear highly stochastic and time-varying throughout the horizon. The goal is to learn a policy т [(a s) that maximizes the expected return V(T) = E,, E r(sh, a1.))
Offline Reinforcement Learning considers the problem of learning a policy from a fixed dataset D = [(s, r, s')} without further environment interaction. A central challenge is distributional shift, where the learned policy may query actions outside the support of the data, leading to erroncous value estimates and thus poor performance (Kumar et al. 2019 Levine al. 2020 Koh et al. 2021).
2

---

Published as a conference paper at ICLR 2026
Implicit Diffusion Q-Learning (IDQL) Glansnn-Estucee al.) 2023) builds on IQL Kostrikov et al. (2021). which can be viewed as an actor-critte method (Konda Tsitsiklis 1999). where the critic objective induces implicit, Shaaioo-reeularizze actor to prevent the value overestimation problem in offline RI.. In this framework, the value function (*)(s) is obtained by minimizing a convex loss dataset actions:
(8) = min Ba-n(als) [s(Qa(s", (s())))
where is chosen as an asymmetric convex function (e.g., expectiles (Kostrikov et al. 2021), quantiles (Koenker & Hallock 2001), exponential (Beirlant et al., 1999)). determining how the implicit policy a6 deviates from the behavior policy μ. The Q-function is trained with Bellman backups:
(o) = (o" at 2 (r(ss,ac + (o((stt) Qo(s", Qo(s",a"))
To recover the policy, IDQL employs T (als) trained via advantage weighted regression (aasaii & Yamashina 2020):
[f d²) a00 1Qts a²) (*(")) 1 -
- VV-(=)))
Cwcightbp (Ø)
Σ H 1=1
(il-To(Vaa" + +VI- ae,
where Na (0, 1) denotes Gaussian noise, t is the noising timestep, Oe is the noise schedule parameter in diffusion training, and I' ov(s) denotes the derivative of with respect to V(s).
0f
3 CHALLENGES IN WHOLE-BODY CONTROL
In this section, we argue that whole-body control presents two key challenges: the increased DoF and non-stationary observation dynamics. We examine the impact of this phenomenon on policy performance from both theoretical and empirical perspectives.
Egocentric view
Third-pe Wevv
Success Rate (962
50
40
20
21-Dof, Dual uaharms
7-Dol ngl
WE-WWA
Whole-body experiment setup
Arm-only experiment setup
W2-IMA fihore body.
(Whhie-body)
600
DP (lem-rkky)
NE-VHN (Arm-nnl)
Figure 1: Performance comparison between 21-DoF whole-body policies and 7-DoF arm-only policies trained on 50 expert demonstrations.
3.1 THEORETICAL ANALYSIS
In the context of behavior cloning (BC), theoretical studies have established that the expert sample complexity, defined as the number of expert trajectories required to learn policy with a desired level of performance, scales poorly with the size of the policy set II. This has been focus of recent work We begin by formally
RRjaraman et al. 2020 Tu et al. 2022 Foster et al. 2024 Xu al. 2024a).
introducing the policy covering number of policy class.
Definition 3.1 (Policy covering number), For a policy class II C {mp X + A(A) we set that II' C (TA X + A(A)) is an s-cover if for all 11, there exists n' 11' such that for all x E X, a E A, and E [H).
Th (a(aaa)
log
Th (a(x) Se.
We denote the size of the smallest such cover by Np (TT,
'While diffusion policies are typically implemented as implicit generative models, they theoretically induce an explicit probability density function x (()z) via the probability flow ODE formulation (Song al. (021)). This bijective mapping ensures that the log-density log x (a(ar is well-defined and computahle, rendering diffusion policies compatihle with this definition.

---

Published as a conference paper at ICLR 2026
We then formalize the dependence of the expert sample complexity of BC on this measure:
Proposition 3.2 (Expert sample complexity of behavior cloning (Foster et al. 2024)). For any expert
policy x- E II, to ensure that the suboptimality gap of the learned policy ß satisfies ((=))- S
with probability at least 1 - 6, the number of expert trajectories required for behavioral cloning needs to satisfy
n n= n=ō(
Log.Npuc (1,, Sx)
(2)
E²
Here, H is the task horizon, is the target suboptimality gap, and (LL Ex) denotes the Sx-policy covering number of the policy class II.
This result reveals critical quantitative limitation of BC: the expert sample complexity of BC is fundamentally tied to the log-covering number the policy class II. As the complexity or size of the action space N En) grows, which in turn necessitates larger number of expert trajectories to learn an opptimal policy.
3.2 EMPIRICAL VALIDATION
Figure I provides an empirical comparison of arm-only and whole-body policies applied to the same task. The leftmost and middle panels show the experimental setups for both cases: the whole-body setup uses a 21-DoF robot with dual arms and an egocentric view, while the arm-only setup uses a 7-DoF robot with a single arm and a third-person view. The rightmost panel compares the success rates of various policies (To. WB-VIMA, and DP) with the same number of expert demonstrations (50) for both the whole-body and arm-only setups. The whole-body policies consistently achieve significantly lower success rates than their arm-only counterparts across all models. This performance gap highlights the challenges posed by nigh-dimensional action spaces and non-stationary transitions.
In summary, the increased DoF in whole-bosd body control leads to an explosion in expert data require- ments, driven both by the complexity of DoF's and the inherent non-stationary transition.
4
METHOD
4.1 WB-50: IMPERFECT DATA FOR WHOLE-BODY RL
The preceding analysis shows that directly relying on expert demonstrations is impractical due to the increased DoF and instability of egocentric observations. Fortunately, in realistic settings, non-expert demonstrations are more abundant, arising naturally from teleoperation and policy rollouts (Zhou al., 2023). To leverage this, we introduce WB-50 as illustrated in Figure 2 a reward-labeled whole- body dataset spanning over 50 hours. WB-50 is intentionally composed of three distinct data sources reflect realistic data distributions: i) expert demonstrations (43.7% ii) imperfect teleoperation (14.6%), and iii) policy rollouts (41.7%) the latter two comprising the majority, mirroring the scarcity perfect supervision in practice. Every trajectory is annotated at the subtask level and labeled with discrete reward signals. More details are listed in Appendix B
direct way to leverage such data offline RI,, which enables policy learning from static and imperfect datasets (Lange et al., 2012 Levine et al.] 2020). However, existing offline RL methods face fundamental limitations when applied to whole-body control. Most prior successes have been restricted to relatively low-dimensional tasks (Mandlekar al. 2022, Sinha al. 2022 Zhou et al. 2023), and current algorithms struggle to scale to the high degrees of freedom inherent in whole-body robots. Compounding this difficulty, whole-body control often involves sparse reward signals, which exacerbate the challenges of temporal credit assignment and policy optimization. Furthermore, prevailing approaches typically confined single-task mingle-modality domains in embodied control, raising concerns about their versatility and scalability.
4.2 HVD: HIERARCHICAL
OFFLINE RL
To address the above issues, we introduce Hierarchical -ietee--eeoppsed Offline Reinforcement Learning (HVD), designed for high- dishninnessonnl,, whole-body control in robotic systems. Unlike conventional approaches that decompose the policy (Sentis & Khatib, 2006). HVD introduces hierarchy directly into the Q-value function through spatial decomposition. This key design allows
4

---

Published as a conference paper at ICLR 2026
Data Collection & Reward Labeiling
Value Learning Objective
Policy Learning Objective
14.0%
Temporal Loss
Valua
Loss Le(
L
Egg (8)
(Φ)
Experi
43.7%
R
Lurra
Carrs
Herarchical Weights Mbas
Phona
Mirr
Sucoptima
Harardhical Decompoaition
Value
Sorre
Pipwrt Denssstrrion
Herarchical Network
V Network
Brbusinndhamm
Polayy
Qa
Action head
General Policy
Pick up pen put ilin the container Bi More farward to the whiieboord and clean Language Instruction
abas atorso Carm
Observation
RGB Image
Point Cloud
Prompt
Proprio.
Mixed-Quality Dataset
Dataset Construction
Value Learning
Policy Learning
Figure 2: Framework of HVD. The proposed HVD framework consists of three stages: (1) Dataset construction and reward labeling from expert, suboptimal, and rollout data; (2) Hierarchical value learning with kinematically decomposed Q-functions and temporal chunking: and (3) Policy learning via hierarchical sountae--wiightee regression. The diagram illustrates the data flow from multi- modal observations through hierarchical value function learning, and ultimately to policy training.
us to maintain unified policy network while enabling fine- fine-graine grained, component-specilft value assessment for different parts of the robot. As a result, HVD achieves more accurate credit assignment across dong-horizon, multi-step behaviors for whole-body control.
Q-value Decomposition. Inspired by Smmthh et al. (2012); ; Pan et al. (2024); Jiang et al. (2025), the action space of whole-body control policies be decomposed into three hierarchical components corresponding to its physical structure, A Anuse x Aueso x Aam- At each time step h, the action h:h++k h.h+t chunk size is defined as h.h+k = base Ojooso
: a.am hihh+R), where each component represents sub-action controlling specific subset of the robot's degrees of freedom. Specifically, Obase A:h+* governs the lower-body motion (e.g., locomotion base movement), Groeso or torso orientation, and aim h.h++ manages the arm movements.
h:h+* controls the upper body
Based on this decomposition, where each level accumulates value estimates conditioned on progressively more complete subsets of the robot's action space. Specifically, for chunk starting timestep h, compute:
define hierarchical Q-values temporal chunks of length k.
thch+ Q0 kzh+k))
O.ase )
Ohch+k = Qo(sn,
hoh+t Ach+k).
O
Ohih+-k Qo(s".
hsh44 hsh44k h.hh+
Aiooro
Qaam
Here, each Q-value corresponds to specific part of the robot, creating layered structure for the value function, enabling more precise, joint-level credit assignment.
Hierarchical Value Estimation. To train the hierarchical Q-function in Equation 3 we employ multi-level TD learning loss that aligns each partial Q-value with its corresponding estimated return:
ch (0) = [B((((" (r(s*) ( hih+k) + Vi (s (+++++1) Q. hh:h+k)?? ?
with i (base, torso, arm),
(4)
where r(s": adothtt)) Ejth r(s), a²) the reward for executing the action chunk lectk on the state sh which aggregates the per-timestep rewards over the sub-episode from time h to h + k. Moreover, (s k++tl) represents the estimated value of the next state predicted by a value network parameterized by v. The Q-learning objective is designed to minimize the temporal difference between the predicted Q-value and the target value, and the total Q loss combines all hierarchical levels defined as below:
H
(0) =
1
Σ h=1
S (8) + Ch Cl (0) + LA (0)
(5)
Н
5

---

Published as a conference paper at ICLR 2026
The result is value decomposition that promotes credit assignment across space with temporal Q chunking, enabling more sample-efficient and coordinated whole-body control.
Implicit Value Learning. Concurrently, we incorporate implicit value learning to align value estimates across the hicrarchical levels of our framework. By leveraging an in-sample learning paradigm (Wainwright 2019 [Hansen-Estruce value overestimation caused by OOD actions, a persistent challenge in Q-learning methods:
al. 2023), HVD effectively mitigates the risk of
H
(v) = H
1
Σ
exp (Q (c (Q, (Quatt - (6)
vo(("))
(base (base, arm)
where > 0 is temperature parameter controlling the strength of the constraint.
By optimizing this loss, we establish soft lower bound on the value estimates across all hierarchical Q-heads, ensuring that limb-level value predictions remain aligned with global, whole-body goals.
Policy Learning. We train the policy network To using a hierarchical variant of advantage- regression (AWR) (Peters & Schaal 2007 Peng et al. 2019 Nair et al. 2020). Rather than uniformly imitating all actions in the dataset, our method assigns importance weights to action chunks based on estimated advantages, encouraging the policy to prefer high-value behaviors while down-weighting low-return trajectories. This weighting is applied separately at each hierarchical level, allowing critical sub-actions to emphasized even when other components generate lower returns.
The per-level advantage weight for action chunk a; hih+k executed from state s is defined as:
wi (s a a lexp (a(Quatt Vy (w(s)))) -1
Q (s()
where a > 0 controls the sharpness of advantage-based reweighting.
This formulation ensures that actions with higher relative advantage receive exponentially increasing weight, while preserving gradient flow even near the decision boundary. Furthermore, to learn robust policies from limited demonstrations, we combine two loss terms: i) an RL term trained on an offline dataset D weighted by the advantage scores; and ii) a BC term trained on smaller set of expert trajectories D providing a stabilizing prior:
CRL
1
E Eo
w
wh a. d TT [eToooVVaat vI ae, se,
(8)
H
CBC (0) = H
ED || To + +V1 - de, sh
(9)
h=1
(Ø) CRL + 1. ()),
(10)
where e (0, 1) denotes Gaussian noise, t the noising timestep, â represents the noise schedule parameter in diffusion training, and 0 controls the off between reinforcement learningdriven exploration and expert imitation.
4.3 GENERAL ALGORITHMS AND PRACTICAL IMPLEMENTATION
Algorithm I offers overview of the HVD approach, which operates in two phases. The first phase focuses on hierarchical value learning, where both the value network V., and Q-network Qo are updated using the hierarchical deloe-ee learning loss (Equation and TD loss (Equation 5) respectively. The second phase performs policy extraction, where the policy network n. is trained maximize the cumulative returns through advantage weighted regression (Equation 10).
Model Architecture. As illustrated in Figure 21 our hierarchical Q-network adopts unified multi-modal architecture centered around a Transtormer -oassoreeer-ee-edddddd-ee backbone. The model can optionally process rich set of sensory modalities input by processing them into token embeddings, including egocentric RGB images, point cloud data from depth sensors, natural language task instructions, and proprioceptive state. More detailed implementation be found in Appendix

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
Our experiments aim address three core questions. Q1: Does HVD consistently surpass imitation
learning baselines acToss diverse policy architectures? (Section 5.2 Q2: Does hierarchical value decomposition yield more accurate credit assignment, and does this improve the policy performance? (Section 5.3) Q3: Can HVD effectively scale multi-task settings, leveraging shared value structure to improve overall performance? (Section
Figure 3: Illustration evaluated tasks: (a) Pen Insert, (b) Cup Upright, (c) Wipe Board, (d) Basket
Carry, (c) Trash Dispose.
5.1 EXPERIMENT SETTINGS
Robot Platform. We conduct all experiments on the Galaxea RI, real-world wheeled humanoid robot with a 21-DoF whole-body system. Task demonstrations are collected using JoyLo, teleopera- tion interface developed by (Jiang et al., 2025). Guidelines are provided constrain operators to generate demonstrations that are easier for the robot to learn. Nevertheless, operator skill levels vary, resulting in substantial number suboptimal demonstrations during data collection.
Task Design. We design suite of five representative office tidying tasks (see Figure 3 with details in Appendix B). The tasks require navigation, dexterous manipulation, and bimanual coordination, with durations from 40-second single-arm actions to over 120-second multi-step sequences involving coordinated locomotion and dual-arm cooperation. Additionally, we assess task difficulty from temporal complexity, kinematics, control, and coordination complexity detailed in Appendix B3.6 This diversity enables rigorous testing both precision and long- long-horizon whole-body control.
Baselines. We develop HVD framework based three baselines with different input modal- ities, including the state-of-the-arr VLA model To (Black et al. (024), the 3D-input model WB- VIMA (Jiang al. 2025). and the Diffusion Policy (Chi et al. 2023). We evaluate the performance of policies trained using the original methods on expert datasets and compare them with policies trained using HVD on quxiddquulity datasets.
Evaluation Metrics. To enable fine-grained assessment of policy performance, each task is de- composed into distinct logical stages. We report two primary metries: success rate for task-level evaluation and normalized stage score for stage-level analysis. Moreover, we introduce perturbations to the task environment background, initial task region, and robot's initial pose to further challenge robustness and evaluate the model's ability to generalize under diverse and realistic variations. Each policy was evaluated over 50 independent rollouts per task. Reported success rates and stage scores averaged over these rollouts to ensure statistical consistency and fair comparison across methods.
5.2 BENCHMARK RESULTS
We present the main experimental results of our study, evaluating each method under its optimal training regime to assess peak performance. HVD is trained the full mixed dataset, while imitation learning baselines are trained on the expert-only subset, consistent with their reliance on high-quality demonstrations.
7

---

Published as a conference paper at ICLR 2026
As shown in Table D our proposed HVD consistently outperforms standard imitation learning across all five tasks, yielding higher average success rates across policies. The gains are especially pronounced in challenging manipulation tasks such Wipe Board and Basket Carry, where robustness to imperfect initial states and partial observability is essential. Moreover, Siows that HVD's benefits extend heyond task-level success: it achieves higher normalized stage scores nearly all subtasks. This demonstrates that HVD not only improves final outcomes but also enhances policy reliability throughout the entire execution trajectory.
shows
Method
Tasks
Avg SR
(IL/HVD)
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
0.64/0.86
0.82/0.90
0.18/0.32
0.26/0.44
0.28/0.36
0.44/0.58
WB-VIMA
0.52/0.78
0.58/0.82 0.66/0.72
0.12/0.26
0.10/0.10 10
0.20/0.32 0.08/0.16
0.30/0.46
DP
0.54/0.64
0,00/0.00
0.00/0.08
0.26/0.32
Table 1: Task-level success rate (SR) of IL and HVD across baseline methods on 5 tasks.
1.00
Pen Insert
Cup Upright
Trash Dispose
0.75
Scores
0.25
0.00
st.1
st1
st1
st2
st3
S14
1.00
Wipe Board
Basket Carry
0.75
Scores
0.50
0.25
0.00
HVD+HVD
pio+Il
HVD
VIMA+
DP+HVD
DP+1I
Figure 4: Stage-level scores of IL and HVD across baseline methods across tasks.
5.3 VALUE DECOMPOSITION ABLATION
First, we investigate whether the observed performance gain is primarily attributable to the hicrarchical value decomposition merely to the application of offline RL. To this end, conduct ablation study comparing our model with the shared Q-value (named HVD w/o hierarchy). Both models trained the same mixed-quality dataset with the same hyperparameters, ensuring that the only architectural difference is the presence of hierarchical value decomposition. Table 2 shows that removing hierarchical decomposition consistently harms performance across tasks, confirming that the improvements not only stem from the training paradigm, but also from the proposed structure.
Method
Tasks
Avg Diff
w/o hierarchy
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
DP
-0.02
0.00
0.00
-0.08
-0.06
-0.03
WB-VIMA
-0.02
0.00
-0.12
-0.08
-0.1 12
-0.07
No
+0.04
-0.02
0.11
-0.10
-0.04
-0.05
Table 2: Ablation study results on hicrarchical value decomposition across 5 tasks. The value here indicates the success rate changes when removing the hierarchy.
8

---

Published as a conference paper at ICLR 2026
To further analyze how decomposition impacts credit assignment, visualize the advantage weights wi several key frames in the Basker Carry task (Figure At the second key frame, where the robot prepares stand and hold the basket, HVD assigns higher weights to the arm and torso, reflecting the importance of these components. In contrast, HVD w/o Hierarchy produces uniformly high weights across all frames, failing to differentiate subcomponents. These findings indicate that HVD enables more precise credit assignment, which directly contributes to more accurate and reliable action generation. The more visualization examples are shown in Appendix D.I
Basket Carry
Weight
HVD Base
HVD Torso
HVD Arm
HVD w/o Hiemarch
Figure 5: Credit Assignment Comparison between HVD and HVD w/o hicrarchy.
5.4 MULTI-TASE LEARNING
We evaluate HVD's ability to scale to multi-tass multi-task learning by training single policy on data from all five tasks and comparing its success rate against specialist (single-task) policies. As shown Table standard II. suffers from negative transfer in the multi-task setting, leading to degraded performance on most tasks. contrast, HVD mitigates such interference and even surpasses single-task specialists on several tasks. We observe that these gains primarily stem from enhanced torso robustness and more generalizable grasping behaviors acquired during multi-task training. Together, these results show that HVD effectively leverages shared knowledge across tasks while maintaining specialization, making it more scalable to diverse multi-task settings.
Method To expertise
Tasks
Avg SR
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
0.64
0.82
0.26
0.28
0.44
multi-task IL
0.50
0.60
0.18
0.24
0.30
0.36 (-0.08)
expertise HVD
0.86
0.90
0.32
0.44
0.36
0.58
multi-task HVD
0.92
0.94
0.32
0.50
0.30
0.60 (+0.02)
Table 3: Task-level success rate comparison of multi-task and expertise policies across 5 tasks.
RELATED WORK
Whole-body Control Policy Learning. Whole-body control is central challenge in robotics due to the high dimensionality of articulated bodies and the lack of inherent self-stabilization (Hirai
lt 1998 Grizzle et al.) 2009). Classical model-based planning emphasizes kinematic feasibility,
stability, and reactive regulation lation (eetiis & Khatib 2006 Dietrich et al. 2012 Burget et al.) 2013
Kaelbling Lozano- Pérez 2013 Dai al. 2014), but struggles with adaptability and scalability
in unstructured tasks. Learning- based approaches optimize control policies from data (Sickmann
al. 202T [i et al. 2021 Dao al. 2022 Radosavovic al. 2024 Cheng et al. 2024). enabling dynamic behaviors difficult to engineer manually (Xia et al., 2021 et al. [0224] Fu et al. 2024
Arm et al. 2024). Recent advances include generative policies for capturing multimodal action distributions (Fu et 2003 Jiang et al. 2025). VLA models that ground control in language and perception (Xu et al.) 2002bb Ding et al. 2025). and hierarchical policy architectures for managing the complexity of humanoid whole-bonl body control (Hansen et al.) 20255 WWii et al. [025 Fu et al.) (2025).
Despite these advances, most methods still require high-quality demonstrations, limiting scalability complex real-world tasks.

---

Published as a conference paper at ICLR 2026
Offline RL for Embodied Control. Offline RL has made significant strides in embodied control tasks, enabling robots to learn complex behaviors from pre-collected datasets without requiring expert demonstrations (Levine et al. 2020; Lin et al. 2025| 2026). Previous works have attempted to learn
policies from trajectories generated by human failures or during policy evaluation (Kumar et al. 2021
Mandlekar al. 2022 [Sinaa et al.) 2022 Bhateja et al. 2023 Luo al., 2023] /hou al. 2023
Zhang et al. 2023b Ma et al. 2024;] Zhang et al. 2024). However, these methods are largely confined to arm-based manipulators, and their effectiveness in high-DoF whole-body control tasks remains unexplored. Recently, there has been an effort to adapt RL algorithms to mobile manipulators (Hu al., 2023; Pan et al. 2024). However, these approaches often rely on single-modality inputs. limiting their ability to integrate with generalist policies like VLA and world models (Kim et al.) 2024. Black et al., 2024. Zhang al. 2203aa] 2024). Furthermore, most of these methods focus on single-task training, raising concerns about their scalability and generalization to multi-task scenarios.
7 CONCLUSIONS AND LIMITATIONS
Conclusions. In this paper, we present HVD, framework for learning whole-body robotic control from imperfect, real-world demonstrations. By introducing kinematically aware value decomposition within a multi-modal Transformer architecture, HVD enables stable and scalable policy learning in high-din dibe-iinnnional action spaces using suboptimal offline data. Together with the release of WB-50, 50-hour dataset of realistic teleoperation and rollout trajectories, we demonstrate that structured offline RL can effectively leverage partial successes and failures to achieve robust, generalizable control. The information about resource cost is listed in Appendix P
First, HVD relies on aunan--nnotated rewards, which can be costly. Second, we have not yet explored using failed data for pretraining, which could become valuable paradigm for improving robustness and scalability in open-ended environments. Future work could also investigate leveraging VLM for automated reward labeling reduce human effort and enhance scalability.
Limitations.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grants 62495090, 62495093, U23B2059, 62506159, U24A20324, and the Natural Science Foundation of Jiangsu under Grants IKK2024119, BK20243039.
REFERENCES
Philip Arm, Mayank Mittal, Hendrik Kolvenbach, and Marco Hutter. Pedipulate: Enabling manip- ulation skills using a quadruped robot's leg. In IEEE International Conference on Roboties and Automation, pp. 5717-5723. IEEE, 2024.
Jan Beirlant, Goedele Dierckx, Yuri Goegebeur, and Gunther Matthys. Tail index estimation and an exponential regression model. Extremes, 2(2):177-200, 1999.
Richard Bellman. Dynamic programming. Science, 153(3731):34 1966.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024.
Chethan Anand Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline rl from internet videos via value-function pre-training. In NeurIPS 2023 Foundation Models for Decision Making Workshop. 2023.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. z0: A vision-language-action flow model for general robot control. arXiv, 2410.24164, 2024.
Kianté Brantley, Wen Sun, and Mikael Henaff. Disagreement- reguulariee imitation learning. In Proceedings of the 8th International Conference Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2026
Felix Burget, Armin Hornung, and Maren Bennewitz. Whole-body motion planning for manipulation of articulated objects. In IEEE International Conference on Robotics and Automation, pp. 1656- 1662. IEEE, 2013.
Xuxin Cheng, Yandong Ji, Jumming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16790 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668, 2023.
Hongkai Dai, Andrés Valenzuela, and Russ Tedrake. Whole-body motion planning with centroidal dynamics and full kinematics. In JEFF-RAS International Conference on Humanoid Robots, pp. 295-302. IEBE, 2014.
Jeremy Dao, Kevin Green, Helei Duan, Alan Fern, and Jonathan Hurst. Sim-to-real learning for bipedal locomotion under unsensed dynamic loads. In International Conference on Robotics and Automation, pp. 10449-10455. IEEE, 2022.
Alexander Dietrich, Thomas Wimbock, Alin Albu-Schaffer, and Gerd Hirzinger. Reactive whole- body control: Dynamic mobile manipulation using a large number of actuated degrees of freedom. IEEE Robotics & Automation Magazine, 19(2):20-33, 2012.
Pengxiang Ding, Jianfei Ma, Xinyang Tong, Binghong Zou, Xinxin Luo, Yiguo Fan, Ting Wang, Hongchao Lu, Panzhong Mo, Jinxin Liu, et al. Humanoid-vla: Towards universal humanoid control with visual integration. arXiv preprint arXiv:2502. 14795, 2025.
Zhaohan Feng, Ruiqi Xue, Lei Yuan, Yang Yu, Ning Ding, Meigin Liu, Bingzhao Gao, Jian Sun, Xinhu Zheng, and Gang Wang. Multi-agent embodied AI: Advances and future directions. Science China Information Sciences, 69(5):151200 2026.
Dylan J Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon imitation learning. arXiv, 2407.15007, 2024.
Zipeng Fu, Xuxin Cheng, and Deepak Pathak. Deep whole-body control: Learning a unified policy for manipulation and locomotion. Conference on Robot Learning, pp. 138- 149. PMLR, 2023.
Zipeng Fu, Tony Z, Zhao, and Chelsea Finn. Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02/17 2024.
Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. HumanPlus: Humanoid shadowing and imitation from humans. Conference on Robot Learning, 2025.
Jessy W Grizzle, Jonathan Hurst, Benjamin Morris, Hae- Park, and Koushil Sreenath. MABEL, a new robotic bipedal walker and runner. In American Control Conference, pp. 2030-2036. IEBE, 2009.
Nicklas Hansen, Jyothir V, Vlad Sobal, Yann LeCun, Xiaolong Wang, and Hao Su. Hierarchical world models visual whole-body humanoid controllers. In International Conference Learning Representations, 2025.
Philippe Hassen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 100.:2334.1073.. 10573, 2023.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kazuo Hirai, Masato Hirose, Yuji Haikawa, and Toru Takenaka. The development of honda humanoid robot. In Proceedings. 1998 IEEE international conference on robotics and automation (Cat. No. 98CH36146), volume 2, pp. 1321-1326. IEFE, 1998.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 68000-6811111 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. International Confer- ence on Learning Representations, 1(2)3, 2022.
Jiaheng Hu, Peter Stone, and Roberto Martin. Causal policy gradient for whole-body mobile manipulation. arXiv preprint 23..v:::355048666, 2023.
Yucheng Hu, Yanjiang Guo, Pengehao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint urXiv:24122 14803, 2024.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. Nu.s a ssioonlanguage-action model with open-world generalization. arXiv preprint arXiv:2504. 100::2544.6004,, 2025.
Yunfan Jiang, Ruohan Zhang, Josiah Wong, Chen Wang, Yanjie Ze, Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, and Li Fei-Fei. Behavior robot suite: Streamlining real-world whole-body manipulation for everyday household activities. arXiv preprint arXiv:2503.0563. 2025.
Zhenyu Jiang, Yuqi Xie, Jinhan Li, Ye Yuan, Yifeng Zhu, and Yuke Zhu. Harmon: Whole-body motion
generation of humanoid robots from language descriptions. arXiv preprint arX1v:2410.12773.
2024.
Leslie Pack Kaelbling and Tomás Lozano-Pérez. Integrated task and motion planning in belief space. The International Journal of Robotics Research, 32(9-10)) 192- 2013.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision- robotic manipulation. In Conference robot learning, pp. 651-673. PMLR, 2018
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, al. Open VLA: An open-source laroon--nnaageaattioon model. arXiv preprint 2.Xv:22406.09246, 2024.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 12(1))122881124 2013.
Roger Koenker and Kevin F Hallock. Quantile regression. Journal of Economic Perspectives, 15(4): 143-156, 2001.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 1999.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learming arXiv preprint arXhv:2110.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In 5th Annual Conference Robot Learning, 2021.
Aviral Kumar, Anikait Singh. Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. arXiv preprint arXiv:2210.08 10.05188, 2022.
12

---

Published as a conference paper at ICLR 2026
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: tatt--ofthh-arre pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 01643,, 2020.
Chengshu Li, Ruohan Zhang. Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang. Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, KounY Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria. Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR- A human- centered, embodied Al benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.09227, 2024.
Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In IEEE International Conference on Robotics and Automation, pp. 2811-2817. IEEE, 2021.
Haoxin Lin, Yu- Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jia, Junyin Ye, Jiaji Zhang, and Yang Yu. Any-step dynamics model improves future predictions for online and offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Haoxin Lin, Siyuan Xiao, Yi-Chen Li, Zhilong Zhang, Yihao Sun, Chengxing Jia, and Yang Yu. ADM-v2: Pursuing full-horizon roll-out in dynamics models for offline policy learning and evaluation. In The Fourteenth International Conference on Learning Representations, 2026. URL https: opwnneeeeee net htt.:eettttooeeeeeeeeeeeee ICKXExapaa
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-IB: A diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864 2024.
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action- quantized offline reinforcement learning for robotic skill learning. In Conference on Robot Learning, pp. 1348-1361. PMLR, 2023.
Chengzhong Ma, Deyu Yang, Tianyu Wu, Zeyang Liu, Houxue Yang, Xingyu Chen, Xuguang Lan, and Nanning Zheng. Improving offline reinforcement learning with in-sample advantage regularization for robot manipulation. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, pp. 1678-1690. PMLR, 2022.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
Guoping Pan, Qingwei Ben, Zhecheng Yuan, Guangqi Jiang, Yandong Ji, Jiangmiao Pang, Houde Liu, and Huazhe Xu. Roboduet: A framework affording mobile-manipulation and crossembodiment. arXiv preprint 2.3vv 6, 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Adyintage-weightte regression: Simple and scalable off off-policy reinforcement learning, arXiv preprint 19100:1110.00177, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of International Conference on Machine learning, pp. 745-750, 2007.
Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.
13

---

Published as a conference paper at ICLR 2026
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning point sets for 3D classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660,, 2017.
Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, :89))cad,9 2024.
Nived Rajaraman, Lin F. Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. In Advances in Neural Information Processing Systems 33, pp. 2914-2924, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 627-635, 2011.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- tional Conference on Learning Representations, 2020.
Luis Sentis and Oussama Khatib. A wholc-body control framework for humanoids operating in human environments. In Proceedings of IEEE International Conference Robotics and Automation, pp. 2641-2648. IEEE, 2006.
Jonah Siekmann, Yesh Godse, Alan Fern, and Jonathan Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In JEEE International Conference on Robotics and Automation, pp. 3309-7315 IEEE, 2021.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning robotics. In Conference on Robot Learning, pp. 907-917. PMLR, 2022.
Christian Smith, Yiannis Karayiannidis, Lazaros Nalpantidis, Xavi Gratal, Peng Qi, Dimos Dimarogonas, and Danica Kragic. Dual arm manipulation-A survey, Robotics and Autonomous Systems,
2012.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021.
Galaxea Team. Galaxea diffusion policy. 2025.
Stephen Tu, Alexander Robey, Tingnan Zhang, and Nikolai Matni. On the sample complexity of stability constrained imitation learning. In Learning for Dynamics and Control Conference, pp. 180-191. PMLR, 2022.
Martin J. Wainwright, Variance-reduced q-learning minimax optimal. arXiv, 1906.04697, 2019.
Lai Wei, Xuanbin Peng, Ri-Zhao Qiu, Xuxin Cheng, and Xiaolong Wang. HMC: Learning het- erogeneous meta-control for contact-rich loco-manipulation. Robotics: Science and Systems, 2025.
Fei Xia, Chengshu Li, Roberto Martín-Martin, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. In IEEE International Conference on Robotics and Automation, pp. 4583-4590. IEEE, 2021.
Tian Xu, Zhilong Zhang, Ruishuo Chen, Yihao Sun, and Yang Yu. Provably and practically efficient adversarial imitation learning with general function approximation. Advances in Neural Information Processing Systems, 37:66108-66 00--61144 46, 2024a.
14

---

Published as a conference paper at ICLR 2026
Xinyu Xu, Yizheng Zhang, Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards lision-language directed object rearrangement by physical humanoid. Advances in Neural Information Processing Systems, 37:18633-18659, 2024b.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations, arXiv preprini 0rXxi:2403.03954 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Ennrgy-weightee flow matching for offiie reinforcement learning. In The Thirteenth International Conference Learning Representations.
Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. STORM: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36:27147-27166, 2023a.
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better: Offline preference-based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference on Learning Representations, 2023b.
Zhilong Zhang, Ruifeng Chen, Junyin Ye, Yihao Sun, Pengyuan Wang, Jingcheng Pang, Kaiyuan Li, Tianshuo Liu, Haoxin Lin, Yang Yu, al. Whale: Towards generalizable and scalable world models for embodied decision making. arXiv preprint arXiv:2411.0561,, 2024.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa. Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7176-7183 83. IEFF, 2023.
15

---

Published as a conference paper at ICLR 2026
A HARDWARE COMPONENTS
Figure 6: Low-cost JoyLo system and Galaxea R1 robot.
As illustrated in Figure 6 the hardware setup comprises a JoyLo system and a Galaxea RI robot. The JoyLo system integrates 3D-printable arm links, low-cost Dynamixel actuators, and commodity Joy-Con controllers; its control loop runs at 100 Hz while data are recorded at 10 Hz. Functional buttons on the right Joy-Con are used to start, pause, include RGB and depth images, point clouds, joint states, odometry, and action commands. The Galaxea R1 platform is equipped with ZED 2i stereo camera, two Intel RealSense D435i cameras, and two Galaxea GI parallel grippers.
and discard recordings. Logged modalities
B TASK DEFINITION
B. PEN INSERT
Task Description This task requires the robot to grasp marker pen lying table and insert it vertically into a fixed pen holder (diameter 8 cm). The core challenge lies seamlessly executing the entire sequence from grasping to insertion. For each trial, both the marker pen and pen holder are randomly positioned on the tabletop within the robot's operational workspace and field of view. To further validate generalization, the robot's starting position and torso pose are also randomized within a constrained range.
Evaluation Rubric The task is evaluated single, continuous stage focusing the successful transfer the pen to the holder.
Stage 1: Grasp and Insert Pen into Holder
0.0 points:
The robot fails to grasp the marker pen (left of Figure 7
The robot successfully grasps the marker pen but fails to place it in the holder, for instance, by dropping the pen outside the holder due to an insecure hold (middle of Figure 7
0.5 points:
1..0 points: The robot firmly grasps the marker pen and successfully places it into the holder (right of Figure 7p.
Prompt: pick the pen and put it into the helder
16

---

Published as a conference paper at ICLR 2026
Figure Scoring rubric visualization for Pen Insert. Stage I: Grasp and Insert Pen into Holder
B.2
CUp UPRIGHT
Task Description. This task requires the robot to grasp horizontally lying plastic cup (diameter cm) and place it in a stable, upright position onto a target plate. The primary challenges involve dexterous reorientation of the cup during manipulation and ensuring steady final placement. For each trial, the cup is randomly placed on the tabletop, and the robot's starting pose is randomized within constrained range to test for policy generalization. The task demands combination of precise grasping and controlled, stable placement.
Evaluation Rubric The task is evaluated as single, continuous stage that assesses the entire sequence from grasping to successful upright placement.
Stage 1: Grasp and Place Cup Upright
0.0 O points: The robot fails to secure the cup with its gripper (left Figure 8)
0.5 points: The robot grasps the cup but fails place it upright on the plate, either due to losing its grip or improper reorientation (middle of Figure 8).
1. 0 points: The robot firmly grasps the cup and places it steadily in an upright position the plate (right of Figure 8).
Figure 8: Scoring rubric visualization for Cup Upright. Stage I: Grasp and Place Cup Upright
Prompt:
pick the cup and put it onto the coaster
B.3 WIPE BOARD
Task Deseription. This task is composed of three sequential stages. In the first stage, the robot navigates its base to position in front of the whiteboard and grasps an eraser. The second stage involves wiping the designated markings from the board. In the third and final stage, the robot places the eraser back into its designated slot. To promote policy generalization, each trial is initialized with randomized starting poses for robot and varied positions for the writing on the whiteboard, both within predefined area. This task demands precise physical interaction, as successful execution hinges on delicate force control: excessive pressure may cause the whiteboard to tilt, while insufficient force will fail to clean the markings completely.
Stage 1: Approach Whiteboard and Grasp Eraser
0.0 0 points: The robot fails to navigate to a position where any eraser is reachable (left of Figure 9.
0.5 points: The robot successfully navigates to the whiteboard but fails to establish stable grasp on an eraser (middle of Figure
1.0 points: The robot successfully navigates to the whiteboard and executes a stable grasp eraser, suitable for the wiping motion (right of Figure
Stage 2: Wipe Markings
17

---

Published as a conference paper at ICLR 2026
+0
+0,5
Figure 9: Scoring rubric visualization for Wipe Board. Stage Approach Whiteboard and Grasp Eraser
0 points: s: The robot fails make effective contact with the markings. due to dropping the eraser or significant positioning errors (left of Figure 10).
0.5 5 points: s: The robot partially crases the markings, or causes the whiteboard to tilt due to imprecise force control positioning error (middle of Figure TOp.
1.00 points: The robot completely erases the markings while maintaining stable contact with the whiteboard (right of Figure TOp.
0.00
+1.0
Figure 10: Scoring rubric visualization for Wipe Board. Stage 2: Wipe Markings
Stage 3: Return Eraser
0 0 points: The robot releases the eraser prematurely before reaching the designated slot (left of Figure I1
0.5 points::
The robot reaches the slot but fails to place the eraser correctly due to
positioning inaccuracies, causing to be dropped (middle of Figure [I].
1.0 0 points: The robot successfully and stably places the eraser back into its slot (right of Figure
0.0
0.5
+ 1.0
Figure 11: Scoring rubric visualization for Wipe Board. Stage 3: Return Eraser
Prompt:
move to the whiteboard and clean the whi shioebdrrd
B.4 BASKET CARRY
Task description. This task consists of four distinct stages. First, the robot navigates to cuboid basket placed on the ground. In the second stage, it bends down and lifts the basket using coordinated movements of both arms. For the third stage, the robot turns while holding the basket and places onto nearby table. In the final stage, the robot utilizes its left and right arms respectively to pick up markers from the table and place them into the basket. To promote policy generalization, each trial initialized with the robot at randomized starting pose, the basket at varied position the ground, and the markers at randomized locations on the table, all within a predefined area. This task demands long-horizon planning and effective bimanual coordination.
Stage 1: Approach Basket
points: The robot navigates to an incorrect position, rendering the basket unreach- able for lifting (left of Figure 12).
0.55 points: The robot navigates to a misaligned position, preventing a symmetric, bimanual grasp required for stable lift (middle of Figure 12..
18

---

Published as a conference paper at ICLR 2026
11.0 O points: The robot successfully navigates centered position directly in front of the basket, enabling a symmetric, bimanual lift (right Figure 12).
Figure 12: Scoring rubric visualization for Basket Carry. Stage 1: Approach Basket
Stage 2: Lift Basket
0.0 points: The robot fails to lift the basket off the ground (left of Figure 13
0.5 points: The robot lifts the basket with both hands but fails to keep it level or properly centered with its body (middle of Figure 13).
points: The robot successfully lifts the basket, maintaining a level and centered orientation relative to its body (right of Figure 13
Figure 13: Scoring rubric visualization for Basket Carry. Stage 2: Lift Basket
Stage 3: Place Basket on Table
0 0 points: The robot fails to place the basket onto the table surface (left of Figure 14p.
0.5 points: The basket is placed on the table but is either dropped from height or left significantly misaligned with the table's edge (middle of Figure 14).
1.0 points: The robot smoothly and squarely places the basket onto the table (right of Figure 14)
Figure 14: Scoring rubric visualization for Basket Carry. Stage 3: Place Basket on Table
Stage 4: Place Markers Basket
0 0.0 0 points: The robot fails to place any of the markers into the basket (left of Figure 15).
0.5 points: The robot successfully places the marker from one side into the basket (middle of Figure
1.0 poi nts:: The robot successfully places both markers into the basket (right of Fig- ure 15
Prompt: move to the storage box, pick up the storage box and place it on the table, then put the pen on the table into the box
B.5 TRASH DISPOSE
Task Description This task is composed of four sequential stages, designed to evaluate the robot's capability in long-horizon planning and whole-hody coordination within practical cleanup scenario.
19

---

Published as a conference paper at ICLR 2026
+0.0
Figure 15: Scoring rubric visualization for Basket Carry. Stage 4: Place Markers in Basket
In the first stage, the robot presses the top of tabletop trash can to open its spring-loaded lid. In the second stage, it turns its body to the right and bends down to grasp a crumpled paper towel placed on nearby chair, action requiring substantial whole-body coordination to maintain balance. For the third stage, the robot turns back to the left, moves its arm above the trash can opening, and releases the paper towel. In the final stage, the robot must close the lid and press it down again to lock it in place.
To promote policy generalization, each trial initializes with the robot in randomized starting pose and the paper towel at a varied position on the chair, both within predefined areas. This task presents significant challenge due its extended, multi-stage nature. It demands seamless transitions between pressing, grasping, placing, and locking sub-tasks, all while executing complex, coordinated movements.
Stage 1: Open Trash Can Lid
0.0 0 points: The robot fails to open the trash can lid (left of Figure T6..
0.5 points: The robot opens the lid but does not succeed on the first attempt (middle of Figure 16..
* 1.00 points: The robot successfully opens the lid on the first attempt (right of Figure 16].
+0.0
+ 0.5
+1.0
Figure 16: Scoring rubric visualization for Trash Dispose. Stage 1: Open Trash Can Lid
Stage 2: Grasp Paper Towel
0.0 points: The robot fails to grasp the paper towel from the chair (left of Figure 17
0.5 pointa: The robot grasps the paper towel but not on the first attempt (middle of Figure 17).
1.0 points: The robot successfully grasps the paper towel on the first attempt (right of Figure
+0.0
0.5
+
+1.0
Figure 17: Scoring rubric visualization for Trash Dispose. Stage 2: Grasp Paper Towel
Stage 3: Dispose of Paper Towel
0.0 points:
The robot fails to dispose of the paper towel into the trash can (left of
Figure 18)
0.5 points:
The paper towel lands the rim or gets stuck at the edge of the trash during disposal (middle of Figure 18).
1.0 points: The robot successfully disposes of the paper towel into the trash can (right of Figure 18).
20

---

Published as a conference paper at ICLR 2026
+ +0.0
0.5
+ +1.0
Figure 18: Scoring rubric visualization for Trash Dispose. Stage Dispose of Paper Towel
Stage 4: Close and Lock Trash Can Lid
0.0 0 points: The robot fails to close the lid of the trash can (left of Figure 19
0 points: The robot pushes the lid down but fails press it again lock causing
the lid to remain unlatched (middle of Figure 19).
1.0 points: The robot successfully closes the lid and presses it to ensure it is securely locked (right of Figure 19)
+0.0
0.5
+1.0
Figure 19: Scoring rubric visualization for Trash Dispose. Stage 4: Close and Lock Trash Can Lid
Prompt: Open the trash bin, turn around, pick up the trash on the chair and put it into the bin, then close the bin.
B.6 TASK COMPLEXITY METRIC
To provide quantitative and objective measure of difficulty for task suite, compute composite complexity score for each task from the collected expert demonstration data (Dexpen). This score is derived from four distinct metries, each capturing a different aspect of task complexity, Let single expert trajectory be sequence of 21-DoF joint states {qt Y The metrics are calculated for each trajectory and then averaged across all demonstrations for a given task.
Temporal Complexity (Ctmm) This metric captures the temporal length of the task and is calculated as the average duration in seconds over all expert demonstrations.
1 N
Crime
Σ
T Al,
(11)
where is the number of expert trajectories and At the time step duration.
Kinematic Complexity (Caimematie) This metric quantifies the total magnitude of motion required, which is distinct from temporal duration. is calculated the average sum of the & norm of joint displacements between consecutive timesteps, capturing the spatial extent of the behavior.
N
T
Ckinematic
1
Σ
( 1
qe 1
(12)
N
==1
t=1
(8) is the joint state vector for the i-th trajectory time t. where di
Control Complexity (Ccontrot) This metric serves as proxy for control difficulty by measuring the lack of smoothness in the motion. We approximate the average total jerk using the third-order finite difference the joint positions.
T ΣΣ
N
Ocontrol
1
3 + 3q. (6)
112))12.
(13)
N
1=1
1=2
Higher values indicate more frequent changes in acceleration, suggesting a higher demand on the controller.
21

---

Published as a conference paper at ICLR 2026
Coordination Complexity (Ccoora) This metric estimates the number of joints actively involved in the task. For each trajectory, we compute the variance of for each the 21 joints over time. The effective dimensionality is the average number of joints whose variance exceeds a small threshold (e.g., 10-1).
N
21
Cooord =
Σ
1(0}
> c).
(14)
N
where I(()) is the indicator function.
synthesize these individual metrics into a single, comparable score for each task, we first perform min-max normalization on each of the four metrics across the entire task suite. This procedure scales the values of each metric to the range of [0, 1], ensuring that each component contributes equally to the final score regardless of its original units scale. The final composite complexity score, Chinat is then computed as the unweighted average these four normalized scores, providing holistic and unified measure of task difficulty. The results of this analysis presented Table
Complexity Score
Table 4: Quantitative analysis of task complexity. All presented metric scores (C²) have been min-max normalized to the range [0, for direct comparison. The final score is the unweighted average of these normalized values, confirming our task suite spans graduated range of difficulty.
Task
Temporal
Kinematic
Control
Coordination
Final Score (Cimal
(Crime)
(Ciinemttc)
(Ceuntra))
(Cooor))
Pen insert
0.00
0.00
0.05
0.01
0.02
Cup upright
0.06
0.51
0.00
0.00
0.14
Wipe board
0.63
0.73
1.00
0.06
0.61
Basket carry
0.47
1,00
0.86
0.78
0.78
Trash dispose
1.00
0.92
0.36
1.00
0.82
B.7 DATASET DESCRIPTION
The WB-50 dataset is reward-labeled whole-body manipulation dataset spanning over 50 hours of diverse robot experience. WB-50 contains three data sources to reflect realistic data distributions: (i) expert demonstrations (43.7%), (ii) imperfect teleoperation (14.6%), and (iii) policy rollouts (41.7%) the latter two comprising the majority, mirroring the scarcity of perfect supervision in practice. The proportion of the five task data frames the total number of frames in the dataset is shown in the Figure 20 It also shows the successful and failed trajectories generated during the expert data collection process of each task, and the number of frames of successful and failed trajectories generated by imitation learning strategy reasoning is also shown. Moreover, assign rewards of O, 0.5, and at the end each subtask according to degree of task completion, and apply step penalty of -0.001 for all other steps.
C IMPLEMENTATION DETAILS
C.I ALGORITHM PIPELINE
present the pseudocode our method as in Algorithm I
C.2 IMITATION LEARNING BASELINES
WB- VIMA. Our implementation of WB-VIMA is based the official policy codebase (Jiang et al. 2025) and applies minor adjustments to the model and training hyperparameters. The detailed parameters summarized in Table
Diffusion Policy, We build our diffusion policy on the official GalaxeaDP codebase TTam 2025). which demonstrates strong compatibility with the Galaxea R1 robot, also developed by Galaxea. As
22

---

Published as a conference paper at ICLR 2026
Dataset Source
Task Duration
14.6%
14.7%
22.9%
43.7%
12.6%
41.7%
15.3%
34.5%
Expert Demonstrations
Pen Insert Cup Upright
Basket Carry
Policy Rollouts
Trash Dispose
Suboptimal Human Data
Wipe Board
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
3.0
Frame number (x109)
2.5
2.0
1.5
1.0
0.5
0.0
joylo success
loyle fail
rollout success rollout fail
loylo success
joylo fall
rollout success
rollout fall
joylo success
joylo fail
rollout success
rollout fall
joylo success
joylo fail
rollout success
rollout fail
Joylo success
joylo fall
rolouut success
rollout fail
Figure 20: Data distribution of WB-50. Distribution of WB-50. The plot shows the proportional composition of the dataset by source, including expert demonstrations, suboptimal expert trajectories and rollout data, together with counts of frames of successful and failed episodes collected during data acquisition.
Algorithm Hierarchical Value- Decomposed Offline Reinforcement Learning (HVD)
Input: Offline dataset D, action hierarchy: Abuse Awroo X Aurm
2: Initialize value network Vg. Q-network Qe, policy network Ts
(Phase 1: Hierarchical Value Learning)
3: for each gradient step do
4: (Update value network)
5: v &- & Av V Vv by Equation equation 6
6: (Update Q-network)
7: 0 x 8 \y (o) by Equation equation 5
8: end for
(Phase 2: Policy Extraction)
9: for each gradient step do
(Update policy network)
11: à 6 X₂ V Ca (Ø) by Equation equation 10
end for
13: Output: Trained policy T
23

---

Published as a conference paper at ICLR 2026
Table Hyperparameters of WB-VIMA model.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameler
Value
PointNet
Prop. MLP
Transformer
N pod
4096
Input Dim
Embed Size
512
Hidden Dim
256
Hidden Dim
256
Num Layers
8
Hidden Depth
Hidden Depth
3
Num Heads
8
Output Dim
256
Output Dim
256
Drop Rate
0.1
Activation
GELU
Activation
ReLU
Activation
GEGLU
Table 6: Hyperparameters of WB-VIMA training process.
Hyperparameter
Value
Learning Rate
10-4
Weight Decay
0.1
Learning Rate Warm Up Steps
1000
Learning Rate Cosine Decay Steps
300,000
Minimal Learning Rate
x 10
the original implementation supports only 14-DoF dual-arm tasks, we extend it by incorporating additional control for the torso and mobile base enable 21-DoF whole-body control tasks. The hyperparameters and model architectures used in our experiments are summarized in Table 7
Hyperparameter
Value
Batch Size
32
Chunk Size
20
History Size
2
Learning Rate
le-4
LR Scheduler
cosine
Optimizer
Adam
AdamW Betas
[0.9, 0.95]
Weight Decay
le-4
Max Training Steps
100,000
Image Type
RGB
Egocentric Perception Type
Joint
Observation Encoder
ResNet-18 (He et al. 2016
Diffusion Model
DDPM (Ho all 2020)
Diffusion Steps
20
Diffusion Network
U-Net Ronneberger et [256, 1024, 4096]
2015
U-Net Structure
Table 7: Hyperparameters of Diffuusion Policy.
To- We adopt the official implementation of To Black rameters are listed in Table
2024 as our codebase. Key hyperpa-
C.3 HVD IMPLEMENTATION DETAILS
Model Architecture. HVD is implemented using a transformer backbone, which naturally accommo- dates multiple input modalities. Specifically, the model processes observations through specialized encoders for each modality:
RGB Input: Three egocentric RGB views (front, left, and right) are independently processed using SigLIP (Zhai et al. 2023), producing sequences of visual tokens that capture spatial context and object semantics.
24

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Batch Size
32
Chunk Size
20
Learning Rate
le-4
LR Scheduler
cosine
Optimizer
AdamW
AdamW Betas
[0.9, 0.95]
AdamW Epsilon
1e-8
Weight Decay
1e-10
Max Training Steps
50,000
Fine-tune Method LoRA Hu et al. 2022
Table Hyperparameterss of mo-
Point Cloud Input:
Depth-derived point clouds encoded with PointNet (Qi al. 2017) modules, enabling robust perception of 3D geometry and scene layout, which is particularly beneficial for navigation and object manipulation.
Task Instruction: Natural language commands (e.g.. "clean the whiteboard") tokenized and embedded to provide high-level goal guidance.
Proprioception: Joint angles, velocities, and end-effector poses are concatenated and normalized form compact state vector representing the robot's internal configuration.
The tokens produced by all modalities concatenated along the sequence dimension form unified representation. Attention masks regulate moosll interactions, after which the integrated
token sequence processed by a pretrained PaliGemma model (Beyer al.) 2024). The resulting
representations are then passed through an MLP-bbseed value decoder to estimate Q-values for the different hierarchical components. The overall model architecture is depicted in 21
NOOOOO
Vallss Decodar
Qaaae
Transformer
Valuue Decoder
+ Quaree
VIT
VIT
VIT
Point Net
Point Net
"cloan the whiteboard"
statt
8:1l
Value Cecode:
Qurm
Egocentric RGB
Egocentric Point Gloud
Task Instruction
Proprioception tion
Figure 21: Overall Model Architecture of Hierarchical Q-Network.
Hyperparameters. In our implementation, the parameter a in equation 6 controls the relative weighting between the TD loss and the BC loss, effectively balancing value estimation and policy imitation, while 3 in equation 10 is used in the exponential weighting of advantages when computing hierarchical action weights, modulating the sensitivity set to their default values of 1.0. Importantly, we did not perform any hyperparameter tuning, our method already achieves strong performance, highlighting the effectiveness and robustness of the proposed HVD approach. Configurations and hyperparameter settings are listed in Table 9
sigh-advantage actions. Both a and
D ADDITIONAL EXPERIMENTS
D.1 QUALITATIVE VISUALIZATION OF HIERARCHICAL CREDIT ASSIGNMENT
The remaining weight visualization results presented in Figure 22
We observe that the HVD w/o Hierarchy weights exhibit a trend similar to the HVD arm weights across all tasks. This indicates that while HVD w/o Hierarchy is able capture key frames in which the arms about to move,
25

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Value Network
MLP[128 x 256 x 64] 30,000
Max Training Steps
BC Loss Weight a
Exponential Weight
Image Encoder
SigLIP Zhai et al. 2023
Point Cloud Encoder
PointNet Qi et al.
2017
Transformer Model
PaliGemma
(Beyer et al. 256
2024
Width
Depth
MLP Dim
1024
Number of Heads
Number of KV Heads
Table Hyperparameters of HVD.
fails to effectively recognize the contributions the base and torso. These results further support our conclusion that HVD provides more accurate credit assignment.
Pen Insert
Weight
Cup Upright
Wipe Board
Weight
Trush Dispose
Weightt
HVD Base
HVD Torso
HVD Arm
HVD wio Hiemarcyy
Figure 22: Remaining Credit Assignment Comparison between HVD and HVD w/o hierarchy.
D.2 QUANTITATIVE ANALYSIS OF HIERARCHICAL CREDIT ASSIGNMENT
To validate the interpretability of our hierarchical framework, analyzed the component weights during policy training across different task stages. The quantitative results in Table I0 reveal distinct stage-aware credit assignment patterns that align with functional requirements. These results
26

---

Published as a conference paper at ICLR 2026
demonstrate that our hierarchical framework successfully differentiates functional requirements across task stages and allocates appropriate credit to different robot components.
Table 10: Hierarchical component weights across task stages.
Component Weight
Task
Stage
Base
Torso
Arm
Stage I: Approach and Grasp Eraser
0,962
0.884
1.321
Wipe Board
Stage 2: Wipe Markings
0.946
0.890
1.054
Stage 3: Return Eraser
0.857
0.826
1.149
Stage 1:
Approach and Grasp Basket
0,980
0,999
1.405
Stage 2: Lift Basket
0.953
0,909
1.254
Basket Carry
Stage 3: Place Basket Table
0.837
0.807
1.063
Stage 4: Place Markers Basket
0.972
0,980
1.243
Stage 1: Open Trash Can Lid
0.933
0.918
1.382
Stage 2:
Grasp Paper Towel
0.930
0.960
Trash Dispose
Stage 3:
Dispose of Paper Towel
0.944
0,902
1,356
Stage 4:
Close and Lock Trash Can Lid
0.804
0.813
1.323
D.3 ANALYSIS OF TASK COMPLEXITY AND POLICY PERFORMANCE
By correlating the difficulty metrics presented Table with the actual success rates learned policies in [ we can identify several key bottlenecks in current whole-body control approaches and point out the primary sources performance gains introduced by HVD. Specifically, our analysis reveals that high control complexity and high kinematic coordination demands are the primary failure modes for standard IL, which HVD successfully mitigates.
Overcoming the Smoothness Bottleneck in High- Higgh-Contro Complexity Tasks. Ocontol serves the most correlated predictor of failure for baseline methods. For instance, the Wipe Board task exhibits the maximum Control Complexity (Cevanol 1.00), indicating requirement for frequent acceleration changes and high jerk. Standard IL baselines collapse on this task. In contrast, HVD significantly improves To performance with +77% relative improvement, demonstrating that HVD effectively models the non-smooth, high-frequency dynamics often lost in standard training.
Robustness in Bligh-Dimensionn Coordination. IL baselines often fail the tasks characterized by high Ckinematic and Cooord due to the difficulty of coordinating nigh-dimensional joints over large spatial displacements. The Basket Carry task, which possesses the highest Kinematic Complexity (Oxinematic = 1.00) and very high Coordination Complexity (Cooord 0.78), illustrates this barrier. While the WB-VIMA and DP baselines struggle significantly (with DP failing almost completely), HVD provides its most robust improvement here, lifting the To success rate from 0.26 to 0.44. This indicates that HVD acts as a superior regularizer in high-variance regimes, maintaining structural integrity over wide state spaces where standard IL fails generalize.
D.4 SIMULATOR EXPERIMENTS
introduce a simulation experiment on the BEHAVIOR-IK platform to empirically validate our approach, directly addressing the reviewer's query (W5) for concrete simulated environment.
D.4.1 SIMULATOR PLATFORM
We conduct our empirical evaluation using the BEHAVIOR-IK simulation environment (Li et al. 2024). This platform is high-fidelity, standardized benchmark focused on humanoid roboties tasks grounded in real-work human needs. The simulated robot model features base, torso, and arm structure that necessitates complex whole-body coordination, aligning directly with methodological requirements for addressing the high- Diggeniimensii the action space. Figure provides overview of the simulator setup.
27

---

Published as a conference paper at ICLR 2026
BEHAVIOR
Figure 23: Overview of the simulator in the BEHAVIOR-
Task Design and Decomposition. We selected the Picking Up Trash task from the official BEHAVIOR- 1K benchmark as our test environment. This task highly demands robust whole-body body control capabilities. Given the computational intensity of long-horizon rollouts in this complex simulator, we constrained each episode to 10,000 control steps. While this limitation prevents full task completion, is sufficient to the most critical phases. To facilitate focused evaluation and learning, we decomposed the task into two distinct, sequential stages with clear success criteria:
Successful if the robot positions itself directly front of the trash bin, such that the bin is visible in at least one camera view.
Approaching:
Grasping: Successful if the robot lifts the trash bin off the ground using either arm.
For reward labeling, successful trajectories (meeting stage criteria) receive a terminal reward of +1.0, provided at the end of the trajectory. Additionally, every intermediate frame incurs small step penalty of -0.001 to encourage efficient execution. The overall task pipeline, illustrating this two-stage structure, is presented in Figure
Figure 24: Task pipeline for the Picking Up Trash task in BEHAVIOR-IK, illustrating the approaching and grasping stages evaluated in our experiments.
Dataset and Evaluation. Our experimental analysis utilized training dataset of 300 trajectories: 200 expert demonstrations from the official BEHAVIOR- benchmark, supplemented by 100 policy rollouts collected using the pre-trained -l baseline. During evaluation, we tested each policy on 100 trials across 20 unseen scenarios to ensure robustness and generalizability. Table 12 summarizes the performance of the baseline Imitation Learning (To-IL) policy and our Hierarchical Value Decomposition (TT-HVD) method across both evaluation stages. HVD consistently demonstrates significant performance improvement over the baseline:
Implementation Details. The HVD implementation built directly on top of the official mo codebase provided by the BEHAVIOR- 1K benchmark, ensuring fair and consistent comparison with the Ny. IL baseline. All implementation details, including the hierarchical architecture, network configurations, and hyperparameter settings, remain the same as those introduced in Section with only minor adjustments as summarized in Table
Task Results. Table 12 summarizes the evaluation outcomes, showing that HVD substantially outperforms the IL baseline. Stage 1, HVD achieves a 77.0% success rate than IL's 45.0% %. The advantage becomes nearly doubles the success rate, reaching 22.0% compared to IL's 5.0%. These results highlight HVD's effectiveness in handling complex, high-dimensional whole-body coordination tasks.
pronounced in the more challenging grasping stage, where HVD
28

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Max Value Training Steps
10,000
Max Policy Training Steps
80,000
Hierarchical Order
[[0-2], [3-6], [7-22]]
Table 11: Adjusted hyperparameters of HVD BEHAVIOR-IK.
Method
Stage (SR)
Stage 2 (SR)
To -T-IL IL
45.0%
5.0%
To HVD
77.0%
22.0%
Table 12: Results for Picking Up Trash task in BEHAVIOR-IK.
D.5 ABLATION STUDY ON DATA DISTRIBUTION
To further empirically validate the data distribution robustness of HVD, we conduct an additional ablation on the Pen Insert task under varying expert ratios. During training, we keep the total number of demonstrations fixed at 100 with expert ratios of 20%, 50%, and 80%. During training, To+ uses only the expert demonstrations, while HVV uses all available demonstrations, including both expert and suboptimal data. The results are shown in Table 13 where HVD consistently outperforms II. confirming that HVD maintains its effectiveness across a wide spectrum of data compositions. Due to time constraints, each policy is evaluated over 20 trials.
Table 13: Success rates the Pen Insert task under varying expert-to-imitation data ratios.
Method
20 exp 80 imp
50 exp + 50 imp
80 exp 20 imp
0.15 HH
To+IL
0.35
0.50 0.75
0.55
0.60
D.6 COMPARISON WITH OFFLINE RL BASELINES
To further benchmark HVD against other offline RL methods, we implement QIPO (Zhang et al.) using the same base policy To and train multi-task learning setting. For fair comparison, QIPŌ and HVD share identical network architectures, optimizer configurations, and training hyperparam- eters. The only differences lie in: the policy update mechanism, and (ii) the use of hierarchical value decomposition. Specifically, QIPO initializes its policy and Q-networks using BC and TD losses, respectively, followed by iterative policy improvement steps. We adopt the hyperparameter recommendations from the original QIPO paper as our starting point and perform minimal tuning to ensure stable convergence. The final configuration uses 3 = 1, M 16 (number of sampled actions), and Krenow 10 (policy renewal interval).
As shown in Table 14 HVD consistently outperforms QIPO across all five tasks in terms of success rate (SR). The improvement is most significant in long-horizon, whole-body control tasks, such as Basket Carry and Trash Dispose, where accurate credit assignment over extended action sequences is critical. We attribute this advantage HVD's hierarchical value decomposition, which enables more precise reward propagation and subgoal-aware policy learning.
D.7 ABLATION STUDY ON DECOMPOSITION ORDER
To validate the impact of different decomposition orders and validate our design choice, we conduct an ablation experiment. Specifically, we compare two alternative sequencing strategies: (i) Arm-Torsc. Base and (ii) Base-Arm- Aans--ArTTToooo Torso, using the same base policy To across all tasks within multi-task learning setup. Due to time constraints, the baseline policy evaluated over 20 trials per task. The results are summarized in the Table [5] clearly demonstrating that our proposed decomposition order outperforms the alternatives, highlighting the importance of the hierarchical structure in approach.
29

---

Published as a conference paper at ICLR 2026
Method
Tasks
Avg SR
To MultiTask
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
QIPO 0.90 0.92
0.90 0.94
0.30 0.32
0.20 0.50
0.25 0.30
0.51
HVD
0.60
Table 14: Task-level success rate HVD and QIPO across tasks.
Table 15: Success rates (SR) for different decomposition orders across tasks.
Method
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
Avg SR
Arm-Torso-Bee Torso-Base
0.55
0.60
0.15
0.20
0.10
0.32
Torso-Arm-Bass
0.80
0.80
0.20
0.35
0.20
0.47
Base-Torsu-Arm (ours)
0.92
0.94
0.32
0.50
0.33
0.60
D.8 COMPARISON WITH RESIDUAL VALUE DECOMPOSITION
To further validate the residual approach versus our proposed independent decomposition, we add an additional baseline across all tasks within a multi-task learning setup, which learns the residual Q-function Qresidual using the formulation defined above. Due to time constraints, the baseline policy evaluated over 20 trials per task. The results are as shown in Table [6] showing that our independent decomposition outperforms the residual variant baseline, achieving higher average success rate.
Table 16: Comparison between residual and independent decomposition variants of HVD.
Method
Pen Insert
Cup Upright
Wipe Board
Basket Carry
Trash Dispose
Avg SR
HVD (residual)
0.75
0.80
0.25
0.50
0.35
0.53
HVD (ours)
0.92
0.94
0.32
0.50
0.33
0.60
E RESOURCE COST
E. REWARD LABELING CoST
Reward signals manually annotated according to the task definitions provided in Appendix For each demonstration, operators carefully review the recordings from multiple camera views (head, left wrist, and right wrist) and mark the key frames where the robot's actions satisfy predefined scoring criteria. This process is highly labor-intensive: labeling the entire WB-50 dataset required two well-trained operators working for nearly hours. The labeled data are subsequently used to train HVD, yielding improved policy performance.
E.2 COMPUTATIONAL COST
All trainings are conducted on RTX 4090 GPU platform. During the value learning stage, Hierarchical networks are trained 4xRTX 4090 GPUs for approximately hours. For policy learning, LoRA fine-tuning of To requires around 40 hours on 4xRTX 4090 GPUs, while training the DP takes I XXTX 4090 GPU for 30 hours. WB-VIMA policies are trained on 2x RTX 4090 GPUs for around 24 hours. For deploying policies, 1 XRTX 4090 GPU is required to load model parameters and run action inference.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large language models (LLMs) are used in the preparation of this manuscript for sentence-level editing, including improving grammar, clarity, and readability.
30


Reviews:
[{"review_id": "BXrRIes7fS", "reviewer": "Reviewer_qv84", "summary": "This paper proposes Hierarchical Value-Decomposed (HVD) Offline Reinforcement Learning, a framework designed for high-dimensional, whole-body control in robotic systems. The authors motivate their approach through empirical and theoretical challenges in whole-body control. The authors then introduce the value decomposition framework, utilized within a transformer architecture, and achieve more accurate credit assignment across long-horizon behaviors. The authors present numerous results that show HVD significantly improves imitation performance, credit assignment, and can scale to multi-task settings.", "questions": "Please address the weakness noted above.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ Paper presents ample contributions, including a novel offline RL method for whole-body control, a dataset of whole-body behavior, and an implementation that support multi-modal and multi-task learning.\n+ Paper is very well-written.", "weaknesses": "- The authors note that HVD relies on human-annotated rewards. Could you provide further information on where these come from and whether these are noisy reward signals? Further, does the framework depend on a specific distribution of data? For example, if 80% were imperfect demonstrations, would the framework still be able to learn behaviors well?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QgzJ2AxUaf", "reviewer": "Reviewer_87vt", "summary": "The authors aim to apply offline RL methods to whole-body robotic control. In general, whole-body control is more complicated than only manipulation control, due to larger degrees of freedom, increased variation from adjusting robot position, etc. These increase the complexity of the RL problem.\n\nTo address this, the authors propose using a hierarchical offline RL approach. Instead of treating the action $a$ as a monolithic action, it is segmented into three components, $a_{base}, a_{torso}, a_{arm}$. A Q-function is trained with each conditioning on all prior modalities, giving 3 Q-functions $Q(s, a_{base})$, $Q(s, a_{base}, a_{torso})$, and $Q(s, a_{base}, a_{torso}, a_{arm})$. Each is fit with its own TD loss term  but shares base architecture. For learning the Q-function and extracting a policy at the end, the authors use IDQL from prior work (fit Q-function with TD-learning then apply diffusion AWR to extract a policy). Each action component $a_{base}, a_{torso}, a_{arm}$ is extracted using advantage weights computed from the corresponding Q-function.\n\nTo evaluate this, the authors collect a 50 hour dataset of whole body control on the Galaxea platform, made of a mix of expert demonstrations, suboptimal demonstrations from operators that are worse at teleoperation, and policy rollouts from learned policies. The authors show their offline RL outperforms pure IL, and that the multi-level Q-function performs better than using no hierarchy.", "questions": "Are expert demonstrations vs suboptimal rollouts evenly split among tasks, or are some tasks more biased towards one or the other?\n\nHow is inference speed affected by dividing the action space this way?\n\nWhy is so much of the appendix spent defining quantitative measures of task complexity that don't seem to be used anywhere else?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important problem of learning from suboptimal offline data. Collecting and labeling a 50 hour dataset is potentially quite useful. The paper acknowledges some of the weaknesses of the approach as well (specifically the need to do this labeling). Ablations suggest there are gains from doing this more hierarchical setup, and visualization of advantage weights over the subtask are compelling.", "weaknesses": "The theoretical analysis seems entirely unnecessary to me. The cited Rajaraman et al 2020 bound is based on a worst case assumption of environment dynamics and generalization, and given that empirical dataset sizes are significantly smaller than this bound, it doesn't really seem to apply and just seems like an overly complicated way to state that higher dimensionality problems may need more data to fit.\n\nThe specific decomposition into base, torso, and arm is pretty specific to whole-body control. It seems like a special case of autoregressively fitting + extracting a Q-function 1 action dimension at a time, similar to Q-Transformer (although Q-transformer does not do any AWR weighting / diffusion step).\n\nIt was unclear to me how many trials were used to generate the task success rate tables of Table 1 / Table 2, or how much the hierarchical setup affects action inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEwwPODfeu", "reviewer": "Reviewer_ruzg", "summary": "The paper indicates that imitation-learning approaches suffer from state-action-space explosion when applied to whole-body control, as a consequence it's unrealistic to collect the exponentially-growing expert data.\nOn the other hand, suboptimal trajectories are more readily available, motivating the use of offline reinforcement learning.\nThe paper proposes to use hierarchical value deomposition based on the embodiment (body, torso, arm) to improve credit assignment.\nTo demonstrate the improvement, the paper provides a curated dataset consists of optimal and suboptimal demonstrations, and policy rollouts, for multiple tasks such as pen insert, cup upright, wipe board, basket carry, and trash dispose.\nOn this dataet, the paper shows that hierarchical-value deomposition outperforms the standard imitation-learning approach on three vision-language-action models.", "questions": "See weakness above", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clear and easy-to-follow.\n- The experiments are conducted on a real-world robot that requires substantial efforts.", "weaknesses": "- Motivation\n\t- In the abstract, the paper states that scaling imitation learning (IL) to high DoF is limited by non-stationary observation transition. My concern with this statement is that if this is indeed true, the paper should not be focusing on formulating whole-body control as a Markov decision process (MDP), since MDPs do assume stationary transitions.\n\t- The paper indicates, on lines 38-42, that scaling IL approaches is fundamentally limited by the number of state-actions exploding exponentially. I agree with this statement, but I also agree that offline reinforcement learning (RL) approaches also suffer from this problem. In fact, one can argue that offline RL scales worse without data coverage and appropriate realizability assumptions [1].\n- Method\n\t- Eq. 1: Why this choice of ordering the base, torso, and arm? Is there any ablation to demonstrate that this is the \"best\" way to decompose the Q-function?\n\t- Eq. 2: It seems like for the later Q-functions, we can simply regress based on, e.g.,  $Q_{base} - Q_{torso}$. That is, why not $Q(s, a) - Q(s, a, a')$? From one perspective we can view $Q_{base}$ as the baseline for $Q_{torso}$ and so on.\n- Experiments\n\t- While I appreicate the challenges in producing results on a real robot, I strongly believe reproducibility is important---as a result, it would be great if the paper can provide even a single simulated environment, perhaps MuJoCo humanoid, that this approach can indeed improve upon existing methods.\n\t\t- I am unsure where the dataset, WB-50, will be provided---that will be greatly appreciated for reproducibility purposes as well.\n\t- What is the number of rollouts for evaluation?\n\t- How is \"better credit assignment\" analyzed? I was hoping for analysis on Q-values being more accurately propagated with the decomposition.\n\t- While the theory suggests the exponential explosion due to increasing DoF, it would be nice if empirically the paper demonstrates this problem with ablation on dataset size vs performance on increasing number of DoF.\n\nReferences:  \n[1] Zhan, Wenhao, et al. \"Offline reinforcement learning with realizability and single-policy concentrability.\" Conference on Learning Theory. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mM87Yf2v1K", "reviewer": "Reviewer_xcm5", "summary": "This work presents HVD (Hierarchical Value-Decomposed Offline RL), a framework that learns high-DoF whole-body robot control from imperfect, reward-labeled data. By decomposing value functions along the robot’s kinematic hierarchy and temporal structure, HVD enables precise credit assignment in long-horizon tasks. Built on a Transformer architecture, it supports multi-modal, multi-task learning. The authors also release WB-50, a 50-hour teleoperation and policy rollout dataset capturing natural imperfections. Experiments show HVD significantly outperforms baselines, demonstrating that effective whole-body control can emerge from structured learning on imperfect data.", "questions": "1. Could you revise Figure 2 and refine the theoretical analysis section to strengthen the justifications and improve clarity?\n\n2. Could you include additional comparisons with offline RL methods that also utilize reward signals for a fairer evaluation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses an interesting and challenging problem of humanoid whole-body control in high-dimensional settings. It proposes a hierarchical decomposition of the Q-function to effectively handle the high degrees of freedom.\n\n2. The empirical validation includes real-world experiments across multiple tasks, demonstrating the effectiveness of the proposed HVD approach. In addition to the method, the paper also introduces a new dataset used for training.\n\n3. The experimental details are clearly presented and appear to be sufficient for reproducibility.", "weaknesses": "1. **Figure clarity**: The pipeline figure (Figure 2) lacks captions, making it difficult to follow and interpret. Clear labeling of modules would improve readability.\n\n2. **Theoretical analysis**: The analysis in Proposition 3.1 is limited to the tabular case and cannot be directly applied to humanoid whole-body control, which involves continuous, partially observed states (e.g., image inputs). The justification should be revised, possibly using covering number arguments for general function approximation. The overall intuition—that higher observation dimensionality increases the need for expert data—is valid. However, the proposed hierarchical method decomposes the action space, not the state space, which may not align with the theoretical bound based solely on state-space cardinality.\n\n3. **Experimental fairness**: The experiments compare the proposed offline RL method (which uses reward information) with pure imitation learning, which does not. This comparison is potentially unfair. The authors should include more offline RL baselines based on generative policies [2, 3] that also utilize rewards for a fair evaluation.\n\n4. **Related work**: Several relevant studies on humanoid whole-body control that also exploit hierarchical structures (e.g., [1]) are missing from the discussion and should be cited for completeness.\n\n\n[1] Hansen, N., Jyothir, S. V., Sobal, V., LeCun, Y., Wang, X., & Su, H. Hierarchical World Models as Visual Whole-Body Humanoid Controllers. In The Thirteenth International Conference on Learning Representations.\n[2] Lu, C., Chen, H., Chen, J., Su, H., Li, C., & Zhu, J. (2023, July). Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (pp. 22825-22855). PMLR.\n[3] Zhang, S., Zhang, W., & Gu, Q. Energy-Weighted Flow Matching for Offline Reinforcement Learning. In The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)