H$^3$DP: Triply‑Hierarchical Diffusion Policy for Visuomotor Learning
Authors: Yiyang Lu
Yufeng Tian
Zhecheng Yuan
Xianbang Wang
Pu Hua
Zhengrong Xue
Huazhe Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Imitation Learning
Representation Learning
Diffusion Model
Primary area: applications to robotics, autonomy, planning

Abstract: Visuomotor policy learning has witnessed substantial progress in robotic manipulation, with recent approaches predominantly relying on generative models to model the action distribution. However, these methods often overlook the critical coupling between visual perception and action prediction. In this work, we introduce Triply-Hierarchical Diffusion Policy (H$^3$DP), a novel visuomotor learning framework that explicitly incorporates hierarchical structures to strengthen the integration between visual features and action generation. H$^3$DP contains $\mathbf{3}$ levels of hierarchy: (1) depth-aware input layering that organizes RGB-D observations based on depth information; (2) multi-scale visual representations that encode semantic features at varying levels of granularity; and (3) a hierarchically conditioned diffusion process that aligns the generation of coarse-to-fine actions with corresponding visual features. Extensive experiments demonstrate that H$^3$DP yields a $+ \mathbf{27.5}$% average relative improvement over baselines across $\mathbf{44}$ simulation tasks and achieves superior performance in $\mathbf{4}$ challenging bimanual real-world manipulation tasks. Project Page: https://h3-dp.github.io/.

Published as a conference paper at ICLR 2026
H TRIPLY-HIBRACHHIAAL DIFFUSION POLICY FOR VISUOMOTOR LEARNING
Yiyang Lu Yufeng Tian Zhecheng Yu.n 1.2,3
Xianbang Wang", Pu Hual.2.3 Zhengrong Xue Huazhe Xu3.2,3
Institute for Interdisciplinary Information Sciences, Tsinghua University, 2 Shanghai Qi Zhi Institute, Shanghai Al Lab, Harbin Institute of Technology 1uyy24@ma 1s.... tsi tainghua edu.cr, cn, hua zhe xuêma .tsinghua edd..cn
5 Sim Benchmarks
40.1
4 Real-World Tasks
Instance
Spatial
Figure I: DP can not only achieve superior performance across 44 tasks on simulation bench- marks, but also handle long-horizon challenging manipulation tasks in cluttered real-world scenar- ios.
ABSTRACT
Visuomotor policy learning has witnessed substantial progress in robotic manip- ulation, with recent approaches predominantly relying on generative models to model the action distribution. However, these methods often overlook the critical coupling between visual perception and action prediction. In this work, we intro- duce Triply-Hierarcchcal Diffusion Policy (H$DP), novel visuomotor learn- ing framework that explicitly incorporates hierarchical structures to strengthen the integration between visual features and action generation. HDDP contains 3 levels of hierarchy: (1) depth-aware input layering that organizes RGB-D obser- vations based on depth information; (2) multi-scale visual representations that encode semantic features at varying levels of granularity; and (3) hierarchically conditioned diffusion process that aligns the generation of coarse-to-fine actions with corresponding visual features. Extensive experiments demonstrate that DP yields + +2755% average relative improvement over baselines across simula- tion tasks and achieves superior performance in 4 challenging bimanual real-world manipulation tasks. Project Page: https::///// //h3 dp. github.io.
"qqual Contribution

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Visuomotor policy learning has emerged as prevailing paradigm in robotic manipulation 6
70
5 [5] 64]. Existing approaches have increasingly adopted powerful generative methods, such as
diffusion and auto-regressive models, model the action generation process [36; 60 13
46
24
However, these predominant methods have focused primarily on separately refining either the rep- resentation of perception or actions, often overlooking establishing tight correspondence between perception and action. In contrast, human decision- Decision-making inherently involves hierarchical pro- cessing information from perception to action 20 3. The visual cortex extracts features a layered fashion and performs hierarchical inference based on visual motion perception, ultimately resulting in the generation of structured motor behaviors (25] 4 Inspired by this, we argue that enabling learned visuomotor agents to emulate such hierarchical behavior patterns is also critical for enhancing their decision- decision-making capabilities.
Prior works have primarily focused hierarchically modeling the action generation process alone [5T [3], without explicitly incorporating hierarchical structure throughout the whole visuo- motor policy pipeline. In this paper, present DP, novel visuomotor policy learning frame- work grounded in three levels of hierarchy: input, representation, and action generation. This design reflects the hierarchical processing mechanisms that humans the visual cortex to perceive envi- ronmental stimuli to guide motor behavior.
At the input level, to better leverage the depth information in modern robotic benchmarks and datasets [21 29 45 [4], DDDP moves beyond prior 2D approaches that primarily rely on RGB or simple RGB-D D concatenation, which has shown limited effectiveness in prior work [65] 72]. We introduce depth-aware layering strategy that partitions the RGB-D input into distinct layers based on depth cues. This approach not only enables the policy to explicitly distinguish between fore- ground and background, but also suppresses distractors and occlusions (39 1. thereby enhancing the understanding and reasoning of spatial structure in the cluttered visual scenarios.
For visual representation, to address the limitations of flattening image features into single vector, which can discard some spatial structures and semantic information [17 42 27), DP employs multi-scale visual representation, where different scales capture features at varying granularity levels, ranging from global context to fine visual details.
In the action generation stage, DP incorporates key inductive bias inherent to the diffusion process: the tendency to progressively reconstruct features from low-frequency to high-frequency components [11 9 58), by hierarchical action generation. Specifically, coarse visual features guide initial denoising steps to shape the global structure (low-frequency components) of action, while fine-grained visual features inform the later steps refine precise details (high-frequency components). This establishes a tighter coupling between action generation and visual encoding, enabling the policy generate actions that semantically grounded in multi-scale perceptual features.
We validate DP through extensive experiments on 44 simulation tasks across diverse bench- marks, where it surpasses state-of-the-an methods by relative average margin of +27. 5%. Further- more, in real- evaluations, we deploy bimanual robotic systems to tackle four challenging tasks situated in cluttered environments, involving high disturbances and long- nonghoorzoon objectives. DP achieves a 72.4% relative performance improvement over baselines real-world scenarios.
2 RELATED WORK
Visual imitation learning. Numerous studies have proposed efficient policy learning algorithms from different aspects 66 70 61]. As representative approach, to endow the learned policy multi- modality ability, Diffusion Policy (6) incorporates the diffusion process to better represent the action distribution. Based on DP, methods like DP3 (65 66) and 3D-Actor (2)) designed for point cloud inputs, enhance the policy's 's scene understanding by refining the visual representation. Consistency Policy [66] and ManiCM [32 modify the inference process to achieve the inference acceleration. However, these approaches focus solely on enhancing either the action generation or the visual fea- ture extraction, without explicitly modeling the relationship between them. To address this issue,
2

---

Published as a conference paper at ICLR 2026
propose hierarchical framework that couples multi-scale visual representations with the diffusion process, enabling a more structured integration between visual features and action generation.
Leveraging hierarchical information for policy learning. In the computer vision community, numerous studies have leveraged hierarchical information to address a variety of downstream
tasks [55]
59
26
40
31
43
For example, standard diffusion models 449]
19
48
50 and flow
matching 28
30
adopt the U-Net framework 422 71,, which exploits multi-scale feature rep-
resentations to retain rich contextual information throughout the denoising process. VAR [53] inno- vatively employs multi-scale visual representations with quantization to perform image generation in an auto-regressive manner. In robot learning, recent works [15 35 69
have also begun to adopt hierarchical paradigms for policy learning. Dense Policy [51] leverages a bidirectional extension strategy to enable hierarchical action prediction. ARP 669 predicts a sequence of actions at differ- ent levels of abstraction in hierarchical way. CARP [5] draws inspiration from VAR by employing a multi-scale VQ- VAE [55] 40 to construct action sequences and subsequently generating residual actions autoregressively using GPT-style architecture (37]. However, these algorithms model only the hierarchical structure of the action generation process, without explicitly addressing the crucial linkage between visual representation and action in visuomotor policy learning. In contrast, DP not only incorporates multi-scale visual representations but also leverages the inherent strengths of diffusion models seamlessly integrate coarse-to-fine action generation into the diffusion process itself. Furthermore, by adopting a depth-aware layering strategy, HPDP maximizes the utilization of hierarchical feature information across the input, latent, and output stages, thereby enriching the policy learning pipeline in a structured and semantically aligned manner.
3 METHOD
We employ three hierarchical structures to enhance the policy's understanding of visual input and predict more accurate action distributions as shown in Figure 3 At the input level, the RGB-D image is discretized into multiple layers improve the policy's ability distinguish and interpret variations. Upon this, we adopt a multi-scale visual representation, wherein Searse-graines features capture global contextual information, while fine- mne-ggained features encode detailed scene attributes. On the action side, correspondingly. the representations at different scales are utilized to generate actions in coarse-to-fee manner, thus strengthening the correlation between action and visual representations. Details of each component are presented in the following sections.
3.1 DEPTH- LAYERING
Effective robotic manipulation requires a strong grasp of 3D structure. RGB data provides texture and color, while depth encodes geometry such as object positions and distances. Combining them powerful, but naive concatenation of RGB and depth
de
M2
RGB
d2
mask
Depth-Aware Layering
da
often fails to improve performance [65 (72). To
better exploit depth, we partition the RGB-D im- age into N non-overlapping layers based on depth values, as illustrated in Figure 2 Specifically, de-
My
dx
Discretize
Layer
fine (do dmin, di,...,de dy = } as the depth boundaries for each layer. Image layer Im is formed by selecting pixels with depth in [dm-1, dm), i.e.,
Depth
Layer
M Iig (SD(s))dd..))
In = I (1)
Figure 2: Depth-aware layering, We de- compose the RGB-D image into N non- overlapping layers based on depth, and en- code each layer independently for better spa- tial understanding.
where / and 1 are the RGB-D image and depth map, respectively, II is the indicator function. This repre- sentation separates the scene into meaningful fore- ground and background regions while preserving all visual details. It allows the policy to selectively at- tend to different depth planes, thereby improving both spatial perception and planning. We empha- size its effectiveness in Section 443.2 and further compare against alternative discretization schemes in Section 4.4 Details for setting depth boundaries {dm m= provided in Appendix A
3

---

Published as a conference paper at ICLR 2026
1. Depth-Aware Layering
2. Multi-Scale Visual Representation
Layer
Codebook
fR
fa
fz
Layer
Encoder
Layer N
Hierarchical
Action Generation
o. Visual Input
Diffusion Process
Output Action
Figure 3: Overview of DP. DPPP integrates three hierarchical design principles across the per- ception and action generation pipeline. At the input level, RGB-D images are partitioned into multi- ple layers based on their depth values. Then, employ multi-scale visual representations to capture features at varying levels of granularity. During the action generation, denoising process is divided into several stages guided by multi-scale visual representations.
3.2 MULTI-SCALE VISUAL REPRESENTATION
In visuomotor policy learning, visual representation plays crucial role embedding input images and mapping them to actions. An effective visual encoder should capture various granularity features of the visual scenarios and guide the policy to predict the action distribution. However, existing methods typically extract features a single spatial scale or compress them into fixed-resolution representation, limiting the expressiveness of learned features 417 27]. To address this problem, we hierarchically partition the feature map into multiple scales, enabling the capture both coarse structural information and detailed fine- ime-grained local cues.
Interpolation and Quantization. After applying depth- aware layering the input image I, each layer 7m is independently encoded into multi-scale feature maps {Sm,k]Sm, E h xC ) k=1 via encoder Em, where {(hę, wr) == denotes the spatial resolutions across scales. Adopting the
quantization design in VQ-VAE [55 [55:0] 40), these feature maps {fm,*] are quantized into discrete vectors drawn from a learnable codebook Z,n- Specifically, each feature vector f(i.)) is mapped
to nearest neighbor Euclidean distance, i.e., f(3,)) + arg arg/mn||z - ,(1.)) - [|2. By applying =E
differentiable interpolation and lightweight convolution to the quantized features fm,ks we then obtain the multi-scale visual representations {m.,) 1K for each layer Im. The pseudocode of full encoding procedure is detailed in Algorithm [ Appendix
Training. To ensure consistent representations across scales, we aim to minimize the consistency loss between the original feature fm Em and the representation fm,k at different scales:
ΣΣ
(/m.A-ss((m))
3||5m.
Cconsistency
where sg(-) the stop gradient operator and |3 balances the gradient flow between two terms. The visual encoder {E.) J and codebook {2m 1.v M are trained end-to-end, as described in detail in Appendix B Notice that although the theoretical minmizer of the consistency loss leads to identical features across scales, in practice. due the limited capacity of the codebook and downsampling operations, each scale captures distinct information. Coarse scales tend to retain global context, while finer scales preserve local details, serving as strong inductive bias for the subsequent section.
4

---

Published as a conference paper at ICLR 2026
3.3 HIERARCHICAL ACTION GENERATION
To match the inherent inductive biases of denoising process 41 2 58 we leverage multi-scale visual representations model action generation in coarse-to-fine manner. The early stage actions are derived from representations that capture global scene information, while fine-graines represen- tations are responsible for generating detailed action components. This approach couples the visual representation and the action generation process via reinforeing their correspondence at the same hierarchical levels.
Inference.
Our action generation module is a denoising diffusion model conditioned on multi-scale ) and robot poses q. The denoising process unfolds over T features F = (fs {mm,k) m=1
steps partitioned into stages ( Tk]. When E (Tk-1, Tkl, the denoising network cg conditioning on the corresponding feature map fa and robot poses predicts the noise component c (a' (fk. q). then generates a from a² via:
d' a' aga' + B.e² + o
gradually transforming the Gaussian noise a into the noise-free free action a", where ats St, o are fixed parameters depending on the noise scheduler, and Et ~ (0, I) is Gaussian noise, see Appendix for more details.
Training,
(t) we randomly sample an observation acciion pair To train the denoising network Eg ) E DD and noise e ~ (0, I). The network optimized to predict given a noisy action
(1, a²) a' Yoa + +VI 7/6, via the objective:
Lailtusion = Eao,e, [|e]] (a') - -cl?]
(4)
More implementation details can be found in Appendix A
By conditioning the final feature fK
during training, gradients from the loss propagate through the entire hierarchical encoder, implicitly optimizing all (fx K This design promotes consistency of representations at each scale for action generation while enhancing training efficiency.
Discussions. Diffusion models inherently aim to predict the posterior average of the target distri- bution conditioned on the provided features [ [2] i.e., the optimal denoising network (4) follows (t) 'g- (a')f..) g) Er,,,a" +
[cla". f. Earlier stages of the denoising process, char- acterized by higher noise levels, tend to have posterior average with lower frequency, while later stages, with reduced noise, correspond to higher frequency components.
Our design of hierarchically conditioned diffusion aligns well with this property. Features at varying resolutions retain information across distinct frequency domains. Consequently, they provide robust guidance for generating specific frequency components of the action during relevant stages of the denoising process. Related experiments are shown in Section 4.1.3] By using lower-resolution features for earlier stages and gradually refining the predictions with higher-resolution features, the model benefits from both the stability of coarse representations and the precision fine details.
EXPERIMENTS
In this section, we present extensive experiments across simulated and real-world settings to demon- strate the efficacy of HdDP. In addition, perform thorough ablation analyses evaluate the tribution of each hierarchical design, and further investigate the efficiency and effectiveness of our method in extracting visual representations.
4.1 SIMULATION EXPERIMENTS
4.1.1 EXPERIMENT SETUP
Simulation benchmarks and baselines: To sufficiently verify the effectiveness of D2,,, we evalu- ate DD on simulation benchmarks, encompassing a total of 44 tasks. These tasks span a variety
of manipulation challenges, including articulated object manipulation 2 38 63 deformable object

---

Published as a conference paper at ICLR 2026
Table I: Simulation task results. Across 5 simulation benchmarks with various difficult levels, DP obtains + +27.5% relative performance gains on average over 44 tasks.
MetaWorld
MetaWorld
MetaWorld (Hude+
ManiSkill
ManiSkill
Adroit
DexArt
RohiTwin
Average
Method
Tasks
(Medium 11)
()
(Defomuble
(Rigia
(3)
(4)
(44)
H DP
98.3
87.8
95.8
59.3
65.3
87.3
53.3
57.4
76 6 18.0
DP DP (w/ depth)
78.2
52.6
58.0
22.3
27.5
79.0
41.3
22.8
77.7
57.2
71.2
44.5
40.8
76.0
42.0
12.6
=2.8
DP3
89.1
52.6
88.4
26.5
33.5
84.0
54.8
59.3 20,8
manipulation [T6. bimanual manipulation [33, and dexterous manipulation 2 38. The details of
the expert demonstrations can be found in Appendix To comprchensively assess the performance of DP, we compare it against three baselines: Diffusioo Policy [6]. one of the most widely used visuomotor policy learning algorithms: Diffusion Policy (w/ depth), which extends Diffusion Policy to incorporate RGB-D input to bridge the information gap; and DP3 [65], an enhanced version of Diffusion Policy that leverages efficient encoder for point cloud input.
Each experiment is conducted with different seeds to mitigate performance variance. For each seed, we evaluate 20 episodes every 200 training epoches. In simpler Meta World, Adroit and Arx tasks, we compute the average of the highest five success rates as its success rate, while in other environments, only the hightest success rate is recorded.
Evaluation metric:
4.1.2 SIMULATION PERFORMANCE
As shown in Table the simulation experiment results exhibit that HPDP outperforms or achieves comparable performance among the whole simulation benchmarks. Our method outperforms DP3 by relative average margin of 2.%. Notably, DP3 requires manual segmentation of the point cloud to remove background and task-irrelevan elements. This process introduces additional human effort and renders performance susceptible segmentation quality. Relevant experimental results are provided in Appendix E.3
In contrast, benefiting from our design, HDDP obtains superior performance using only raw RGB-D input, without the need for any pre-processing or segmentation. Furthermore, on the Adroit and DexArt benchmark, while DP3 leverages multi-view cameras restore the complete point clouds, DP attains comparable performance using only one single-camera RGB-D image. The whole simulation results in each task can be found in Appendix [E.1] Notably, the hierarchical design in DP introduces negligible overhead relative to DP3, while being significantly more efficient than DP as detailed in Appendix E.4
4.1.3 SPECTRAL ANALYSIS OF ACTIONS
MetaWorkd
Adroit
To gain more comprehensive un- derstanding of the action generation, we apply Discrete Fourier Trans- form (DFT) to examine how the frequency composition of actions evolves throughout the denoising process. Specifically, we conduct the analysis across 4 benchmarks and visualize the spectral character- isties of action chunks during gen- eration. As shown in Figure [ the results consistently indicate that the denoising process begins with the synthesis of low-frequency fea- tures, which are incrementally com- plemented by higher-frequency fea- tures in later stages. This observa- tion not only shows that action, akin to image, exhibits an intrinsic inductive bias in the diffusion process, but also elucidates the action generation mechanism of DD, wherein actions are hierarchically composed to captured features across varying levels of granularity.
Magnitude
10
10a
DexArt
RoboTwin
Magnitude
Buclet
Texl Adjust
10f
10a
2013
10a1
0
Frequency
0
Frequency
Figure 4: Action DFT results. As the denoising process progresses, the Gaussian noise (t = 74) is gradually trans- formed into the predicted action (f = m)) Timesteps t; is arranged in descending order of noise levels. The results re- veal a consistent frequency evolution pattern: low-frequency components predominantly emerge during the early stages of denoising, whereas high-frequency features are progressively introduced in the latter phases of the process.

---

Published as a conference paper at ICLR 2026
Real-World Experiment Results
65
63
Ours
Success Rate
51
2-
50
48
52
DP3
A
41
2-1
2-5
33
24
13 X><
12 >-
2-5
15
2-1
CF
PJ
ST
PB
Figure 5: Success rate in real-world. We evaluate the success rate of H2DP, DP and DP3 across challenging real-world tasks. HPDP outperforms DP and DP3 in all 4 tasks.
4.2 REAL-WORLLD EXPERIMENTS
In terms of real-world experiments, we choose Galaxea RI robot as our platform. We design four diverse challenging real-world tasks to evaluate the effectiveness of our method:
Clean Fridge (CF): In cluttered refrigerator environment, the robot is required to relocate trans- parent bottle from the upper compartment the lower one. The bottle randomized within 30 em x 5 cm region on both the upper and lower shelves of the refrigerator.
Pour Juice (PJ): This is long-horizon task. The robot required to place cup front of a water dispenser, scoop spoonful of juice powder, then fill the cup with water, and finally put straw in the cup. The cup is placed within a cm x 7 cm area, and both the color of the juice powder and the position of the water dispenser subject to variation across trials.
Sweep Trash (ST): This long-horizon task entails picking up a broom, sweeping scattered debris table into dustpan, and subsequently emptying the contents into trash bin. The trash is randomly distributed across the entire table surface, approximately within a 40 cm x 40 cm area. Place Bottle (PB): The robot must place bottle, initially located a random position, onto des- ignated coaster. The bottle is placed within a 15 cm x 15 cm region, while the coaster is positioned within an around 25 em X 25 em area.
4.2.1 EXPERIMENT SETUP
Galaxea R1
ZED camera
Meta Quest3
We the ZED camera acquire the RGBD image. The demonstrations are collected by Meta Quest3. Each task evaluated 20 randomly sampled posi- tions within the defined randomization range. We record the success trials and calculate the corre- sponding success rate. We compare DP with 2 baselines: Diffusion Policy [6] and DP3 (65]
Figure 6: Experiment Setup.
In addition. during policy deployment, we adopt asynchronous design to obtain an approxi- mately double inference speed compared to baseline. We also introduce temporal ensembling and p-masking to improve temporal consistency and alleviate overfitting to the proprioception state. More details can be found in Appendix
4.2.2 EXPERIMENT RESULTS
Spatial generalization: As shown in Figure DP significantly outperforms the baselines across all four real-world tasks, achieving an average relative improvement of + 77.44% It should be noted that CF and PJ tasks, the policy required not only identify target objects in cluttered vi- sual environments but also perform long-horizon reasoning to accomplish the tasks. While DP and DP3 struggle to complete either task, HdDP achieves substantial improvements. Therefore, H&DP demonstrates superior perceptual and decision-making capabilities compared to alternative algo- rithms. Furthermore, HDDP surpasses DP and DP3 even when trained with only 20% the expert data, detailed in Appendix E.5 This finding highlights the efficiency of DD in learning from limited data.
Meanwhile, it should be noted that in terms of the point cloud based method DP3, requires precise segmentation and sigh-iiclity depth sensing, resulting in it being less effective in handling our four cluttered real-world scenes that designed.

---

Published as a conference paper at ICLR 2026
Table 2: Instance generalization results. HDPP achieves +21.0% relative performance gain average.
Place Bottle
Sweep Trash
Method
Tasks
Average
coke bottle
sprite
8 cmm
cm
216cm²
DP
67
49
53
75
86
67
66.2
DP
25
16
52
72
60
42.2
DP3
43
30
37
72
79
67
54.7
Instance generalization: Regarding instance generalization, we evaluate the model on ST and PB by varying the size and shape of bottles trash items. As shown in Table 2 after replacing the objects with variants of differing sizes and shapes, DDDP maintains strong generalization capabil- ities attributable its ability to hierarchically model features at multiple levels of granularity, and consistently outperforms baseline approaches across all settings.
4.3 ABLATION STUDY
In this section, we ablate each key component our framework and conduct experiments on three benchmarks further exhibit the effectiveness of HDDP. Furthermore, fully demonstrate the advantages of DSD,, we also conduct additional experiments to analyze the efficiency and effec- tiveness of our method, especially in extracting task-relevant visual representations.
4.3.1 HIERARCHICAL DESIGNS
Table 3: Ablation on hierarchical features.
Methods Benchmarks
MW
MS
RT
Average
Hierarchical design. We ablate the three hierarchical components intro- duced in our framework and compare them against DP with RGB-D input. As shown in Table 3 each hierar- chical component independently con- tributes to performance improvement, consistently outperforming the DP (w/ depth). Furthermore, Table also also demonstrates that the integration of all three hierarchical designs leads to a substantial enhancement in overall performance.
DD
65.7
68.0
45.0
59.6
w/o depth layering
55.0
52.5
32.0
46.5
w/o hierarchical action
57.0
50.0
40.0
49.0
w/o multi-scale representation
53.7
52.5
40.0
48.7
DP (w/ depth)
46.7
47.5
32.0
42.1
Impact of N in depth-aware layering. For the depth-aware layering component, we inves-
Table 4: Ablation on number layers N.
Methods \ Benchmarks
MW
MS
RT
Average
tigate whether the policy's 's performance is sen- sitive the choice of the number of layers N.
HDDP (N 1)
55.0
52.5
32.0
46.5
As presented in Table 4 our policy achieves optimal and comparable performance when N is set to 3 4, trend consistently observed across all evaluated benchmarks. When N be- comes excessively large, the image is over- partitioned, thus reducing the representation capacity of the policy. Nevertheless, in such cases, the performance remains better than non-layered baseline. The results highlight the critical role of depth-aware layering in enhancing the policy's 's performance.
HDDP (N 2)
55.7
60.0
35.0
50.2
HDPP (N 3)
65.7
68.0
45.0
59.6
HDPP (N =
67.0
61.5
50.0
59.5
H'DP (N = 5)
58.7
55.0
50.0
54.6
H*DP (N =
56.0
51.0
40.0
49.0
4.3.2 VISUAL REPRESENTATIONS
Efficiency and Effectiveness of H' DP Encoder. Prior work suggests that pre-trained visual repre- sentation may enhance spatial generalization of policy [6]]. Hence, we investigate the impact of inte- grating a pre-trained visual encoder with the original DP. We specifically replace the standard ResNet encoder [18] in DP with DINOv2 [34] and evaluate randomly selected tasks from the Meta MetaWorld benchmark. The comparative results are presented in Table 5] Although DP-DINOv2 shows marginal improvement on some tasks compared to the original DP baseline, this comes with longer training time, inference latency and larger number parameters due to the DINOY2. In contrast, DDDP utilizes efficient visual encoder with less than 0.7M parameters, which achieves strong performance improvements over the original DP without incurring the aforementioned overheads. We show the effectiveness of adopting separate encoders for each depth layer in Appendix F.2
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparison with DP with pre-trained visual encoder. While DP-DINOv2 yields small improvement after paying additional cost, DP demonstrates superior performance.
Meta World
Method
Tasks
Average
Hand Insert
Pick of Hole
Disassemble
Stick Pull
Soccer
Sweep Into
DP
100
40
96
83
85
100
84.0
DP
73
13
81
64
43
74
58.0
DP-DINOv2
91
24
77
72
41
78
63.8
Table 6: Performance comparison demonstrating the effectiveness of depth-aware layering. Tasks with significant depth variations show great improvement only with depth layering compared to DP (w/ depth), surpassing the point cloud baseline (DP3).
MetaWorld
Method|
Tasks
Average
Push
Shelf
Disassemble
Pick Wall
Peg Side
DP (only w/ depth layering) DP (w/ depth)
100
100
86
79
80
53
59.0
DP3
96
86
57
97
92
87.7
begin
w/o noise
end
Trajectory Visualization
begin
w/ noise
end
wio noise
w/ noise
DPDP
H DP
begin
end
begin
end
noise wi noise
DDDP w/o VQ
Ha w/o VQ
Figure 7: Visualization of DP's robustness noisy depth. Under the noisy depth input, DP (blue) successfully finish the soccer while keeping the similar action trajectory w/o noise (red). In contrast, DP w/o VQ (yellow) produces a deviated trajectory compared to the original (green).
Effectiveness Significant Depth- Tasks. As introduced in Section 3.1 our depth-aware layering mechanism partitions the image into distinet layers. This layering offers crucial advantage in scenarios with significant depth variations by providing structured representation that preserves separation. As in Table
visual detail while emphasizing breground-backgrron vations reveal a consistent pattern: in tasks involving significant depth variations, point cloud-based policy initially demonstrated superior performance compared standard RGB-D processing, rep- resented by DP (w/ depth). However, upon integrating the depth-aware layering mechanism, H consistently outperforms the baseline on these tasks, which strongly supports our claim.
obser-
Robustness to Noisy Depth. To simu- late low-quality depth input, we add Gaus- sian noise with standard deviation of 0.1 to the normalized depth images (w/ noise in Table ( during training and evalua- tion. As shown Table 7 DP demon- strates strong robustness degraded depth quality, maintaining high performance. contrast, DP3 is highly sensitive to depth noise, exhibiting a significant drop per- formance.
Table Ablation study on the effectiveness of VQ and robustness to noisy depth. HDDP consistently outperforms DP3 especially under noisy depth input, demonstrating strong robustness to noisy depth input. VQ design further enhances the model's robustness to noisy depth input, leading improved performance.
MetaWorld (w/o noise) Box Close
MetaWorld (w/ noise)
Method Tasks
Soccer
Box Close
Soccer
DCDP HPP w/o VQ DP3
98
85
98
75
98
82
90
65
78
57
28
17

---

Published as a conference paper at ICLR 2026
Table 8: Performance comparison with extensions of diffusion policy. We compare D& with flow-based, equivariance-bet based, and 3D representation--aase based approaches, respectively. D& sistently outperforms these baselines across all evaluated tasks, highlighting its superior capability over various diffusion-based policy learning approaches.
MetaWorld
Method Tasks
Average
Asvembly
Shelf
Hand
Hole
Push
DP
99
100
40
100
89.8
Flow Policy
67
100
12
42
29
100
61.0
ET-SEED
54
100
23
79
89
37
96
70.7
3D Diffuser Actor
22
100
33
26
100
54.0
Table 9: Performance comparison with other policy with hierarchical structures. We compare DP with policies that utilize hierarchical designs. Our method outperforms these baselines across all evaluated tasks, demonstrating the effectiveness of our hierarchical designs.
Method \
Tasks
Bin Picking
Box Close
Hummer
Meta World
Insert Sidk
Disusemble
Shell Place
Door
Adroit
Pon
DexArt Laptop
Average
D'
100
98
100
98
96
100
79
83
81
92.8
Policy
25
51
71
56.0
3D Dense Policy
47
69
100
82
98
77
72
61
85
76.7
To further understand the source H'DP's robustness to noisy depth, we compare it with a variant of DP that does not use the feature vector quantization module (H"DP w/o VQ). As shown in Table 7 this ablated version suffers from a substantial performance drop under the same noisy depth imput. This indicates that the robustness of DP can be largely attributed to the design of the feature vector quantization module. As shown in Figure 7 by mapping visual features to the nearest neighbor in learned codebook, the codebook effectively projects representations back to the in- an-distribution space, thereby enhancing robustness to noise.
4.4 COMPARISON WITH MORE BASELINES
Comparison with extensions of diffusion policy. Previous discussions have primarily focused on comparing H DP with original diffusion formulation, i.e., based DP [19]. Recently, flow- 54, 62), and other 3D representation- based [65] 22 policy
based 7 I0 equivariance-bast
have been proposed to enhance the efficiency and effectiveness of diffusion-based policy learn- ing. To provide a more comprehensive evaluation, we extend our comparisons to include additional state-of-the-arr baselines. Specifically, compare DdDP with FlowPolicy [67]] (flow-based) ET-
SEED (54) (cquivariance-basel) and 3D Diffuser Actor [22] (3D representation--assed on selected tasks. As shown in Table DP consistently outperforms these baselines across all evaluated tasks, demonstrating its superior capability over various cestuusion-basee policy learning approaches.
Comparison with other hierarchical methods. To further validate the effectiveness of our hier- archical design, we compare DD with Dense Policy (51). which predict action chunks coarse to fine via autoregressive modeling. As shown in Table 9 DDDP outperforms Dense Policy by rel- ative average margin of +21.0% across the evaluated tasks, demonstrating the superiority of our hierarchical designs over simple coarse-to-fin action modeling.
CONCLUSION
In this paper, we introduce DP,,, an efficient generalizable visuomotor policy learning framework that can obtain superior performance in a wide range of simulations and challenging real-world tasks. Extensive empirical evidence suggests that establishing a more cohesive integration between visual representations and the action generation process can enhance the generalization capacity and learning efficiency policies. The proposed three hierarchical designs not only facilitate the effective fusion of RGB and depth modalities, but also strengthen the correspondence between visual features and the generated actions at different granularity levels. In the future, we expect to extend the applicability of DP more intricate and fine- mne-grained dexterous real-world tasks.
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
The proposed method in this paper is intended to enhance the capabilities of robotic systems in per- forming manipulation tasks. While the advancements in robotic manipulation can lead to significant benefits in various fields, including manufacturing, healthcare, and service industries, it is crucial to consider the ethical implications associated with the deployment of such technologies. Potential concerns include job displacement due to automation, privacy issues related to data collection, and the safety of human-robot interactions. It is imperative that researchers and practitioners in this field adhere to ethical guidelines and regulations to ensure that the development and application of robotic technologies are conducted responsibly and with consideration for societal impacts.
This paper has also benefited from the use of large language models (LLMs) to aid in refining and polishing the writing. LLMs were employed to enhance the clarity and coherence the manuscript, ensuring that the ideas and contributions communicated effectively. However, all technical con- tent, experimental results, and conclusions were developed independently by the authors without reliance on LLMs.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we will release the complete codebase, including the implementation of DPD,, training scripts, and pre-trained models upon publication. The code will be well-documented to assist researchers in understanding and utilizing the framework effectively. Additionally, we will provide detailed instructions for setting up the environment and running experiments, along with the specific configuratio inffiuuaations used in our evaluations. All datasets and simulation environments referenced in this work are publicly available, and we will include links to these resources in the released code. Furthermore, we will share the hyperparameters and training protocols employed in our experiments to enable others to replicate our results accurately.
REFERENCES
[1] Stefan Ainetter, Christoph Böhm, Rohit Dhakate, Stephan Weiss, and Friedrich Fraundorfer. Depth-aware object segmentation and grasp detection for robotic picking tasks. arXiv preprint arXiv:2111.11114 2021.
[2] Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipulation with articulated objects. In CVPR, 2023.
[3] Johannes Bill, Hrag Pailian, Samuel J Gershman, and Jan Drugowitsch. Hierarchical structure is employed by humans during visual motion perception. Proceedings of the National Academy of Sciences, 2020.
[4] Johannes Bill, Samuel Gershman, and Jan Drugowitsch. Visual motion perception as online hierarchical inference. Nature communications, 2022.
[5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xi- aoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. To: A vision-language-actio flow model for general robot control. In RSS, 2025.
[6] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023.
[7] Eugenio Chisari, Nick Heppert, Max Argus, Tim Welschehold, Thomas Brox, and Abhinav Valada. Learning robotic manipulation policies from point clouds with conditional flow match- ing. In CoRL, 2024.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
11

---

Published as a conference paper at ICLR 2026
[9] Sander Dieleman. Diffusion is spectral autoregression, 2024. URL BRLLtt::///sadde aai/ 2024
html html
[10] Haoran Ding, Noémie Jaquier, Jan Peters, and Leonel Rozo. Fast and robust visuomotor rie- mannian flow matching policy. IEEE T-RO, 2025.
[11]
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
[13] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In ICLR, 2025.
[14] Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Char- lie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiageng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhao Zhang. Jialiang Zhang, Chengyang Zhao, Hao- ran Lu, Yufei Ding, Ran Gong, Yuran Wang, Yuxuan Kuang, Ruihai Wu, Baoxiong Jia, Carlo Sferrazza, Hao Dong, Siyuan Huang, Koushil Sreenath, Yue Wang, Jitendra Malik, and Pieter Abbeel. Roboverse: Towards a unified platform, dataset and benchmark for scalable and gen- eralizable robot learning. In RSS, 2025.
[15] Zhefei Gong, Pengxiang Ding, Shangke Lyu, Siteng Huang, Mingyang Sun, Wei Zhao, Zhaoxin Fan, and Donglin Wang. Carp: Visuomotor policy learning via coarse-to-fine au- toregressive prediction, In ICCV, 2025.
[16] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskil12: unified benchmark for generalizable manipulation skills. In ICLR, 2023.
[17] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In 7PAMI, 2015.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[20] David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology. 1962.
[21] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment, In RA-L, 2020.
[22] Tsung-Wer Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. In CoRL, 2024.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[24] Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. PMLR, 2024.
[25] Tai Sing Lee and David Mumford. Hierarchical bayesian inference in the visual cortex. Journal of the Optical Society of America A, 2003.
[26] Siyuan Li, Rui Wang, Minxue Tang, and Chongjie Zhang. Hierarchical reinforcement learning with advantage-bases auxiliary rewards. In NeurlPS, 2019.
12

---

Published as a conference paper at ICLR 2026
[27] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, 2017.
[28] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023.
[29] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng. Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, In NeurlPS, 2023.
[30] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In ICLR, 2023.
[31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[32] Guanxing Lu, Zifeng Gao, Tianxing Chen, Wenxun Dai, Ziwei Wang, and Yansong Tang. Maniem: Real-time 3d diffusion policy via consistency model for robotic manipulation. arXiv preprint arXiv:2406.00 01586, 2024.
[33] Yao Mu, Tianxing Chen, Shijia Peng. Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins. In CVPR, 2025.
[34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024.
[35] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforce- ment learning: A comprehensive survey. CSUR, 2021.
[36] Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation. In RSS, 2024.
[37] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
[38] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In RSS, 2018.
[39] Deepak Rao, Quoe V Le, Thanathorn Phoka, Morgan Quigley, Attawith Sudsang, and An- drew Ng. Grasping novel objects with depth segmentation. In IROS, 2010.
[40] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2q-vae-2. In NeurIPS, 2019.
[41] Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In ICLR, 2023.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, 2015.
[43] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A hierarchical vision transformer without the bells- and-whistles. In JCML, 2023.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xivv1707.000 2017.
[45] Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Hu- manoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. In RSS, 2024.
13

---

Published as a conference paper at ICLR 2026
[46] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers:
Cloning modes with one stone. In NewrIPS, 2022.
[47] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In AI/ML for MDO Applications, 2019.
[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[49] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data dis- tribution. In NeurIPS, 2019.
[50] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[51] Yue Su, Xinyu Zhan, Hongjie Fang, Han Xue, Hao-Shu Fang, Yong-Lu Li, Cewu Lu, and Lixin Yang. Dense policy: Bidirectional autoregressive learning of actions. ICCV, 2025.
[52] Qiao Sun, Zhicheng Jiang, Hanhong Zhao, and Kaiming He. Is noise conditioning necessary for denoising generative models? In ICML, 2025.
[53] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurIPS, 2024.
[54] Chenrui Tie, Yue Chen, Ruihai Wu, Boxuan Dong, Zeyi Li, Chongkai Gao, and Hao Dong. Et-seed: Efficient trajectory-level (3) equivariant diffusion policy. In ICLR, 2025.
[55] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017.
[56] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vr13: A data-driven framework for visual deep reinforcement learning. In NeurIPS, 2022.
[57] Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark Yeatman. Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant diffusion policy. In CoRL, 2024.
[58] Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.0574), 2025.
[59] Xin Wang, Geoffrey Oxholm, Da Zhang, and Yuan-Fang Wang. Multimodal transfer: A hier- archical deep convolutional neural network for fast artistic style transfer. In CVPR, 2017.
[60] Zhendong Wang, Zhaoshuo Li, Ajay Mandlekar, Zhenjia Xu, Jiaojiao Fan, Yashraj Narang. Linxi Fan, Yuke Zhu, Yogesh Balaji, Mingyuan Zhou, Ming- Yu Liu, and Zeng Yu. One-step diffusion policy: Fast visuomotor policies via diffusion distillation. In ICML. 2025.
[61] Zhengrong Xue, Shuying Deng, Zhenyang Chen, Yixuan Wang. Zhecheng Yuan, and Huazhe Xu. Demogen: Synthetic demonstration generation for data-efficient visuomotor policy learn ing. In RSS, 2025.
[62] Jingyun Yang, Zi-ang Cao, Congyue Deng, Rika Antonova, Shuran Song, and Jeannette Bohg. Equibot: Sim (3) e3)-vvuivt diffusion policy for generalizable and data efficient learning. In CoRL, 2024.
[63] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforce- ment learning. In CoRL, 2020.
[64] Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learning to manipulate anywhere: A visual generalizable framework for reinforcement learn- ing. In CoRI., 2024.
14

---

Published as a conference paper at ICLR 2026
[65] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In RSS, 2024.
[66] Yanjie Ze. Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng. and Jiajun Wu. Generalizable humanoid manipulation with 3d diffusion policies. In IROS, 2025.
[67] Qinglun Zhang, Zhen Liu, Haoqiang Fan, Guanghui Liu, Bing Zeng. and Shuaicheng Liu. Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation. In AAAI, 2025.
[68] Renrui Zhang, Han Qiu, Tai Wang. Ziyu Guo, Ziteng Cui, Yu Qiao, Hongsheng Li, and Peng Gao. Monodetr: Depth-guided transformer for monocular 3d object detection. In ICCV, 2023.
[69] Xinyu Zhang, Yuhan Liu, Haonan Chang, Liam Schramm, and Abdeslam Boularias. Autore- gressive action sequence learning for robotic manipulation. RA-L, 2025.
[70] Tony 7. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained biman- manipulation with low-cost hardware. RSS, 2023.
[71] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In TMI, 2018.
[72] Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces on robot learning. In NeurlPS, 2024.
15

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameters used for MetaWorld, DexArt.
Hyperparameter
Value
Observation Horizon (T,)
Action Horizon (Ta)
2
Prediction Action Horizon (Tp)
Optimizer
AdamW 23
Betas (31, 32)
[0.95, 0.999)
Learning Rate
1.0e-4
Weight Decay
1.0e-6
Learning Rate Scheduler
Cosine
Training Timesteps (7)
50
Inference Timesteps
20 e-prediction
Prediction Type
Image Resolution
128 128
Scale Number (K)
4
Multi-Scale Representation Resolutions (f(h, wx)]
(0.:),6.33,655577777))
Stage Boundiaries ((T=/T)E=))
(0.0.4.0.6600881.0))
Codebook Size
64
Table 11: Hy Hyperparameters used for Adroit
Hyperparameter
Value
Observation Horizon (T,)
2
Action Horizon (Ta)
2
Prediction Action Horizon (Tp)
4
Optimizer
AdamW
Betas (81, (22)
[0.95, 0.999]
Learning Rate
1.0e-4
Weight Decay
1.0e-6
Learning Rate Scheduler
Cosine
Training Timesteps (T)
50
Inference Timesteps
20 e-prediction 84 x 84
Prediction Type
Image Resolution
Scale Number (K)
4
Multi-Scale Representation Resolutions (f(hg, wk) w
(0.)).G.3))
Stage Boundiaries ((Tx/T)E<o)
(0.004.0.6.088.1.0))
Codebook Size
64
APPENDIX
A HYPERPARAMETERS
To effectively address the varying levels of difficulty and distinct properties inherent to different benchmarks, we adapt our hyperparameter settings for each specific dataset. The chosen configura- tions, detailed Table 10
12
are selected based on previous works 6 65 72 33).
In addition to the hyperparameters reported in the table, the choice of the number of layers N demonstrates great importance, as shown in Table Empirically. we choose N in Adroit, MetaWorld Hard and Hard++, and N = 3 in other benchmarks.
For the reverse process, we employ different formulations depending on the environment. The noise scheduler for diffusion process is determined by a,, defined using function f())
f(t ((t-1) 02 f(t)
π t
where f(t) = cos
(5)
2 T
16

---

Published as a conference paper at ICLR 2026
Table 12: Hyperparameters used for ManiSkill.
Hyperparameter
Value
Observation Horizon (T,)
Action Horizon (Ta)
8
Prediction Action Horizon (Tp)
16
Optimizer
AdamW
Betas (31, 32)
[0.9, 0.95]
Learning Rate
1.0e-4
Weight Decay
1.0e-4
Learning Rate Scheduler
One Cycle LR 47 100
Training Timesteps (7)
Inference Timesteps
100
Prediction Type
e-prediction
Image Resolution
128 128
Scale Number (K)
4
Multi-Scale Representation Resolutions (f(h, wx)]
Stage Boundaries (=o))=)oTT))
(0.004.0.6600..1.0)))
Codebook Size
64
Table 13: Hyperparameters used for RoboTwin.
Hyperparameter
Value
Observation Horizon (T))
3
Action Horizon (Ta)
2
Prediction Action Horizon (Tp)
Optimizer
Adam W
Betas (81, (2)
[0.95, 0.999]
Learning Rate
1.0e-4
Weight Decay
1.0e-6
Learning Rate Scheduler
Cosine
Training Timesteps (7)
100
Inference Timesteps
100
Prediction Type
e-prediction
Image Resolution
180 320
Scale Number (K)
4 (01.))G.5))6.))66..)))
Multi-Scale Representation Resolutions (f(h,, k
Stage Boundaries (TT))
(0.044.0.6,008.110))
Codebook Size
64
where is the total number diffusion timesteps. In MetaWorld, Adroit and DexArt, we follow the DDIM [48] approach, formulating the reverse process ODE, which corresponds to setting a, = 0 for all f. In ManiSkill and RoboTwin, we follow the design of DDPM [19 and formulate the reverse process Variance Preserving (VP) SDE (50). Correspondingly,
Tt
8
V 7-- of VI
70-1
(6)
7
The boundary in Equation 1 set following linear-increasing discretization 68
which is
i(i + 1) d, N(N +1) (dmax dmin)) dmin: i i= O....,
where duin and dmax the minimum and maximum depth values in the input depth image, spectively. This discretization strategy allocates finer depth resolution to closer objects, which are typically more relevant for manipulation tasks, while providing coarser resolution for distant objects.
17

---

Published as a conference paper at ICLR 2026
B METHOD DETAILS
This section outlines the implementation details of our multi-scale encoding. The encoder Em for each depth layer m adopts the architecture from VQGAN [[]] ensuring strong representational ca- pacity while preserving spatial information. We use interpolate to denote differentiable resizing operation (e.g. bilinear or nearest-neighhor interpolation), which is crucial for enabling gradient flow during training. The function Q represents the quantization process detailed Section 3.2 Finally, after interpolating feature map fm. & to the highest resolution, apply lightweight con- volutional network @m, designed to help restore fine details from the potentially lower-resolution source features.
The pseudocode for this process is outlined in Algorithm
Algorithm 1: Multi-scale Encoding
Inputs: raw image 7
Hyperparameters: depth layer number N, scale number K, resolutions {(he, wx) for m = 1,... .....,N
Partition image / into N images {4mJm=1 according to Equation
do
fm f ( E x=xxxxxX
for k = 1,..., K do
Jm,k +- interpolate( I hy: wx) E RhexxuxX fm.k + k(fm,k)
fm,k + &m,k (interpolatoo((mm.s hK, wx)) X
fi,k <- Eick Smik'
fm x fm fm,k
12 Return: multi-scale features F= = (m J
All trainable parameters, including the visual encoders { N-1 the codebooks [ the CNN parameters {f0m, 1 1k=1 and the denoising network cø. are trained jointly in an end-to- end manner. The optimization minimizes the combined objective function L, defined as weighted sum of consistency loss (Equation 2 and the diffusion loss (Equation 4):
Laiffusion + a Coonsistenc:
where a is hyperparameter balancing the two loss terms.
C EXPERT DEMONSTRATIONS
Regarding the MetaWorld 663 and the RoboTwin [33 benchmarks, we utilize scripted policies generate expert demonstrations. In the case of ManiSkill [16] tasks, we employ the officially provided demonstrations. Trajectories for other simulation benchmarks are collected with agents 56 The expert policies are evaluated over 200 episodes, and
trained by RL algorithms 65
their success rates are detailed in Table 23
Given the varying difficulty levels across benchmarks, we provide a different number of demon- strations for each. Specifically, we provide 50 trajectories per task for MetaWorld, Adroit, and RoboTwin. For DexArt, we follow the setup in [65] and provide 100 trajectories per task. For ManiSkill, use all official demonstrations: 1000 for rigid tasks and 200 for deformable tasks.
In real-world experiments, we collect demonstrations of varying quantity, depending on the com- plexity and horizon length of the tasks. For short-horizon tasks, the number of collected trajectories relatively limited - 100 for Clean Fridge and 200 for Place Bottle. In contrast, Long-horizon tasks demand more comprehensive data coverage. We collect more demonstrations: 270 for Pour Juice and 500 for Sweep Trash. These demonstrations play a crucial role guiding the training process, especially in scenarios where exploration is challenging unsafe.
18

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of real-world inference speeds for different methods. The asynchronous version of our method demonstrates significant speed-up by decoupling inference from action execution.
Method
DP
DP3
HdDP
DdP (asynchronous)
Inference Speed (FPS)
12.4
12.7
12.1
24.2
D REAL-WORLD TRAINING DETAILS
As mentioned in 15 basb methods often suffer from low inference speed, which can cause the inference process to stall. Prior approaches, including DP3 [5], attempt to address this by increasing action horizon (e.g. Ta = or Ta 8) or reducing the number of model parameters (e.g. Simple DP3). However, these strategies often compromise manipulation accuracy and dexterity. A further complication is that increasing Ta widens the temporal gap between consecutive inference steps, leading to greater discrepancies observed information, and consequently, divergence in predicted actions. This often results in noticeable jitter and discontinuities in manipulation.
In general, DP-based methods are hindered by low inference speed, temporal inconsistency and overfitting to proprioceptive information. To address these challenges and improve real-world per- formance, we employ several empirical techniques.
D.1 HIGHER INFERENCE SPEED
To mitigate slow inference rooted in DP, adopt asynchronous design, achieving final infer- ence frequency of 10-15 Hz. Instead of waiting for the execution of all predicted actions before initiating the next inference cycle, our method performs inference concurrently with action execu- tion. The predicted action is stored in a queue to be executed at a fixed inference speed (10-15 Hz in practice, 12 Hz average).
The inference speeds achieved in real-world scenarios are presented in Table 14 DP (asyn-
chronous) demonstrates a superior inference speed compared to standard DP 6 and DP3 (65]. as well our synchronous H&DP implementation. In addition to this speed advantage, DP fea- tures shorter action sequence length (T, = 2), which contributes more dexterous manipulation capabilities.
D.2 TEMPORAL. CONSISTENCY
Having adopted the asynchronous design, we have obtained action sequences with overlapping time intervals. To ensure temporal smoothness and reduce discontinuities, incorporate temporal sembling mechanism from ACT [70]. As in ACT, DD performs a weighted average of actions with the same timestep across multiple overlapping sequences. This ensembling mitigates the gap between actions inferred from slightly different observations and effectively reduces jitter.
D.3 ALLEVIATE OVERFITTING
Similar other real-world robotic systems. HdDP susceptible overfitting on proprioceptive inputs, often neglecting the RGB-D information. This is evidenced by that the model generates similar actions regardless variations in object positions. We hypothesize that this occurs because the simple. wwwpprrmeterr MLP used to encode proprioception is casier to optimize than the more complex CNN used for RGB-D input, leading to reliance on the former.
To mitigate this, we introduce p-masking strategy during training. This mechanism stochastically masks all proprioceptive inputs with probability p. which decays linearly over the training process. Specifically, for training timestep t in total horizon T, p(t) the model to rely more on RGB-D features early in training, helping avoid early-stage overfitting and develop stronger visual grounding.
t/T. This schedule encourages
19

---

Published as a conference paper at ICLR 2026
Table 15: Effectiveness of separate encoders for each depth layer. Using separate encoders for each depth layer yields better performance compared to sharing a single encoder across layers with negligible increase model size.
MetaWorld Pick Place
Tasks \ Method
Average
Box Close
Stick Pull
DP
98
99
83
93.3
DP (shared encoder)
92
89
76
85.7
Table 16: Comparison of DP3 under different segmentation qualities. We compare DP3 success rates selected tasks when provided with different segmentation qualities, highlighting significant performance degradation.
MetaWorid Soccer
Method Tasks
Average
Push
Shelf Place
Stick Pull
Bin Picking
Pick Place Wall
DP
100
100
83
85
100
100
94.7
DP3
96
86
57
100
97
82.8
DP3 (w/o ideal segmentation)
89
26
48
29
84
54.3
E ADDITIONAL EXPERIMENT RESULTS
E. SIMULATION RESULTS FOR EACH TASK
We present the simulation results for each task in Table 22 which serves as supplement to Table 1 For each experiment, we report the average success rate over three different random seeds. The final average result is obtained by averaging across all benchmarks.
We also provide the training progress of 4 algorithms on 12 various tasks across 3 different bench- marks in Figure 9 The selected tasks span range of difficulties and are included without cherry picking to provide an unbiased view of each algorithm.
E.2 EFFECTIVENESS OF SEPARATE ENCODERS
As mentioned in Section 3.2 we adopt separate encoders for each depth layer to allow specialization across spatial regions. While this may appear redundant, whole encoder is still lightweight (< 0.7M parameters), and the overall model remains smaller than DP. We found that sharing single encoder across layers led to performance degradation, as shown in Table [1] This suggests that separate encoders help capture the distinct characteristies of each depth layer more effectively, while the negligible increase model size does not impact efficiency.
E.3 IMPORTANCE OF SEGMENTATION IN DP3
As highlighted Section 4.1.2. DP3 relies on manual segmentation of point cloud for optimal performance. To demonstrate this dependency, we evaluate DP3's performance under two distinct segmentation conditions using randomly selected tasks from the World benchmark.
We compare the following two scenarios: DP3 with ideal segmentation, which utilizes clean seg- mented point clouds containing only the robot and task-relevant objects, as implemented in the original DP3 algorithm; DP3 without ideal segmentation, which utilizes point clouds that are inten- tionally processed to include desk surface upon which objects rest, while other background elements still removed. This configuration simulates common real-world scenarios where simple auto- mated segmentation rules might fail to perfectly isolate the task-relevant objects.
As shown in Table 16 DP3's performance degrades substantially when operating point clouds without ideal segmentation. This result confirms that DP3 is highly sensitive to the quality of the input point cloud segmentation.
20

---

Published as a conference paper at ICLR 2026
Table 20: Validation on Special Tasks. lighting (place bottle).
Success rates of different dimmed light levels dimmed
Dimmed light level
0%
10%
20%
30%
40%
50%
Success rates
52
55
50
40
45
40
contrast, DPP operates directly on raw image without requiring such pre-processing, thereby avoiding such failure mode and the associated need for careful, potentially manual, segmentation tuning, especially common in real-world scenarios.
In setup, the head-mounted camera is ZED, which produces relatively low-quality visual in- puts. This hinders the direct application of DP3 in our experimental setting. To ensure a fair com- parison, evaluate both DD and DP3 four real-world tasks with same visual inputs.
E,4 INFERENCE SPEED AND MODEL. SIZR
As shown in Table 17 we evaluate the in- ference speed of different methods within simulated environments. The results indi- cate that the primary bottleneck of the in- ference speed of HPD lies in the diffusion process itself, whereas the additional op- erations introduced for processing visual inputs and managing multi-scale represen- tations incur only minimal computational overhead. A corresponding analysis of in- ference speed in real-world scenarios is available in Appendix D.I
Table 17: Comparison of inference speeds for DP, DP3 and D in simulation tasks. The result indi- cates that additional operations introduced in DP are lightweight compared to the diffusion process.
Method
DP
DP3
DP
Inference Speed (FPS)
11.1
12.2
12.0
Table 18: Comparison of model sizes for DP, DP3 and DP. H introduces negligible overhead com- pared to DP3 while being smaller than DP.
As shown in Table 18 although DPDP in- troduces multi-scale representation learn- ing, it remains lightweight with negligible overhead in size compared to DP3, and is smaller than the DP baseline.
Method
DP
DP3
H'DP
Model Size (M)
389
255
261
E.5 RESULTS IN LOW-DATA REGIME
To further demonstrate the sample ef- ficiency of our approach, evalu- ate the performance D&DP under limited expert demonstrations in real- world settings. In particular, we train HPDP using only 20% of the avail- able training data. As shown in Table 19 DP consistently achieves supe- norr results even in such low-data sce- narios, highlighting its strong sample efficiency.
Table 19: HdDP low-data regime. HDDP demonstrates strong performance with only 20% expert data compared to DP and DP3.
Method Tasks
CF
PJ
ST
PB
Average
DP
51
65
63
52
57.8
HPDP (w/ 20% expert data)
37
44
58
33
43.0
DP3
12
41
48
33
33.5
DP
13
24
50
15
22.5
E.6 VALIDATION SPECIAL TASKS
To fully validate the effectiveness of H conduct additional experiments on two special tasks: occulsion- -heay cluttered task (tool adjust), where several objects are stacked together, to show the advantage of depth-aware layering in handling complex spatial arrangements; dimmed lighting task (place bottle), where the ambient light is significantly reduced during inference, to demonstrate the robustness of our bierarchical visual representation.
21

---

Published as a conference paper at ICLR 2026
The results summarized in Table 20 and Table 21 In the occulsion-heary clut- tered task, DP outperforms both DP and DP (w/ depth) by significant margin, highlighting its ability effectively utilize depth information in complex scenes. In the dimmed lighting task, DP maintains robust performance even lighting con- ditions deteriorate, demonstrating the re- silience of its hierarchical visual representation.
Table 21: Validation on Special Tasks. Success rates
of different methods occlusion-heavy cluttered (tool adjust).
Method
DP
DP (w/ depth)
DPP
Success Rate
32
45
E.7 VISUALIZATION
We provide more visualization of our depth-aware layering results in Figure
Apple Cabinet Storage
Dual Bottles Pick
Block Hammer Beat
Figure 8: Visualization of depth-aware layering on different tasks. From top to bottom: RGB image, depth image, and four layers obtained via depth-aware layering. Each layer captures different depth ranges, effectively segmenting the scene based on distance from the camera.
22

---

Published as a conference paper at ICLR 2026
Table 22: Success rates on 44 simulation tasks. Results of four different methods for each task are provided in this table. The summary across domains shown in Table
MetaWorke
633
(Medium)
Method
Tasks
Baskethall
Picking
Close
Coffee Pull
Coffee Push
Push Wall
DOP
100
100
100
100
100
85
100
DP
100
96
83
82
84
64
43
76
DP (w/ depth)
100
98
79
64
37
70
DP3
100
100
100
100
97
57
95
MetaWorld (Medium)
MetaWorke (Hard)
Method
Tasks
Insert Side
Sweep
Sweep
Assembly
Hand Insert
Pick Out Hole
Pick Place
Push
DP
98
100
100
100
100
40
99
100
DP
62
96
74
100
73
0
77
DP (w/ depth)
53
98
100
100
75
32
0
79
DP3
92
100
61
100
37
30
0
96
Meta MetaWorld (Hard++)
DexArt 2
Method
Tasks
Shelf Place
Dinssemble
Stick Pull
Stick Push
Pick Place Wall
Laptop
Faucet
Toilet
Bucket
DP
100
96
83
100
100
81
34
70
28
DP
20
81
64
70
55
69
23
58
27
DP (w/ depth)
29
76
71
100
80
63
20
62
23
DP3
86
98
61
100
97
80
33
79
27
Adroit (38
ManiSkill [16 (Rigid)
Method
Tasks
Hammer
Door
Pen
Peg Insertion (Grusp)
Peg Insertion (Align)
Pick Cube
Tur Faucet
D'
100
79
83
88
15
85
73
DP
95
73
78
7
DP (w/ depth)
100
66
62
93
12
33
23
DP3
81
ManiSkill (Deformable)
RoboTwin
[33
Method
Tasks
Excavate
Hang
Pour
Fill
Apple Cabinet Storage
Dual Bottles Pick (Easy)
Dual Bottles Pick (Hard)
DP
38
93
98
98
48
53
DP
2
52
36
73
53
28
DP (w/ depth)
78
25
DP3
15
80
12
55
55
42
RoboTwin
Method
Tasks
Average
Block
Pick Apple Messy
Toal Adjust
DPDP
70
85
25
35
45
75.6 6 18.8
DP
28
0
0
0
48.1
DP (w/ depth)
0
2
7
52.8 22.2
DP3
85
47
30
8
45
59.3
59.3=241 26.9
23

---

Published as a conference paper at ICLR 2026
Table 23: Success rates of experts on 44 simulation tasks. We evaluate 200 episodes for each task. For ManiSkill tasks, the demonstrations are provided officially, and we record the success rates as 100%. The final average result is obtained by averaging across all benchmarks.
MetaWorld 663
(Medium)
Method
Tasks
Basketball
Bin Picking
Bex Close
Coffee Pull
Coffee Push
Hammer
Soccer
Push Wall
Expert
100.0
97.0
90.0
100.0
100.0
100.0
90.5
100.0
MetaWorld (Medium)
Meta World (Hard)
Method Tasks
Peg Insert Side
Sweep
Sweep Into
Assembly
Hand Insert
Pick Out of Hole
Pick Place
Push
Expert
92.0
100.0
90.0
100.0
100.0
100.0
100.0
100.0
World (Hard++)
DexArt
Method Tasks
Shelf Place
Diassemble
Stick Pull
Stick Push
Pick Place Wall
Laptop
Faucet
Toilet
Bucket
Expert
99.5
92.5
95.0
100.0
99.5
86.5
58.0
66.5
80.0
Adroit [38
ManiSkill [6 (Rigid)
Method Tasks
Hammer
Door
Pea
Peg Insertion (Grasp)
Peg Insertion Side (Aliga)
Cube
Tum Faucet
Expert
99.0
100.0
97.0
100.0
100.0
100.0
100.0
ManiSkill (Deformable)
RoboT win
33
Method Tasks
Excavate
Hang
Apple Cabinet Storage
(Fasy)
(Hard)
Expert
100.0
100.0
100.0
100.0
96.0
97.0
55,5
Robo Twin
Method Tasks
Average
Block Handover
Block
Diverse Bottles Pick
Apple Messy
Tool Adjust
Expert
98.0
97.0
72.0
88.5
86.5
93.9
24

---

Published as a conference paper at ICLR 2026
DexArt
DexArt Faucet
DexArt
Bucket
Laptop
0.6-
Success Rate
DP (W/ depth)
0.3
0.3
0.5
0.0
21
0.0
0.0g
Adroit Door
Adroit
Adroit
Hammer
Pan
1.0
Success Rate
0.0
0.0
0.0
MetaWorid
MetaWorid Hand Insert
MetaWorld Pick Place
Box Close
Success Rate
0.5-
0
MetaWorld
MetaWorld
MetaWorid
Success Rate
0.5
0.0
0.0
0.0
23
23
23
Training Staps (10k)
Training Steps (10k)
Training Steps (10k)
Figure 9: Learning curves of the four methods on 12 randomly sampled diverse simulation tasks. In most tasks, DDDP demonstrates faster convergence, higher final success rates, and lower variance compared other three methods.
25


Reviews:
[{"review_id": "6iuw3083AE", "reviewer": "Reviewer_CE83", "summary": "This paper proposes H3DP (Triply-Hierarchical Diffusion Policy), a hierarchical generative framework for visuomotor policy learning in robotic manipulation. The method integrates three levels of hierarchy: depth-aware input layering that partitions RGB-D observations, multi-scale visual representations that preserve global-to-local semantics, and a coarse-to-fine action generation diffusion process. Extensive experiments are conducted across 44 simulated tasks (5 benchmarks) and 4 challenging real-world bimanual manipulation tasks, demonstrating significant performance gains over major baselines and ablation variants.", "questions": "1. See weaknesses.\n2. How were the depth maps obtained? It would be interesting to evaluate how well the method performs when the depth maps are generated by a neural model from RGB images.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Ablation and analysis depth:**  The authors ablate each hierarchical design aspect (see Tables 13–15), show model behaviors under low-data and noisy depth regimes (Tables 7, 22), and compare to closely-related algorithmic variants (DP3, CARP). The connection between model robustness and architectural choices (quantization, depth-layering) is examined through concrete experiments and Figure 7.\n2.  **Efficiency:**  H3DP achieves results rivaling or surpassing point-cloud and multi-view methods but with only single-view RGB-D as input, reducing computational and engineering demands. Model size and inference speed are competitive (Tables 20–21).\n3. **Empirical evaluation:**  The paper covers a broad suite of tasks: 44 simulated and 4 real-world challenges, while using established benchmarks (MetaWorld, ManiSkill, Adroit, etc.) and challenging bimanual/long-horizon settings. The use of both instance and spatial generalization metrics is appreciated. Table 1 and Figure 1 provide clear, accessible summaries of the breadth and magnitude of gains.", "weaknesses": "1.  **Incomplete baselines:** I think the main weakness lies in the incomplete set of baselines used for comparison. The model is primarily evaluated against DDPM-based approaches, making it unclear how well it performs relative to flow-based [1, 2, 3] or equivariance-based designs [4, 5, 6].\n\n[1] Zhang et al., \"FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulation\". 2024. https://arxiv.org/abs/2412.04987\n\n[2] Chisari et al., \"Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching\". 2024. https://arxiv.org/abs/2409.07343\n\n[3] Ding et al., \"Fast and Robust Visuomotor Riemannian Flow Matching Policy\". 2025. https://doi.org/10.1109/TRO.2025.3601293\n\n[4] Wang et al., \"Equivariant Diffusion Policy\". 2024. https://openreview.net/forum?id=wD2kUVLT1g\n\n[5] Yang et al., \"EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning\". 2024. https://arxiv.org/abs/2407.01479\n\n[6] Tie et al., \"ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policy\". 2024. https://arxiv.org/abs/2411.03990", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyCWWCtn0X", "reviewer": "Reviewer_wXbT", "summary": "**Motivation**\n\nPrevious methods have focused on refining inputs, perception, and action separately. In contrast, humans process information hierarchically at all stages.\n\n**Proposal**\n\nThe authors present H3DP – Triply‑Hierarchical Diffusion Policy. An agent with hierarchical processing at three stages of action prediction: inputs, representation and the action prediction.\n\nThe agent uses RGB-D inputs, where the images are segmented into multiple layers using the depth maps.\n\nFor representation, multiple scaled input images are used to provide information from all levels of granularity.\n\nFor action, the diffusion is conditioned progressively from coarse to fine features.\n\n**Results**\n\nThe agent is evaluated in sim benchmarks (44 tasks in 5 sims) and real world tasks (4 bimanual manipulation).\n\nThe agent performs well in the simulation tasks (75.6%), outperforming the next performing baseline (DP3) by 27.5% (relative) in sim tasks.\n\nThe agent also performs well in real world tasks (57.75% overall, +72.4% relative to next baseline).\n\nAdditionally, the method performs better than baselines with 20% data.", "questions": "- How susceptible is the performance to the hyperparameter choice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good integration of multiple complex architectures: multi-layered depth separated visual inputs, multi-scale image features, and scale-specific feature conditioned diffusion policy.\n- Method works in the real world under cluttered conditions (move the bottle in the fridge, pour water and a scoop of powder, sweep trash into the dustbin using a broom and dustpan, and move the bottle to the designated spot).\n- Extensive evaluation and ablation study.", "weaknesses": "- The evaluation uses only 3 seeds per task. It should ideally be at least 5.\n- In some environments, the top 5 success rates are used, and only the top success rate in some (with only 20 episodes in each). This makes it difficult to judge the variance in performance. The number of episodes should be increased (at least 50), and maybe a plot with evaluation scores averaged across all seeds against training step should be reported for a few tasks.\n- While the method is tested in general/standard settings, if possible, validation on tasks where the contributions must be required to complete would be great. For example, testing on an occlusion-heavy task, where the depth-wise layering is needed for success.\n- The paper does not explicitly mention failure modes or scalability issues. Given the number of hyperparameters (different $\\beta$s, learning‑rate schedules, codebook sizes per simulator), a brief discussion on sensitivity and potential deployment challenges would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXvU07KJon", "reviewer": "Reviewer_ija8", "summary": "This paper proposes Triply-Hierarchical Diffusion Policy (H³DP), a variant of Diffusion Policy designed to improve the coupling between perception and action in robotic manipulation. H³DP introduces three hierarchical components: depth-aware input layering that organizes RGB-D observations by depth, multi-scale visual representations that encode semantic features at varying granularities, and a hierarchically conditioned diffusion process that generates actions in a coarse-to-fine manner aligned with corresponding visual cues. Experiments show that H³DP achieves substantial performance gains over baselines across multiple simulated manipulation tasks and demonstrates strong results in challenging bimanual real-world settings.", "questions": "Could the authors provide additional analysis or experiments to address the points raised in the weakness section? In particular, it would be helpful to (1) compare depth-aware layering with a segmentation-based alternative and (2) explore extensions of the current hierarchical diffusion process toward higher-level action decomposition across longer horizons.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents extensive and well-designed experiments, including both simulation and real-world evaluations. The tasks cover a diverse range of object types, manipulation actions, and embodiments, such as pick-and-place, articulated/deformable object manipulation—using both parallel grippers and dexterous hands. The ablation studies thoroughly analyze the contribution of each hierarchical component, examine robustness to noisy depth inputs, and compare different vision encoders, resulting in a comprehensive experimental analysis.\n\n2. The hierarchical design that jointly structures visual encoding and action generation is clear, well-motivated, and conceptually sound, and the experimental results demonstrate consistent and significant improvements over strong baselines.\n\n3. The figures and visualizations are clear, informative, and effectively convey the hierarchical framework and experimental results.", "weaknesses": "1. The paper lacks a comparison of segmentation-based layering as an alternative to the proposed depth-aware layering. While the authors highlight the importance of point cloud segmentation for DP3, they do not explore or justify why semantic segmentation was not considered. Specifically, we can use a tool such as Grounded-SAM to segment objects and backgrounds based on semantic grounding. In some scenarios, semantically similar target objects may appear at different depths but play equivalent roles in the task, suggesting that semantic segmentation layering could potentially yield more meaningful representations than purely depth-based ones.\n\n2. Hierarchical actions remain limited. The current hierarchy in the diffusion process focuses on coarse-to-fine action refinement within a short temporal window, but it does not extend to high-level task decomposition or subtask sequencing. It would be interesting to explore whether the framework could be extended to generate hierarchical action chunks. For example, first predicting subtask-level actions and then refining them into low-level motor commands.\n\n3. Minor writing issues: Some abbreviations (e.g., MW, MS, RT in Tables 3 and 4) are not defined when first introduced. Clarifying these at the start of the experimental section would improve readability and presentation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YoLWx609Wb", "reviewer": "Reviewer_YfHx", "summary": "This paper introduces H3DP, a triply-hierarchical diffusion policy that seamlessly couples visual perception with action generation in visuomotor learning. The framework systematically incorporates three key innovations: depth-aware input layering for foreground/background disentanglement, multi-scale visual representations for granular spatial semantics, and a coarse-to-fine diffusion process conditioned on these hierarchies. Extensive evaluations on 44 simulated tasks and 4 challenging real-world bimanual scenarios demonstrate that H3DP consistently outperforms strong baselines (e.g., DP and DP3), achieving performance gains of +27.5% and +72.4% respectively, while maintaining efficient inference.", "questions": "Q1. Capacity-controlled ablation\nIn Table 3, the reported ≈14% performance gain from depth layering is based on a comparison with a smaller, single-stream encoder. To more rigorously isolate the effect of the proposed architecture, could you conduct a capacity-matched ablation by widening the RGB-only baseline to match the FLOPs or parameter count of your model (e.g., by increasing channel width by 1.5×)? If the performance lift then diminishes to, say, below 7%, how would you substantiate the claim that the improvement stems from the depth-layered design itself, rather than merely increased model capacity?\n\nQ2. Out-of-distribution robustness\nThe claim of \"superior generalization\" would be strengthened by evaluating under modest domain shifts. Could you provide a simple robustness curve—for instance, by taking an existing H3DP checkpoint and testing it under altered conditions, such as with 50% dimmed lighting or a handheld iPhone RGB-D stream (with ±10 cm vertical motion)? A simple bar plot showing the performance drop would be sufficient. If accuracy declines by more than 20%, openly reporting and explaining this result would be more informative than an unqualified generalization claim.\n\nQ3. Statistical significance evaluation\nAll results are currently reported as “mean ± std over 3 seeds.” For your two key real-world tasks (PJ and ST), could you supplement this with either (a) p-values from a bootstrap test across at least 5 seeds, or (b) 95% confidence intervals? In recent CV meta-reviews, results based on fewer than 5 seeds without confidence intervals have been noted as providing insufficient statistical evidence. Providing such information would help preempt concerns regarding result stability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "H3DP offers a genuinely original reframing of visuomotor diffusion policies by hierarchizing the entire pipeline—from depth-layered inputs to coarse-to-fine denoising—a conceptual contribution not previously articulated. This idea is validated through massive, well-documented experiments (44 simulated and 4 real-robot tasks), which demonstrate clear and substantial improvements over strong baselines without requiring extra sensors or manual preprocessing. With the code set to be open-sourced, the community can immediately build upon this work. Although the writing is at times dense with numerical details and the notation could be streamlined, the core contribution remains easily accessible through a single overview figure and is poised to influence not only robotic manipulation but also broader domains employing diffusion-based decision-making.", "weaknesses": "The performance gain attributed to depth-layer processing appears confounded by increased network capacity. While Table 3 reports an average improvement of ≈ +14% with depth layering, the baseline model (\"w/o depth layering\") uses a smaller, single-stream encoder. In computer vision, such a design would typically prompt the question: is the observed improvement truly due to the proposed method, or simply a result of additional parameters?\n\nFurthermore, the paper claims \"superior generalization,\" yet all images were captured under identical lab conditions—consistent lighting and a fixed tripod-mounted camera. A rigorous evaluation would require testing under at least one domain shift, such as variation in illumination, camera viewpoint, or background texture.\n\nLastly, all results are presented as \"mean ± std over 3 seeds.\" Recent methodological standards in machine learning have raised concerns that fewer than 5 seeds, without supporting confidence intervals or p-values, may provide insufficient statistical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0rXG5ntlGt", "reviewer": "Reviewer_L9a5", "summary": "This paper, $\\text{H}^3\\text{DP}$, proposes several modifications to diffusion policy. The authors propose binning RGBD pixels by its depth value into 3-4 bins. Then, for each layer, they train a separate VQGAN-like encoder for multi-scale visual representation. These representations are fed into a diffusion policy head which sequentially uses coarse to fine feature maps for action denoising. Experiments on real and sim show that the proposed method works better than prior state of the art, and ablations show the necessity of each component.", "questions": "- Clarification: is temporal ensembling and p-masking on proprio also done for the baselines for the real robot experiments?\n- Baselines: it would be nice to see comparisons with other works on visual representation like 3D Diffuser Actor.\n- Ablation: how well does the method perform if you only train the downstream policy with the finest feature map?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The ideas of binning depth, hierarchical feature maps, and coarse-to-fine denoising are well-motivated.\n- Various experiments on sim and real environments show that the proposed method outperforms prior state of the art.\n- Extensive ablations in the main text and appendix validate the necessity of each component.\n- Minimal inference speed overhead compared to vanilla formulation.", "weaknesses": "- Complicated setup. While the reviewer is impressed by the performance improvement, the proposed modifications are quite involved. Specifically, the reviewer would like to see an ablation where the downstream policy only denoises on the finest feature map.\n- Lack of baselines. As the paper largely deals with improved representation and policy learning, it would be nice to see comparisons with recent 3D-aware representation and policy learning works. Right now the paper only compares with DP and DP3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ehmZha3eE", "reviewer": "Reviewer_cnwD", "summary": "This work proposes a visual Imitation Learning (IL) method built on Diffusion Policy (DP) which incorporates additional structure in the perception modules and aligns this structure with the action diffusion process, leveraging depth input in addition to RGB. The method incorporates 3 forms of structure: (1) Scene decomposition based on depth. (2) Coarse-to-fine-grained visual features are encoded separately using multi-scale feature maps. (3) Coarse-to-fine-grained visual features are aligned with coarse-to-fine-grained generated action details by conditioning the action diffusion process with increasingly fine feature maps with the denoising timestep progression. The method is thoroughly evaluated on various simulated and real-world robotic manipulation tasks, comparing with both standard and 3D point-cloud conditioned DPs, clearly demonstrating the efficacy of the approach.", "questions": "**Questions**\n\n- Line 196 refers to the “original features f_m”, what are those features? I believe I have an understanding of what the authors are referring to after reading the appendix but it should be clear from the main text.\n- What is the purpose of line 10 in Algorithm 1?\n- Section 4.1.3. The authors claim that the coarse-to-fine frequency progression in action generation is inherent to the diffusion process. It is therefore not a trait of your specific method, am I correct? If so, what is the purpose of this section? If your method somehow facilitates this, why is there not a comparison to the spectral decomposition in a standard DP? \n- Lines 473-475: do you apply the noise after training? How about during training? Is the noise not meant to simulate real depth input which is also noisy during training?\n- Can you provide more visualizations of the depth segmentation on the different environments? Would you say the resulting representation roughly results in an agent-object-bg segmentation in the evaluated envs? Is it an underlying assumption of your method that depth corresponds to agent-object-background decomposition?\n- How are the real world demonstrations collected? Are there any special techniques in the data collection that facilitate the generalization behavior observed in the website videos?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Overview**\n- Well written, clear and easy to follow.\n- The method seems novel.\n- Interesting use of depth for scene decomposition.\n- Interesting alignment of visual feature granularity with action generation detail.\n- Evaluated on diverse simulated environments.\n- Evaluated generalization scenarios.\n- Real-world experiments.\n- Thorough method analysis and ablation study.\n- Website with demonstration videos.\n- Detailed appendix.", "weaknesses": "**Overview**\n- Some aspects and design choices of the method are not thoroughly discussed or motivated.\n- No limitations section.\n\n**Codebook**\n\nWhy did the authors choose to have a separate codebook for each image layer?\n\nWhy are the codebooks shared across the depth layers and not the visual frequency? My intuition is that features in the same scale have more in common than features in the same depth image segment.\n\nI believe the reader would benefit from a discussion about the motivation for this design choice and/or an ablation study. If this is present in the appendix, please refer to it from the main text.\n\n**Diffusion Policy Conditioning**\n\nLines 230-232 state that conditioning on f_K during training implicitly optimizes all f_k for k=1,...,K. Why is this the case? Can the reader understand this without reading the appendix?\n\nWhy does this design choice promote consistency of representations at each scale? Can’t it just disregard certain scales?\n\nThere is a clear discrepancy between the diffusion condition during training and inference. Why does this work? Have you attempted training with the hierarchical conditioning mechanism used for action generation?\n\n**Missing Architectural Details**\n\nThere are some architectural details that I find missing in the paper:\n- What architecture is used for the diffusion policy?\n- How exactly is the diffusion model conditioned on the different feature maps?\n- Can the authors give some basic details about the image encoder architecture other than citing VQGAN so the paper is more self-contained?\n\n**Limitations**\n\nWhat are the limitations and underlying assumptions of your method? These are not discussed in the paper. In my opinion, acknowledging limitations and assumptions are very important for the reader to acquire a deeper understanding of the method and how various aspects of it can be applied to other methods in future work.\n\n**Misc**\n- Although it makes for a nice name, the depth-aware layering does not exactly constitute a hierarchy, I would say it is more a decomposition. It seems that it is treated as a set (i.e., there is no ordering) in the later stages of the pipeline.\n- Standard deviations across the evaluated seeds should be reported alongside the mean.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)