RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Tasks
Authors: Mingxuan Yan
Yuping Wang
Zechun Liu
Jiachen Li
Venue: NeurIPS 2025 poster ()
Keywords: Sub-task Discovery
Robotic Learning
Primary area: reinforcement_learning

Abstract: To tackle long-horizon tasks, recent hierarchical vision-language-action (VLAs) frameworks employ vision-language model (VLM)-based planners to decompose complex manipulation tasks into simpler sub-tasks that low-level visuomotor policies can handle. Typically, the VLM planner needs finetuning to learn to decompose a new task, which requires target task demonstrations segmented into sub-tasks by either human annotation or heuristic rules. However, without prior knowledge, the heuristic sub-tasks can deviate significantly from the visuomotor policy's training data, thereby degrading task performance. To address these issues, we propose a Retrieval-based Demonstration Decomposer (RDD) that automatically decomposes video demonstrations into sub-tasks with prior by aligning the visual features of the decomposed sub-task intervals with those from the training data of the low-level visuomotor policies. RDD outperforms the state-of-the-art sub-task decomposer on both simulation and real-world tasks, demonstrating robustness across diverse settings. Code and more results are available at https://rdd-neurips.github.io

RDD; Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Tasks
Mingxuan Yan'
Yuping Wang 1.2
Zechun Liu
Jiachen Li
Uniiersity of California, Riverside University of Michigan fmyan035, yuping wang, jiachen 1ucr edu zechunliu@meta com
& Meta AI
Abstract
To tackle long-horizon tasks, recent hierarchical Aasion-language-actiion (VLAs) frameworks employ vision-language model (VLM)-based planners decompose complex manipulation tasks into simpler sub-tasks that low-level visuomotor poli- cies can handle. Typically, the VLM planner needs finetuning learn decompose new task, which requires target task demonstrations segmented into sub-tasks by either human annotation or heuristic rules. However, without prior knowledge, the heuristic
can deviate significantly from the visuomotor policy's training data, thereby degrading task performance. To address these issues, we propose a Retrieval-basee Demonstration Decomposer (RDD) that automatically decomposes video demonstrations into sub-tasks with prior by aligning the visual features of the decomposed sub-task intervals with those from the training data of the low-level visuomotor policies. RDD outperforms the state-of-the-at sub-task decomposer both simulation and real-world tasks, demonstrating robustness across diverse settings, Code and more results are available rdd-neurips github.io
Introduction
Developing generalist robots that are capable of executing complex, long-horizon tasks in unstructured environments has become one of the central goals of current robotics research. Traditional robotic programming and learning methods often struggle with the variability and complexity inherent in real- world scenarios. Building upon the success of Vision-Language Models (VLMs) and Large Language Models (LLMs), new class of multi-modal foundation models known as Vision- LiioonLanguage-ACoe models (VLAs) I
4 5 has emerged specifically for embodied AI applications. As recent IO 12 have shown, integrating high-level planners above the low-level visuomotor policies vastly improves the performance for long- honghoorzzon robotic tasks. This has led to the hierarchical VLA paradigm 14
2
3
studies 6
9
15
13
16
17
19. 20). The planner, often a powerful 18
VLM, performs task planning and reasoning to break down complex tasks into simpler sub-tasks with step-by-step language instructions. A learning- short-horizon sub-tasks and conditioned on the generated sub-task instructions, performs precise manipulation to complete the sub-tasks one by one, thereby completing long-houe horizon tasks.
visuomotor policy, trained on datasets with
Despite its versatility, vanilla VLM planner typically needs to be finetuned with human demon- strations when deploying to given task [18 [4 [6]. To build the dataset for planner finetuning.
15] or heuristics 13 15 22 24 25). However, these methods are neither scalable nor efficient, and,
demonstrations are temporally decomposed to sub-tasks by human annotation [14] 16
[8]
19
most importantly, they could generate sub-tasks that deviate significantly from the training data of the low-level visuomotor policy. Figure [aarreeilltttttt this dilemma. The state-of-the-at sub-task de- composer UVD (25). which uses heuristic decomposition rule based on visual feature change-point
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sub-tasies appeurs in the truining set visuomotor policy
Retrieve simillar from policy training set
RDD
Sub-task
Sub-task
Sub-dask Instructions
Hive 3od junti alore the sode euplly grlppe cliendd
LORA
Visuomoter Policy
VI.M-Based Task Planner
(b)
OO
Sub-task..?
X
Figure 1: The core idca of RDD. (a) Two sub-tasks appear in the visuomotor policy's training set, on which the policy has been optimized. (b) Existing sub-task decomposers, such as UVD 25 use heuristic decomposition rules and may generate "unfamiliar" sub-tasks that difficult handle for the low-level visuomotor policy. (c) contrast, RDD decomposes the demonstration into sub-tasks that are visually similar the ones in the training set of the visuomotor policy. The sub-tasks are then used to finetune the high-level planner, which gives task instructions to the low-level visuomotor policy and guides it to finish the task step-by-step.
detection, generates sub-tasks that significantly deviate from the training data of the visuomotor policy. Finetuning the planner with these sub-tasks could make the planner generate sub-task instructions that the visuomotor policy is not optimized for, leading to compromised performance.
This gap motivates to develop automatic, training-frees free, and computationally efficient approach that identifies sub-tasks from a video demonstration with prior, i.e. the decomposed sub-tasks should be aligned with the training data of the low-level visuomotor policies. To achieve this, we propose Retrieval-base Demonstration Decomposer (RDD) that decomposes the demonstration into sub-tasks visually similar the ones in the training set of the visuomotor policy, illustrated in Figure (c). Inspired by previous work 25 we employ existing visual encoders [26] 27 28 29 30 that encode images into compact latent space where distance metrics (c.g., angular distance)
effective in describing the semantic relationship between images. To align the sub-tasks to the training data the low-level visuomotor policy, we build sub-task visual feature vector database with the visuomotor training set and design an effective sub-task similarity measure to ensure similar sub-task samples can be efficiently retrieved. We formulate sub-task identification as an optimal partitioning problem and employ a dynamic programming- -badg solver to optimize the sub-task partitioning strategy efficiently. The experiments show that RDD consistently outperforms state-of-the-ar methods on both simulation and real-world benchmarks.
The main contributions of this paper as follows:
This work is the first to coordinate the high-level planner and low-level visuomotor policy in the hierarchical VIA framework by generating the planner's finetuning dataset that is well aligned with the visuomotor policy to improve the long-horizon task performance.
We propose RDD, a retrieval- based video tub-task identification algorithm with sub-task prior. Specifically, model sub-task task identification an optimal partitioning problem, which can be solved efficiently with a dynamic programming solver.
We evaluate RDD on both simulation and real-world benchmarks. Experimental results show that RDD outperforms the state-of-the-arr heuristic decomposer and is robust across various settings.
2 Related Work
Hierarchical VLAs. While single-stage VLAs I 2
5 achieve promising performance in
chort-horizon manipulation tasks, long-horizon tasks need an in-depth understanding of the task and general planning ability, which is hard to handle by a single-stage model. To this end, hierarchical structures have emerged as a compelling solution for long-horizon manipulation tasks [4 15 13 [6
17 18
As representative examples, Hi Robot [14] and mo.5 [8] enhance their previous work on visuomotor policy 4 3 with VI.M-based planner. According to image observation and the overall task goal, the planner provides sub-task instructions at each time step. The low-level
19
20 10

---

policy, conditioned on the instruction, outputs the final actions. Hierarchical structures also enable error correction and human intervention [33 T6 14] However, these methods rely on either human annotation or heuristic rules to identify sub-tasks when finetuning the planner, which is less efficient and could generate sub-tasks that are hard to handle by the visuomotor policy.
Sub-Task Identification. Finetuning the high-level planner in hierarchical VLAs requires demon- strations broken down into sub-tasks with associated labels. Manually performing this segmenta- g
is slow and expensive. Human subjectivity also leads to inconsistencies. 24, such as segmenting based on contact changes or end- effector velocity profiles, require task specific knowledge for carefully designed rules. In contrast, UVD [25] leverages general visual representation and identifies sub-tasks by detecting frame-by by- frame temporal change points of visual embedding distances. However, when applying to hierarchical VL.As, UVD can still sub-optimally decompose sub-tasks, which may deviate significantly from the training data of the visuomotor policy. In contrast, with the sub-task prior, RDD identifies sub-tasks by explicitly aligning the sub-tasks with the training set of the visuomotor policy, enabling seamless coordination between the planner and visuomotor policy.
tion 14 16
19
15
Heuristic methods 13
21 23
Visual Representations. Considerable efforts have been made develop visual encoders that embed RGB frames into compact latent vector spaces (26] 27 28 29 30). Some of these efforts are specially designed for robotics and manipulation scenarios. For instance, R3M 27 uses contrconrr learning on large datasets of human videos; LIV (26) learns a value function conditioned on both language instructions and images. These visual representations are designed to capture meaningful information about the scene, objects, and potentially their relationships or temporal dynamics.
3 Retrieval-Based Demonstration Decomposer (RDD)
3.1 Problem Statement
Visuomotor-Planeee Dataset Alignment. Hierarchical VLAS typically follow an imitation learning framework that trains a low-level visuomotor policy no ma(ay)ss St, ot, l,, L) and high-level planner po (lee)s,, 0r, 4-11 L). The latter is usually a VLM. ag denotes the waypoint action at timestep t, including 6-DoF pose and binary gripper state. Both policy mo and planner Pa are conditioned on the RGB image observation o,, proprioceptive states 8t, and the overall task objective description L in natural language, such as "put the cube in the drawer". The policy we is additionally conditioned sub-task instruction 4 like "first, pick up the cube", which is determined by the planner Pe at time t.
During the policy training phase, the raw training dataset Dirain = ((S*, L')) is composed of Nuai demonstrations where S = {(at st, E and I,' represents the corresponding task objective description. To break the complex long-horizeg tasks down to simple instructions required by the low-level policy Ta a demonstration St is decomposed into a set of partitionn pi = ( B, based on task-specific rules or human annotations. The j-th interval Zj = corresponds to single coherent sub-task, where b/. ej are indexes of the starting and ending frames. All time steps within the same interval share the same sub-task instruction 4 fune (prompt,) ) labeled manually or generated by a powerful language model. As such, the demonstration is augmented with language descriptions Ii to Siug = {(ai, st, of, 12) 12 and the augmented training set is denoted as Diman aug = {(Sing: It) ),) The policy aa(a (st, oe, L,, L)) is then optimized on Dtman
S'(e,]] (b) <ee
During the high-level planner finetuning phase, given M demonstrations (M & Nuain) for each task, we construct planner finetuning dataset Ddemo {(S*, Ar and predict the sub-task partitioning strategy P E I(S') for St, where IIS) denotes all possible partitioning over a sequence S:
{ P [I, ..... IR}
II(S)
U - - S, I III, = for + j
Sub-task Identification as Optimal Partitioning Problem. Finding the optimal sub-task partitioning strategy can be formulated as an optimal partitioning problem, as illustrated in Figure 2
p = arg max J ((P),
(3.1)
PeIT(S))
3

---

v(Z))= concat(E(os)) 8(o.)) 3q.3.5
V(2j)
sim
J= J (I)) + J
(I
+ +
J
(I)
sim (z), (Z),T)
Eq.3.2
3..
3..3.6
Visuomotor
pt
arg [
Training Set
Polley
LL
v(T))
1
ANNS (VIZ)), Ding
Solve with Dynamic Programming
E
Retrieve I, with
Figure RDD formulates sub-task identification optimal partitioning problem. Intervals colored in green are proposed segments the demonstration St, and ones colored blue are retrieved from the visuomotor policy's is training set Ditain
where J(P) the partitioning strategy scoring function defined on that evaluates how close the strategy is to the low-level visuomotor policy's s training dataset Dirain Given the partitioning found, Deleoo is augmented by finng and arranged Dileno following the same procedure Dirain A pre-trained planner po 2o(4e)1,, ot, 44-1, L) is then finetuned on Deemo with supervised learning to learn aug
to decompose the new task.
3.2 Dynamic Programming Solver
Brute-force search of p requires 022 times of evaluation for a N frame's demonstration, which is computationally intractable. Fortunately, (31] show that when J is interval-wise additive (as illustrated Figure 2) i.e:
J(P)
Σ
(3.2)
IEP
which implies J(P) J(P)) (P1, ((PP,P))PPPPPP Pin = =0}, where is the scoring function of a single interval. The following optimality holds:
function Theorem 3.1 (Principle of Optimality (3I). Given an additive scoring J, any subset P' of
an optimal partition Pa is the optimal partitioning strategy of the intervals covers.
This implies that if we find the partial optimal partitioning strategy for S : j). it must be subset of the global optimal pi+ This optimality structure allows dynamic programming algorithm [3] to find the optimal partition with O(N2) evaluations of the interval scoring function J.
In real-world robot learning scenarios, the duration of sub-task is limited (typically tens of sec- onds) [32 33 34 [4], thus the complexity of the algorithm can be further improved by ignoring intervals excessively long. We show that: if the length of the interval is bounded, the complexity can be fürther reduced O(N). We provide the algorithm implementation Appendix A.1 Algorithm| and draw the following conclusion:
Corollary 3.1.1. the length of every interval is in the range [Imins Limax), O < Lomin < Limax S N, Algorithm finds the optimum with O ((Lmax Lmmaa) max Limax Limins N Lemax)) evaluations of the interval scoring function J.
We defer the proof to Appendix A.2 When the maximum sub-task interval length Tomax is bounded,
which is common in robotics learning scenarios, linear complexity O(N) is achieved. Considering general cases, in this work, make assumption Lmax and only mildly assume Lmin = 2 for

---

sanity (a valid interval must have both the starting and ending frame). We additionally remark that Algorithm| supports parallel evaluation of the scoring function, the intervals be evaluated are determined at the beginning.
Interval Scoring Function. Recall that J should reflect how well the proposed interval aligns with the intervals in the training set Daug,, we define the interval scoring function J as:
Definition 3.1. The scoring function J for an interval I is defined as:
((]))=I=)le (T,, ANNS(V(T)) Dang)) )) = [T) (T),
(3.3)
where V maps interval I into d-dimensional vector representation, II is the duration of the I, and ANNS(I, " represents the approximate nearest neighbor of the interval proposal I in the training set Demii under some distance metric 5 in Rd. sim an interval similarity measure. For simplicity, we denote the resudt of approximate nearest neighhor search for I3 as TT.
Eq. 3.3 essentially evaluates how close the proposed interval is to the training set of the visuomotor policy the training set D Moreover, Def. 3.1 ensures the following notable property:
Proposition 3.1. Suppose interval Ij can be split into K consecutive parts {Ij Tj2 I Ij
all of which have the same training set similarity score, i.e., sim(T), T2) sim (T,1, T1, =
sim(Li k: I2 Given the interval scoring function J of Eq. 3.3 and an additive J, the following
equality holds:
J(Hz))) J((),, I2k]).
(3.4)
The proof is in Appendix B This equality implies that J is ignorant of the number of intervals when evaluating nested partitionings with the same similarity score. An alternative way interpret is that, in Eq. 3.3 sim assigns scores to the sub-task assignment of each timestamp in an interval instead of assigning to the interval as a whole, thus the score summation is irrelevant to the number of intervals in the partitioning strategy.
3.3 Interval Similarity and Overall Objective
Interval Similarity Measures. As introduced in Section 2
one can embed the RGB image observa- tion of into compact latent vector space for similarity measures. We define V as:
V(Z) concat (E(n)) (E(o)),
(3.5)
As illustrated in Figure 2 0b, 0e are image observations at the beginning and end of I, and the embedding function. This formulation is inspired by former studies [66 35 25] that the ending frame (i.e., the goal frame) contains rich information about the sub-task goal and thus can be a distinguishable representation. Eq. 3.5 also includes the starting frame, which is essentially the goal state of the previous sub-task, to aggregate context-related information into the vector representation.
Let the approximate nearest neighbor of Ij be 1j = ANNS(I), aug measure sim between I3 as:
we define the similarity
sim am,(,,,)= 100=-((VV)). ((V(T)). +
E T
(3.6)
where the first term is the distance between the vector representations of Σj and 73: the second evaluates the relative difference between the temporal durations two intervals. a a hyperparameter that controls the weights between temporal and visual similarity.
Considering OOD Sub-tasks. While the primary objective of RDD is align the planner with the visuomotor policy's 's existing capabilities, in real-world applications, out-o--distribution (OOD) sub- tasks not learned by the low-level visuomotor may exist. In such scenarios, the objective changes to: aligning sub-task intervals to both existing visuomotor sub-tasks and general sub-tasks, and the newly identified sub-tasks will be used to finetune both the visuomotor and the planner. Firstly, to detect the existence of new sub-tasks in demonstrations, one can quantify the novelty of a demonstration by 1 Ezer J(],, the average similarity score of the optimal partition P found by the standard A = [P|
5

---

RDD algorithm. A low value of A indicates a low averaged similarity, which signals novel tab-tasks. An alternate interval similarity measure sim for the OOD setting is defined:
(im((),T)) = (I). + (G(
(3.7)
retrieval
general
where Vz( = E(o)) and only the ending frame is used to calculate the semantic distance due to unpredictable OOD sub-task durations; G evaluates how well a proposed interval aligns with "general" sub-tasks. The hyperparameter balances the trade-off between aligning with visuomotor sub-tasks and discovering novel, generalizable sub-tasks. G can be implemented using heuristic general sub-task identification functiins like UVD 25 generic change-poin detection heuristics:
to measure how well an interval conforms to
1 7 abs(b UVD(e,, I)),
G(Z)
(3.8)
where b, e represent the index of the beginning and ending frame of interval I. UVD(e,,) gives the index of the UVD predicted beginning frame, given the goal frame
Approximate Nearest Neighbor Search. Considering the vast number of intervals in Dtrain and the ligh-dimensional vector space, we adopt approximate nearest neighbor search (ANNS) to implement the nearest neighbor searcher. We choose the popular random-pre wwwoon--poeeeeoo-teeee tet-tttreeeeeeee method Annoy (36] as the ANNS implementation, which is computationally efficient and shows good robustness on various data (37 RDD can also work with GPU-accelerated ANNS libraries like FAISS (8
aug
for further acceleration.
Overall Optimization Objective. By substituting Eq. 3.2 and Eq. complete definition of the optimization problem as:
3.3
into Eq.
3.1
we have the
p
arg max PEII(S))
sim [Z I)),
(3.9)
Ijer
where sim is defined by Eq. 3.6 or alternatively Eq. 3.7 for 00D settings. The optimal partitioning strategy pix of demonstration S can be solved by Algorithm
4 Experiments
Implementation and Parameter Settings. We adopt RACER 13 as the base hierarchical VI.A framework, which uses RVT (39] as the low-level visuomotor policy my and the recent I.La Va-based 44 as the pre-trained base model for planner pá- We use the pre-trained
VLM llama-
RVT policy Te provided by RACER [13 trained mug and the validation set of RLBench (labeled with the same decomposition rule as in Dinge) During the deployment phase, the planner is finetuned for two epochs on Ddomo using LoRA 41, with the rank 128 and a scaling factor of 256 following RACER. The finetuning process takes about 5 minutes with 4 NVIDIA 6000 Ada GPUs. For base parameter settings, we set the weighting factor a = 1 and interval similarity measure sim in Eq. 3.6 for non-OOD scenarios, and use LIV (26] as the visual encoder E that is specifically designed for manipulation tasks. We use Gemini- 1.5--lash [42 to generate sub-task language instructions for proposed intervals in Deemm.
aug
Visuomotor Policy Training Dataset and Vector Database. We evaluate RDD on the RLBench robot manipulation benchmark. The visuomotor policy training set Dirain is adapted from 13
Dirain originally consists of 1908 teleoperated demonstrations from the RLBench's 's training set. When generating Dirain RACER additionally augmented it with heuristic failure-recovery samples, resulting in a training dataset with 10,159 demonstrations. In this work, we only use the original 1908 demonstrations to exclude interference. Demonstrations are decomposed into 12700 sub-task intervals using a task-specific heuristic decomposer based on motion and gripper states. Generally, the decomposer will mark a goal state of sub-task whenever: 1) the gripper state closes or opens, 2) the arm stops for pre-defined duration, and 3) the end of the demonstration. More details about this heuristic can be found in Section III. of [13]: RACER uses GPT-4-turbo as the language labeling function Stang to annotate the sub-task intervals, given the language descriptions of the robot movement and initial environment setup.

---

Table 1: Multi-task success rates (%) on RIBBench.
Avg. Succ. (f)
Avg. Rank (L)
Close Jar
Install Bulb
Meat off
Open
Place
Push
Method
Grill
Drawer
Wine
Buttons
w/o Finetune
52.6 182
4.5 112
27.6 1264
34,8 142
46.4 1 266
95.6 11
83.2 180
54.8 191
Uniform
71.3
3.1 112
46.4
51.2 1922
76.4 224
100.0 100
80.8
82.0 ass
UVD
71.4
213
44.0 228.7
54,8 =000
85.2 20.6
100.0 200
80.8 183
67.2 1116
RDD (Ours)
74.9 .0.
2.2 any
46.0 2
52.8
84.4
99.2
86.4
84.0
Expert
75.1 +47
2.2 +
50.4 3M
50,4
94.4 =97
99.2 224
81.6 150
85.6 =00
Put in
Put in
Put in
Drag Stick
Slide Block
Sweep
Tum
Method
Cupboard
Drawer
Safe
Dustpan
Tap
w/o Finetune
41.2 2011
58.8 2204
36.0
57.2
22.8 135
89.2
Uniform
36.8 154
98.0 =17
92.4 10s
64.8 167
(4.4=s9
34.8 31777
98.8 =3
UVD
35.2
90.4 =k6
96.8 106
74,4
66.8 a21
43.6
89.6
RDD (Ours)
41.2 27.1
97.2 #3.1
98.4 222
68.0 =5.0
65.2 143
57.2 2w0
94.0 =3
Expert
39.6
91.2
97.6
75.2 =346
66.4 22.
48.8 255
96.0
Given Diran, we build a vector database following Eq. 3.5 and employ Annoy 336 as the ANNS algorithm to retrieve the approximate nearest neighbor. För each frame, to exclude the inference of occlusion, we concatenate the representation vectors of the front-view and gripper-view images into one. We apply the same configuration to UVD for fair comparison. For Annoy, we set the number of sundom-projection trees to 10 and let the searcher search through all trees at runtime. We empirically find that the choices of the ANNS algorithm impact on the performance. We use angular distance as the distance measure ó, which is written as /2(11 cos(u, u)) for normalized vectors u, v. The finetuning dataset aug is built on RLBench'ss 's validation set following the same procedure except that the decomposition strategy is replaced by RDD. Each task has three demonstrations.
search parameters have minor
Evaluation Metries and Baselines. We evaluate the performance of RDD and baselines in terms of multi-task success rates and corresponding rankings across 13 RLBench tasks We compare our approach with variety of baselines that adopt different sub-task identification strategies:
Expert
13 : The same expert heuristic decomposer used in Dimin as performance upper bound.
aug
UVD 25): A task-agnostic decomposer that detects change points of learning-based visual features.
Uniform: A decomposer that divides each demonstration into 10 partitions with equal duration.
w/o Finetune: The planner ps is the pre-trained VLM model without finetuning on Ddemo
4.1
Quantitative Results and Analysis
Multi-Task Performance on RLBench. Table shows the overall performance of RDD and baseline methods multiple manipulation tasks using the base setting in Section 4 Results are averaged over 10 random seeds. RDD achieves a near-oracle performance and only compromises the success rate of merely 0.2% compared with the expert decomposer, our performance upper bound. On the other hand, we observe that UVD performs similarly to the naive uniform sampling strategy. It implics that the change points of learning- based visual features are not always aligned with the samples in Dimin. By aligning the high-level planner to the knowledge of low-level policy, RDD outperforms the baseline methods that blindly decompose the demonstrations without this knowledge. It also suggests that finetuning is necessary for VLM-based planners. All finetuning-bast based methods achieve over 35% improvement over the vanilla Llama model.
Choice of Visual Representation. As an important building block of RDD, the choice of visual
representation is of great importance. Table the performance of RDD when adopting different
visual encoders E, including robotics specialized encoders: LIV [26), R3M (27), VIP [35], VC-1 (28];
and encoders for general vision tasks: CLIP [43]. DINOv2 (29) and ResNet 44 pre-trained for ImageNet- Ik classification. Results are averaged over three random seeds.
2 Tasks on which the low-level visuomotor policy has a decent performance (success rate > 35% with expert planner). It excludes the interference of poorly optimized visuomotor when evaluating planners. Performance on all 18 tasks can be found in Appendix C
7

---

It can be seen that RDD shows good robustness with various visual encoders and consistently outperforms baselines with the majority of encoders except for VC-1 and VIP, which demonstrates the strong robustness of RDD. VC-1 and VIP, on the other hand, are the only models that do not involve any form of language integration during training and perform the worst among all encoders. This implies the importance of language integration for visual encoders in VLA perception for semantic information retrieval. For instance, subtle pixel differences, such the change of gripper state, may have a significant difference in language description. Surprisingly, ResNet, whose training does not explicitly involve language supervision, demonstrates a strong performance. The reason may be that its training dataset, ImageNet-1k Ik, implicitly correlates its latent space with the language image labels.
Table 2: Results when using dif- ferent visual encoders E. Full re- sults on all tasks can be found in Table 9 in the appendix.
Visu.
Avg. Suec. (1) Rank (J)
Avg.
Repr.
LIV
81.1 200
3.7 Ls
R3M
+1.
3.9 +17
VIP
75.3 vs4
4,1 inn
VC-1
75.5
3.8
CLIP
78.2
4.7 220
DINOv2
78.4
4,5
ResNet 81.1
Weighting Parameters. Table shows the impact of a on the performance of RDD. Results are averaged over three random seeds. When a = 0, i.e., there is no temporal alignment, and the algorithm confused about sub-tasks whose beginning and ending frames are similar (e.g., reciprocating motion). On the other hand, overly relying on the temporal similarity ignores the semantic relationship between intervals and leads to performance degradation. We also evaluate the impact of the 3 in Table 5 for OOD scenarios, and the result shows that RDD is less sensitive to 3. The choice of 3 depends on specific applications and user needs.
Table 3: Results when tuning the weighting parameter a. Full re- sults on all tasks can be found in Table TO
Avg.
Avg. Rank
Suce.
0
3.0 75.0
0.5
75.7 =24
2.5 207
1
81.1 Los
2.3 114
2
76.2 100
2.2
Number of Demonstrations in Ddemo To explore the data effi- ciency of RDD, Table shows its averaged success rates under differ- ent numbers of demonstrations in Daig. Results are averaged over three random seeds. Specifically, we break the droonstemonttottion base setting dataset into three non-overlapping datasets with one demonstration per task to avoid bias induced by varying demon- stration qualities. This result shows a high data efficiency RDD. We credit this efficiency to the less-noisy keyframes provided by RDD, which are more informative for VLM to learn the underlying decomposition rules.
Table 4: Results with different numbers of demonstrations per task in Dekgoo Full results on all tasks are in Table 11
Demo.
Avg. Suce. (t)
Avg. Rank (4)
Num.
1 (RDD)
77.9
2.0 100
3 (RDD)
81.1 =n9
1.6 1.6 =06
(UVD)
75.6
206
Performance on Real-worid and OOD sub-tasks. Here we demonstrate RDD's performance on both real-world and settings where the OOD sub-task appears. We first evaluate RDD on the real-world manipulation benchmark AgilotWorld- Alpha 33.. We test RDD and UVD on the "supermarket" task, using 152 demos to build the RDD database and 37 demos for testing. For OOD sub-tasks, we test RDD on the human- operated demonstration dataset from RoboCerebra [34], which features highly diverse demonstrations in terms of objects, task goals, and arrangements. We use 560 demos to build the RDD database and test on the remaining 140 demos. We use the similarity measure sim in Eq. 3.7 for the OOD setting.
Table 5: Performance on real-world and OOD sub-tasks (IoU).
AgiBot. (Real World) (OOD)
LIBERO
Method
UVD
0.506
0.598
RDD
0.706
RDD (3 0.25)
0.624
RDD (3 0.10)
/
0.630
RDD (,3 0.05)
/
0.614
We evaluated the quality of the decomposition against ground-truth segmentations using the mean intersection over union (mloU). As shown in Table RDD outperforms UVD on real-world data. Under OOD settings, RDD consistently outperforms UVD by leveraging potential similarity between sub-tasks.
Speed and Scalability. We test the running time of Algorithm [ with different numbers of frames on AMD EPYC 9254 using one CPU core. Figure 3 plots the running time with/without the prior knowledge of the maximum length of interval Lmax. The results show that the complexity with Laux grows linearly with the number of frames, which aligns with our conclusion in Corollary which indicates that when Laxx is determined, the complexity of Algorithm I will be O(N).
8

---

Note that Algorithm I supports parallel evaluation of the scoring function J. and the latency can be significantly reduced with multi- processing. Also, we demonstrate the scalability of RDD when working with GPU-accelerated ANNS algorithms like FAISS (38 in Appendix(D
500-
W/ Leus
400-
w/o Leus
Runtin 60
300
200
Necessity of Finetuning on Target Tasks: One may ask if the planner transfer to an unseen new task in zero-shot. We thus build a new planner finetuned before deployment on the training set of the following tasks: "Close Jar", "Insert Peg", and "Install Bulb" as the baseline, which learns the visual features but not the task decompositions. Then, test its performance on the remaining tasks. Results in Table are averaged across 10 random seeds, and we also exclude tasks where the visuomotor fails. The results prove the necessity of fine-tuning on target tasks.
100-
200
400
Number Fromes Decompose
Figure 3: Linear scaling of ning time Algorithm T with Loax. Tested with one CPU core.
Table 6: Vanilla Planner without finetuning on the target task. Full results all tasks are Table T2
VLMS pretrained on internet-scale data are promising to process a variety of video understanding tasks. In Table we compared RDD with Gemini-2.5-pp 42 based decomposer with the following prompt:
Decompose with VLMss:
Avg. Succ. (1) Rank (U)
Avg.
Method
There is a robot doing a task, which can be segmented into multiple steps. keyframe is where the robot finishes the previous step and begins the next. Can you help me find ALL indexes of keyframes? Please return list of indices, for example: [15, 65, 105, ...]. Note that the frame index starts from O instead of 1.
w/o finetuning
77.9 =433 1.6
on target task
RDD (Ours) 79.6 112 1.4
As shown, RDD outperforms 10mini--25--pm despite its powerful general video understanding abilities. This result highlights the necessity of the planner aligning and the effectiveness of RDD.
Table 7: Comparing RDD with Gemini-2.5-pme Full results on all tasks are in Table
Extended Evaluations and Discussions. We provide extended eval- uations results in C and further discussions in Appendix [] We also provide a conceptual speed evaluation of RDD when working with the GPU-accelerated ANNS method FAISS (38 in Appendix D.I
Method
Avg.
Avg.
Suec. Rank (4)
Gemini-2.5-pro 72,6 +47 1.7 404 RDD (Ours)
74.9 269 1.3
4.2 Qualitative Results and Analysis
Figure visualizes the decomposition results of RDD and UVD on both real-world and simulation benchmarks. We can observe that RDD is robust to task-irrelevant interference and can robustly identify the sub-tasks that close to the expert sub-task division. Also, RDD demonstrates strong robustness to nuanced arm movements, where the keyframe localization is challenging precisely. Conversely, UVD fails locate keyframes precisely, and the generated sub-tasks largely deviate from expert sub-tasks.
Discussions and Future Works
Visuomotor Training Data Generation based on Source Dataset: While this work applies RDD to planner-visuomotor alignment, it can also be used to generate additional sub-task training data for visuomotor aligned with a labeled source dataset. By aligning the sub-task interval visual features with the existing source dataset, RDD may make the newly labeled data easier learn, allowing the visuomotor reuse learned knowledge from the source dataset.
Specific Sub-task Interval Features: RDD measures sub-task interval similarity in the single-frame image feature space. Some applications, such as hierarchical lasguaelanguage navigation [19), which require the planner use historical landmark images, may necessitate specialized designs of the similarity score function.
Data Quality of the Source Dataset and Data Curation: As retrieval-based sub-task identification method, RDD's primary objective to the high-level planner effectively utilize the skills that the low-level visuomotor policy already possesses. Therefore, RDD is agnostic to the "optimality" of the

---

Expert
UVD
RDD (Ours)
Figure 4: Qualitative results of RDD and UVD functioning on both real-world (AgiBotWorld) and simulation (RLBench and LIBERO) benchmarks. Blocks outlined in black are sub-tasks decomposed by the same task- specific heuristic used in the visuomotor policy's training set; blocks outlined in green are sub-tasks found by RDD; and blocks outlined in red sub-tasks found by UVD.
skills themselves. This ensures the planner generates commands that the policy can reliably execute, rather than potentially "better" ones cannot handle.
On the other hand, in scenarios where the visuomotor policy's training data contains significant noisy samples that the policy fails to learn, RDD can be easily integrated with dataset curation techniques 45 [46]. These methods can serve as pre-processing step to filter the visuomotor training set. For instance, CUPID [45] computes "action influence" score for state-action pairs that can be used to evaluate each segment's contribution to the policy's final behavior. By applying a simple threshold, low-influence or flawed segments can be pruned from the dataset before RDD it as a reference. This would prevent catastrophic failures by ensuring RDD aligns demonstrations only with hugh-tyaltt, influential sub-tasks.
Conclusion
In this work, present the Retrieval-based Demonstration Decomposer (RDD), training-free decomposition method that aligns the high-level task planner and low-level visuomotor policy in hierarchical VLAs. By retrieving and aligning sub-task segments with the low-level policy's training data, RDD enables an effective planner that fully exploits the capability of the visuomotor policy. We formally formulate the sub-task identification task into optimal partitioning problem, which can be efficiently solved by dynamic programming with our novel sub-task interval scoring function. Experiment results demonstrate that RDD outperforms state-of-the -ice----the-att demonstration decomposers. RDD offers a scalable and promising solution for sub-task identification, opening new avenues for planner-policy coordination in hierarchical robot learning systems.
10

---

References
[1] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-lngguageactionn models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165-2183. PMLR, 2023.
[2] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source Vision-language-actiinn model. arXiv preprint arXiv:2406.0922 2024.
[3] Karl Pertsch, Kyle Stachowicz, Brian lchter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-languag--actinn models, arXiv preprint arXiv:2501.09747 2025.
[4] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. T0: A ssiio-language-action flow model for general robot control. arXiv preprint arXiv:2410.244 24444..444,.... 2024.
[5] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt- a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07555
2024.
[6] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[7] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision- asioonlangguae planning. arXiv preprint 1rX.42231..1884. 2023.
[8] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang- Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained language models. arXiv preprint arXiv:2303.0094 2023.
[9] Hongyi Chen, Yunchao Yao, Ruixuan Liu, Changliu Liu, and Jeffrey Ichnowski. Automating robot failure recovery using ussionllanguage models with optimized prompts. arXiv preprint 2rXiv:2409.03366, 2024.
[10] Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.0188 2024.
[11] Peiqi Liu, Yaswanth Omu, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401. 12202, 2024.
[12] Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: manipulation through mark- visual prompting. In First Workshop Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024.
robotic
[13] Yinpei Dai, Jayjun Lee, Nima Fazeli, and Joyce Chai. Racer: Rich language -ugdggegggudddd failure recovery policies for imitation learning. International Conference on Robotics and Automation (ICRA), 2025.
[14] Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-actoon models. arXiv preprint arXiv:2502. 19417, 2025.
[15] Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, al. Hamster: Hierarchical action models for open-world robot manipulation. arXiv preprint arXiv:2502.05485, 2025.
11

---

[16] Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z Zhao, Archit Sharma, Karl Pertsch. Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv:2403. 12910, 2024.
[17] Weiyu Liu, Neil Nie, Ruohan Zhang, Jiayuan Mao, and Jiajun Wu. Learning compositional behaviors from demonstration and language. In 8th Annual Conference Robot Learning, 2025.
[18] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. x0.5: a vision- language-action model with open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
[19] An-Chieh Cheng, Yandong Ji, Zhaojing Yang. Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Biyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-lagguage-actionn model for navigation. arXiv preprint arXiv:24122 1xXiv:24/200444: 2.4453, 2024.
[20] Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots using vision-language models. arXiv preprint arXiv:2406. 18915, 2024.
[21] Changyeon Kim, Minho Heo, Doohyun Lee, Jinwoo Shin, Honglak Lee, Joseph J Lim, and Kimin Lee. Subtask-aware visual reward learning from segmented demonstrations, arXiv preprint arXiv:2502.20663, 2025.
[22] Wensheng Wang and Ning Tan. Hybridgen: VIm-guided hybrid planning for scalable data generation of imitation learning, arXiv preprint 1313111,, 2025.
[23] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang. Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310. 17596, 2023.
[24] Tongzhou Mu, Minghua Liu, and Hao Su. Drs: Learning reusable dense rewards for multi-stage tasks. arXiv preprint arXiv:2404. 16779, 2024.
[25] Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Ja- son Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6973-6980. IEEE, 2024.
[26] Yecheng Jason Ma. William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman, Liv: Language-iman representations and rewards for robotic control. arXiv preprini arXiv:2306.00953. 2023.
[27] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 22022223.2220.. 2022.
[28] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances Neural Information Processing Systems, 36:6556667777 2023.
[29] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniee, Vasil Khalidov, Fernandez, et al. Dinov2: Learning robust visual features without supervision, 2023. 2023.
[30] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
[31] Brad Jackson, Jeffrey D Scargle, David Barnes, Sundararajan Arabhi, Alina Alt, Peter Giou- mousis, Elyus Gwin, Paungkaew Sangtrakulcharoen, Linda Tan, and Tun Tao Tsai. An algorithm for optimal partitioning of data on an interval. IEEE Signal Processing Letters, 1252)1005-108 2005.
12

---

[32] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Roboties and Automation Letters, 5(2):3019-3026, 2020.
[33] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Xindong He, Xu Huang. al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025.
[34] Songhao Han, Boxiang Qiu, Yue Liao, Siyuan Huang, Chen Gao, Shuicheng Yan, and Si Liu. Robocerebra: surge-scale benchmark for long-horizon robotic manipulation evaluation. arXiv preprint Ar0iv 2506.06677 2025.
[35] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training, arXiv preprint arXiv:2210.000:: 2022.
[36] Erik Bernhardsson. ANNOY library. 2025-05-05.
Accessed:
[37] Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. Ann-benchmarks: A bench- marking tool for approximate nearest neighbor algorithms. Information Systems, 87:101374, 2020.
[38] Mathijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
[39] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694-710. PMLR, 2023.
[40] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu. and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild. URL https://llava-vl. gitpei..ll..i.iiiiiie.
2024.
[41] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhin Alen,ZZhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[42] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- Laan-Batistte Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.//88) 2023.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 877488773. PmLR, 2021.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[45] Christopher Agia, Rohan Sinha, Jingyun Yang, Rika Antonova, Marco Pavone, Haruki Nishimura, Masha Itkina, and Jeannette Bohg. Cupid: Curating data your robot loves with influence functions. arXiv preprint arXiv:2506.191211 2025.
[46] Joey Hejna, Suvir Mirchandani, Ashwin Balakrishna, Annie Xie, Ayzaan Wahid, Jonathan Tompson, Pannag Sanketi, Dhruv Shah, Coline Devin, and Dorsa Sadigh. Robot data curation with mutual information estimators. arXiv preprint arXiv:2502.08623. 2025.
[47] Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEFE International Conference on Robotics and Automation (ICRA), pages 6892-6903. IEEE, 2024.
13

---

[48] Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision- laiio-language-actioon model for affordable and efficient robotics. arXiv preprint arXiv:2506.0/ 0844, 2025.
[49] Jumming Zhang, Weijia Chen, Yuping Wang. Ram Vasudevan, and Matthew Johnson- Roberson. Point set voting for partial point cloud analysis. IEEE Robotics and Automation Letters, 6(2):596-603, 2021.
[50] Mingxuan Yan, Ruijie Zhang, Xuedou Xiao, and Wei Wang. Detvpce: Roi-based point cloud sequence compression for 3d object detection. arXiv preprint arXiv:2502.04811 048044 2025.
[51] Zehao Wang, Yuping Wang, Zhuoyuan Wu, Hengbo Ma. Zhaowei Li, Hang Qiu, and Jiachen Li. Cmp: Cooperative motion prediction with multi-agent communication. JEEE Robotics and Automation Letters, 2025.
[52] Yuping Wang and Jier Chen. Eqdrive: Efficient equivariant motion forecasting with multi- modality for autonomous driving. In 2023 8th International Conference on Robotics and Automation Engineering (ICRAE), pages 224-229. IEEE, 2023.
[53] Yuping Wang and Jier Chen. Equivariant map and agent geometry for autonomous driving motion prediction. In 2023 International Conference Electrical, Computer and Energy Technologies (ICECET), pages 1-6. IEEE, 2023.
[54] Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 1001-1009, 2025.
[55] Yuping Wang. Xiangyu Huang, Xiaokang Sun, Mingxuan Yan, Shuo Xing, Zhengzhong Tu, and Jiachen Li. Unioce: A unified benchmark for occupancy forecasting and prediction in autonomous driving. Proceedings the IEEF/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
[56] Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, et al. Generative ai for autonomous driving: Frontiers and opportunities. arXiv preprint 2XXi:2505.088544 2025.
[57] Xu Liu, Tong Zhou, Chong Wang, Yuping Wang, Yuanxin Wang, Qinjingwen Cao, Weizhi Du, Yonghuan Yang, Junjun He. Yu Qiao, et al. Toward the unification of generative and discriminative visual foundation model: A survey. The Visual Computer, pages 1-42, 2024.
[58] Shuo Xing, Yuping Wang, Peiran Li, Ruizheng Bai, Yueqi Wang. Chengxuan Qian, Huaxiu Yao, and Zhengzhong Tu. Re-align: Aligning vision language models via eetrieval-augmente direct preference optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025.
[59] Shuo Xing, Zezhou Sun, Shuangyu Xie, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, and Zhengzhong Tu. Can large vision language models read maps like a human? arXiv preprint 250.::2503.14601 2025.
[60] Congrui Hetang and Yuping Wang. Novel view synthesis from a single rgbd image for indoor scenes, In 2023 International Conference on Image Processing, Computer Vision and Machine Learning (ICICML), pages 447-450. IEEE, 2023.
[61] Xiangbo Gao, Yuheng Wu, Xuewen Luo, Keshu Wu, Xinghao Chen, Yuping Wang, Chenxi Liu, Yang Zhou, and Zhengzhong Tu. Airv2x: Unified air-ground vehiele-to-everyhhing collaboration. arXiv preprint arXiv:2506. 2w..2252.6........ 19283, 2025.
[62] Peiran Li, Xinkai Zou, Zhuohang Wu, Ruifeng Li, Shuo Xing, Hanwen Zheng, Zhikai Hu, Yuping Wang, Haoxi Li, Qin Yuan, et al. Safeflow: A principled protocol for trustworthy and transactional autonomous agent systems. arXiv preprint 2rXiv:2506.07564, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Assumptions are explicitly list in the conclusions, and proofs are listed in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the training and test details the main text, appendix, and source code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that Algorithm
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the code and relevant data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubblic/guuueee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details the main text, appendix, and source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeated all experiments for multiple rounds and multiple random seeds and provided the statistics in the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
cc. http://sss..ppulleeeaeecccseddda
Answer: [Yes]
Justification: The research conducted in the paper conformed, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We do not release any new datasets or models. Our work only uses existing pre-trained models, which are widely available and do not introduce additional safety risks. Therefore, safeguards are not applicable in our case.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators original owners of assets (e.g., code, data, models), used in the paper and conformed the license and terms.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We attach the details of the dataset/code/mmodel along with the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our method uses LLMs for dataset annotation, and have described details in the main text.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
A
Algorithm Details
A.1 Dynamic Programming Solver to Problem 3.1
Algorithm [ shows the dynamic programming solver. Lmax and Liin user-specified parameters that determine the minimum and maximum length of proposed sub-task intervals. J is the interval scoring function.
Algorithm 1 MaxSumPartition
Require: Sequence u [ui, "2..... un). scoring function J, integer Lmin, integer Lmxx Ensure: Maximum score and partition of
1: Initialize
n| t -o, parts(0 (.. .n] +-
2: dp[0] +-0
3: for Lmin + to do
4: bestScore 6 00
5: bestPartition t Ø
6: for j to do
7:
if Limin Si- S Tmma then segment < ulj :1] 8 + (segmnen)) if dplp)++33 bestScore then bestScore + dpli] + bestPartition < parts[j] U {segment)
8:
9:
> can be evaluated in parallel before loops
10:
11:
12:
13:
end end if end for
14:
15:
if bestPartition f 0 then
17:
dpli] t bestScore parts[i] bestPartition
18:
19:
else
dp[i] dpli 1] parts|i] [i] + parts 1]
21:
end if
23:
end for
return ddpin].partst
A.2 Proof of Correctness and Complexity of Algorithm
Proof. The correctness when Lmin Lmax |S (we denote the algorithm under this case as Algorithm 0) has been proven in Proof Jackson et al. [3I]. is sufficient to prove the cases when special case of Algorithm by constructing an adapted scoring function defined as Algorithm 2 where the score of invalid intervals
1 < Limin Tamax < |S"). Notice that Algorithm
[ equivalent
Algorithm 2 AdaptedScoocFune
Require: Sub-sequence u' [u1, U2, um], scoring function J, integer Lmin- integer Lmax Ensure: Adapted score of u' 1: if Lmin < |u'| < Lmux then
2: return Juu')
3: else
4:
return
5: end if
oo. Note that ADAPTEDSCOREFFUN preserves the additiveness of J, because if any interval in
strategy P violates the length assumption, P is also invalid, i.c.. J(P) -oo. Given the facts: I)
22

---

the correctness of Algorithm 0 has been proven by Proof 2 [31]; 2) Algorithm is equivalent to special case of Algorithm by implication, the correctness of Algorithm proved.
I
As for complexity, let N be the length of the demonstration and M be the number of evaluations to J, we have:
Lan Tegin +-1
M =
j ++ (N
( - Limin + 1)
j=2
(Lmax Liin + 3) Limin) + (N Lemax) ) (Lmmx Lmmin + 1)
O ((Tmmax T<min) max (mmx Tominy N Trnax)
B Proof of Proposition 3.1
Proof. Let the identical similarity scores equal 8, and let Uj. ej be the starting and ending indexes of interval Tj. respectively. By applying Eq. 3.2 and Eq. 3.3 we rewrite the left side Eq. 3.1 to:
J((T))) = J(zz) = (e) b3)s
And the right side:
K
J(I T.,.... x=+))= = Σ J(Tz
km1
(e)]
b)1 + e/2 b)22
+ Cj b)6)s
(e)
bj + e/2 - 6/1 +... + oj 1))-1)
Since intervals are consecutive.
= (e) - bi)s
J(IT))
C Additional Quantitative Results
Tables 13 provide the complete multi-task performances of the results in Section 4.1 including ones where the visuomotor policy fails.
D Discussions
D.1 Work with GPU-accelerated ANNS
The nearest neighbor (NN) search in RDD can be significantly accelerated using GPU-accelerated
typical database of 10 million entries (mainstream policy training dataset scale, as shown in Section D.2) of 2048 dimensions (same dimension
libraries like FAISS 38) We conduct experiments
achieve > 300 NN queries per second on one NVIDIA 4090 GPU. Under this setting, RDD only needs < 2 minutes decompose 500-frame video (5 fps). with max interval length of 100 frames. (44549 NN queries in total). In other words, as part of the offline dataset building process, RDD can decompose demonstrations high speed of 4.3 fps, which shows the high scalability of RDD.
main experiment in Table Th As shown in Table 14 FAISS
D.2 Scale of Mainstream Robotics Datasets
To support the aforementioned experiment settings, here we provide the scale of some of the most popular open-soureed robotics datasets. In summary, assuming each demonstration can be
23

---

Table 8: Main results with all RLBench Tasks.
Avg.
Avg.
Close
Insert
Install
Meat off
Open
Place
Sort
Place
Method
Suce. (t)
Rank (1)
Jar
Peg
Bulb
Grill
Drawer
Cups
Shape
Wine
w/o Finctune
39.7 165
4.3 13 27.6 1264 5.6 167 34,8 144 46.4 1268 95.6 1.1 3.2 143 16.0 an 83.2 136
Uniform
54.5 141
3.1 11
46.4 220 8.8 51.2 192 76.4 100.0
0.8 25.6 192 80.8 144
UVD 25
539
3.2 122
2 10.4 141 54.8 200 85.2 2016
100.0 ano 1.2 L. 25.2 ano 80.8 183
Expert 13
53.6
200
20.4 12.0+mh 50.4 =33 94.4 99.2 =24 3.2 =19 26.0 =0 81.6
RDD (Ours)
57.3
253
=1
46.0 =x2 16.8 18 52.8 104 84,4 99.2 =24 2.0 =20 32.4 + wa 86.4 154
Push
Put in Cupboard Drawer
Put in
Put in
Drag Stick
Slide
Stack
Stack Sweep Turn Cups
Method
Buttons
Safe
Block
Blocks
Dustpan
Tap
w/o Finetune
54.8 291 41.2 auu 36.4 58.8 =3 36.0 57.2
2.8 =26
2.8 22.8 a +3.
Uniform
82.0 =78 36.8 1154 98.0 22 92.4 tos
1ox 64.8 1 64.4 =99 13.6 5.2 e47 34.8 a37 98.8 =30
UVD
25
61. 35.2
90.4 96.8 an 74.4 zw 66.8 2212
9.6 =62 1,6 43.6 +30 89.6
Expert
13
85.6 260 39.6 186
91.2 27
97.6 =s. 75.2 24. 66.4 =m0
14.8 =112 5.2
244 48.8
235 96.0 257
RDD (Ours)
84.0 278 41.2 17. 97.2 234 98.4 a2 68.0 25.0 65.2 243 5.2 #36 1.6 =27
57.2 2207 94.0
Table 9: Multi-task performances with different visual representations.
Visu.
Avg.
Avg. Rank (1)
Close
Insert
Install
Meat off
Open
Place
Sort
Place
Repr.
Succ. (t)
Jar
Peg
Bulh
Grill
Drawer
Cups
Shape
Wine
LIV
26
61.0 aus
3.6
4.0 68.0 Lss
41.3 96.0
100.0 san 1,3 119 32.0
96.0 43.1
R3M
27
59.2 a2 4.2 am
65.3 an
450 44.0
97.3 aa 98.7 0.0 AnD 12.0
86.7 AmS
VIP
35
56.5 am
2.7 72.0 4.0
38.7 uss 93.3 aM
100.0 Lan 5.3 s.o 22.7 mo 89.3 Ax2
VC-1
28
56.9 416
427
73.3 4ea 1.3 419
4
186 93.3 494 100.0 can 8.0 433 20.0 aM 86.7 mmn
CLIP
43
58.4
4.3 430
62.7 Am 4.0 453
46.7 iss 96.0 457 100.0 can 0.0 ano 16,0
82.7 1156
DINO
29
58.3 414
4.4 a1
65.3 18
41.3 am 98.7 1M 100.0 can 1.3 19 13.3
80.0 .M
ResNet
60.5 =20
3.8 3.8 212
68.0
2.7 283
46.7 10s 96.0
100.0 zan 0.0 200 13.3 a50
84.0 163
Visu.
Push
Put in
Put in Drawer
Put in
Drag
Slide
Stack
Stack Sweep to
Turn
Repr.
Buttons
Cupboard
Safe
Stick
Block
Blocks
Cups
Dustpan
Tap
LIV
26
78.7 =22 57.3 238
97.3 =9 97.3 =3a 88.0 73.3 =3a
4.0 1.3 66.7 =50 94.7 250
R3M
27
89.3
50.7 100 85.3 =50 94.7 =50
94.7 =50 82.7
8.0
1.3 53.3 a*a
97.3 a38
VIP
35
92.0.a3
64.0 93.3 =38 89.3 100 10.7 27 46.7 237
2.7
2.3 92.0
97.3 21
VC-1
28
93. 65.3 =a2 93.3 268 92.0
9.3 sas 52.0
4.0
9.3
92.0 100.0
CLIP
89.3
46.7 132 81.3 238 94.7 =50 94.7 =50 81.3
10.7
5,3
$2.0
88.0
DINOv2
29
88.0
50.7 154 85.3 =50
94.7 a50
94.7 =50
78.7 268
2.3
4.0
46.7 =50
94.7 415
ResNet
44
93.3 61.3 294 98.7 219 90.7 2ta 86.7 ios 73,3 264
115 119 56.0 100,0 200
Table 10: Multi-task performance with diffcrent weighting parameter a.
Avg.
Avg.
Close
Insert
Install
Meat off
Open
Place
Sort
Place
0:
Suec. (t)
Rank (L)
Jar
Peg
Bulb
Grill
Drawer
Cups
Shape
Wine
57.3 421
74.7 am 0.0 auD 32.0 anz 52.0 142 98.7 2I9 6.7 a. 29.3 ssu 81.3 aMo
0.5 57.6 421 2.7 aux
73.3 ams 0.0 aup 33.3 ama 49.3 ama 100.0 and 5.3 as 29.3 Lox 92.0 aso
1
61.0
2.3 414
68.0
4.0 ass 41.3 and 96.0
100.0 aud 1.3 11 32.0 2s7 96.0 a52
2 58.0
2.2
76.0
0.0 ann 33.3 48.0 an 100.0 ano 8.0 29.3 aa 88.0
Push
Put Cupboard
Put in
Put in
Drag
Slide
Stack
Stack Sweep to Turn Cups
@ Buttons
Drawer
Safe
Stick
Block
Blocks
Dustpan Tap
90.7
62.7 124 96.0 77.3 76.0 4n
58.7 294 18.7 1.3 a1 78.7 96.0
0.5 85.3 203
62.7 2122 96.0 233 80.0 76.0 444 58.7 24x
17.3 186 40 80.0 97.3 11
1 78.7 лна
57.3 a33
97.3 219
.1a
414 88.0 73.3 2as
4.0
1.3 11 66.7 =so 94.7 ase
2 88.0 a86
60.0 ...
100.0 on
81.3 46a 77.3 1144 58.7 c6s
14.7 in 1.3 1 84.0 96.0
decomposed into 10 sub-tasks, the mainstream policy training datasets typically have 10 million sub-tasks. (<z 10 million entries in the database). The Open X-Embodiment (OXE) Dataset [47]: A landmark collaboration among 21 institutions, OXE provides over million robot trajectories from 22 different robot embodiments. Its explicit goal is to foster the development of generalist models, demonstrating that the community is actively removing the proprietary data barriers of the past. The explicit purpose of OXE to provide a standardized, large-scale resource to train generalist models that have demonstrated significant performance gains by training on this diverse data. Hugging Face
24

---

Table 11: Multi-task performance with different numbers of demonstrations.
Demo.
Avg.
Avg.
Close
Insert
Install
Mcat off
Open
Place
Sort
Place
Num.
Succ. (1)
Rank (4)
Jar
Pog
Bulb
Grill
Drawer
Cups
Shape
Wine
(RDD)
59.1
1.8
75.6
6.2
35.6 65.8 100.0 5.8 247 25.8
91.1
(RDD)
21.0
1.8 2us
an.
4.0
41.3 22i7
26.0 100.0 2n0 1.3 a19 32.0 237 96.0 233
3 (UVD (25)
57.1 20a
2.3
66.7 a22
4.0 37.3 asa
93.3 294
100.0 2.7 419 21.3 ans 77.3 2115
Demo.
Push
Put in
Put in
Put in
Drag
Slide
Stack
Stack Sweep
Tum
Num.
Buttons
Cuphoard
Drawer
Safe
Stick
Block
Blocks
Cups
Dustpan
Tap
(RDD)
86.7 AnD
60.4 97.8 and 78.2 ans 61.8 a. 79.6 182
11.6 1x1 2.2 x1z 87.1 as 92,9 a01
3 (RDD)
78.7 Ax2 57.3 11
97.3 a19 97.3 axx 88.0 as. 73.3 asx
4.0 433 1.3 419 66.7 ase 94.7 aso
3 (UVD (25
62.7 4124 44.0 aes 84.0 96.0 157 85,3 82.7
16.0 433 1.3 439 60.0 183 93.3 an0
Table 12: Multi-task performance of Vanilla Planner without finetuning on target task.
Avg.
Avg.
Meat off
Open
Place
Push
Put
Method
Succ.
Rank (1) Grill
Drawer
Wine
Buttons Cuphoard
w/o finetuning
77.9 243
1.6 99.2 224 99.6 86.4 2a8 70.4 61.2 168
on target task
RDD (Ours)
79.6
1.4 1os
84.4 cm 99.2 344 86.4 154 84.0 41.2 171
Put Drawer
Put Sufe
Drag Stick
Slide
Sweep to
Tum Dustpan
Method
Block
Tup
w/o finetuming = 94.8 and 74.0 223 62.4 160 30.0 92.4 on target task
RDD (Ours) 97.2 23 98.4 +32 68.0 =20 1143 57.2 297 94.0
Table 13: Comparing RDD with Gemini-2.5-pre 2.5-pro.
Avg.
Avg.
Close
Install
Meat off
Open
Place
Push
Method
Suce. (t) Rank (4)
Jar
Bulb
Grill
Drawer
Wine
Buttons
Gemini-2.5-pro 72.6 247
41.2 mm 40,8 =26 83.2 115 99.6 212 86.4 in. 82.4 1,3 46.0 2212 52.8 104 84.4
1.7 204
RDD (Ours)
74.9 200
#211 +24 86.4 154 84.0
Put in Cupboard
Put in
Put in
Drag Stick
Slide
Sweep to Turn
Method
Drawer
Sufe
Block
Dustpan
Tap
Gemini-2.5 20s 94.0
=68 93.6 292 23,6 63.6 144 48.4 99.2
RDD (Ours) 41.2 97.2 =1.1 98.4 =3 68.0 65.2 143 57.2 2M17 94.0=s. =5a
Table 14: Performance of FAISS nearest neighbor search and RDD time NVIDIA 4090.
Hardware
Dim
Vec Num
QPS
Linox
lg
RDD Time (s)
NVIDIA 4090
2048
10M
386
100
500
115 (4.3 fps)
SmoIVLA Dataset [48]: The emergence of models like SmoIVLA, capable vision-languagg-action
model trained entirely on 23k episodes from 487 open open-sourced community datasets through the LeRobot framework, outperforms the flosed-source-datuse policy no [4]. AgiBot World [33]: AgiBot World provides not just datasets but complete open-source toolchains and standardized data collection pipelines, further enriching the public ecosystem. It has collected over 1 million trajectories
on over 100 homogeneous robots. Their proposed model GO-1, entirely trained on this open-sourced dataset, outperforms the closed-souree dataset policy mo [4.
E Broader Impacts
The potential negative societal impacts of our work are consistent with those commonly observed in roboties research, including risks related to privacy, labor displacement, and unintended misuse in sensitive contexts. While our method is primarily designed to enhance the scalability and efficiency
of robotic systems, such advancements may accelerate deployment in real-world settings, amplifying both positive and negative consequences. In parallel, advances in point cloud analysis [49 50].
25

---

cooperative motion prediction [ST] autonomous driving frameworks [22 53 54 55 and generative AI for driving [56] highlight both the promise and the risks of deploying increasingly capable vision- action models. Broader surveys of visual foundation models [57] and new work multimodal alignment [58 [59 further strengthen the importance of trustworthy design and governance, especially for safety-critical applications such as transportation and human-robot interaction [60] 61]. To mitigate these risks, we emphasize alignment with ethical guidelines, including fairness, accountability, transparency, and safety, and encourage interdisciplinary collaboration to monitor societal impacts as these technologies evolve [62].
26


Reviews:
[{"review_id": "YcAr8Tra9s", "reviewer": "Reviewer_9Wsv", "summary": "This paper introduces RDD (Retrieval-based Demonstration Decomposer), a training-free method for decomposing long-horizon robotic demonstrations into sub-tasks for finetuning hierarchical Vision-Language-Action (VLA) models. The key insight is that existing decomposition methods (human annotation or heuristics) often generate sub-tasks that deviate from the training data of low-level visuomotor policies, leading to suboptimal performance. RDD addresses this by formulating demonstration decomposition as an optimal partitioning problem that explicitly aligns decomposed sub-tasks with the visuomotor policy's training set through visual feature retrieval. The method uses dynamic programming to efficiently solve the optimization problem and demonstrates 6.7% improvement over state-of-the-art methods on 18 RLBench tasks.", "questions": "1. How does RDD handle demonstrations containing sub-tasks that are fundamentally novel and have no close visual or temporal analogues in the policy's training set? Could you discuss this potential failure case and how it might impact the fine-tuning of the planner and overall task success? A change in my score would depend on a convincing explanation of how the system avoids catastrophic failure in such out-of-distribution scenarios.\n2. Your method's success relies on the quality of the pre-existing visuomotor training set. How sensitive is RDD to the quality of the sub-task segmentation within this dataset? For example, if $\\mathcal{D}^{train}_{aug}$ was segmented using a poor or noisy heuristic, would RDD simply learn to replicate this flawed strategy, or does the optimization process offer some resilience?\n3. Can you provide examples where RDD fails to decompose demonstrations correctly? Specifically, are there cases where visually similar intervals have very different semantic meanings that confuse retrieval?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "=== Strengths ===\n\n1. The paper identifies an important gap in hierarchical VLAs - the misalignment between planner training data and visuomotor policy capabilities. This is a practically relevant insight that is overlooked in prior work.\n2. The formulation as an optimal partitioning problem with interval-wise additive scoring is mathematically sound. The dynamic programming solution with O(N²) complexity is efficient, and the theoretical analysis (including Corollary 3.1.1) is rigorous.\n3. Using only start/end frames for interval representation (Eq. 3.5) is clever, avoiding variable-length representations while maintaining efficiency for nearest neighbor search.\n\n=== Weaknesses ===\n\n1. The method is only evaluated on RLBench in simulation. Other more challenging manipulation benchmarks (e.g., LIBERO) and real-world robotic manipulation often involve more complex visual variations, occlusions, and dynamics that may challenge the visual similarity-based retrieval approach.\n2. RDD requires access to the full training dataset of the visuomotor policy to build the retrieval database. This may not always be feasible, especially for proprietary or large-scale policies.\n3. The experiments show that performance degrades when the retrieval database is down-sampled to 25% of its original size. This suggests that a large and comprehensive database is necessary for good performance, which could pose a scalability challenge as the number of tasks and skills grows.", "was_revised": true, "final_justification": "The rebuttal mostly solved my concerns that RDD can be extended to OOD scenarios and prevent catastrophic failures. I therefore maintain my initial rating of borderline accept.", "rebuttal": "First of all, we sincerely thank the reviewer for the insightful feedback. We are encouraged that the reviewer pointed out the major strengths of our paper: 1) the significance of our problem formulation; 2) the mathematical rigor of our solution; and 3) the efficiency of key design choices. We address the reviewer’s comments and questions point by point in the following.\n\n# Response to Weakness 1 & Question 1\n\nWe thank the reviewer for this insightful question regarding RDD’s handling of out-of-distribution (OOD) demonstrations with novel sub-tasks. While the primary objective of RDD is to align the planner with the visuomotor policy’s existing capabilities, this is a critical aspect for real-world applicability. **In such OOD scenarios, the objective changes to: aligning sub-task intervals to both existing visuomotor sub-tasks and potentially new sub-tasks.**\n\nThanks to the flexible design of RDD's scoring function $\\\\tilde{J}$, we do have a simple yet effective mechanism to gracefully handle these OOD scenarios and prevent catastrophic failures. Our approach involves two stages: OOD identification and OOD-aware decomposition.\n\n**OOD Identification**\n\nFirst, we must determine if a given demonstration is, in fact, OOD. We can quantify the novelty of a demonstration by calculating $\\\\Delta$, the average similarity score of the optimal partition $\\\\mathrm{P}$ found by the standard RDD algorithm:\n\n\n$$\n\\\\Delta=\\\\frac{1}{|\\\\mathrm{P}|}\\\\sum_{\\\\mathcal{I} \\\\in \\\\mathrm{P}} \\\\tilde{J}(\\\\mathcal{I}).\n$$\n\nA low value of *$\\\\Delta$* indicates that even the best possible decomposition has a low similarity to the sub-tasks in the training data, which signals an OOD demonstration. A simple threshold can be used to trigger our adaptive mechanism.\n\n**OOD-aware Decomposition** \n\nOnce an OOD demonstration is identified, we introduce a modified similarity measure, $\\\\mathbf{sim}_{OOD}$, which incorporates a general interval scoring function $\\\\mathbf{G}$:\n\n\n$$\n\\\\mathbf{sim}_{OOD}(\\\\mathcal{I}_j^i,\\\\tilde{\\\\mathcal{I}_j^i})=\\\\mathbf{sim}(\\\\mathcal{I}_j^i,\\\\tilde{\\\\mathcal{I}_j^i})+\\\\beta \\\\mathbf{G}(\\\\mathcal{I}_j^i).\n$$\n\nHere, $\\\\mathbf{G}$ evaluates how well a proposed interval aligns with \"general\" sub-tasks, independent of the planner's specific training set. The hyperparameter $\\\\beta$ balances the trade-off between aligning with known sub-tasks (the original RDD objective) and discovering novel, generalizable sub-tasks. $\\\\mathbf{G}$ can be implemented using functions like UVD [A] to measure how well an interval conforms to generic change-point detection heuristics:\n\n$$\n\\\\mathbf{G}(\\\\mathcal{I})=-\\\\frac{1}{|\\\\mathcal{I}|}\\\\mathbf{abs}(b-\\\\mathbf{UVD}(e,\\\\mathcal{I})),\n$$\n\nwhere $b,e$ represent the index of the beginning and ending frame of interval $\\\\mathcal{I}$. $\\\\mathbf{UVD}(e,\\\\mathcal{I})$ gives the index of the UVD predicted beginning frame, given the goal frame on $e$. \n\n## Additional Experimental Validation on LIBERO\n\nFor OOD scenarios, we conducted additional experiments on LIBERO. To simulate a highly OOD scenario, we used the human-operated demonstration dataset from RoboCerebra [B] where demos are highly diverse in objects, task goals, and arrangements. We evaluated the quality of the decomposition against ground-truth segmentations using the mean Intersection-over-Union (mIoU) (see response to Reviewer DAU1 for details). We use 560 demos to build the RDD database and test on the remaining 140 demos.\n\n| Methods             | mIoU (LIBERO OOD Test) |\n| ------------------- | ----------------- |\n| UVD                 | 0.598             |\n| RDD ($\\\\beta=0.25$) | 0.624 ( ↑ 4.4%)   |\n| RDD ($\\\\beta=0.1$)  | 0.630 ( ↑ 5.3%)   |\n| RDD ($\\\\beta=0.05$) | 0.614 ( ↑ 2.6%)   |\n\nThe results clearly show that our OOD-aware **RDD not only avoids catastrophic failure but also outperforms the baseline UVD decomposer in OOD settings.** RDD can successfully leverage even limited similarities in novel scenarios to produce a superior decomposition. \n\n## Additional Experimental Validation on Real-World Dataset\n\nFor in-distribution alignment (original goal of RDD), we test RDD on the real-world manipulation dataset AgiBotWorld-Alpha [C] to show the generalization ability of RDD visual features.  We test RDD and UVD on the task \"supermarket\" where 152 demos are used to build the RDD database and 37 demos are used for testing.\n\n| Methods | mIoU (AgiBotWorld Alignment Test) |\n| ------- | --------------------------- |\n| UVD     | 0.506                       |\n| RDD     | 0.706 ( ↑ 39.6%)            |\n\nRDD significantly outperforms the baseline method UVD, proving its generalization ability.\n\nThese results demonstrate the robustness and adaptability of the RDD framework. We will add the above discussions to the revised paper.\n\n# Response to Weakness 2\n\nWe thank the reviewer for the insightful feedback. We acknowledge that RDD requires access to the visuomotor policy’s training data, which could be a limitation for proprietary policies. However, we argue that the strong and growing trend in the robotics community toward open-sourcing models and datasets makes RDD increasingly relevant and applicable.\n\nFor instance, the recent release of powerful, open-sourced models like OpenVLA is often trained on large-scale public datasets such as the Open X-Embodiment dataset. RDD’s core contribution is bridging the gap between high-level planners and low-level policies in hierarchical VLAs.\n\nAs the community continues to embrace and build upon open-source resources, RDD is well-positioned to be a valuable tool for effectively leveraging these powerful models. We will clarify this application scope in the revised paper to highlight RDD’s utility in this emerging research landscape.\n\n# Response to Weakness 3\n\nWe thank the reviewer for raising the important point about scalability. Since the offline vector dataset construction is performed only once, the core efficiency concern lies in the online decomposition process. Here, we highlight that the online decomposition process is efficient and scalable in two ways:\n\n**1. Linear Time Complexity**: As formally stated in **Corollary 3.1.1** in the paper, with a maximum sub-task duration (a mild assumption for real-world robotics), our dynamic programming-based solver has a linear time complexity of **O(N)** with respect to the number of demonstration frames. Our experimental results in Figure 3 confirm this linear growth.\n\n**2. Efficient Search & Parallelization**: The nearest neighbor search can be significantly accelerated using GPU-accelerated libraries like **FAISS [D]** or **Milvus [E]**. Furthermore, our decomposition algorithm is inherently parallelizable, allowing for substantial speed-ups through distributed computing.\n\n\n# Response to Question 2\n\nWe thank the reviewer for their insightful question regarding RDD’s sensitivity to the quality of sub-task segmentation. We would like to address this by highlighting two key aspects of our approach:\n\nFirst, we wish to clarify RDD’s primary objective. The core contribution is not to learn new skills or correct flawed ones, but to enable a high-level planner to effectively utilize the skills that the low-level visuomotor policy already possesses. **Therefore, RDD is largely agnostic to the \"optimality\" of the skills themselves. This ensures the planner generates commands that the policy can reliably execute, rather than potentially \"better\" ones it cannot handle.**\n\nSecond, in scenarios where the visuomotor policy's training data contains significant noisy samples that the policy fails to learn, **RDD can be easily integrated with dataset curation techniques [F, G].** These methods can serve as a pre-processing step to filter the visuomotor training set. For instance, CUPID [F] computes an \"action influence\" score for state-action pairs that can be used to evaluate each segment's contribution to the policy's final behavior. By applying a simple threshold, low-influence or flawed segments can be pruned from the dataset before RDD uses it as a reference. This would prevent catastrophic failures by ensuring RDD aligns demonstrations only with high-quality, influential sub-tasks.\n\nWe will add these clarifying discussions to the revised version of our paper.\n\n# Response to Question 3\n\nWe thank the reviewer for this insightful question. This issue is most apparent in sub-tasks with exceptionally long horizons (e.g., spanning several minutes, whereas typical sub-tasks in our experiments span tens of seconds). In such cases, the intermediate states and temporal evolution, which are not fully captured by our default start-end frame similarity, become crucial for disambiguating the sub-task’s semantic meaning.\n\nHowever, a key strength of our RDD framework is its modularity. The interval similarity measure, **sim**, is a plug-and-play component that offers a convenient and flexible interface to incorporate domain-specific knowledge or more powerful representations without altering the core dynamic programming algorithm. For instance, one could readily implement a more sophisticated **sim** function using powerful video encoders like VideoCLIP [H], which are designed to capture richer temporal and semantic information from video segments. Our novel, general problem formulation opens up exciting avenues for future research.\n\n# References\n\n[A] Zhang, et al. \"Universal visual decomposer:\" ICRA, 2024.\n\n[B] Han, et al. \"RoboCerebra\" arXiv preprint (2025).\n\n[C] Bu, et al. \"Agibot world colosseo.\" arXiv preprint (2025).\n\n[D] Johnson, et al. \"Billion-scale similarity search with GPUs.\" *IEEE Transactions on Big Data* 7.3 (2019). \n\n[E] Wang, et al. \"Milvus: A purpose-built vector data management system.\" SIGMOD. 2021.\n\n[F] Agia, et al. \"CUPID: Curating Data your Robot Loves with Influence Functions.\" *RSS.* 2025.\n\n[G] Hejna, et al. \"Robot data curation with mutual information estimators.\" arXiv preprint (2025).\n\n[H] Xu, et al. \"VideoCLIP\" arXiv preprint (2021)."}, {"review_id": "kLyD1HG7iQ", "reviewer": "Reviewer_DAU1", "summary": "This paper identifies a critical alignment issue between high-level planners and low-level visuomotor policies in hierarchical vision-language-action (VLA) agents. To address this, the authors propose a Retrieval-based Demonstration Decomposer (RDD), a novel training-free method to automatically segment long-horizon demonstrations for planner fine-tuning. The core mechanism of RDD is to formulate the task as an optimal partitioning problem, which is solved by retrieving visually and temporally similar sub-tasks from the low-level policy's training data to guide the decomposition.", "questions": "1. Given RDD's reliance on the D_aug_train database, how does its performance depend on the quality and style of the upstream heuristic used to create that database? Is RDD robust to different or lower-quality source heuristics?\n2. What is the expected behavior of RDD when encountering a truly novel sub-task not present in its retrieval database? Have you considered fallback mechanisms for such low-confidence retrieval scenarios?\n3. To better isolate the contribution, have you considered a more direct evaluation of decomposition quality itself？", "limitations": "Yes, the authors have acknowledged the existence of a limitations section in the appendix. However, given the significance of certain limitations (e.g., the reliance on the quality of the policy's training data), it would improve the paper to briefly discuss the most critical ones in the main text for better transparency.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper tackles a highly relevant and significant problem in hierarchical robot learning. Ensuring that the high-level planner generates instructions that the low-level policy can reliably execute is crucial for the success of long-horizon tasks. The proposed direction of aligning the planner's fine-tuning data with the policy's existing capabilities is a valuable contribution to the field.\n2. The central concept of RDD is elegant and highly intuitive. Instead of relying on hand-engineered heuristics or expensive human annotations, RDD leverages the policy's own training data as a \"template\" for what constitutes a good, executable sub-task. \n3. The experimental evaluation is comprehensive, featuring strong baselines on a standard benchmark and extensive ablations that validate key design choices and demonstrate the method's effectiveness.\n\nWeakness\n1. The sub-task representation, using only start and end frames, is simplistic. It discards crucial intermediate dynamics and may fail to distinguish between sub-tasks with different motions but similar boundary states.\n2. The technical novelty is somewhat limited. The method is primarily a novel application and combination of existing techniques (dynamic programming, visual similarity search) rather than a fundamental algorithmic contribution.\n3. The retrieval-based nature of RDD raises questions about its robustness to novel, out-of-distribution sub-tasks. The paper lacks discussion and experiments for scenarios where a new demonstration contains actions not well-represented in the training database.", "was_revised": true, "final_justification": "The authors' response has addressed some of my concerns, and I encourage the authors to include these experiments and discussions in the paper. I will raise my score to borderline accept.", "rebuttal": "First of all, we sincerely thank the reviewer for the insightful feedback. We are encouraged that the reviewer pointed out the strengths of our paper: 1) the significance of the problem we address; 2) the elegant and intuitive nature of our proposed method; and 3) the comprehensive experimental validation with extensive ablations. We address the reviewer’s comments and questions point by point in the following.\n\n# Response to Weakness 1\n\nWe thank the reviewer for this insightful question. A key strength of our RDD framework is its modularity. The interval similarity measure, **sim**, is a plug-and-play component that offers a convenient and flexible interface to incorporate domain-specific knowledge or more powerful representations without altering the core dynamic programming algorithm. For instance, one could easily implement a more sophisticated **sim** function using powerful video encoders like VideoCLIP [A], to capture richer temporal and semantic information. Our novel, general problem formulation, along with the flexible design of our scoring function, is ready to integrate various designs of similarity measures. \n\nIn \"Response to Question 2 & Weakness 3\" we provide experimental validation of an implementation of this idea.\n\n# Response to Weakness 2\n\nWe thank the reviewer for their valuable feedback. We respectfully emphasize that our work's primary contribution is the novel formulation and solution to the critical, previously unaddressed alignment gap between a hierarchical VLA's high-level planner and its low-level visuomotor policy. Our novelty is threefold:\n\n1. **Novel Problem Formulation:** As noted by Reviewer 9Wsv, we are the first to formally identify and address the misalignment between a planner's generated sub-tasks and the visuomotor policy's capabilities. We introduce a principled mathematical framework by formulating this as an optimal partitioning problem.\n2. **A Provable and Linear Complexity Algorithm:** Crucially for robotics applications, we prove that it is possible for the dynamic programming algorithm to achieve linear time complexity, **O(N)**, when the maximum interval length is bounded. (Corollary3.1.1.)\n3. **Unbiased Scoring Function:** We propose a novel interval scoring function to measure the semantic alignment of sub-tasks with the policy's training data. Its additive property not only enables the dynamic programming solution but, as proven in Proposition 3.1, it also prevents segmentation bias, ensuring the quality of the partitioning.\n\nWe believe the identification of this fundamental problem, novel mathematical formulation and an efficient, theoretically-grounded algorithm, constitutes a significant contribution. We will highlight these points more clearly in the revised paper.\n\n# Response to Question 1\n\nWe thank the reviewer for their insightful question regarding RDD’s sensitivity to the quality of sub-task segmentation. We would like to address this by highlighting two key aspects of our approach:\n\nFirst, we wish to clarify RDD’s primary objective. The core contribution is not to learn new skills or correct flawed ones, but to enable a high-level planner to effectively utilize the skills that the low-level visuomotor policy already possesses. Therefore, **RDD is largely agnostic to the \"optimality\" of the skills themselves. This ensures the planner generates commands that the policy can reliably execute, rather than potentially \"better\" ones it cannot handle.**\n\nSecond, in scenarios where the visuomotor policy's training data contains significant noisy samples that the policy fails to learn, **RDD can be easily integrated with dataset curation techniques [B, C].** These methods can serve as a pre-processing step to filter the visuomotor training set. For instance, CUPID [B] computes an \"action influence\" score for state-action pairs that can be used to evaluate each segment's contribution to the policy's final behavior. By applying a simple threshold, low-influence or flawed segments can be pruned from the dataset before RDD uses it as a reference. This would prevent catastrophic failures by ensuring RDD aligns demonstrations only with high-quality, influential sub-tasks.\n\nWe will add these clarifying discussions to the revised paper.\n\n# Response to Question 2 & Weakness 3\n\nWe thank the reviewer for this insightful question regarding RDD’s handling of out-of-distribution (OOD) demonstrations. While the primary objective of RDD is to align the planner with the visuomotor policy’s existing capabilities, this is a critical aspect for real-world applicability. **In such OOD scenarios, the objective changes to: aligning sub-task intervals to both existing visuomotor sub-tasks and potentially new sub-tasks.**\n\nThanks to the flexible design of RDD's scoring function $\\\\tilde{J}$, we do have a simple yet effective mechanism to gracefully handle these OOD scenarios and prevent catastrophic failures. Our approach involves two stages: OOD identification and OOD-aware decomposition.\n\n**OOD Identification**\n\nFirst, we must determine if a given demonstration is, in fact, OOD. We can quantify the novelty of a demonstration by calculating $\\\\Delta$, the average similarity score of the optimal partition $\\\\mathrm{P}$ found by the standard RDD algorithm:\n\n\n$$\n\\\\Delta=\\\\frac{1}{|\\\\mathrm{P}|}\\\\sum_{\\\\mathcal{I} \\\\in \\\\mathrm{P}} \\\\tilde{J}(\\\\mathcal{I}).\n$$\n\nA low value of *$\\\\Delta$* indicates that even the best possible decomposition has a low similarity to the sub-tasks in the training data, which signals an OOD demonstration. A simple threshold can be used to trigger our adaptive mechanism.\n\n**OOD-aware Decomposition** \n\nOnce an OOD demonstration is identified, we introduce a modified similarity measure, $\\\\mathbf{sim}_{OOD}$, which incorporates a general interval scoring function $\\\\mathbf{G}$:\n\n\n$$\n\\\\mathbf{sim}_{OOD}(\\\\mathcal{I}_j^i,\\\\tilde{\\\\mathcal{I}_j^i})=\\\\mathbf{sim}(\\\\mathcal{I}_j^i,\\\\tilde{\\\\mathcal{I}_j^i})+\\\\beta \\\\mathbf{G}(\\\\mathcal{I}_j^i).\n$$\n\nHere, $\\\\mathbf{G}$ evaluates how well a proposed interval aligns with \"general\" sub-tasks, independent of the planner's specific training set. The hyperparameter $\\\\beta$ balances the trade-off between aligning with known sub-tasks (the original RDD objective) and discovering novel, generalizable sub-tasks. $\\\\mathbf{G}$ can be implemented using functions like UVD [D] to measure how well an interval conforms to generic change-point detection heuristics:\n\n$$\n\\\\mathbf{G}(\\\\mathcal{I})=-\\\\frac{1}{|\\\\mathcal{I}|}\\\\mathbf{abs}(b-\\\\mathbf{UVD}(e,\\\\mathcal{I})),\n$$\n\nwhere $b,e$ represent the index of the beginning and ending frame of interval $\\\\mathcal{I}$. $\\\\mathbf{UVD}(e,\\\\mathcal{I})$ gives the index of the UVD predicted beginning frame, given the goal frame on $e$. \n\n## Additional Experimental Validation on LIBERO\n\nFor OOD scenarios, we conducted additional experiments on LIBERO. To simulate a highly OOD scenario, we used the human-operated demonstration dataset from RoboCerebra [E] where demos are highly diverse in objects, task goals, and arrangements. We evaluated the quality of the decomposition against ground-truth segmentations using the mean Intersection-over-Union (mIoU) (see response to Reviewer DAU1 for details). We use 560 demos to build the RDD database and test on the remaining 140 demos.\n\n| Methods             | mIoU (LIBERO OOD Test) |\n| ------------------- | ----------------- |\n| UVD                 | 0.598             |\n| RDD ($\\\\beta=0.25$) | 0.624 ( ↑ 4.4%)   |\n| RDD ($\\\\beta=0.1$)  | 0.630 ( ↑ 5.3%)   |\n| RDD ($\\\\beta=0.05$) | 0.614 ( ↑ 2.6%)   |\n\nThe results clearly show that our OOD-aware **RDD not only avoids catastrophic failure but also outperforms the baseline UVD decomposer in OOD settings.** RDD can successfully leverage even limited similarities in novel scenarios to produce a superior decomposition. \n\n## Additional Experimental Validation on Real-World Dataset\n\nFor in-distribution alignment (original goal of RDD), we test RDD on the real-world manipulation dataset AgiBotWorld-Alpha [F] to show the generalization ability of RDD visual features.  We test RDD and UVD on the task \"supermarket\" where 152 demos are used to build the RDD database and 37 demos are used for testing.\n\n| Methods | mIoU (AgiBotWorld Alignment Test) |\n| ------- | --------------------------- |\n| UVD     | 0.506                       |\n| RDD     | 0.706 ( ↑ 39.6%)            |\n\nRDD significantly outperforms the baseline method UVD, proving its generalization ability.\n\nWe will add the above discussions to the revised paper.\n\n# Response to Question 3\n\nWe sincerely thank the reviewer for this insightful suggestion. To achieve this, we adopt the mean Intersection-over-Union (mIoU) as the metric to evaluate decomposition accuracy. We treat the segments generated by the task-specific decomposer, which was used to create the visuomotor policy’s training set, as the ground truth. The accuracy is formulated as\n$$\naccuracy=\\\\frac{1}{|\\\\mathrm{P}|}\\\\sum_{\\\\mathcal{I}\\\\in\\\\mathrm{P}} \\\\mathbf{IoU}(\\\\hat{\\\\mathcal{I}},\\\\tilde{\\\\mathcal{I}}),\n$$\nwhere $\\\\tilde{\\\\mathcal{I}}$ is the ground truth interval with maximum IoU with the predicted one $\\\\hat{\\\\mathcal{I}}$. To validate this metric, the table below shows the measured mIoU and end-to-end success rate of UVD and RDD.\n\n| Methods | mIoU    | Success Rates |\n| ------- | ------- | ------------- |\n| UVD     | 0.522   | 57.1          |\n| RDD     | 0.623 ↑ | 61.0 ↑        |\n\n\n# References\n\n[A] Xu, et al. \"VideoCLIP\" arXiv preprint (2021).\n\n[B] Agia, et al. \"CUPID: Curating Data your Robot Loves with Influence Functions.\" *RSS.* 2025.\n\n[C] Hejna, et al. \"Robot data curation with mutual information estimators.\" arXiv preprint (2025).\n\n[D] Zhang, et al. \"Universal visual decomposer\" ICRA, 2024.\n\n[E] Han, et al. \"RoboCerebra\" arXiv preprint  (2025).\n\n[F] Bu, et al. \"Agibot world colosseo.\" arXiv preprint (2025)."}, {"review_id": "VgkseFmvVV", "reviewer": "Reviewer_wuNw", "summary": "This paper proposes a method to decompose the demonstration into \"sub-tasks\" to improve the learning of VLA models. The authors formulate the decomposition as an optimal partition problem, where the partition objective relates to the learning data of low-level policies. The decomposition method is applied to a baseline hierarchical VLA and tested on 18 RLBench tasks for evaluation.", "questions": "See the strength and weakness section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "[+] The overall idea is well-motivated and shares a lot of insights with recent hierarchical VLA models, where we can use long-horizon task demonstrations to facilitate the learning of plan and action, and also perhaps task augmentation.\n\n[+] The overall description of the framework is clear and illustrative, with major points of the design easy and clear to follow.\n\n[-] One major concern lies in the marginal improvement of this method to the tasks selected in the evaluation phase. The current result is a bit hard to validate the effectiveness of the current method, especially given several ties between the proposed and the Uniform or Heuristic methods.\n\n[-] Another point is that I do feel the current framework can be extended to form many more interesting attempts and analyses, e.g., cross-domain or sim2real transfer (as the current decomposition should ideally support skill reuse). The current discussion on only RLBench is a bit narrow in my opinion, especially given the marginal improvement.", "was_revised": true, "final_justification": "The author rebuttal has addressed most of my concerns. Therefore, I'm increasing my score to an accept.", "rebuttal": "First of all, we sincerely thank the reviewer for the insightful feedback. We are encouraged that the reviewer pointed out the strengths of our paper:  the strong motivation for our idea and the clarity of our paper. We address the reviewer’s comments and questions point by point in the following.\n\n# Response to Weakness 1\n\nWe sincerely thank the reviewer for the comment. We respectfully clarify that our work's significance is threefold:\n\n1. **Near-oracle Performance and State-of-the-art Advancement:** We would like to emphasize that the Heuristic baseline, which relies on manually-defined rules, represents a strong, near-oracle method for these tasks. Our proposed RDD, without any task-specific knowledge, **achieves comparable performance to this oracle**, as shown in Table 1. This is a significant result, demonstrating the effectiveness of our approach. Furthermore, RDD improves the average task success rate by 6.7% over the state-of-the-art general-purpose decomposer, UVD. **To better demonstrate the effectiveness of RDD, we provide extended experiment results at the end of this rebuttal.**\n\n2. **Scalability and Growing Importance:** More importantly, we argue that the true advantage of RDD also lies in its scalability and adaptability. The current trend in robotics is toward leveraging large-scale, diverse visuomotor datasets. In such scenarios, designing effective, task-specific heuristics will become increasingly complex and impractical. RDD is designed to automatically and efficiently align the planner with the visuomotor policy’s training data, **a feature that will become crucial as the complexity of tasks and datasets grows.** In section 6 of Hi Robot [A], the state-of-the-art hierarchical vision-language-action research, especially highlights this problem:\n\n   > \"The training process decouples the high-level and low-level models, and they are not aware of one another’s capabilities except through the training examples. Coupling these two layers more directly, e.g., by allowing the high-level policy to be more aware of how successfully the low-level policy completes each command, would be an exciting direction for future work.\"\n\n   RDD provides an automated and efficient solution to align demonstrations with a policy's training data, which is a crucial capability for future complex hierarchical VLAs.\n\n3. **Novel Problem Formulation and Efficient Solution:** As recognized and highlighted by Reviewer 9Wsv, **we are the first to identify and formally address the crucial alignment gap between high-level planner training data and the capabilities of the low-level visuomotor policy in hierarchical VLAs.** As we note in our paper, sub-tasks generated by heuristics can deviate significantly from what the visuomotor policy is optimized for, leading to degraded performance. Our work is the first to tackle this specific misalignment problem. **We introduce a novel formulation of this problem as an optimal partitioning task.** This provides a principled mathematical framework for demonstration decomposition. Based on this, we develop an efficient dynamic programming solver and analyze its complexity. We prove that **when the maximum interval length is bounded, our algorithm achieves linear time complexity O(N), which is a vital property for general robotics applications**. This theoretical analysis of the algorithm’s efficiency is a key algorithmic contribution.\n\n# Response to Weakness 2\n\nWe thank the reviewer for this insightful question regarding RDD’s applications in broader scenarios. Both cross-domain and sim2real transfer require RDD to handle the out-of-distribution (OOD) cases, where fundamentally novel sub-tasks appear in the demonstrations to be decomposed. **In such OOD scenarios, the objective changes to: aligning sub-task intervals to both existing visuomotor sub-tasks and potentially new sub-tasks.** In this response, we show how RDD can extend to identify cross-domain novel sub-tasks and its performance in the real-world dataset.\n\nThanks to the flexible design of RDD's scoring function $\\tilde{J}$, we do have a simple yet effective mechanism to gracefully handle these OOD scenarios and prevent catastrophic failures. Our approach involves two stages: OOD identification and OOD-aware decomposition.\n\n**OOD Identification**\n\nFirst, we must determine if a given demonstration is, in fact, OOD. We can quantify the novelty of a demonstration by calculating $\\\\Delta$ , the average similarity score of the optimal partition $\\\\mathrm{P}$ found by the standard RDD algorithm:\n\n\n$$\n\\\\Delta=\\\\frac{1}{|\\\\mathrm{P}|}\\\\sum_{\\\\mathcal{I} \\\\in \\\\mathrm{P}} \\\\tilde{J}(\\\\mathcal{I}).\n$$\n\nA low value of *$\\\\Delta$* indicates that even the best possible decomposition has a low similarity to the sub-tasks in the training data, which signals an OOD demonstration. A simple threshold can be used to trigger our adaptive mechanism.\n\n**OOD-aware Decomposition** \n\nOnce an OOD demonstration is identified, we introduce a modified similarity measure, $\\\\mathbf{sim}_{OOD}$, which incorporates a general interval scoring function $\\\\mathbf{G}$:\n\n\n$$\n\\\\mathbf{sim}_{OOD}(\\\\mathcal{I}_j^i,\\\\tilde{\\\\mathcal{I}_j^i})=\\\\mathbf{sim}(\\\\mathcal{I}_j^i,\\\\tilde{\\\\mathcal{I}_j^i})+\\\\beta \\\\mathbf{G}(\\\\mathcal{I}_j^i).\n$$\n\nHere, $\\\\mathbf{G}$ evaluates how well a proposed interval aligns with \"general\" sub-tasks, independent of the planner's specific training set. The hyperparameter $\\\\beta$ balances the trade-off between aligning with known sub-tasks (the original RDD objective) and discovering novel, generalizable sub-tasks. $\\\\mathbf{G}$ can be implemented using functions like UVD [B] to measure how well an interval conforms to generic change-point detection heuristics:\n\n$$\n\\\\mathbf{G}(\\\\mathcal{I})=-\\\\frac{1}{|\\\\mathcal{I}|}\\\\mathbf{abs}(b-\\\\mathbf{UVD}(e,\\\\mathcal{I})),\n$$\n\nwhere $b,e$ represent the index of beginning and ending frame of interval $\\\\mathcal{I}$. $\\\\mathbf{UVD}(e,\\\\mathcal{I})$ gives the index of the UVD predicted beginning frame, given the goal frame on $e$. This $\\\\mathbf{G}$ essentially measures how $\\\\mathcal{I}$ aligns with the general sub-task interval proposed by UVD. This function penalizes intervals that deviate from what a general-purpose decomposition model would propose, effectively acting as a robust prior. The newly discovered sub-tasks can then be used to fine-tune both the planner and the visuomotor policy, which allows the system to learn and adapt to the new skills.\n\n## Additional Experimental Validation on LIBERO \n\nTo validate this approach, we conducted additional experiments on a challenging LIBERO benchmark. To simulate a highly OOD scenario, we used the human-operated demonstration dataset from RoboCerebra [C], where demos are highly diverse in objects, task goals, and arrangements. We evaluated the quality of the decomposition against ground-truth segmentations using the mean Intersection-over-Union (mIoU) (see response to Reviewer DAU1 for details). We use 560 demos to build the RDD database and test on the remaining 140 demos.\n\n| Methods             | mIoU (LIBERO OOD Test) |\n| ------------------- | --------------- |\n| UVD                 | 0.598           |\n| RDD ($\\\\beta=0.25$) | 0.624 ( ↑ 4.4%) |\n| RDD ($\\\\beta=0.1$)  | 0.630 ( ↑ 5.3%) |\n| RDD ($\\\\beta=0.05$) | 0.614 ( ↑ 2.6%) |\n\nThe results clearly show that our OOD-aware **RDD not only avoids catastrophic failure but outperforms the baseline UVD decomposer in OOD settings.** By balancing retrieval with a general sub-task prior, RDD can successfully leverage even limited similarities in novel scenarios to produce a superior decomposition. This demonstrates the robustness and adaptability of the RDD framework.\n\n## Additional Experimental Validation on Real-World Dataset\n\nFor in-distribution aligning (original goal of RDD), we test RDD on the real-world manipulation dataset AgiBotWorld-Alpha [D] to show the generalization ability of RDD visual features.  We test RDD and UVD on the task \"supermarket\" where 152 demos are used to build RDD database and 37 demos are used for testing.\n\n| Methods | mIoU (AgiBotWorld Alignment Test) |\n| ------- | --------------------------- |\n| UVD     | 0.506                       |\n| **RDD**     | **0.706 ( ↑ 39.6%)**            |\n\nRDD significantly outperforms the baseline method UVD, proving its generalization ability.\n\nWe will add the above discussions to the revised paper.\n\n# References\n\n[A] Shi, et al. \"Hi robot: Open-ended instruction following with hierarchical vision-language-action models.\" arXiv preprint (2025).\n\n[B] Zhang, et al. \"Universal visual decomposer.\" ICRA, 2024.\n\n[C] Han, et al. \"RoboCerebra.\" arXiv preprint  (2025).\n\n[D] Bu, et al. \"Agibot world colosseo.\" arXiv preprint (2025)."}, {"review_id": "PuDxVkd7ve", "reviewer": "Reviewer_G7RJ", "summary": "This work introduced RDD, which can decompose demonstrations into sub-tasks without relying on human annotation or heuristics rules. RDD aligns the visual features of sub-task intervals with the training data of visuomotor policy. The author provides detailed demo video and code, which significantly enhances the reproducibility of the proposed method. RDD achieves good performance on RLBench and robustness.", "questions": "See Strengths And Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength: \n1. The author provides detailed demo video and code, which significantly enhances the reproducibility of the proposed method.\n2. The author conduct ablations and present detailed analysis for the proposed method.\n3. The paper is well-written.\n\nWeaknesses:\n1. How does the RDD perform compared to the decomposer based on code-generating LLM/VLMs, e.g., instruct2act [1], robotics programmer [2] and Code as policies [3].\n\n[1] Huang, Siyuan, et al. \"Instruct2act: Mapping multi-modality instructions to robotic actions with large language model.\" arXiv preprint arXiv:2305.11176 (2023).\n\n[2] Xie, Senwei, et al. \"Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation.\" arXiv preprint arXiv:2501.04268 (2025).\n\n[3] Liang, Jacky, et al. \"Code as policies: Language model programs for embodied control.\" ICRA, 2023.", "was_revised": true, "final_justification": "The authors’ responses have addressed my concerns. I will maintain my scores, i.e, borderline accept.", "rebuttal": "First of all, we sincerely thank the reviewer for the insightful feedback. We are encouraged that the reviewer pointed out the major strengths of our paper: 1) the paper’s clarity; 2) detailed analysis; and 3) reproducibility. We address the reviewer’s comments and questions point by point in the following.\n\n# Response to Weakness 1\n\nThanks for providing the reference papers about VLM-based sub-task decomposition. **We would like to clarify that these works use code scripts as low-level control policies, which is different from our settings, where a learning-based visuomotor policy is the target for planner alignment.** Instead, we provide extended experiments by using the state-of-the-art large multi-model language model Gemini-2.5-pro as a sub-task decomposer for comparison. In the following table, we compared RDD with a Gemini-2.5-pro-based decomposer with the following prompt:\n\n> There is a robot doing a task, which can be segmented into multiple steps.\n> A keyframe is where the robot finishes the previous step and begins the next.\n> Can you help me find ALL indexes of keyframes?\n> Please return a list of indices, for example: [15, 65, 105, ...].\n> Note that the frame index starts from 0 instead of 1.\n\n| Method         | Avg. Success Rate (↑) | Avg. Ranking (↓) |\n| -------------- | --------------------- | ---------------- |\n| Gemini-2.5-pro | 72.6 ± 4.7            | 1.7 ± 0.4        |\n| RDD (Ours)     | **74.9 ± 6.9**        | **1.3 ± 0.4**    |\n\nAs shown in the table, RDD outperforms Gemini-2.5-pro despite its powerful general video understanding abilities and significant computing resources. **These results highlight the necessity of planner alignment, as well as the effectiveness of our approach RDD.**\n\nWe will also include and discuss the reference papers provided by the reviewer in the related work section of the revised paper."}]

Author rebuttal:

Meta-review: This submission focuses on the alignment between high-level planners and low-level visuomotor policy for long-horizon robotic manipulation. The authors propose a Retrieval-based Demonstration Decomposer, which a a training-free method to  automatically decomposes demonstrations into sub-tasks. Experiment on RLBench demonstrate the performance under various settings. All reviewers vote for borderline acceptance.
AC recommend accept as poster.
Decision: Accept (poster)