Efficient Robotic Policy Learning via Latent Space Backward Planning
Authors: Dongxiu Liu
Haoyi Niu
Zhihao Wang
Jinliang Zheng
Yinan Zheng
Zhonghong Ou
Jianming HU
Jianxiong Li
Xianyuan Zhan
Venue: ICML 2025 poster (2025)
Keywords: Planning
Embodied Agents
Goal-Conditioned Policy
Primary area: applications->robotics

Abstract: Current robotic planning methods often rely on predicting multi-frame images with full pixel details. While this fine-grained approach can serve as a generic world model, it introduces two significant challenges for downstream policy learning: substantial computational costs that hinder real-time deployment, and accumulated inaccuracies that can mislead action extraction. Planning with coarse-grained subgoals partially alleviates efficiency issues. However, their forward planning schemes can still result in off-task predictions due to accumulation errors, leading to misalignment with long-term goals. This raises a critical question: Can robotic planning be both efficient and accurate enough for real-time control in long-horizon, multi-stage tasks?
To address this, we propose a **B**ackward **P**lanning scheme in **L**atent space (**LBP**), which begins by grounding the task into final latent goals, followed by recursively predicting intermediate subgoals closer to the current state. The grounded final goal enables backward subgoal planning to always remain aware of task completion, facilitating on-task prediction along the entire planning horizon. The subgoal-conditioned policy incorporates a learnable token to summarize the subgoal sequences and determines how each subgoal guides action extraction.
Through extensive simulation and real-robot long-horizon experiments, we show that LBP outperforms existing fine-grained and forward planning methods, achieving SOTA performance. Project Page: [https://lbp-authors.github.io](https://lbp-authors.github.io).

Efficient Robotic Policy Learning via Latent Space Backward Planning
Dongxiu Liu 12*
Haoyi Niu 1' Zhihao Wang 13 Jinliang Zheng" Yinan Zheng'
Zhonghong Ou" 21
Jianming Hu Jianxiong Li Xianyuan Zhan 14f
Abstract
1. Introduction
Current robotic planning methods often rely on predicting multi-frame images with full pixel de- tails. While this fine-gri fine-grained approach can serve as generic world model, it introduces two sig- nificant challenges for downstream policy leamn ing: substantial computational costs that hinder real-time deployment, and accumulated inaccu- racies that can mislead action extraction. Plan- ning with coarse- coarse-grained grained subgoals partially alle- viates efficiency issues. However, their forward planning schemes can still result in off-task pre- dictions due accumulation errors, leading misalignment with long-term goals. This raises critical question: Can robotic planning be both efficient and accurate enough for real-time con- trol in long-hori horizon, multi-stage tasks? To ad- dress this, we propose a Latent Space Backward Planning scheme (LBP), which begins by ground- ing the task into final latent goals, followed by cursively predicting intermediate subgoals closer to the current state. The grounded final goal en- ables backward subgoal planning to always re- main aware of task completion, facilitating on- task prediction along the entire planning hori- policy incorpo-
Accurately predicting future states is crucial for many robotic planning methods in solving nag-horizon, multi- stage tasks, where models must anticipate outcomes over extended temporal sequences. However, this requires bal- ancing two conflicting objectives: (1) capturing sufficiently rich and accurate future information for task completion, and (2) maintaining computational efficiency for real-time decision making. Current methods often struggle with this trade-off. Those that prioritize long-term capability typ- ically predict multi-step future states to provide detailed guidance, but are prone to high computational costs and rapid error accumulation. In contrast, efficiency--rreeneed methods often sacrifice the semantic richness needed to han- dle complex, long- long-horizon tasks. This creates fundamen- galemca-balancing
tal trilemma- balancing efficiency, accuracy and sufficient future guidance- thadanceethat remains unresolved and presents a significant challenge in real-world robotic planning.
To model future outcomes, one category of existing robotic planning methods (Du et al., 2023; Ajay et al., 2024; Hu et al., 2024) resorts to predicting an episode of future video as policy guidance. However, predicting consecutive frames can propagate inaccuracies that compound over time, result- ing in significant deviations from the intended final goal or physically inconsistent frames that confuse the downstream policies. Furthermore, modeling entire future videos re- quires high computational costs and puts a heavy burden on real-time inference. Obviously, predicting every detail in the future often unnecessary for task execution, and the fine- grained predictions hinder both computational efficiency and task-oriented consistency.
zon. The cubgoal-conditionas rates learnable token to summarize the sub- goal sequences and determines how each sub- goal guides action extraction. Through exten- sive simulation and real-robot long-horizon ex- periments, we show that LBP outperforms ex- isting fine-grained and forward planning meth- ods, achieving SOTA performance. Project Page: https: / /tppppppppaap authors. g. github. io
The second category of robotic planning methods focuses on predicting future subgoals (Nair & Finn, 2020; Huang et al., 2024). These coarse maare--gaained subgoals improve planning ef- ficiency and reduce computational burden. However, it still adheres to the forward planning paradigm, often leading to plans that are less aligned with distant goals, which in turn cause off-task behavior (Kang & Kuo, 2025). To address this, recent methods have introduced reachability or optimal- ity checks (Eysenbach et al., 2019; Nasiriany al., 2019; Fang et al., 2022; Huang et al., 2024) to correct deviations and improve on-task accuracy. However, these post-hoc adjustments also add lots complexities and do not really
Equul contribution. Work done during internship Institute for Al Industry Research (AIR), Tsinghua University. 'Tsinghua University 'Beijing University of Posts and Telecommunica- tions 'Peking University "Shanghai AI Lab. Correspondence to: Zhonghong Ou <<zhonghong.om @ @bup..dd..cn>> Jianming Hu <hujm @ mail. wwwigsua.nnnuu...mmm anyuan @ air wwwignhhaac.uu.n......... edu.en"..
Xianyuan Zhan <zhanxi-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
address the fundamental challenges.
The two aforementioned categories of methods both have some pros and cons. The video planning methods provide rich future guidance but suffer from heavy training demands and inefficient inference. The subgoal planning methods enjoy efficient planning but trade off long-horizon task progress guidance. Apart from failing to strike a desirable balance across different considerations, all previous efforts fall short in maintaining on-task prediction accuracy. How can we address the above limitations and enable robots to plan efficiently and effectively through long-horizon tasks?
zc
F2
Fg
25
Forward Fine iine-graieed Planning
22
Fs
F
Z₁
Z
Forward Coarse- -eree--rallll grained Planning
In this paper, we propose a Backward Planning approach in Latent space (LBP) for language-guidn guided robotic control in Figure 1. LBP first trains latent goal predictor that maps the current state and language description to distant final goal, grounding the task objective in latent image space to enforce task progression. Second, LBP recursively predicts intermediate subgoals that are closer to the current state, ensuring that each subgoal remains aligned with the task progression and completion. These two steps mirror how humans plan in complex tasks: we begin by envisioning the desired outcome based on the task objective, and then break it down into smaller, gradually manageable subgoals that are closer to the present stage. The subgoal sequences in LBP track the path to the goal with less redundancy, providing denser guidance in near terms while preserving task progres- sion information over the entire planning horizon. Lastly, LBP incorporates a subgoal fusion technique that enables the chhgoal-conditioned policy to adaptively determine how to best utilize subgoals at varying distances. Collectively, LBP effectively addresses the triplet of challenges of off- task planning, insufficient guidance, and high computational costs inherent in previous methods.
Zz
Z1
Z+
Zz
Za
Z/
ZA
Z-
Z/
Latent Backward Planning (Ours)
2
1-th frame latent feature
I-th frame latent feature/Image
latent goal feature
latent goal feature/Immage
Language latent feature
Figure 1. Illustration of latent space backward planning.
2. Related Works
Video Planning. A significant body of research has ex- plored video generation as planners for visuomotor con- trol (Pertsch et al., 2020; Du et al., 2023; Ajay et al., 2024; Hu et al., 2024; Wu et al., 2024; Bharadhwaj et al., 2024). Approaches such as UniPi (Du et al., 2023) and HiP (Ajay et al., 2024) generate actions using inverse dynamics models from predicted consecutive frames, while Seer (Tian et al., 2025) and GR-1 (Wu et al., 2024) jointly predict actions and subsequent image frames. Although some methods op- erate in latent space (Nair et al., 2020; Hu et al., 2024; Li et al., 2024), this line of work faces significant challenges, including high computational demands and limited real-time capabilities, primarily due to the need to generate every con- secutive frame the future. Most of these methods operate in a forward autoregressive manner (Wu al., 2024: Tian et al., 2025), which are prone to rapid error accumulation over time, significantly complicating policy learning. In summary, these approaches attempt to plan with excessive detail that is often unnecessary to visuomotor control, result- ing in computational inefficiency, compounded prediction errors, and challenges in effective action extraction.
LBP provides a lightweight planning framework for robotic policy learning with on-task subgoal generation guarantee. combines the strengths of latent planning (Wang et al., 2023; Wen et al., 2024) and coarse-grained subgoal plan- ning (Wang et al., 2025; Black et al., 2024), drastically reducing computational costs and enabling real-time deploy- ment. Unlike previous methods that struggle with subgoal horizon selection, LBP provides an informative subgoal sequence spanning the entire planning horizon toward the final goal. This offers flexibility that allows the downstream policy leverage subgoal signals at varying distances. The backward planning approach further enhances on-task ac- curacy by ensuring that the predicted subgoal sequence remains aligned with the overall task progression and the ultimate objective. Through extensive evaluations in both simulation and real-robot experiments, we demonstrate that LBP significantly outperforms existing methods, especially excelling on long-horizon, multi-stage tasks.
Coarse-grained Planning. Coarse- Coarse-grained planning ap- proaches focus on predicting intermediate subgoals (Nair & Finn, 2020; Wang et al., 2023; Black et al., 2024; Wang et al., 2025), improving computational efficiency by avoid- ing the need to prediet every frame of details. Goal- conditioned supervised (GCSL) (Ghosh et al., 2021; Em- mons et al., 2022; Wang et al., 2025) and reinforcement learning (GCRL) (Chane-Sama et al., 2021; Xu et al., 2022; Park et al., 2024), have demonstrated that planning with intermediate goals can alleviate downstream policy learning burdens while enhancing long-horizon capabilities in sim-
2

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
ulation benchmark tasks. However, this paradigm faces unresolved challenges, particularly in subgoal selection, such 1) determining appropriate prediction horizons, and 2) balancing the number of subgoals for effective policy guidance (Levy et al., 2019; Nachum et al., 2019). Dis- tant subgoals provide limited actionable information, while nearby subgoals may misalign with final task objectives. Similarly, excessive subgoals increase model complexity, whereas sparse subgoals fail to capture task progression semantics. Existing methods lack principled treatment to balance planning efficiency and long-term reliability. Fur- thermore, the forward planning paradigm inherently suffers from error accumulation over time, leading to fasstask be- havior (Kang& & Kuo, 2025). Recent attempts to mitigate this through post-hoc corrections, such as reachability or optimality checks (Eysenbach et al., 2019; Nasiriany et al., 2019; Fang al., 2022: Huang et al., 2024), which add extra complexity without addressing the fundamental limitations of forward planning.
We use the expectation over all contexts because some video segments are annotated with multiple types of task- relevant information, which can be utilized as guidance during policy learning. For instance, f((/7) represents the distribution of language descriptions, f() [[)) = ff 44, (f((T)) models the joint distribution of goal im- ages and language descriptions, and [(w; Ig, [|=] - l(w = L++* 44, 1g. (), ().(())))) captures the distribution of k-step future subgoals, goal images, and language descrip- tions. Each context provides a different level of guidance: language serves as basic task identifier, goal images indi- cate task completeness, and subgoals reflect task progression toward completion. By exploring different combinations of these contexts, we can adapt the level of guidance to meet varying demands for granularity in policy learning.
4. Latent Backward Planning
Overview. We propose latent space backward planning (LBP),, an efficient and robust planning framework for long- horizon visuomotor tasks, built upon the idea of backward subgoal prediction. We observe that existing long-horizon planning with predicted subgoals struggles with (1) plan- ning inefficiency and (2) off-task prediction. Generating high-dimennionall subgoal images poses significant chal- lenges of computation loads, while modeling every future frame sequentially further deteriorates temporal efficiency, collectively hindering real-time real-world planning.
Summary. Both fine- aeeeggaaied (video) and coarse grarneeg (subgoal) planning approaches fail to resolve the fundamen- tal trilemma of robotic planning: achieving computational efficiency, maintaining mong-horizon consistency, and en- suring prediction accuracy. These limitations highlight the need for novel approach that appropriately balances these objectives in long-horizon, multi-stage visuomotor tasks. Inspired by the "coarse-to-fine" paradigm in natural lan- guage processing (Wei et al., 2022), computer vision (Tian et al., 2024), and decision making (Wang et al., 2025), we propose backward planning framework that predicts sub- goals in reverse temporal order-from coarse to fine hori- zons startingg from the final goal. At every control step, this approach generates a subgoal sequence that spans the entire task horizon, providing sufficient actionable guidance efficiently while minimizing error accumulation by ensuring consistent task alignment.
Thus, one of our key insights is planning in latent space with coarse-graine grained subgoals, enhancing planning efficiency in both spatial and temporal dimensions. While latent subgoal planning has been explored in existing works (Veerapaneni et al., 2020), they typically adopt forward planning that often fail to align subgoals with ultimate task objectives.
Without accounting for task completion, subgoals can easily deviate from desired task progression, causing downstream policies suffer from compounding errors snowballing along the planning process. Existing approaches have to introduce additional subgoal quality checks on reachability or optimality to combat the error accumulation (Nair & Finn, 2020; Eysenbach et al., 2019; Nasiriany et al., 2019; Fang et al., 2022; Huang et al., 2024), at the cost of adding un- necessary complexity and trading off efficiency, but without fundamentally resolving the underlying off- issues.
3. Preliminaries
We consider the problem of learning visuomotor policy conditioned on different contexts e that reflect task objec- tive or completeness. These contexts e can include goal images I, E 9 C I, language deseriptions E L, in- termediate subgoals W; E W C I, and etc. Each video segment is defined as T. {[Ir, H. with H; frames, where I, I represents the image observation and a, A denotes the action at time step t. Given dataset of video segments D= = {TI, T2, . .N } and a distribution over con- texts (c]T), a conditioned policy To Ta c) is trained to generate control signals in a closed-loop manner, achieving the desired behaviors aligned with the task deseription or future goals. The policy learning objective can be given as:
Another key insight of ours is that we can learn a final goal predictor that grounds the ultimate task objective (i.e. lan- guage description) into latent image space (Section 4.1). Latent image space encapsulates much richer task progres- sion information than language space, enabling backward planned subgoal sequences grounded on the predicted final goal to ensure on-task consistency (Section 4.2). The sub- goal sequences can effectively capture task progression and provide flexibility for downstream policy learning to lever- age envisioned subgoals varying distances. To facilitate
max Σ Σ TED
Ee- [log To (a(((Te..))
(1)
ISISH
3

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
efficient policy training, introduce subgoal fusion tech- nique that non-trivially compresses subgoal information and adaptively determines how to best utilize subgoals across different distances (Section 4.3).
Latent Space Projection
Latent Backward Planning
Subgoal Predictior
Predictor
4.1. Grounding Task Objective as Latent Goals
"Shift Cupe"
Goal- Goal-Fusion Attention
Previous research suggests that visual instructions can com- plement language descriptions, significantly enhancing guid- ance performance in conditioned visuomotor policy learn- ing (Radford et al., 2021; Shah al., 2023; Li et al., 2024; Zheng et al., 2024; Li et al., 2025). This synergy proves to be crucial in long- mogghooizon, multi-stage tasks, where lan- guage descriptions often reduce to task identifiers due to their limited semantic information. In contrast, latent vi- sual representations provide richer information about task progression, with latent visual goals offering precise specifi- cations of the desired final scenario. However, while latent visual goals can be easily obtained through hindsight la- beling during training, their test-time specification presents challenges (Lynch & Sermanet, 2021): it inherently de- pends on the current scenario configurations -or example, in the task "place the brown cup in front of the white cup", the precise goal state variably depends on the initialized relative spatial locations of the two cups. Crucially, this relationship is not fixed: if the position of the white cup changes at test time, the semantic meaning of "in front of" must be re-evaluated, requiring corresponding adjustments to the target visual goal. To address this, we learn a goal prediction model fg that estimates the latent visual goal 2g from the current observation I, and language instruction I. Given a dataset of video segments D2- = [T,}A,, with latent visual state 2; in Fi {2,, at) Hi and language feature Φi encoded by some pre- tre-trained encoder (zz, (b)) = $b1,, !), we optimize:
Description
ResNet
Concat
Image Observations
FILM
Context Conditioned Policy
Figure Overall framework architecture of LBP.
reflect task progression, while long subgoal sequences are prone to compounding prediction errors that lead to off-task behaviors deviating from the intended task goals. To address this, we begin by predicting the first subgoal wi from the current state 21, final goal 2g. and language instruction Φ in latent space, with the optimization objective:
max ΣΣ 8t
E Fptw., log Lu (wi [es Eg. 21) (3)
TED: 1SISH
To ensure sufficient long-term information, we set the first subgoal relatively close to the final goal, which maintains better alignment with task objectives yet provides less im- mediate guidance for policy learning. To bridge this gap, recursively predict intermediate subgoals closer to the current state. Specifically, each subsequent subgoal wy is predicted from the previous subgoal W;-1, current state 21, and instruction di, forming a backward chain from coarse to fine temporal resolutions. The optimization objective for predicting subgoal w; is given by:
visual
maxx Σ
Σ
Ep(enoort)) log (2gg((z) Φh)
fa TED. 1SISH
max Bb
Bptwenn 1.01)) log f (wil W; W;-1, Φ1) TED IStSH
Σ
where p(zg. 1/7) represents the conditional context distri- bution (p(c(T)) of latent goals and language instructions derived from trajectory т. This approach enables dynamic goal specification while ensuring the planning process oper- ates within the semantically rich latent visual space.
(4)
For convenience, let I(w;) denote the corresponding time step of subgoal w; in the trajectory. We thus define recursive planning coefficient À T(w;-1)-r' (wo zg).
[(w:)=
4.2. Predicting Subgoals with a Backward Scheme
to govern the recursive subgoal generation for = 2, which represents the ratio of the temporal distance between the predicted subgoal and the current state 2; relative the distance between the previous-leve subgoal w;-1 and the current state 2T-
While the final visual goal specifies the condition of task completion, it provides limited guidance about task pro- gression-the sequence of states required to achieve the ultimate objectives. To better capture long-horizon task progression, we predict intermediate subgoals. However, subgoal selection presents a fundamental dilemma (Park al., 2024; Levy et al., 2019): balancing the sufficiency of subgoals for task progression against the accuracy of their prediction. Sparse subgoal predictions fail to adequately
By inspecting Eq. (3) and (4), we observe that it is possible to use a single unified model fo for all different levels of subgoal predictors fü, as they all share the same structure. This unified model is expected to predict the intermediate subgoal 2 := [[e] between any start latent state 2, and final latent state 2k, where S f < k.

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
The objective is given by:
outputs the context embeddings Zç- This design compresses all contextual tokens e into lower-dimensional token &c while enabling the adaptive extraction of the most relevant context information. This enables dynamic balancing of short- and long-term guidance throughout task execution, maximally leveraging the flexibility of predictions at vary- ing distances and granularities.
max Σ
Σ B pl fw r ISKCH pllechharlt)
log for (22x)/2,,,20 ZAp 2x (,) Σ
Σ Σ ISECH ((la)))ii11)
E plemadylt)
Σ log (2x (1)
(5)
where Zạz := 2(0-N))44M CT, [l..... n| denotes the ground truth subgoal and Exp denotes its predicted coun- terpart by fw. The first term in Eq. (5) fits subgoal prediction with the ground truths 2x² in т, capturing the actual task progression. The second term optimizes the subgoal predic- tor fw given its own previous predictions Σx-- 1 as inputs, ensuring the consistency of the recursive prediction of fw at test-time. This recursive mechanism will suffer much less compounding error as the X-recursion effectively reduces the planning steps, and the training of fw incorporates su- pervision of groundtruths in every recursion level.
4.4. Practical Algorithm
In the training phase, we learn final goal predictor fa with Eq. (2), a unified subgoal predictor fwr with Eq. (5), and conditioned policy with Eq. (6). At each step & at test time, LBP processes the current observation It and language instruction into latent state 2, and language feature di, and then generates latent (sub)goal plans {wx..... W2, W1, &g} by fg and fw- Then we use the contexts e including pre- dicted goal plans and language features Φi to condition the policy T(a,) Je, e) for action extraction.
The detailed architecture of our model is present in Fig- ure 2. We implement the goal predictor fy and the subgoal predictor fw using two-layer MLPs and employ cross- attention block for the goal-fusion module. Compared to recent planning- level generative models, LBP demonstrates significant effi- ciency. For the low-level policy, we use a shared ResNet- 34 (He et al., 2016) as the backbone to extract visual features from different camera views, where the language embed- dings are injected via FiLM conditioning layers (Perez et al., 8a). The current visual features encoded by ResNet are then integrated with the contexts to generate actions. The policy is optimized with diffusion loss to model complex distributions (Chi et al., 2023), with the denoising step fixed at 25. More details are provided in Appendix A.
As illustrated in Figure 1, this backward planning scheme generates asymmetric to-se-to-finne grained latent subgoal sequences spanning the entire task horizon, offering three key advantages over conventional methods: (1) compre- hensive task progression information in subgoal sequences, providing rich and flexible guidance for policy learning: (2) improved prediction consistency with task objectives in backward manner, reducing error accumulation com- pared to forward planning; and (3) computational efficiency by adopting recursion, avoiding the need for ine-graiined frame-by-frame prediction.
methods that rely on complex pixel-
4.3. Learning Context Conditioned Policy
The generated subgoal sequence provides rich contextual information for policy learning. Given the complete context set e {Way..., W1, 2g. Φ1} E [x(+2)xNN, derived from dataset Dz, we optimize the conditioned policy through:
5. Experiments
5.1. Experimental Setup
max Σ TED
Σ 15ISH
Ee-p(ehr) log (0el)22)))) e)
(6)
Using previously validated evaluation recipes for embod- ied Al (Black et al., 2024; Kim et al., 2024; Tian et al., 2025; Zheng et al., 2025), we assess the performance of the proposed LBP. Specifically, we assess LBP on both the LIBERO-LONG simulation benchmark and real-robot enviromment with long-horizon multi-stage tasks. For all methods involving subgoal prediction, the planning process is solely applied to the LONG experiments and the top view in the real-world ex- periments. Unless otherwise stated, we adopt a three-step planning scheme (predicting a final goal and two interme- diate subgoals) of LBP and set the planning coefficient A = 0.5. Ablation studies on key framework designs and different choices of À are provided in Section 5.3.
However, even in latent space, the aggregated context di- mensions can burden policy learning. Moreover, the pol- icy should adaptively leverage (sub)goal information rather than treating all predictions equally, as different task exe- cution stages require varied focus between short-term and long-term guidance. For instance, tasks requiring large movements intuitively benefit more from distant subgoals to prevent actions that hinder future progress, while precision- oriented tasks require stronger emphasis on nearby subgoals.
view in the LIBERO-
To address these challenges, we introduce a goal-fusion module with a Perceiver-style cross-attention (Jaegle et al., 2021) that performs both correlation discovery and dimen- sionality reduction. Specifically, the contexts e are queried by a trainable latent vector of size Dz: E IRIXN which
LIBERO-LONG experiments. LIBERO-LONG (Liu et al., 2024) consists of 10 distinct long-horizon robotic
5

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
Table LIBERR-LOOG results. For each task; we present the average performance of top-3 checkpoints. The metric "Avg. Success" measures the average success rate across 10 tasks. LBP outperforms baselines with higher Avg. Success and hetter results on most tasks. The best results are bolded. LIBERO-LONG tasks include: (1) put soup and sauce in basket; (2) put box and butter in basket; (3) turn on stove and put pot: (4) put bowl in drawer and close it; (5) put mugs on left and right plates: (6) pick book and place it in back; (7) put mug on plate and put pudding to right; (8) put soup and box in basket; (9) put both pots on stove; (10) put mug in microwave and close it.
Task ID
2
4
5
6
7
9
10
Avg. Suc 1
Method
MTACT
0.00
50.0
75.0
85.0
20.0
75.0
0.00
30.0
10.0
65.0
41.0
MVP
78.3
90.0
80.0
88.3
46.6
63.3
45.0
83.3
60.0
46.6
68.2
MPI
86.6
86.6
96.6
95.0
83.3
83.3
56,6
66.6
40.0
78.3
77.3
Open VLA
45.0
95.0
65.0
45.0
40.0
80.0
60.0
35.0
20.0
55.0
54.0
Seer
88.3
90.0
91.6
81.6
85.0
65.0
86.6
80.0
51.6
66.6
78.6
SUSIEt
83.3
63.3
96.6
100.0
83.3
83.3
83.3
39.9
53.3
76.6
76.3
LBP SigLIP
86.6
100.0
93.3
100.0
63.3
73.3
86.6
80.0
73.3
93.3
85.0
LBP DecssioNN
90.0
100.0
100.0
100.0
76.6
86.6
90.0
86.6
60.0
96.6
88.6
? Since the original SuSIE only supports single-view input, we incorporate a wrist view to reproduce it fair comparison.
manipulation tasks that require diverse skills such as pick- ing up objects, turning stove, and closing microwave. These tasks involve multi-stage decision makinng and span variety of scenarios, making them particularly challenging. All models are trained on 50 unique expert demonstrations for each task. More details of LIBERO-LONG benchmark are provided in Appendix B.
vision- sation-laggaae-actiin policy OpenVLA (Kim et al., 2024), an image-editing based subgoal planner SuSIE (Black et al., 2024), and the end-to-end predictive inverse dynamics model Seer (Tian et al., 2025). For real-world experiments, we compare LBP with SuSIE, one of the most competitive methods against LBP the LIBERO-LONG benchmark. Also, we deploy vanilla Language Conditioned Behavior Cloning (LCBC) and Goal-and- Loal-and-Language Conditioned Be- havior Cloning (GLCBC) for comprehensive comparison.
To investigate the effectiveness of LBP in real world, we specifically design four long- horizon tasks: Stack 3 cups, Move cups, Stack 4 cups and Shift cups. Each task is decomposed into multiple sequential stages, illustrated in Figure 3, requiring the robot to per- form fundamental pick-and-place operations. These tasks establish a critical dependency where progress in subsequent stages is contingent on successful exccution of preceding ones. We assess task performance using a stage-based scor- ing system with discrete values {0, 25, 50, 75, 100} for each stage, where each score corresponds to the completion progress of the current stage. A stage is assigned 100 only upon successful completion of the entire stage. All exper- imental evaluations are conducted with 6 DoF AIRBOT robotic arm, together with three different views provided by Logitech C922PRO cameras. The overall environmen- tal setups and task illustrations are shown in Figure All models are trained using 200 expert demonstrations for the task Move cups and Shift cups, and total of 200 expert demonstrations for Stack 3 cups and Stack 4 cups. More details of experimental setups can refer to Appendix C.
Real-world experiments.
Metries for long-horizon multi-stage tasks. Following Seer (Tian et al., 2025), we evaluate model performance on the LIBERO-LONG benchmark by averaging the results of the top three checkpoints, each evaluated over 10 rollouts per task. For real-world experiments, we evaluate the last three checkpoints, with each checkpoint being tested across 10 rollouts per task to provide an average score at each stage, offering a thorough evaluation of long-horizon capabilities. Since each task consists of multiple stages, we design a fine- grained scoring system to evaluate performance at the stage level. Specifically, a score of 25 is awarded when the robot shows clear intent to approach the correct target object. If the target is successfully picked up, the score increases to 50. Carrying the object toward the correct destination yields 75 points, and placing it successfully at the desired location results in a full score of 100. For multi-stage tasks, we enforce a strict rule: the robot can proceed to the next stage only if the current stage achieves full 100 points. This rule makes our real-robot experiments a rigorous benchmark for evaluating long-horizon capability.
Baselines. For the LIBERO-LONE benchmark, we im- plement the multi-task policy MTACT (Zhao et al., 2023), the general image-based pre-trained policy MVP (Xiao et al., 2022), the interaction-oriente representation learn- ing method MPI (Zeng et al., 2024), large-scale pretrained
5.2. Main Results
Simulation Experiment Results. Table presents the quantitative comparison on the LIBERO-LONG benchmark.
6

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
[1] Move
cups
Top view
Stage
Wrist view
(2) Stack
cups
DeF AIRBOT
Stage
Stage
Stage
(3) Shift
cups
Stage
Stage II
Stage III
Stage IV
Stage
Figure Left: the entire desktop environment setups of real-world experiments contains a 6 DoF AIRBOT arm and three Logitech C922PRO cameras with different views: Right: (1) Move cups: both brown cups in front of the white ones; (2) Stack cups: stack all paper cups together; (3) Shift cups: shift all the paper cups to another plate, a clockwise direction.
Stack cups
cups
Stack 4 cups
Shift cups
100
LEBC
SUSE
Avg. Score
60
40
Stage
Stage
Stage
Stage
Stage
Stage
Stage
Figure Real-workd main results. We evaluate LCBC, GLCBC, SuSIE and LBP aforementioned 4 tasks. The metrie " "Avg. Score" measures the average score for each stage. We observe that while LBP slightly outperforms other strong baselines at the early stages, LBP wins by a fairly large margin the final stages all tasks. This shows LBP significantly excels handling long-horizon tasks.
LBP outperforms all baselines, achieving higher success rates across the majority of tasks. Specifically, LBP at- tains average success rate of 85.0% in SigLIP (Zhai et al., 2023) latent space and 88.6% DecisionNCE (Li et al., 2024) latent space, demonstrating its flexibility in leveraging different latent representations. Compared to SuSIE and Seer, which rely on heavy generative models for high-level planning, LBP demonstrates that lightweight MLPs can achieve comparable even better performance in long-horizon tasks. This improvement stems from the backward planning paradigm adopted in LBP, which main- tains long-horizon horizon consistency by recursively generating subgoals that preserve alignment with the final objective, ultimately enhancing both overall performance and compu- tational efficiency on long-horizon multi-stage tasks.
formance throughout. The results also show that GLCBC sometimes initially outperforms LCBC by incorporating additional visual goal features but suffers sharp decline in later stages. This drop is likely due to misalignment between the given final goals and current states, which mis- guides the policy in long-horizon tasks, highlighting the importance of dynamically predicting the final latent goal in LBP. Additionally, observe that SuSIE often generates hallucinated and incorrect subgoal images that confuse the low-level policy. While this issue may be less pronounced in relatively deterministic simulation environments, it sig- nificantly impacts performance inherently complex disturbances and stochasticity. In con- trast, LBP enables easy prediction and efficient planning in latent space with its backward philosophy.
weal-world settings with
Real-world experiment results. In Figure 4, we present the quantitative comparison on the real-world AIRBOT tasks. LBP consistently achieves the best performance at each stage in long-horizon tasks. Notably, in the early stages, the performance gap between different methods is relatively small. However, as the task progresses, other methods strug- gle due to insufficient and inconsistent guidance, leading to failures in later stages, whereas LBP maintains strong per-
Comparison to the forward paradigm. To evaluate the effectiveness of the backward planning paradigm, we com- pare it with the conventional forward planning, with both using the same hyperparameter configurations to ensure fair comparison. Specifically, the forward planner predicts the subgoal 10 steps ahead at each iteration, autoregressively generating the entire subgoal sequence. We randomly sam- ple 3,000 data points representing the current state from

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
sacovars cups
stack cups
move cups
Recoware cupS
0.2
Tooword planting
kacioars furwert plansing
Rncovers
Trevers plan
Prrvert stansry
3.5
Mean Squared Enorr
Pouan Squarted Errer
2.4
2.00
Menn Squared
-3
Mean
2.24
01
Tock Progress
Tisk Progress
Tock Progress
Thili Progress
Figure 5. Mean Squared Errors (MSE) between predicted subgoals and corresponding ground truths under forward and backward paradigm.
Table Ablations on different hyperparameter choices of LBP on LIBERO-LONG.
Table 3. Ablations on key design components of LBP on LIBERO- LONG.
A
(Sub)Goals
Avg. Suc f
Variant
Avg. Suc 1
77.3
Effectiveness of the planner
w/o planner
77.3
Zg
83.3
ours
88.6
0.5
=g. WF
85.6
The strategy of goal-fusion
average pooling
79.0
0.5
Zg. W1, W2
88.6
ours
88.6
0.5
Zg> W W22; W3
83.0
0.75
Zg WF
84.6
correspond to predicting more subgoals. Additionally, we vary the planning coefficient X, which controls the temporal sparsity of the subgoal more densely packed subgoals, closer to the final goal. The main findings are: (1) Without grounding the task objective in the latent visual goal zn. the approach reduces to LCBC, achieving an average success rate of 77.3%. When 2g is provided as additional context, the variant shows a signifi- cant improvement of 6.0%, demonstrating the effectiveness of leveraging the visual goal. (2) Adding subgoals w as additional contexts leads to obvious performance im- provement since it provides downstream policy with more about the future, but it is unnecessary to predict a large number of subgoals to achieve optimal results. This reflects the efficiency of our approach- unlike many planning meth- ods that rely on generating numerous continuous waypoints, our method achieves high performance with fewer subgoals. This advantage likely arises from the backward planning philosophy of LBP, where subgoals are predicted recursively in reverse from the final goal, providing efficient yet relevant planning information closely aligned with task progression. (3) We test À with 0.5 and 0.75, observing that LBP is robust and relatively insensitive to this hyperparameter choice.
0.75
Eg. W1, W22
85.0
paaueence-arge values result in
0.75
Zg> , W22; W3
84.0
our real-robot datasets and compute the mean squared error (MSE) between the predicted subgoals and their correspond- ing ground truths. The results are visualized in Figure 5, with normalized task progress shown on the x-axis.
We observe that the compounding errors of forward plan- ning increase rapidly across all tasks. In particular, for the most challenging task, Shift Cups, the prediction error becomes unacceptably large when predicting distant sub- goals. This issue is further exacerbated approaches that attempt predict continuous future image frames, where compounding errors can be even more severe. In contrast, our backward planning method maintains consistently low error across the entire planning horizon. These results high- light the advantages of our approach, which enables both efficient and accurate subgoal prediction.
5.3. Ablation Studies
In this section, we conduct ablation studies to evaluate the impact of different design choices of LBP on long-horizon performance. All models adopt NecisionNCE latent space and are tested on LIBERO-LONG.
A blation on key architectural components. We ablate the impact of the LBP planner and goal-fusion strategy, with results presented in Table 3. Removing the planner and re- lying solely on the low-level policy reduces the model to LCBC, resulting in a 11.3% performance drop. underscor- ing the necessity of subgoals predicted by LBP. Besides, replacing our goal-fusion strategy with simple average pool-
Ablation on key hyperparameters. We perform an ab- lation study on two key hyperparameters: planning steps and the recursive planning coefficient À in Table 2. We test different numbers of planning steps, where more steps
8

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
ing causes 9.6% decline in performance, showing that naively compressing subgoals across different horizons dermines the low-level policy. This highlights the role of our goal-fusion strategy in adaptively leveraging subgoals at dif- ferent distances in a way that effectively enhances planning performance.
archical planning. In Advances in Neural Information Processing Systems, 2024. 1,2
Bharadhwaj. H.,, Dwibedi, D., Gupta, A., Tulsiani, S., Do- ersch, C., Xiao, T., Shah, D., Xia, F., Sadigh, D., and Kirmani, S. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16283 2024. 2
6. Conclusion and Future Direction
Black, K., Nakamoto, M., Atreya, P., Walke, H. R., Finn, C.,
We present LBP, novel and efficient robotic planning framework that features backward planning in the latent space to break the critical trilemma among planning effi- ciency, long-horizon temporal consistency, and prediction accuracy. By leveraging visual latent space for planning. LBP achieves computational efficiency while maintaining sufficient information to capture task progression. More- over, by adopting the recursive "coarse-to- toneeett-eee"" backward prediction paradigm, LBP fundamentally mitigates the com- pounding prediction errors inherent in traditional forward planning approaches, particularly addressing the challenges of off-task prediction in long-horizon scenarios. Exten- sive evaluations across diverse simulated and real-world environments, including complex Long-horizon and multi- stage robotic tasks, consistently demonstrate LBP's supe- rior performance and robustness. One promising research direction is the integration of advanced subgoal selection mechanisms, such as key-frame detection methods, to en- hance the identification of informative subgoals. Another is the incorporation of more sophisticated robotic encoders to construct better-structured latent spaces for more efficient and accurate planning.
Kumar, A., and Levine, S. Zero-shot robotic manipula- tion with pre-trained image-editing diffusion models. In International Conference Learning Representations, 2024. 2, 5, 6, 12
Chane-Sane, E., Schmid, C., and Laptev, I. Goal- conditioned reinforcement learning with imagined sub- goals. In International Conference Learning Repre- sentations, 2021. 2
Chi, C.,, Feng. S., Du, Y., Xu, 7.,, Cousineau, E.,, Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Roboties: Science and Systems, 2023. 5, 12
Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems, 2023. 1,
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, Rvs: What essential for offline RL via supervised learn ing? In International Conference on Learning Represen- tations, 2022. 2
Acknowledgement
Eysenbach, B., Salakhutdinov, R. R., and Levine, S. Search on the replay buffer: Bridging planning and reinforce- ment learning. In Advances in Neural Information Pro- cessing Systems, 2019. 1,3
This work is supported by funding from Wuxi Research In- stitute of Applied Technologies, Tsinghua University under Grant 20242001 120, Beijing Academy of Artificial Intelli- gence (BAAI).
Fang, K., Yin, P., Nair, A., and Levine, S. Planning to practice: Efficient online fine-tuning by composing goals in latent space. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022. 1,3
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There might be some potential soci- etal consequences of our work, none which we feel must be specifically highlighted here.
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C. M., Eysenbach, B., and Levine, S. Learning to reach goals via iterated supervised learning. In International Conference
References
on Machine Learning, 2021. 2
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. Is conditional generative modeling all you need for decision-making?? In International Conference on Learning Representations, 2023. 13
Hansen-Estree Estrucc, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, Idql: Implicit q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023. 12
Ajay, A., Han, S., Du, Y., Li, S., Gupta, A., Jaakkola, T., Tenenbaum, J., Kaelbling, L., Srivastava, A., and Agrawal, Compositional foundation models for hier-
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. 5, 12

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
Hu, Y., Guo, Y., Wang, P., Chen, X., Wang, Y.-.,, Zhang, J., Sreenath, K., Lu, C., and Chen, J. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412. 2024. 2
Nachum, O., Tang, H., Lu, X., Gu, S., Lee, H., and Levine, S. Why does hierarchy (sometimes) work well in re- inforcement learning? arXiv preprint arXiv:1909.1060 2019. 3
Nair, S. and Finn, C. Hierarchical foresight: Self-supervised learning of tion. In International Conference on Learning Represen- tations, 2020. 1, 2,
Huang, Z., Lin, Y., Yang, F., and Berenson, D. Subgoal dif- fuser: Coarse-to-fine subgoal generation to guide model predictive control for robot manipulation. In IEEE Inter- national Conference Robotics and Automation, 2024. 1,3
tasks via visual subgoal genera-
Nair, S., Savarese, S., and Finn, C. Goal-aware predic- tion: Learning to model what matters. In International Conference on Machine Learning, 2020. 2
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, Perceiver: General perception with it- erative attention. In International Conference on Machine Learning, 2021. 5
Nasiriany, S., Pong, V., Lin, S., and Levine, S. Planning with goal-conditioned policies. In Advances in Neural Information Processing Systems, 2019. 1,3
Janner, M., Du, Y., Tenenbaum, J., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022. 13
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offine goal-conditioned rl with latent states actions. In Advances in Neural Information Processing Systems, 2024. 2,4
Kang, X. and Kuo, Y.-L. Incorporating task progress knowledge for subgoal generation in robotic manipulation through image edits. In JEEE/CVF Winter Conference Applications of Computer Vision, 2025. 1,3
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 2018a.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, F. P., Sanketi, P. R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. Open- VLA: An open-source vision- lasion-language-attin Conference on Robot Learning, 2024. 5, 6,6,1 12
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 2018b. 12
model. In
Pertsch, K., Rybkin, O., Ebert, F., Zhou, S., Jayaraman, D., Finn, C., and Levine, S. Long-horizon visual plan- ning with coal-conditioned hierarchical predictors. In Ad- vances in Neural Information Processing Systems, 2020.
Levy, A., Platt, R., and Saenko, K. Hierarchical reinforce- ment learning with hindsight. International Conference on Learning Representations, 2019. 3,
2
Li, J., Zheng. J., Zheng, Y., Mao, L.. Hu, X.. Cheng, S., Niu, H., Liu, J., Liu, Y., Liu, J., et al. DecisionNCE: Embodied multimodal representations via implicit preference leam- ing. In International Conference on Machine Learning, 2024. 2, 4, 7, 12
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Learning Representations, 2021. 4, 12
Li, J., Wang, Z., Zheng, J., Zhou, X., Wang, G., Song, G., Liu, Y., Liu, J., Zhang, Y.-Q., Yu, J., and Zhan, X. Robo-mutual: Robotic multimodal task specification via unimodal learning. In IEEE International Conference on Robotics and Automation, 2025. 4
Shah, R., Martfn-Martin, R., and Zhu, Y. Mutex: Learning unified policies from multimodal task specifications. In Conference on Robot Learning, 2023. 4
Tian, K., Jiang, Y., Yuan, Z., PENG, B., and Wang. L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Advances in Neural Information Processing Systems, 2024. 3
Liu, B., Zhu Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. Libero: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Informa- tion Processing Systems, 2024. 5
Tian, Y., Yang, S., Zeng, J., Wang, P., Lin, D., Dong, H., and Pang, J. Predictive inverse dynamics models are scal- able learners for robotic manipulation. In International Conference on Learning Representations, 2025. 2, 5, 6, 12
Lynch, C. and Sermanet, Language conditioned imitation learning over unstructured data. In Robotics: Science and Systems, 2021.
10

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
Veerapaneni, R., Co-Reyes, J. D., Chang, M., Janner, M., Finn, C., Wu, J., Tenenbaum, J., and Levine, S. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning. 2020. 3
for enhanced embodied foundation models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025. 5
Wang, C., Fan, L., Sun, J., Zhang, R., Fei-Fei, L., Xu, D., Zhu, Y., and Anandkumar, Mimicplay: Long- horizon imitation learning by watching human play. In Conference Robot Learning, 2023. 2
Wang, G., Niu, H., Li, J., Jiang, L., HU, J., and Zhan, X. Are expressive models truly necessary for offline RL? AAAI Conference on Artificial Intelligence, 2025. 2,
Wei, J., Wang, X.. Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
Wen, C., Lin, X., So, J., Chen, K., Dou, Q. Gao, Y., and Abbeel, P. Any-point trajectory modeling for policy learn- ing. In Robotics: Science and Systems, 2024. 2
Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, Li, X., Liu, M.. Li, H., and Kong, T. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024. 2
Xiao, T., Radosavovic, Darrell, T., and Malik, J. Masked visual pre-traie tre-training for motor control. arXiv preprint 2003v:2203.06173, 2022. 6
Xu, H., Li, Li, J., and Zhan, X. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022. 2
Zeng. J., Bu, Q., Wang, B., Xia, W., Chen, L., Dong, H., Song, H., Wang, D., Hu, D., Luo, P., et al. Learning ma- nipulation by predicting interaction. In Robotics: Science and Systems, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 7, 12
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13700 2023, 6, 12
Zheng, J., Li, J., Cheng, S., Zheng, Y., Li, J., Liu, J., Liu, Y., Liu, J., and Zhan, X. Instruction-guided visual masking. In Advances in Neural Information Processing Systems, 2024. 4
Zheng, J.. Li. Liu, D., Zheng, Y. Wang, Z., Ou, Z., Liu, Y., Liu, J., Zhang, Y.-Q., and Zhan, X. Universal actions

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
A. Implementation Details
LBP (Ours). For high-level planner, we implement the goal predictor fg and the subgoal predictor f using two-layer MLPs and employ two cross-attention blocks to realize the goal goal-fusion attention model. We employ DecisionNCE (Li et al., 2024) and SigLIP (Zhai et al., 2023) as frozen encoders to project language instructions and images to latent space.
For the low-level policy, we use a shared ResNet-34 (He et al., 2016) as the backbone to extract visual features from all camera view images, where the language embeddings are injected via FiLM conditioning layers (Perez et al., 2018b). The visual features, goal-fused feature, and current proprioception are then concatenated and fed into a residual MLP (Hansen- Estruch al., 2023) to generate actions. The policy is optimized with diffusion loss to model complex distributions (Chi al., 2023), with the denoising step fixed at 25. We employ action chunking with fixed length of 6 and utilize an exponential weighting scheme to ensemble overlapping action sequences, following (Zhao et al., 2023).
For training the high-level planner, we use a batch size of 64 and train for 100k steps with the AdamW optimizer. For the low-level policy LIBERO-LONG, we set the batch size to 64 and train for 200k steps. In the case of the low-level policy for real-world robot experiments, we increase the batch size to 128 and train for 400k steps.
SuSIE (Black et al., 2024). The high-level image-editing diffusion model is trained on video data using four A6000 GPUs. We utilize the official codebase with minimal modifications, altering only the datasets. For simulation experiments, we fine-tune the released SuSIE checkpoint on the LIBERO dataset. In the real-robot experiments, we fine-tune the checkpoint exclusively with collected data on Airbot. Regarding the low-level policy, we adopt the same model architecture with the policy model of LBP, while adopting a channel-wiss concatenation of subgoal images and current images as inputs. Additionally, language instructions removed to maintain consistency with the downstream training in SuSIE.
LCBC. We implement it by directly removing our high-level planner from the architecture of LBP. The language is projected to the latent space by CLIP (Radford et al., 2021), and images are projected by a ResNet-34 (He et al., 2016), then the semantic features are captured by a FiLM (Perez et al., 2018b) module. The low-level policy takes in these semantics, together with current proprioception, and then outputs a predicted diffusion noise.
The only difference between GLCBC and LCBC is the part before entering the FiLM module: we choose a predefined image as the final goal and then project it to a latent space with DecisionNCE image encoder (Li et al., 2024). We concatenate the language embeddings with the final goal image embeddings, then incorporate them as inputs into the FiLM module.
GLCBC.
Others. For LIBERO-LONE benchmark, since our experimental settings and evaluation metrics are the same with Seer, we obtain the scores of MTACT, MVP, MPI, Ope and Seer from the original paper (Tian et al., 2025).
B. LIBERO-LONG Benchmark Details
We follow (Kim al., 2024) to re-render the images at resolution of 256x256. The detailed language instructions and average demonstration lengths for each task of LIBERO-LOON is shown in Table 4.
C. Real-RRoot Experiment Details
We collect 200 expert demonstrations each for tasks Move cups, Stack 3 cups, Stack 4 cups and Shift cups. To enhance the robustness of model trained on this dataset, we manually add some augmentation techniques (Table 5), including Distractor augmentation, Target augmentation. Background augmentation and View augmentation. Examples from the side view can be seen at Table 6. View augmentation always exsits because the side view camera is not a fixed-position view. Distractor augmentation means placing various unrelated distractor objects on the table. Target augmentation refers to replacing the target objects with different ones of this type, i.e., replacing paper cups with cups of different materials. Background augmentation is placing tablecloths with various colors above the clean white table.
D. A Aditionaa Results
Numerical results. Except for the visualized results of real-robot experiments in Figure 4, we also present the numerical results of each stage for each task in Table 7-10.
12

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
Table 4 Language instructions and average lengths of I LIBERO-LONG..
Task
Task
Language instruction
Average demonstration length (frames)
ID
put soup and
put both the alphabet soup and the tomato sauce the basket put both the cream cheese box and the butter in the basket
1
294
sauce in basket
put box and
2
260
butter in basket
turn on stove
turn on the stove and put the moka pot
3
266
and put pot
put bowl
put hlack howl in the bottom
4
249
drawer close
drawer the cahinet and close
put the white mug on the
5
put mugs
left plate and put the yellow
258
left and right plates
and white mug on the right plate
pick book and
pick up the book and place in
6
189
place in back
the back compartment of the caddy
put the white mug
mug on plate and
7
the plate and put chocolate
255
put pudding to right
pudding to the right the plate
put soup and box in basket
put both the alphabel soup and
8
270
the cream choese box in the basket
put both pots stove
put buth moka
9
416
pots the stove
10
put mug mierowave and close
put the yellow and white
305
mug the microwave and close
Table Dataset settings of real robot experiments.
Task
Language
w/ distractor augmentation?
w/ view augmentation?
w/ target augmentution?
w/ background augmentation?
name
instruction
tirst put the right brown cup in front of the right white cup then put the left brown cup front of the left white cup
Move cups
Stack cups
stack the paper cups
move cach to
Shift cups
position clockwise direction
Generalization experiment. We LBP on the longest real-world task (Shift Cups) with different backgrounds and distractor objects, where we find that LBP maintains robust performance in these complex scenarios, still outperforming the strongest bascline LCBC in base setting. The numerical results present in Table 11.
Comparison to the parallel planning paradigm. To evaluate the effectiveness of the backward planning paradigm, we further compare with the parallel planning paradigm (Janner al., 2022; Ajay al., 2023), with both using the same hyperparameter configurations with LBP ensure fair comparison. Specifically, the parallel planner predicts all subgoals simultaneously in single batch at each execution step. We randomly sample 3,000 data points representing the current state from our real-robot datasets and compute the mean squared error (MSE) between the predicted subgoals and their corresponding ground truths. The results are visualized in Figure 6, with normalized task progress shown on the x-axis.
We observe that although parallel planning avoids eттоr accumulation by predicting all subgoals simultaneously, it suffers from consistently inaccurate predictions across the entire planning horizon. This limitation can be attributed to the difficulty of the training objective, which requires simultaneous supervision of subgoals. Such an approach demands greater model capacity and incurs significantly higher computational costs. In contrast, our backward planning method maintains consistently low prediction error across the entire planning horizon. These results highlight the advantages of our approach, which enables both efficient and accurate subgoal prediction.
13

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
Table 6. Demonstrations of augmentation techniques.
Augmentation metrics
Examples
Distractor augmentation
Target augmentation
Background augmentation
Table 7. Numerical results of task: Stack cups.
Method
Stage
Stage II
Avg. Score
LCBC
94.1
63.3
78.7
GLCBC
95.0
74.1
84.6
SuSIE
83.3
37.5
LBP
94.1
75.0
84.6
Table 8. Numerical results task: Move cups.
Method
Stage
Stage
Avg. Score
LCBC
84.1
36.6
60.4
GLCBC
85.8
40.0
62.9
SuSIE
71.6
20.8
LBP
90.0
65.8
77.9
Table 9. Numerical results task: Stack 4 cups.
Methad
Stage
Stage IT
Stage III
Avg. Score
LCBC
90.8
62.5
11.6
55.0
GLCBC
82.5
48.3
5.8
45.5
SuSIE
75.0
15.0
LBP
96.6
77.5
43.3
72.5
14

---

Efficient Robotic Policy Learning via Latent Space Backward Planning
stack
stack cups
shift
tocoann pueced
sccowro zaserng Foreort piasting
ssaward paaves
Bestwen carero Faralial
favard arrho
Femard parvers
26
14-
2.6
0.4
0.3
Moan
Moan
02
04
30
0.2
0.
Tast Progress
Tesk Progress
Tisk Progress
Tadk Progress
Figure 6. Mean Squared Errors (MSE) between predicted subgoals and corresponding ground truths in parallel, forward and backward planning.
Table 10. Numerical results of task: Shift cups.
Method
Stage
Stage
Stage III
Stage IV
Stage V
Avg. Score f
LCBCC
85.0
55.0
48.3
20.8
0.0
41.8
GLCBC
89.1
75.8
15.8
0.0
0.0
36.1
SuSIE
78.3
10.0
0.0
0.0
0.0
17.7
LBP
97.5
87.5
74.1
50.0
26.6
67.1
Table 11. Numerical results of generalization experiment on Shift cups.
Method
Stage
Stage II
Stage III
Stage IV
Stage V
Avg. Score f
LCBC (Base setting)
85.0
55.0
48.3
20.8
0.0
41.8
LBP (Distracting objects)
87.5
75.8
48.3
35.0
9.0
51.1
LBP (Different backgrounds)
91.6
55.8
37.5
13.3
56.4
LBP (Base setting)
97.5
87.5
74.1
50.0
26.6
67.1
15


Reviews:
[{"review_id": "pL1hkI1xOT", "reviewer": "Reviewer_5oF8", "summary": "To enable real-time planning for long-horizon and multi-stage tasks, the paper proposes LBP, a backward planning scheme in the latent space. By eliminating the need for pixel-level generation, the proposed scheme significantly improves inference speed while alleviating compound errors. Additionally, it enhances on-task planning through guidance from the final goal. The evaluation is conducted on LIBERO-LONG and real-world setups.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts and valuable feedback!\n# Claims and Evidence\n> **Missing ablations to forward planning**\n\nWe add an ablation study comparing LBP to latent forward planning. The results demonstrate that LBP significantly outperforms the forward planning paradigm in both **subgoal predicting accuracy** and **final policy performance**. `The updated results of prediction errors are visualized on our project page, linked at the end of our abstract`.\n- **LBP obtains substantially lower prediction error**.\n  - **Train forward planner**: We learn a forward planner in latent space for our real-robot tasks, which predicts the subgoal 10 steps ahead, similar to SuSIE [1]. At each step, the forward planner autoregressively generates latent subgoals towards the final goal.\n  - **Evaluate planning accuracy**: We randomly sample 3000 data points as the current state from our real-robot datasets and compute the mean square errors (MSE) between predicted subgoals and their corresponding groundtruths. \n  - **Visualization of prediction error results**: Please refer to Figure 5 on our website, which illustrates that *forward planning struggles with long-horizon subgoal prediction due to rapid error accumulation*. Given that long-horizon tasks often span hundreds of frames, this error compounding makes forward planning impractical. In contrast, *LBP consistently produces accurate subgoals with significantly lower error magnitude, maintaining reliability throughout planning horizon.*\n- **LBP obtains substantially stronger long-horizon performance**.\nThe tables below show that LBP significantly outperforms latent forward planning in all the long-horizon real-robot and simulation tasks, benefiting from the recursive backward planning scheme and its subgoal generation accuracy. Note that all settings remain the same to ensure a fair comparison.\n\n||stack 3 cups|||stack 4 cups|||\n|-|-|-|-|-|-|-|\n||stage 1|stage 2||stage 1|stage 2|stage 3|\n|latent forward planning|78.3|6.7||71.6|21.6|5.0|\n|LBP (ours)|**94.1**|**75.0**||**96.6**|**77.5**|**43.3**|\n\n||move cups|||shift cups|||||\n|-|-|-|-|-|-|-|-|-|\n||stage 1|stage 2||stage 1|stage 2|stage 3|stage 4|stage 5|\n|latent forward planning|43.3|5.0||95.0|65.0|11.6|0.0|0.0|\n|LBP (ours)|**90.0**|**65.8**||**97.5**|**87.5**|**74.1**|**50.0**|**26.6**|\n\n||libero-long|\n|-|-|\n|LCBC|73.0|\n|latent forward planning|73.6|\n|LBP (ours)|**82.3**|\n\n**Lastly, we are unsure which specific approach the reviewer refers to as \"parallel planning\". We would greatly appreciate any further clarification and descriptions on this. We would be happy to explore the comparison if time allows.**\n# Weaknesses\n> \"Predicting final goals directly may lead to large errors.\"\n- As shown in the above prediction error results, while predicting final goals may introduce some errors, they are negligible compared to the accumulated errors in forward (progressive) planning. \n- Grounding the task objective in final goals also stabilizes subgoal predictions along the horizon, keeping subgoal prediction errors low and demonstrating the effectiveness of error control in our recursive backward planning scheme.\n- Predicting the final goal not only plays a key role in LBP but also is not as difficult as it seems, as it is relatively deterministic given the current state and task description.\n\n> \"It would be beneficial to report the inference frequency.\"\n\nWe present the inference time of LBP and a competitive generative planner, SuSIE. Other baselines either adopt large VLA models or are not planning methods, which are not meaningful to compare on inference latency. **The results show that LBP is significantly more efficient than SuSIE.** Each model is tested on a single GPU.\n\n||Inference time|\n|-|-|\n|SuSIE|28.13s|\n|LBP|0.013s|\n# Questions\n>A little confusion on line 220-221.\n\nWe apologize for confusing the readers. We meant to say:\n- This recursive mechanism suffers from considerably fewer compounding errors, as the λ-recursion effectively reduces the number of planning steps, and *the training of $f_w$ incorporates supervision (of groundtruths) in every recursion level*.\n\n>Differences to DiffuserLite.\n- *LBP is based on real-world task settings*: Unlike LBP, DiffuserLite cannot handle language-conditioned tasks and struggles in high-dimensional spaces due to high computational cost of diffusion.\n- *LBP enjoys simplicity in design*: DiffuserLite trains separate diffusion models at each level, while LBP trains a single MLP to recursively predict subgoals.\n- *LBP enjoys computational efficiency*: DiffuserLite only uses the first prediction in trajectories for next-level trajectory generation, which results in many redundant computations. In contrast, LBP predicts one subgoal at each level with MLP, which is more efficient and proves effective for policy guidance.", "claims_and_evidence": "The main idea of the paper is backward planning in the latent space. However, it fails to provide two key ablations: forward planning and parallel planning. The absence of these comparisons makes it difficult to conclude the superiority of backward planning. In fact, it is possible that the planning order has minor effects and the performance gain is attributed to the informative subgoals in the SigLIP and DecisionNCE latent spaces.", "methods_and_evaluation_criteria": "The proposed methods are aligned with the motivation and evaluated using appropriate criteria.", "theoretical_claims": "I have checked the theoretical claims in this paper.", "experimental_designs_or_analyses": "The paper provides extensive experiments with sufficient details.", "relation_to_broader_scientific_literature": "Recent goal-conditioned robot planning typically uses generative models for goal prediction. Unlike these approaches, the paper suggests that predicting latent goals enhances computational efficiency and achieves better performance. Although the proposed backward planning scheme sounds technically novel, I have concerns about its superiority due to a lack of ablation studies.", "essential_references_not_discussed": "As far as I know, all closely related works are cited appropriately.", "other_strengths_and_weaknesses": "W1) **Missing key ablations.** The proposed backward planning is not compared to forward planning and parallel planning using the same latent subgoals, which compromises the validity of the main contribution. In some cases, leaving causal uncertainty while determining closer steps first could be beneficial for decision making [1].\n\nW2) **Predicting final goals directly may lead to large errors.** There is no evidence suggesting that distant final goal prediction is easier than progressive prediction.\n\nW3) **Inference speed is not reported.** Since the authors highlight the efficiency over previous generative planners, it would be beneficial to report the inference frequency.\n\n[1] Diffusion Forcing: Next-Token Prediction Meets Full-Sequence Diffusion. Boyuan Chen, et al.", "other_comments_or_suggestions": "Please see the questions below.", "questions_for_authors": "Q1) **A little confusion.** I am confused by the sentence on Line 220-221. It states that the proposed mechanism suffers from less compounding error because it is completely supervised with ground truth data. My question is: if ground truth is not used, then what kinds of supervision could be?\n\nQ2) **Differences to DiffuserLite.** The proposed method reminds me of DiffuserLite [2], which also introduces an efficient coarse-to-fine planning process that transits from long horizon to short horizon. Could you elaborate on the differences?\n\n[2] DiffuserLite: Towards Real-Time Diffusion Planning. Zibin Dong, et al.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j1kEBKbE7r", "reviewer": "Reviewer_sA8h", "summary": "This paper focuses on latent space planning to accomplish robotic tasks. It breaks down a long horizon language conditioned manipulation task into predicting the final goal. Then using the final-goal to predict sub-goals moving from the goal state to the initial state. Once these have been learned a sub-goal/final-goal conditioned context policy is learned. During inference time, the approach generates the final subgoal and then other sub-goals, which are used to predict the action which is then rolled out. Experiments are performed on Libero-Long benchmark  and shows some improvments over baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Experimental Designs Or Analyses**\n>Other long horizon tasks for comparison (e.g. AntMaze etc).\n\nThanks for the positive comments to our experimental designs. While other long-horizon tasks like variations of AntMaze exist, we excluded them from our primary benchmark suite for specific reasons aligned with the focus of our work. \n- Firstly, our method is explicitly designed and evaluated for language-guided robotic control, as detailed in the introduction of our paper. AntMaze, lacking language instructions, does not allow for the validation of language-driven task execution, which is a key application of our approach.\n- Furthermore, its low-dimensional state space and explicit coordinate goals make it significantly simpler than the complex image-based tasks that are the current focus of modern robotic research, as evidenced by [1, 2, 3, 4].\n\n[1] Black, et al. Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models. ICLR 2024.\n\n[2] Tian, et al. Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation. ICLR 2025.\n\n[3] Nair, et al. R3M: A universal visual representation for robot manipulation. CoRL 2022.\n\n[4] Chi, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.\n\n## **Essential References Not Discussed**\n>Classical works on backward chaining with skill trees should also be cited.\n\nThanks for the suggestion! We will discuss these relevant references in the final version.\n\n## **Other Strengths And Weaknesses**\n> Lacking baselines \n\nWe wish to emphasize that LBP's primary contribution lies in enhancing language-guided robotic control, a significant challenge in the field. Our experimental evaluation includes comparisons against the most relevant SOTA robotic methods, such as SUSIE, Seer, and OpenVLA.\n\n1. Other hierarchical approaches for control tasks\n\nThe hierarchical RL methods mentioned by the reviewer are not appropriate baselines for LBP due to their fundamentally different application scope. They are tailored for simpler, often customized environments (such as the AntMaze benchmark) and, crucially, do not support language-conditioned tasks, making them incomparable to LBP's core functionality.\n\nAlthough they are not suitable as baselines, **we will include a discussion of these methods in the related work of the final version**.\n\n2. Another interesting baseline is to use denser language labels generated by large multi-modal models. \n\n- As far as we know, **benchmarks with dense language labels are rare in the robotics community**, as collecting reliable and sufficient language annotations is both costly and impractical.\n- Moreover, such methods **typically require significantly larger models** to process diverse and dense language inputs while also handling out-of-domain scenarios at test time.\n\nIn contrast, our LBP method offers a more robust, efficient, and lightweight approach for subgoal specification, which can take advantage of rich observation data without dense language labels, as is the common case.\n\n> **Fixed recursive time for sub-goal generation (Q1) & Inference time action selection (Q2).**\n\nWe would like to address a potential misunderstanding concerning how LBP is used for action selection. \n- **LBP is designed to predict (update) future subgoals at every step of the task execution** as we describe in lines 272-274. This dynamic planning scheme ensures that all the parts in task horizon would be covered during planning, thus addressing the \"challenge part\" (Q1). \n- At test time, we rollout action based on the fusion of all the subgoals generated at this time, as we describe in Section 4.3 (Q2).", "claims_and_evidence": "Yes the claims seem reasonable.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "none", "experimental_designs_or_analyses": "yes, the libero long experiments is a decent choice since the dataset is focused on long horizon tasks. However, there is nothing in the method which is unique to manipulation, hence other common long horizon tasks could also  have been used for comparison (e.g. different variants of AntMaze etc). The real-world experimental design seems good. Finally, the ablation analysis makes sense and the main components of the approach have been validated.", "relation_to_broader_scientific_literature": "Many prior approaches have focused on sub-goal generation and then using goal-conditioned supervised learning, which is what the proposed approach is doing. The only big difference is how sub-goal generation happens. The paper claims following a final-goal to initial state approach should be better. This has also been applied", "essential_references_not_discussed": "see below. Also, classical works such as doing backwards chaining using skill trees should also be cited [1]. There is a large body of work around this which predates a lot of modern deep learning based approaches. None of these approaches are cited or discussed.\n\n[1] Konidaris et al. Robot Learning from Demonstration by Constructing Skill Trees", "other_strengths_and_weaknesses": "**Pros:**\n\nThe paper focuses on an important problem. Developing robust planning approaches would be super useful for robot tasks. Overall, the paper is also well written (although some details are missing see below).\n\n**Cons:**\n\n**Few baselines:** There has just been a tremendous amount of work on heirarchical approaches for control tasks. Many papers have tried techniques similar to the ones proposed in the paper. But these approaches have not been compared against, some of them have not been cited at all [1, 2, 3]. I think some of these alternative approaches which do sub-goal generation differently should be compared against and properly discussed.\n\nAnother interesting baseline would ideally be using denser language labels for the entire task. Here the need for sub-goals is motivated by the language labels not being dense enough for the task and not providing enough semantic value (Line 180 — language descriptions often reduce to task identifiers …). However, given improved understanding of large multi-modal models it may be possible to zero-shot get denser labelings for a task from a long horizon video. In case the policy performance with dense language relabeling is worse then it is possible to conclude that latent image goal/sub-goal embeddings are indeed crucial but it is unclear if this the case with the current set of experiments.\n\n**Fixed recursive time for sub-goal generation:** This seems like a very big assumption. For many tasks the challenging part of the task might be much shorter than the other parts, in this case a fixed approach might just miss the right sub-goal for the task. This will always be a problem in the proposed approach since it apriori relies on no information to do sub-goal generation. This makes the proposed approach quite unscalable to more challenging and interesting tasks. \n\n**Inference time action selection:** How does action selection happen at inference time. Once all the sub-goals are selected does the policy use the first sub-goal to generate action and rollout. When does the policy switch to next sub-goal?\n\n[1] Hierarchical reinforcement learning with timed subgoals\n\n[2] Zhang et al. Generating adjacency-constrained subgoals in hierarchical reinforcement learning\n\n[3] Lei et al. Goal-conditioned Reinforcement Learning with Subgoals Generated from Relabeling", "other_comments_or_suggestions": "please see above", "questions_for_authors": "please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13ICmCI56r", "reviewer": "Reviewer_Zaj7", "summary": "In this work, the author proposed a robotic manipulation method called LBP. This method first grounds the task into final latent goals and then recursively predicts the intermediate subgoals closer to the current state. Compared to previous fine-grained approaches, LBP is more lightweight and less prone to accumulating inaccuracies. For implementation, the goal predictor and subgoal predictor of LBP only use two-layer MLPs and use a cross-attention block to realize the goal-fusion model. The effectiveness of LBP is proven by the experiments on both the LIBERO-LONG benchmark and four real-world long-horizon tasks.\n\n\n## Update after rebuttal\nThe generalization capabilities of LBP are demonstrated by the additional results of shifting cups. And, my misunderstandings about the baseline selection have been well addressed. However, the real-world task settings are still very simple, I'm prone to maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's positive feedback and recognition of our work! Below are our responses to the concerns raised.\n\n## Experimental Designs Or Analyses\n>The presentation of real-world results is great...It could be better to involve more contact-rich tasks.\n\nThanks for your suggestion! We are willing to try different classes of robotic tasks in future version, including contact-rich tasks.\n\n> The baselines selected in real-world experiments are not strong enough. I would like to recommend adding R3M, VC-1, DP, or other policies into your comparisons.\n+ In our experiments, **the LCBC baseline is actually implemented using Diffusion Policies (DP) [1] with language instructions**. More details can be found in Appendix A of our paper.\n+ R3M [2] and VC-1 [3] primarily focus on representation learning, **while LBP is a planning framework that allows flexible representation choices**. For planning in latent space, we adopt DecisionNCE [4] and SigLIP [5], two recent strong methods in robotic representation learning. As shown in [4], DecisionNCE outperforms R3M, making it a sufficiently strong choice for our experiment. SigLIP has been widely adopted in many robotic frameworks like OpenVLA [6].\n\n## Other Strengths And Weaknesses\n> No generalization experiment results are provided to prove LBP’s robustness.\n\n+ We test LBP on the longest real-world task `shift cups` with different backgrounds and distracting objects and find that **LBP maintains robust performance in these complex scenarios**, still outperforming the strongest baseline LCBC in base setting. The corresponding videos have also been updated on our website `(Click the link at the end of our abstract)`.\n+ In our planning framework, the generalization capability also depends on the selected latent space. If adopting stronger latent spaces, the generalization capability of LBP can be further improved.\n\n||shift cups|||||\n|-|-|-|-|-|-|\n||stage 1|stage 2|stage 3|stage 4|stage 5|\n|LCBC (Base setting)|85.0|55.0|48.3|20.8|0.0|\n|LBP (Distracting objects)|87.5|75.8|48.3|35.0|9.0|\n|LBP (Different backgrounds)|91.6|84.1|55.8|37.5|13.3|\n|LBP (Base setting)|97.5|87.5|74.1|50|26.6|\n\n[1] Chi, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.\n\n[2] Nair, et al. R3M: A universal visual representation for robot manipulation. CoRL 2022.\n\n[4] Majumdar, et al. Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? NeurIPS 2023.\n\n[4] Li, et al. DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning. ICML 2024.\n\n[5] Zhai, et al. Sigmoid Loss for Language Image Pre-Training. ICCV 2023.\n\n[6] Kim, et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods make sense, and the evaluations contain both simulation and real-world experiments, which makes the evaluation results comprehensive.", "theoretical_claims": "I have checked the soundness of LBP’s theoretical claims, especially for the derivative of Eq (3,4,5) in the Sec 4.2 “Predicting Subgoals with a Backward Scheme” part.", "experimental_designs_or_analyses": "- In the LIBERO-LONG benchmark, two versions of LBP have achieved almost state-of-the-art performance. However, on certain tasks such as task 6 and 7, LBP still has a large gap behind the best method. Overall, LBP’s average performance is the strongest.\n- The presentation of real-world results is great. Figure 4 delivers a direct impression of each method’s performance at each stage. However, the long-horizon tasks involved in the real-world experiments are simple pick-and-place or stacking tasks. It could be better if the real-world experiments involve more contact-rich tasks, e.g. articulated object manipulation. \n- Besides, the baselines selected in real-world experiments are not strong enough. I would like to recommend adding R3M[1], VC-1[2], DP[3], or other policies into your comparisons.\n\n[1] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In CoRL, 2022.\n\n[2] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, Franziska Meier. Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? In arXiv, 2023.\n\n[3] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In RSS, 2023.", "relation_to_broader_scientific_literature": "Previous methods usually predict consecutive frames to model future outcomes, which could bring the propagation of inaccuracies. LBP is a new scheme that aims to achieve the balance between efficiency and long-horizon processing capabilities.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No generalization experiment results are provided to prove LBP’s robustness.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2LTXBb0VN", "reviewer": "Reviewer_dUcZ", "summary": "The authors introduce LBP (Latent space Backward Planning), a novel approach for robotic planning. LBP works by grounding tasks into final latent goals and recursively predicting intermediate subgoals backward toward the current state. The authors evaluate LBP on simulation benchmarks and real-robot environments, demonstrating its performance over existing methods for long-horizon, multi-stage tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and recognition of our work! If you have any further concerns or questions related to LBP, we would be happy to discuss them.", "claims_and_evidence": "The experimental validation supports the claims by the authors. LBP achieves 82.3% success rate on LIBERO-LONG, outperforming baselines like MPI (77.3%) and Seer (78.6%). Ablation studies validate the contribution of each component, showing significant performance drops when removing key elements.", "methods_and_evaluation_criteria": "The methods are appropriate for the problem.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental designs are comprehensive, evaluating on 10 LIBERO-LONG tasks and 4 real-world tasks against multiple baselines. Ablation studies effectively isolate component contributions.", "relation_to_broader_scientific_literature": "Ok", "essential_references_not_discussed": "References are complete. Perhaps there may be some marginal connections with Hindsight Experience Replay.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for generating latent-space subgoals for long-horizon language-conditioned robotic manipulation tasks by planning backwards from the goal. The evaluation on the LONG subset of the LIBERO benchmark shows the promise of this approach.

Reviewers see merits in the proposed approach, including its novelty and experimental design. The biggest outstanding concerns are the coverage of related work and the scarce experiments on physical robot hardware. Alternatively, as one of the reviewers mentioned, since LBP doesn't seem to rely on any robotics-specific task features, it could be evaluated more extensively by adding non-robotics long-horizon benchmarks. That said, there isn't a pre-existing method that clearly undermines LBP's novelty, so a more extensive coverage of related work, as the authors promised in the rebuttals, should address the former concern. Also, the evaluation on the LIBERO-LONG benchmark is convincing enough, even though a broader evaluation would be more desirable. 

Given the above considerations, on balance the metareviewer recommends this work for publication.
Decision: Accept (poster)