Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion
Authors: Dan Haramati
Carl Qi
Tal Daniel
Amy Zhang
Aviv Tamar
George Konidaris
Venue: ICLR 2026 Poster (2026)
Keywords: Deep Reinforcement Learning
Goal-conditioned Reinforcement Learning
Object-centric Representations
Diffusion Subgoal Generation
Primary area: reinforcement learning
TL;DR: We present a hierarchical entity-centric framework for offline goal-conditioned RL that produces entity-factored diffusion-generated subgoals for an RL agent, yielding consistent performance gains on long-horizon, image-based, sparse-reward tasks.

Abstract: We propose a hierarchical entity-centric framework for offline Goal-Conditioned Reinforcement Learning (GCRL) that combines subgoal decomposition with factored structure to solve long-horizon tasks in domains with multiple entities.
Achieving long-horizon goals in complex environments remains a core challenge in Reinforcement Learning (RL). Domains with multiple entities are particularly difficult due to their combinatorial complexity. GCRL facilitates generalization across goals and the use of subgoal structure, but struggles with high-dimensional observations and combinatorial state-spaces, especially under sparse reward. We employ a two-level hierarchy composed of a value-based GCRL agent and a factored subgoal-generating conditional diffusion model. The RL agent and subgoal generator are trained independently and composed post hoc through selective subgoal generation based on the value function, making the approach modular and compatible with existing GCRL algorithms. We introduce new variations to benchmark tasks that highlight the challenges of multi-entity domains, and show that our method consistently boosts performance of the underlying RL agent on image-based long-horizon tasks with sparse rewards, achieving over $150$% higher success rates on the hardest task in our suite and generalizing to increasing horizons and numbers of entities. Rollout videos are provided at: https://sites.google.com/view/hecrl.

Published as a conference paper at ICLR 2026
HIERARCHICAL ENTITY-CENTRIC REINFORCEMENT LEARNING WITH FACTORED SUBGOAL DIFFUSION
Dan Haramati", Carl Qi Tal Daniel Amy Zhang", Aviv Tamar"1, George Konidaris Brown University, UT Austin, *Carnegie Mellon University, "Technion, Equal advising cen.haramati@broww edu; carlqeutexas edu
ABSTRACT
We propose a hierarchical entity-centric framework for offline Goal-Conditioned Reinforcement Learning (GCRL) that combines subgoal decomposition with fac- tored structure to solve long-horize bag-hhrizonn tasks in domains with multiple entities. Achieving Long-horizon goals in complex environments remains core challenge in Reinforcement Learning (RI.). Domains with multiple entities are particularly difficult due to their combinatorial complexity. GCRL facilitates generalization across goals and the use of subgoal structure, but struggles with high-dimensional observations and combinatorial state-spaces, especially under sparse reward. We employ a two-level hierarchy composed of a value-based GCRL agent and a fac- tored generrainn conditional diffusion model. The RL agent and sub- goal generator are trained independently and composed post hoc through selec- tive subgoal generation based on the value function, making the approach mod- ular and compatible with existing GCRL algorithms. We introduce new varia- tions to benchmark tasks that highlight the challenges of multi-entity domains, and show that our method consistently boosts performance of the underlying RL agent on image-based Long-horizon tasks with sparse rewards, achieving over 150% higher success rates the hardest task in suite and generalizing increasing horizons and numbers of entities. Rollout videos are provided at: nttps://sitest www.ie.essoooee...oooooooooo.o.e com wom/view/hecrr
INTRODUCTION
Planning to achieve complex, long-horizon goals in unstructured environments is defining chal- lenge the core of Reinforcement Learning (RI) and sequential decision-making research. Goal- Conditioned RI. (GCRL) (Kaelbling] 1993) provides a useful framework for training goal-reaching agents by facilitating generalization and knowledge transfer between goals and has proven promis- ing when coupled with deep RL algorithms (Schaul et al. 2015). However, its effectiveness remains limited when faced with complex goal distributions and high-dimensional observation spaces, espe- cially under sparse rewards (Park et al.) 2025a)
Exploiting the subgoal structure in GCRL (i.c., the fact that intermediate states in a trajectory can also be treated goals) has lead to significant progress in handling long-horizon sparse reward tasks via goal relabeling (Andrychowicz et al.) 2017) and hierarchical learning (Levy et al.) 2019) Chane-Sane et al. 2021] Baaaria al. (2021). Park al. (2023) proposed a two-level policy hierar- chy extracted from a single goal-conditioned value function to address compounding approximation errors when learning with temporal difference objectives. The resulting algorithm, HIQL, offers a simple instantiation of hierarchy in the offline GCRL setting, However, as the recent OGBench benchmark (Park al., (025a) reveals, offline GCRL methods (including HIQL) continue to strug- gle with combinatorial state-spaces and image observations.
If we are to make progress towards applying GCRL to real-world problems and in domains such as robotics, our algorithms must scale to complex goal distributions and rich, high-dimensional ob- servations such as images. Particularly challenging are scenarios where the environment contains multiple entities, where the state-space grows combinatorially with the number of entities. Examples for such domains include robotic object manipulation (Haramati et al. 2024), multi-robot path plan- ning (Shaoul et al. 2025), , autonomous driving (Vinitsky al. (018)) and video games (Zambaldi ct al. 2019:Delfosse Delfosse et al. 2024). In this case, the underlying factored structure can be leveraged and incorporated inductive bias significantly simplify learning.

---

Published as a conference paper at ICLR 2026
In this work, seek combine the benefits of subgoal hierarchy with factored structure solve Long-horizon tasks multi multi-entity domains. A natural subgoal decomposition in some cases corre- sponds to modifyyng subset of the entities, but as show, state-of-the-e methods such HIQL do not encourage this kind of sparsity. We propose a hierarchical entity-centric framework for offline GCRL that scales to image observations without requiring supervision. It leverages unsupervised object-centric representations and the factored structure in multi-entity environments to produce entity-factored subgoals, i.e., subgoals with sparse changes to entities compared to the current state.
We employ a two-level hierarchy where the low-level is value-based GCRL agent and the high- level a generrainng conditional diffusion model. We show that the bias induced by entity- centric diffusion encourages aatity-factore subgoals, which simplifies the subtask for the RL policy. Our method is designed be modular and adaptive: the two levels are only related by the datasets they are trained and are combined post-training via subgoal generation procedure, which in- volves selective subgoal generation based on the value function. This allows simple integration with potentially any value-based GCRL algorithm.
We demonstrate agents that can effectively achieve long-horizon goals in combinatorially complex and high-dimensional observation spaces whilst learning from sparse rewards. We test our approach using novel variations of existing benchmarks to highlight the above challenges. Our method con- sistently boosts the performance of flat entity-centric GCRL agents, achieving more than a 150% success increase on the most difficult task our suite, and generalizes to increasing horizons and number of entities.
Unlock Window
Oyen Window
Look Window Unlock Denoer
Upen Deaver
Lock Deover
Lin Cube
Place Cabe in Deswer
Generated
Subgoals
Goazl
States
Figure 1: Scene based factored subgoals. Bottom row: environment image observations at the time of subgoal generation. Top row: Reconstructions of the generated latent factored subgoals. Rightmost image: goal image observation. Circles and arrows: circles on the top row highlight fac- tors (excluding the arm) that are modified by the subgoals compared to the current observation the image below them. Arrows leading to circles on the bottom row highlight the factors that are ma- nipulated by the low-level RL policy, showing that the subgoals were achieved. Subgoal reconstruc- tions only include the foreground of the scene captured by the Deep Latent Particles (DLP, Daniel & Tamar (2022) representation. See App. for details on the task and the DLP model A.3
2 BACKGROUND & RELATED WORK
Goal-conditioned Reinforcement Learning: GCRL (Kaelbling 1993) considers Markov Deci- sion Process M (S, r), where S denotes the state space, A the action space, μ (S) the initial state distribution, p: A (S)) the environment transition dynamics, r 8
IR
the reward function and G the goal space. The objective is to lear policy π S x A that maximizes the expected discounted return E, I2 7're] for given goal distribution, 7 de- noting the discount factor. Our work focuses on the offline setting, where the agent learns from fixed dataset of suboptimal state-action trajectories. We assume that G S and a sparse reward (2(3 g) 1 {s= g) - 11, i.c., 0 when at the goal and -1 otherwise.
Hierarchical condiioond Reinforcement Learning: Hierarchical RL enables reasoning over multiple timescales and levels of abstraction by exploiting temporal structure within sequential data (Klissarov et al. 2025). In the context of GCRL, many algorithms tackle long- monghhoiizon goal-
reaching with various instantiations of subgoal hierarchies (Schmidhuber 1991 Dayan & Hinton
1992 Kulkarni et al. 2016) Vezhnevets et al., 2017 Nachum et 20188) Levy et al. 2019
Chane-
Sane et al. 2021
Bagaria al. (021). Closely related to work is HIQL (Park al. 2023)
2

---

Published as a conference paper at ICLR 2026
which extracts a two-level policy hierarchy from single value function learned from offline data. In contrast to HIQL, we model the high-level policy using a diffusion model which as we show, is crucial for producing high-quality subgoals that correspond to valid states. We additionally incorpo- rate entity-centric structure across the hierarchy to simplify learning in domains with combinatorial multi-entity state spaces and facilitate factored subgoals.
Diffusion for Sequential Decision-making: Diffusion models (Sohl-Dickstein et al. 2015 Ho
al. 2020) learn to generate data by reversing process that gradually adds noise to clean sam- ples, training a neural network to perform step by step denoising. The network is optimized by predicting the noise injected at each step and generates data by gradually denoising sample from an initial noise distribution. These have become widely popular for their ability to capture complex multi-modal data distributions. Work applying diffusion models to decision-making can be coarsely divided to diffusion policies (Chi et al. 2023, Hansen-Estruch et al., 2023) which model distribu- tions of actions (or action "chunks") and diffusion planners (also referred as diffusers) (Janner et al. 2022 Ajay et al.] 2023 Lu et al., 2025) which model distributions of state or state-action trajectories. Li et al. (2023): Chen et al. (2024) propose hierarchical diffusers that generate sub- goal trajectories condition lower-level diffusers for long- mogghorizoon tasks. Compared to the above, our method employs a high-level diffusion model without guidance to generate a single immediate subgoal for GCRL policy and uses its value function for test-time subgoal selection. We adapt the entity-centric diffuser proposed in Qi et al. (202), which was designed for goal-conditioned behavioral cloning, to generate factore subgoals.
Eentty-centric Reinforcement Learning: Entity-centric RL considers environments which can be described as collection of entities i.e., S Si &82 @ ... 2 SN. This factored structure can simplify et al., 2018 Zadaianchuk et al., 2022; Sancaktar et al.) 2022, Zhou et al., 2022) and facilitate composi- tional generalization (Lin et al., 2023). Obtaining factored representations of images that closely approximate the true state of the environment without supervision not trivial and has been a sub- ject of previous work. These either learn representation concurrently with the decision-making modules (Zambaldi et al. 2019 Watters et al.,
learning when incorporated in state representation and agent architecture (Sanchez-Gonzall
2019:
Veerapaneni et al. (0200) or pretrain unsuper- Locatello et al. [2020 Daniel & Tamar 2022)
vised object-centric representations (Lin et al.
2020
for downstream RL (Zadaianchuk et al. 2021]
Yoon et al.] Haramati et al. (024). We present modular hierarchical framework that builds Haramati al. (2024) and enables long-horizon
goal-reaching from sparse reward via factored subgoal diffusion.
See App. A for extended background and related work.
3 HIERARCHICAL ENTITY-CENTRIC REINFORCEMENT LEARNING
Our framework, Hierarchical -enttric RL (HECRL), addresses two main aspects that hinder accurate value learning in multi- eetii--enttty domains: reward propagation over long horizons 3.2 and combinatorial state complexity (.33 both of which increase with the number of entities. We employ an entity-centric approach (Haramati et al. 2024 Qi et al. 2025) and propose a two-level hierarchy composed of a value-based GCRL agent and a factored subgoal-generating diffusion model. Our ap- proach is modular and adaptive, making it compatible with various value-based GCRL algorithms. Additionally, it can handle image-based domains without access to the underlying factored envi- ronment state (e.g., robotic manipulation from pixels and video games) using unsupervised object- centric representations (e.g., DLPv2 (Daniel & Tamar 2024), which we use in our experiments).
3.1 MOTIVATION: SIDESTEPPING COMPOUNDING VALUE APPROXIMATION ERROR
Leveraging factored structure in learned value functions improves performance in combinatorial state-spaces, but does not completely mitigate value approximation error. Temporal Difference (TD) learning (Sutton) 1988) leads to errors which compound over the timestep horizon. This error is more consequential in long-horizon sparse reward tasks where the reward must propagate through many steps of discounted TD updates. While the value function can still be useful for approximating distances between states and goals, it may not be as effective for conl-conditioned policy extraction: for states far away from a given goal, the approximation error can be larger than the fine difference in value between states that are low-level action apart. Park et al. (2023) refer this as the
3

---

Published as a conference paper at ICLR 2026
value Signal-tonnoise ratio, which grows with the distance between states. This ratio defines an "effective radius" of states whose values provide clear learning signal for policy extraction, which will refer to the policy competence radius and denote R R for simplicity of notation. This quantity is not known a priori. Given an offline RI. policy, our method aims to produce an immediate subgoal that is both reachable by that policy, that is, within Ry and leads it closer to the goal. This process can be repeated until the goal is within the policy's ' reach. Two key aspects of our approach distinguish from previous methods: (1) Modularity and Test-time Flexibility- we do not require any modifications to the underlying RL agent. We separately train subgoal- generating conditional diffusion model and enforce value-based reachability constraint test- time (see Fig. 2 left, and App. D.2 for further discussion). (2) Fuctored Subgoal Diffusion-ourr entity-centric approach encourages subgoals with sparse modifications to state factors when the data supports it (see Fig. [ right, and Sec. 4.5 for empirical results). Subgoals that require modifying few state factors are generally casier to reach when these factors (or subsets thereof) are independently controllable, making them favorable for conditioning the low-level policy.
Factered Subgoal
State
Goal
A
Non-factored Subgoal
Figure 2: Left: subgoal sampling illustration (Alg- lines 2-4). Robot agent, Dashed circle competence radius, Flag goal, Red triangle discarded subgoal, Green circle filtered sub- goal, Yellow star chosen subgoal, Background color gradient value landscape. Right: factored subgoal illustration. Factored subgoals make it easy modify small subsets of entities which simplifies the subtask when factors are independently controllable. The subgoal images depict two states that are roughly the same distance from the initial state, where the top requires moving only the blue cube while the bottom requires moving all three.
3.2 THE SUBGOAL DIFFUSER
Our method assumes access trained value-based entity-centric GCRL agent consisting of policy π S X + A and value function V: S X + IR as well as the offline dataset it was trained but we maintain the G notation for readability.
on. We remind the reader that in our setting
We train a conditional diffusion- based subgoal generator D : S x G + G on the offline RL dataset fit the distribution of states that are at most K timesteps away from a state given goal g. where is a state sampled uniformly from future timesteps in the same trajectory as s. That is, given an offline state trajectory of length T: (Sg..... ST), we (1) uniformly sample a timestep from [0. - 1] to select a state = (2) uniformly sample a timestep ty from [t + 1, 1,7] to select goal state y = St,; ; (3) set the training subgoal to ğ *min((+K,t,) (4) train a conditional diffusion denoiser to model the dataset distribution p (ğls, g). This process can be thought of as training a goal-conditioned behavioral cloning subgoal policy, which we refer to as the Subgoal Diffuser.
We do not assume the dataset contains goal goal-directed behavior, which has several implications: (1) p (ū]s,g) g) can be highly multi-modal, motivating our diffusion modeling choice. (2) p (ūl 8) g) may contain diverse states terms of value-distance from 8 and g, i.e., (g.ğ) and (Ø.9) respectively, making K hard to set such that the subgoals guaranteed to be reachable (even if we had access to R. a priori).
(3) The subgoal Diffuser fits the behavior data and thus does not capture any notion of subgoal optimality. We therefore introduce simple and effective test-time subgoal generation procedure, which we describe in the following and is summarized in Alg. [
We sample N subgoal candidates from the subgoal diffuser and filter them for reachability based on value threshold R, i.e., keep subgoals that satisfy V (s, g) >> R. We then select the subgoal that is
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 Subgoal Generation Procedure
Input: current timestep t, current state s, current subgoal g', goal g. subgoal Diffuser D, value
function V. policy competence radius fł, subgoal samples N, subgoal timesteps Tsg.
Output: subgoal 9
1: if % Tag = 0 then Sample new subgoal
2: Sample subgoal candidates {0a ) D (5,g)
3: Filter reachable subgoals {ōz M + (ōr (s. (.) R)
#MSN
4: Select subgoal closest to goal 9 +- arg ma ( (5.9)
5:
if V ( 9)
S (s.g) then
6:
- 9
# If closer to the goal g than the generated subgoal ÿ. go directly to g
7:
end if
8: else
# Keep current subgoal
9:
g g--g g
10: end if
11: return g
closest to the goal, i.e. with the highest value V (ō. g) (see Fig. 2 left). We rollout the low-level RL policy x with this subgoal for a fixed Tag timesteps and then repeat the subgoal generation procedure. This can be viewed as form of constrained sample-basee based planning with receding horizon control or Model Predictive Control (MPC), in which case our model consists of {D, V) and the optimization is performed the state-space S rather than the action-space A (see App. D for discussion). To aid convergence, if the goal is closer to the current state than the chosen subgoal it is replaced with the goal (lines 5-7 in Alg. []. The constraint we impose on the subgoals is appropriate in our setting since
((s.)) g) can be viewed as a discounted (asymmetric) distance metric between states (Wang
al. 2023). and our approach compatible with other test-time constraints on states.
3.3
GENERATING ENTITY-FACTORE SUBGOALS
The previous section describes a subgoal generation and selection mechanism which is agnostic to the structure of the state space. When the state of the environment can be described as a collection of entities. prior work has demonstrated that factored state representations and set-based policy archi- tectures (c.g., Transformers (Vaswani et al. (2017)) better handle the combinatorial complexity of the state-space, yielding improved performance, greater sample efficiency, and facilitates compositional generalization (Zhou et al,, 2022 Haramati et al., 2024). We further exploit this structure to generate autity-factored subgoals. addition to the above benefits, entity-level factorization can simplify the subtask for the low-level policy by facilitating modifications to subset of the entities. While there may be explicit ways to constrain the subgoal generation to encourage sparse modifications from the current state, observe that this emerges naturally from the data with the appropriate inductive bias. our case, it is the bias induced by entity-centric diffusion (Qi et al.) 2025): given sets of diffusion model gradually denoises m=1- where M denotes the set size and r the diffusion
state and goal entities a set of noisy subgoal entities ÿ = (0%,
{sm. Jm= and gr {9m IM m=1
timestep. Importantly, each entity in s, g and g is a separate input to the Transformer denoiser (or other bet-based architecture) encoded with its affiliation to either of the sets.
As discussed in the previous section, the distribution p (ūls, .) can encompass number of ways to reach a given goal, which grows combinatorially with the number of entities. This leads to a wide and multi-modal distribution over the relevant subgoals. Our choice of diffusion enables capturing the multiple subgoal modes present in the data, potentially corresponding to modifying states of different subsets of entities. This in turn allows sampling from distinct modes. By contrast, pro- ducing a weighted average of those modes is more likely to result in modifications to large portions of the state (see Sec. 4.5). As our experiments show, coupling entity-centric representations with Transformer-basse diffusion encourages entity-factored subgoals. We attribute this partly to the Transformer's 's ability to selectively copy its input tokens (fsm /m=1 1 and {gm 1M 1==1) to the output (fgr, 1.M via the attention mechanism (Jelassi et al. 2024). The entity-centric structure addition- ally allows us to train relatively small diffusion models with as few as 10 denoising steps, which reduces computational burden for real-time control.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
Experiments are designed to study: (I) our method's efficacy in handling long-horizon goal- conditioned tasks involving multiple entities (Sec. 4.3 (2) the impact of our design choices on performance (Sec. 4.4); (3) the quality of subgoals generated by our method (Sec. 4.5): (4) our method's compositional generalization capabilities (Sec. (46)). We present our evaluation suite in
Sec. [.. 1.1 and describe implementation and baselines in Sec. 4.2 We provide datasets, checkpoints
and code to reproduce the results this paper at: https: tigttte thub. com www.o//////n.../.........oo....a
PPP-Cube
Stack-Cube
Scene
Push-Tetris
Figure Environment suite. PPP-Cube: Pick, Place and Push cubes to goal positions.
Sta caackbeub Pick-and-place cube stacking. Scene: Manipulate objects (drawer, window, button and cube) to goal configurations. Push-Tetris Push blocks goal positions and orientations.
4.1 ENVIRONMENTS, TASKS AND DATASETS
Our experimental results focus on multi-object manipulation using a suite of environments, tasks and datasets which We adapt from previous benchmarks to highlight challenges in perception and handling combinatorial state-spaces. The environments are visualized in Fig. The majority of our suite builds on the OGBench (Park al. 2025al manipulation environments. specifically Cube and Scene environments. These contain a UR5e robot arm and involve table-top object manipulation. We increase image resolution, add mulli-view, perception and remove simplifying visual components such as arm transparency and end-effector coloring.
Tasks: (1) PPP-Cube: based on the OGBench Cube environment, the agent is required to ma- nipulate cubes between randomly initialized state and goal positions. Distinct colors are assigned each cube at random out of a fixed set at the beginning of each episode. The accompanying dataset contains diverse agent-object interactions including Pick-Place and Push operations. Image observations include two views, front and side. To isolate algorithmic aspects from any particular object-centric image representation, support state-based entity-centric observations that emu- late a "perfect" factored representation. (2) Stack-Cube: identical to PPP-Cube except the dataset contains only Pick-and-Place operations with high probability of stacking and the agent only evaluated on stacking. (3) Scene: We adopt the OGBench Visual Scene environment as with modifications some tasks avoid ambiguity in goal specification that arises due to our more realistic visual setup (see App. B. The environment is observed from a single view. (4) Push-Tetris: we adapt the Push-T environment introduced in Chi al. (2023) multi-object manipulation of Tetris-like blocks. Distinct block types are sampled at random at the beginning of each episode. Each block type has a distinct color which is fixed across episodes. The agent required to push the blocks to goal configurations including position and orientation. A dataset is collected using a random policy restricted to fixed radius around an object, sampled at fixed time intervals, resulting in highly suboptimal behavior. See App. B for extended environment details.
Environment Characterization:
We consider PPP-Cube Cube and Stack- -Cue the most challeng- ing in suite" requiring learning long-horizon 3D manipulation in combinatorial state-space from realistic image observations. Scene presents similar challenges in terms of perception and includes more object types (and corresponding manipulation capabilities) but has much fewer effec- tive state configurations, making it significantly less combinatorially complex. Push-Tet ris is
see App. B.I for literature review highlighting that non-trivial performance these environments with than objects has heen attained prior this work within the data regime consider.

---

Published as a conference paper at ICLR 2026
more visually simple and limited to 2D plane but requires fine- tine-grained control of variable objects in combinatorially complex state-space.
Training datasets contain 3 objects (excluding Scene) and up 3M state transitions. We train agents with a fixed amount gradient updates and report mean and standard deviation of the final checkpoint across seeds for all of our experiments and metries. Best results up stan- dard deviation are highlighted in bold. In contrast to OGBench, we do not terminate upon reaching the goal test-time such that the agent cannot just stumble upon the goal but must reach and stably maintain it. This is critical in tasks such as St ack -Cube where the cubes should remain stacked and S Scene where the effective number of goal configurations allows non-trivial performance by policy that simply explores those configurations without relying on the goal specification.
Evaluation:
4.2 IMPLEMENTATION AND BASELINES
Baselines were carefully chosen to contrast the different aspects and design choices of our method (see App- Table 9 while ensuring fair comparison. The RL agents we consider are based on goal- conditioned variants of IQL. (Kostrikov al. 2022).
EC-SGIQL (Ours):
implement our method on of an ECRL (Haramati et al. 2024) agent trained with IQL, For the Subgoal Diffuser we adapt EC-Diffuser (Qi al. 2025) to generate sub- goals conditioned entity-centric states and goals. We refer to this instantiation of our method as Entity-Centrr SubGoal IQL (EC-SGIQL).
EC-IQL: corresponds to ECRL (Haramati al. 2024) agent adapted to the offline setting by integrating the Entity Interaction Transformer (EIT) architecture with IQI. This baseline represents an entity-centric non-hierarchical method, which is equivalent
without subgoal generation. EC-Difluser (Qi et al. 2025): an entity-centric diffusion-based behavioral cloning method.
HIQL: an agent based on HIQL (Park al., 2003)).
IQL: an agent based on IQL (Kostrikov et 2022).
All entity-centric methods (EC- prefix) are trained from entity-centric state observations (see App. B for details) latent image-based representations extracted using pretrained DLPv2 (Daniel & Tamar 2024) (see App. AA.3 for an overview of DIP). Standard agents are trained from a single- latent image-based representations extracted using pretrained VQ- vector state observation 2017). Image representations were pretrained on the offline RL datasets,
VAE (Van Den Oord et al.
one for each task (see App.
D.7 for reconstruction visualizations). We provide extended implemen-
tation details in App.
4.3 LONG- MANIPULATION
Table 1 summarizes performance of all methods on enviromment suite. We report the state-goal pixel coverage (i.e., overlap) averaged over objects for Fush-Tetris and success rates otherwise. Our method significantly outperforms all baselines with two exceptions in which it performs on par. Notably, EC-SGIQL uses the same low-level policy and value function as EC-IQL yet consistently improves its performance. On PPP-Cube from images, the most challenging task in our suite, achieves more than 150% increase in success rate. EC-IQL is the second most performant method, highlighting the significance of structure in these domains. Figures [ and 4 visualize rollouts of our method following the subgoals generated by the Diffuser in the image-based based environments. See App. D.3 for further discussion and performance results our website for rollout videos.
Table 1: Long-horizon manipulation performance. All values are success rates except for Push-Tet- for which we report state-goal pixel coverage. See Sec, 4.2 for baseline details. Best results up standard deviation are highlighted in bold.
Environments
EC-SGIQL
EC-IQL
EC-Diffuser
HIQL
IQL
PPP-Cube (Stare)
82.5 111
51.5
4.4
44.8 667
48.3 73
34.3 449
Stack Stack-Cube (Sinte
43.5 -
29.0 29
43.8 192
0.0
+3.
PPP-Cube (Image)
64.3 =
25,0
0.3 0.
0,0
0,0 +.0
Scene (Image)
61.5 -
53.0 +55
3.3 =2
8.3
17.5
Push-Tetris (Image)
61.4 3.3
31.6 1.3
7.9 1.5
5.2 1.2.008
3.4 1.08
7

---

Published as a conference paper at ICLR 2026
Generated Subgoals
Gogal
States
Subgoals
Goal
States
Figure 4: PPP-Cube and Push-Tetris Titris image-based factored subgoals. Bottom row: environ- ment image observations at the time of subgoal generation. Positions of latent particles are plotted on the image. Top row: DLP reconstructions goal image observation. Circles and arrows: circles on the top highlight factors (excluding the agent) that are modified by the subgoals compared to the current observation in the image below them. Arrows leading to circles the bottom row highlight the factors that are manipulated by the low-level RL policy, showing that the subgoals were achieved.
the generated latent subgoals. Rightmost image:
4.4 ABLATION STUDY
Table presents results ablating the following aspects of our subgoal generation procedure: (1) use of the value function for subgoal selection (Random Sample), (2) use of the value function for subgoal filtering (line in [lg[[ (Max Value) and (3) the use of diffusion by training a deterministic Transformer-base based subgoal generator with an Serrantage-Wigghee 2019h objective as in HIQI. AWR with entity-centric latent image representations requires special care as the latent entities lack ordering. We adjust the standard regression loss to a loss based on the Chamfer distance (see App. C for details).
Regression (AWR, (Peng et al.)
Our method consistently achieves the best results, both in terms of success rate and timestep ef- ficiency. The Max Value variant performs on par on some of the tasks, indicating that the policy competence radius may be larger than the radius of the generated subgoals in these cases. Another trend we observe is that the Random Sample and AWR variants require more timesteps to reach goals when the success rates comparable. This is expected when selecting random samples from the Diffuser which do not always make direct progress to the goal. The AWR variant achieves the lowest performance overall which we attribute to the quality of the subgoals. Fitting (latent) observation- generating model with weighted regression results in weighted averages of observations, which do generally correspond valid state. We study this further in the following section.
4.5 MEASURING SUBGOAL QUALITY
We hypothesize that our approach performs well compared baselines and ablations part because it encourages generating simpler subgoals with sparse changes to factors compared to the current state when the data and domain support We measure this sparsity in the state-based Cube envi- ronments and report the average number of modified entities in Table 3 Training the subgoal policy with AWR results in changes to all 3 cubes most of the time, while our Subgoal Diffuser modifies close to cube average. We attribute this the diffusion model's ability to produce samples from distinct subgoal modes compared to the deterministic policy which produces a weighted average of those modes, well the factored structure of the Subgoal Diffuser.
8

---

Published as a conference paper at ICLR 2026
Table 2: Subgoal generation ablation performance. Ours: EC-SGIQL. Max Value: Ours w.o. filtering subgoal candidates. Random Sample: randomly samples a subgoal from the Diffuser. AWR: replaces the Diffuser with deterministic Transformer trained with AWR. For Push-Tetris we report pixel coverage (top) and the sum of pixel Chamfer distances across the episode (bottom). For the rest we report success rate (top) and negative returns (bottom) which correspond to the average number of timesteps taken to either complete the task or terminate due to evaluation timestep limit. Best results up to a standard deviation are highlighted bold.
Environments
Ours
Max Value
Random Sample
AWR
82.5 1: 3.1
76.3
73.0 i:
67.8 1
PPP-Cube (State)
422 + 60
464 = 35
759 = 23
773 = 46
43.5 :: 1.9
35,5 7.5
28.8 5.5
9.0 4: 4.2
Stack-Cube (Sture)
706
771 + 54
904 =
985 =
64.3 + 4,9
46.3 + 2.1
57.3 + 8.7
55.5 3.4
PPP-Cube (Image)
587
709 7
750 42
735 37
61.5 + 5.9
61.0 + 6.7
41.0 + 6.2
44.5 + 6.0
Scene (Image)
597 + 26
589 = 43
843 5
719 + 45
61.4 1: 3.3
59.5 3.4
58.6 4.5
56.1 1. 5.6
Push 1 Teerrs (Jmage)
510 + 12
505 = 32
622 = 30
606 = 40
Table 3: Number of modified entities excluding the agent in the generated subgoal compared to the input state. EC-Diffusion: Our subgoal Diffuser, EC-AWR: a Transformer trained with AWR on entity-centie states, AWR: an MLP trained with AWR on single-vector states. An entity is con- sidered modified if its position changed more than a threshold distance. The environments contain 3 cubes and the robotic arm. Values are averaged over 400 randomly sampled initial states and goals. Best results up to standard deviation are highlighted in bold.
Environments
BC-Diffusion (Ours)
EC-AWR
AWR
PPP-Cube (State)
1.36
0.01
2.96 0.01
2.98 0.00
Stack- Cube (State)
1.04 + 0.01
2.82 + 0.02
2.98 + 0.01
Since measuring the subgoal sparsity in the image- based environments not straightforward, we provide qualitative observations obtained by reconstructing the latent subgoal particles with the DLP decoder for the Subgoal Diffuser Fig. and 4 and for the AWR variant in Fig. [ and (App.). The Diffuser subgoals are often composition of the input state and goal images and involve sparse changes from the state. The generated subgoals are not perfect and occasionally include the same entity twice. When this occurs, the entities are more often than not an exact copy from each of the inputs, i.e., one from the state and one from the goal. This supports our hypothesis that the Transformer selectively copies its input entities to the output. We attribute these type of error the nature of the DLP representation and the diffusion objective, which might be remedied with larger diffusion Transformers. The AWR subgoals often contain multiple duplicates of entities representing different futures in a single subgoal, which provides the low-level RL policy with ambiguous goals. See App. for further discussion.
4.6 GENERALIZATION
Incorporating entity-factored structure relaxes the combinatorial complexity of the state space by facilitating compositional generalization (Lin al.) 2023). Learning requires less coverage of the state space long as there is sufficient coverage of the individual factors. Entity-level state gener- alization (e.g., to novel compositions of object position and color) measured by sample efficiency. Indeed, we observe that entity-centric methods reach peak performance with much fewer gradient updates compared to the unstructured ones (when the latter achieve non-trivial performance). Com- positionality with respect the global state of the system be tested by varying the number of state and/or goal entities. We test our method's generalization capabilities in these cases and report the performance in Table 4 (see results including P in App. Table 14). Our method

---

Published as a conference paper at ICLR 2026
showcases non-trivial generalization which degrades with increasing number of objects. We surpass the performance the flat entity-centric agent, showing that the subgoals maintain a level quality sufficient to guide the RL policy. These results hint method's 's scaling potential to increasing numbers of entities via curriculum or offline-to-online finetuning, which is left for future work.
Table 4: Zero-shot compositional generalization. All values are success rates. signifies the training variant in each segment, which was used to evaluate zero-shot capabilities on the other variants. Best results up to a standard deviation are highlighted in bold.
Variants
EC-SGIQL.
EC-IQL
EC-Diffuser
Cuubes-Stace (Stare
69.8 7.0
45.8 + 12.
87.8 +: 5.4
3 4-Cubes-Stack (State)
43.5
29.0 4: 2.9
43,8 +: 9.2
4 4-Cubes-Staan (State)
28.8 3,9
17.3 + 2.5
1,5 1.3
PPP-1-Cube (State)
97.0 1.8
92.3 i: 8.9
85.0 1.9.
PPP-2-Cubes (State)
94.0
75.0 3.8
76.5 7.6
- 3CCbbe (State)
82.5 3.1
51.5 4.4
44.8 6.7
PPP-4 -Cubee (State)
65.3
31.8 4.0
42.8 +
BPP-5-Cube (Stute)
49.0 8.5
19.3
23.8 6.7
PPP- 6-Cubes (State)
25.7
10.5 6.1
8.8 3.5
PPP 1-Cube (Image)
94.5 2.9
91.5 + 1.3
3,8 + 1.3
PPP 2 C (Image)
77.0
47.8 6.2
0.0
309-3--ubbe (Image)
64.3 6.0
25.0 +
0.3 0.5
PPP-4-Cubes (Image)
38.3
11.5 1.7
0.0 1.0
PPP- 5-155-Cubbe (Image)
19.3 6.2
4.0 4.0
0.0 0.0
PPP 6 6-Cubes (Image)
9.5 -
1.5
0.0
5 CONCLUSION
We present Hierarchical Entity- Centtrc Reinforcement Learning (HECRL), offline GCRL frame- work that integrates subgoal hierarchy with factored structure to solve long- hong-horizon tasks in multi- entity domains and scales to image observations. We design our method to be simple, flexible and modular, making it compatible with various value-based GCRL algorithms and test-time subgoal constraints. We empirically demonstrate that our factored Subgoal Diffuser coupled with our simple value-based selection procedure produces high-quality subgoals to guide low-level entity-centric RL agent, consistently boosting performance. We study what aspects contribute the perfor- mance of our method and find that the bias induced by entity-centric diffusion encourages subgoals with sparse modifications from the current state compared commonly used weighted regression objectives. Finally, we display non-trivial zero-shot generalization performance with increasing number of entities (and consequently longer horizons). hinting to potential scaling to environ- ments with more entities.
Limitations and Future Work: our method assumes that the value function provides a sufficient signal for the low-level policy to have non-negligible competence radius well as for guiding the subgoal generator, which held in our challenging domains but may limit its applicability That said. the modularity of our framework facilitates improvements top of advances in flat value- based GCRL algorithms. The Subgoal Diffuser trained with subgoals up to a fixed K steps away from the current state. While our method is time subgoal filtering mechanism (see App. D.5), future work can explore ways to automatically infer K from data. Finally, applying our method to domains with image observations relies on a good factored state estimator. We show that this is possible in our simulated domains without requiring supervision. The potential of scaling our approach to real-world an-the-willd scenarios thus goes hand in hand with advancements in unsupervised object-centric representation learning (Seitzer) al. 2023 Zadaianchuk et al. 2023
others.
robust this hyperparameter due to test-
Daniel Tamar 2024).
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our work advances methods for offline Goal- Conditioned Reinforcement Learning (GCRL). The experiments this paper were conducted in simulated environments, with no involvement of human subjects or privacy concerns, and we do not foresee any ethical or societal risks from this work.
REPRODUCIBILITY STATEMENT
We are committed to enabling full reproducibility of work. We have included extensive im- plementation and evaluation details throughout the paper and in the Appendix. We release code, checkpoints and datasets reproduce all of the results in this paper and facilitate future research https: https://github.com rttp:///.huubcccmm/aaiii.....m 1/HECRL
ACKNOWLEDGMENTS
This research was supported by the ONR under grant N00014-2211-299.. AZ and CQ were funded by NSF 2340651, NSF 2402650, DARPA HR001 1249043,, and ARO I91
AT received funding from the European Union (ERC. Bayes-RL, Project Number 101041250). Views and opinions expressed do not necessarily reflect those of the European Union or the European Research Council Executive Agency, Neither the European Union the granting authority can be held responsible for them.
however those of the authors only and
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? The Eleventh Interna- tional Conference Learning Representations, 2023. URL https: htttps://opente http:///ppeeeeeeeeeeeeeeeeeeeeee /
Forum?id= Fo2K9DFG
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances neural information processing systems, 30, 2017.
Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International conference on machine learning, pp. 521-531. PMLR, 2021.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioeed reinforcement learning with imagined subgoals. In International conference on machine learning, pp. 1430-1440.
PMLR, 2021.
Michael Chang. Alyssa Li Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, and Amy Zhang. Hierarchical abstraction for combinatorial generalization in object rearrangement. In The Eleventh International Conference Learning Representations, 2023. URL hhtpss://
rpenreview net. fentt/forum?id0m http://rrumnndfffccc/////3/
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In The Twelfth International Conference Learning Representationss 2024.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics:
Science and Systems (RSS), 2023.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent particles. In International Conference on Machine Learning, pp. 4644-4665. PMLR, 2022.

---

Published as a conference paper at ICLR 2026
Tal Daniel and Aviv Tamar. DDLP: Unsupervised object-centric video prediction with deep dynamic latent particles. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
/tttps://cpenreview.. net http://www....ttooiiiieiii---- wtw.:rri-mmmimiiii......
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kerst- ing. OCAtari: Object-centric atari 2600 reinforcement learning environments. Rein- forcement Learning Conference, 2024. URL https:://penrevleww
2401w9d5k8
Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, and Bart Dhoedt. Focus: object-centric world models for robotic manipulation. Frontiers in Neurorobotics, 19:1585386, 2025.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018.
Philippe Hansen Esttruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint 2rXiv:2304. 10573, 2023.
Dan Haramati, Tal Daniel, and Aviv Tamar. Entity-centric reinforcement learning for object manip- ulation from pixels. In The Twelfth International Conference on Learning Representations, 2024.
URL petttps://operreviiee
net wwt.tneunfftttteoo forum? id=uDxeSZ w
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-685,, 2020.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
URL htllhttp:// jmmrr.mmi....//.m wwt.o/////////. htpppers/v23//22 1011-133 1342. html
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
Samy Jelassi, David Brandfoobrenerr, Sham M Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models at copying, In International Conference on Machine Learning. pp. 21502-21521. PMLR, 2024.
Jindong Jiang", Sepehr Janghorbani*, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Represen- lations, 2020. URI. htttts:://penrevieews net//wne///ttee ott.:eeweeeeeeoooooooossrrrrrrr
Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, volume 2, pp. 1094-8, 8, 1993.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
1312.61312.61 14,6111140000 2013.
Martin Klissarov, Akhil Bagaria. Ziyan Luo, George Konidaris, Doina Precup, and Marlos C Machado. Discovering temporal structure: An overview of hierarchical reinforcement learning. arXiv preprint ArXiv:2506.1400 2025.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https:// openreview www. forum? idd 68n2 2
Tejas D Kulkami, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
12

---

Published as a conference paper at ICLR 2026
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2019.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. The Exploration in Al Today Workshop at ICML 2025, 2025.
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manip- ulation using relational reinforcement learning. 2020 ieee international conference on robotics and automation (icra), pp. 4051 1-58. IEEE, 2020.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. International Conference on Machine Learning, pp. 10035-20064 PMLR, 2023.
Baihan Lin, Djallel Bouneffouf, and Irina Rish. A survey on compositional generalization in appli- cations. arXiv preprint arXiv:2302.01067, 2023.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
URL
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in neural information processing systems, 33: 33:11525-11538, 2020.
Haofei Lu, Dongqi Han, Yifei Shen, and Dongsheng Li. What makes good diffusion planner for decision making? In The Thirleenth International Conference on Learning Representations, 2025. URL cettps://operreviews
Davide Mambelli, Frederik Träuble, Stefan Bauer, Bernhard Schölkopf, and Francesco Locatello. Compositional multi-object reinforcement learning with linear relation networks. arXiv preprint MxXiv:2201.13388, 2022.
Aditya Mohan, Amy Zhang, and Marius Lindauer. Structure in deep reinforcement learning: A survey and open problems. Journal of Artificial Intelligence Research, 19:1167-, 2024.
Malte Mosbach, Jan Niklas Ewertz, Angel Villar-Corrales, and Sven Behnke. Sold: Slot object- centric latent dynamics models for relational manipulation learning from pixels. arXiv preprint arXiv:2410.08822. 2024.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Higl: Offline goal- conditioned with latent states as actions. Advances Neural Information Processing Systems, 36:34866-34899 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned In International Conference on Learning Representations (ICLR), 2025a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv preprint arXiv:2506.04166 2025b.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 17X00:1910.00177, 2019.
Carl Qi, Dan Haramati, Tal Daniel, Aviv Tamar, and Amy Zhang. EC-diffuser: Multi-object manipulation via entity-centric behavior generation. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: /tttts:://openrvview o3pJU5QCtVV
Cansu Sancaktar, Sebastian Blaes, and Georg Martius. Curious exploration via structured world models yields zero-shot object manipulation. Advances in Neural Information Processing Sys- fems, 35:24170-24188 2022.
13

---

Published as a conference paper at ICLR 2026
Alvaro Sanchez-Gonzalle Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International conference machine learning. pp. 4470-4411 4470-4479. PMLR, 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
Jürgen Schmidhuber. Learning to generate sub-goals for action sequences. In Artificial neural networks, pp. 967-972, 1991.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Lo- catello. Bridging the gap to real-world object-centric learning, The Eleventh International Conference on Learning Representations, 2023, URLhttps: /ttp///pwwweweeettt.tt www./ettfounnno
U
[dd=b9tUk-F_a
Yorai Shaoul, Itamar Mishani, Shivam Vats, Jiaoyang Li, and Maxim Likhachev. Multi-robot mo- tion planning with diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, pmlr, 2015.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9 3:9-44,1 1988.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033 5033. IEEE, 2012.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- lion processing systems, 30, 2017.
Rishi Veerapaneni, John Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Alexander Sasha Vezhnevets, Simon Osindero. Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540-3549. PMLR, 2017.
Eugene Vinitsky, Aboudy Kreidich, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen. Benchmarks for reinforce- ment learning in mixed-autonomy traffic. In Conference robot learming, pp. 399 409. PMLR, 2018.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal ment learning via quasimetric learning. In International Conference on Machine Learning, pp- 3641 16441-3 PMLR, 2023.
reinforce-
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration, arXiv preprint 190iv 1905.0925. 2019.
14

---

Published as a conference paper at ICLR 2026
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object- centric representations for reinforcement learning. In International Conference on Machine Learning (ICML) 2023, pp. 1-28, 2023.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforcement learning with aijecttcentricccppeeeeeeee oat.://eee.......oooooorueaaooor 21opppLLhL1 tations, 2021. URL /tt.Ltttpp///pppeeeiieww. net/ forum
In International Conference on Learning Represen-
Andrii Zadaianchuk, Georg Martius, and Fanny Yang. Self-supervised reinforcement learning with
independently controllable subgoals. In Conference on Robot Learning, pp. 384-394. PMLR, 2022.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. Advances in Neural Information Processing Systems, 36:61514-61545, 2023.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep re- inforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019. URL http::httt/pgttt rettesee//pe..rev.l...tntt htt.:pppe.....e.eeeeeeeeeooooooe -orumpic-HHaaaacCKK
Weipu Zhang, Adam Jelley, Trevor Mclnroe, and Amos Storkey. Objects matter: object-centric world models improve reinforcement learning in visually complex environments. arXiv preprint arXiv:2501. 16443, 2025.
Linfeng Zhao, Lingzhi Kong, Robin Walters, and Lawson LS Wong. Toward compositional gener- alization in object-oriented world modeling. In International Conference on Machine Learning, pp. 26841 126664. PMLR, 2022.
Allan Zhou, Vikash Kumar, Chelsea Finn, and Aravind Rajeswaran. Policy architectures for com- positional generalization in control. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. In Ferty-second International Conference on Machine Learning, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs were used solely to polish writing in some parts of the paper.
A EXTENDED BACKGROUND & RELATED WORK
A.1 MULTI-ENTITY ENVIRONMENTS
Slightly more formally. we define a multi-entity environment as one that can be characterized by factored state-space: S = S1 S2 @ ... 8 Sy. where some attributes are shared across the different factors {S: which constitute the entities. One example is robotic multi-object manipulation where the robot and objects are the entities, each described by shared attributes such as position and visual appearance (and potentially others which are not shared). Another example is robotic locomotion, where each robot joint can be viewed as a separate entity described by shared physical properties. A comprehensive review of structure in (deep) reinforcement learning that defines and discusses different forms of state factorization can be found in Mohan et al. (2024).
A.2 OBIECT-CENTRIC REPRESENTATIONS FOR SEQUENTIAL, DECISION-MAKING
Many previous studies have dealt with the question of how to leverage the factored structure in environments with multiple entities for sequential decision-making Some assume access to the underlying factored state of the system and develop set-based architectures process them in
sample-efficient and generalizing manner both in model-free (Li et 2020 Mambelli et al. 2022 al.] [2018] Saan-
Zadaianchuk et al. 2022 Zhou et al. 2022) and model- based "Sanchez-Gonzalle
caktar et al. 2022) settings. In order to scale these methods to image observations, some methods take an end-to-end learning approach by learning an object-centric representation concurrently with the decision-making
modules (Zambaldi et al. 2019 Watters et al.) 2019 Veerapaneni et al. 2020
Ferraro et al., 2025). Others leverage unsupervised Object-Centric Representations (OCRs) includ- ing patch-based (Jiang* et al., al.,, 2020). slot-based (Locatello et al. 2020) and particle-based (Daniel
Tamar 2022) models. These are commonly pretrained and kept frozen for downstream training of the poccisio-makinge components. These have been applied in both model-free (Zadaianchuk al.
/ Yoon et al. [2033) Haramati et al. (024)). model-based (Zhao et al.)
2021
2022 Chang et al.] 2023
Mosbach et al. 2024 /Zhang al. 2025 and imitation learning (Qi et al. 2025) approaches.
A.3 DEEP LATENT PARTICLES
This section provides an overview of the Deep Latent Particles (DLP, Daniel & Tamar 2022 2024) model which we employ the unsupervised object-centric image representation in our experiments. DLP is a Variational Auto-Encoder (VAE, Kingma & Welling (213)) with a structured latent space consisting of a set of latent vectors referred as particles. Particles encode local regions in the image that ideally correspond to salient factors such as objects or parts of objects. Each particle representation is comprised of the following attributes: pixel-space 2D position 2p E R scale of the box bounding the region represents 2,4 IR2 "depth" attribute used model occlusion be- tween particles E IR, transparency 2 E R and visual latent features that encode the appearance of the particle region 2g IR.. n denoting the per-particle visual latent dimension hyperparameter. A separate particle is allocated to encode the background of the image. The number of particles M a hyperparameter-wwin
has been chosen in previous work as well as this one-to upper-bound the number of entities of interest in the image. It is worth noting that DLP is entirely unsupervised and it is not guaranteed that each particle represent an object in the image nor that an object will be represented by a single particle. Allocating large number of particles increases the likelihood that all entities of interest are captured by the representation with the price of increased dimen- sionality and computational complexity both for DLP training and downstream decision-making. making. See Figures / 10 and II for DLP decompositions of images from the environments we use in our experiments.
16

---

Published as a conference paper at ICLR 2026
B ENVIRONMENTS, TASKS AND DATASETS
We provide detailed descriptions of the environments, tasks and datasets we use in this work for transparency and reproducibility purposes. We highlight the differences from the environments they are based on for the reader's 's convenience. We introduce novel variations of tasks from the OGBench benchmark (Park et al. 2025a) implemented with the MuJoco (Todorov et al. 2012) simulator and novel variant of the Push-T environment introduced in Chi et al. (2023). OGBench implementations com sechongpark/ ogbench). We base our implementation of PushT on a variant introduced in Zhou et al. (2025) using the paper's s official code found in: htttps: battp:://ithuu: github..hhugg. com/ go zhou/ dino wm We hope that these new variants, which highlight challenges less studied in popular benchmarks such as combinatorial state-spaces and compositional generalization, will facilitate further research in the field of entity- centric decision-making and beyond. Datasets and code including environment implementations and : : //sttes..........//...//eeeeete. google com/ wit/vview//////ee website for video demonstrations of the data collection policies.
are based on the official code found in: https::
data collection scripts can be found at pittps:
See our
PPP-Cube
Environment: based the OGBench Cube environment, the agent is required to manipulate cubes between randomly initialized state and goal positions in a 3D space. Distinct colors are assigned to each cube random out of a fixed set of 6 colors at the beginning of each episode. The modified multi-color setup effectively increases the size of the state-space and allows testing generalization increasing number of objects, and consequently longer horizons, without requiring generalization to new colors. Image observations include two views, front and side. To isolate algorithmic as- pects from any particular object-centric image representation, we support state-based entity-centric observations that emulate a "perfect" factored representation. Specifically, each entity's (i.e., robot gripper and cubes) state represented by its 3D position and yaw angle. The rohot arm's state includes gripper opening and the cube state is padded with 0 to match the robot's s state dimension. Each entity's s state is concatenated to a 1-hot identifier e; E {0, 13". en represents the agent and the rest represent the different colored cubes. Pixel observations are multi-view RGB images of dimension 128 x 128. Action space dimensionality is 5 and includes deltas in gripper 3D position, yaw and opening. A goal is considered reached when all cubes are within threshold distance from the specified position.
Dataset: the dataset contains diverse agent-object interactions including Pick, Place and Push oper- ations collected with a noisy scripted policy. The policy randomly selects a cube to either: (1) pick and place in a random location or (2) push from a random face. The dataset contains cubes and a total of 3M transitions, 7500 episodes each with 400 transitions.
Evaluation: Maximum evaluation episode length for 3 cubes is 1000 timesteps. We evaluate each agent with a checkpoint trained for 2.5M gradient steps on 100 randomly sampled initial state and goal configurations.
Differences from Original: (1) Pixel observations are multi-view RGB images of dimension 128 128 compared to OGBench which are single-view RGB images of dimension 64 x 64. (2) OGBench modifies the visual properties of the robot arm to be semi-transparent highlight the end- effector with purple color, which we remove in our version. (3) State observations are minimal compared OGhench and do not contain robot joint positions and velocities, gripper contact and object quaternions. (4) Each cube in our environments can be in 1 of colors (without repetition per instantiation) regardless of the number of cubes where in OGBench these are fixed and depend on the number of cubes in the environment. (5) The dataset collection policy in our setting is different, the major difference being the push operation. (6) Episode length in our dataset is 400 compared to 1000 in OGBench.
Stack-Cube
Environment: see PPP PPP-Cup -Cube.
Dataset: the dataset contains Pick-and- ank--ace operations with high probability of stacking collected by noisy scripted policy. The scripted policy includes recovery mechanism in cases it is in the processes of stacking a block and moves any of the lower blocks in the stack, which involves placing those blocks back in their position before attempting to stack the original one on top of them. This makes the dataset much more goal- goal-directed. We found this to be crucial for learning robust RL policies that achieve non-trivial performance on 3-cube stacking (especially in our evaluation setup that does not terminate upon reaching the goal). To the best of our knowledge, since the release of the OGBench benchmark, no method has achieved non-trivial performance 3-cube stacking
17

---

Published as a conference paper at ICLR 2026
with the OGBench dataset. Our dataset contains 3 cubes and total of 3M transitions, 3000 episodes each with 1000 transitions.
Evaluation: Maximum evaluation episode length for 3 cubes is 1000 timesteps. We evaluate each agent with a checkpoint trained for 3M gradient steps on 200 randomly sampled initial state and goal configurations.
Differences from Original: see (1)-(4 in PPP-Cube (5) The dataset collection policy in our setting is different as described above.
Scene
Environment: We adopt the OGBench Visual Scene environment that involves manipulation of var- ious object types (button, cube, drawer, window) in a 3D space. We make modifications to some test tasks to avoid ambiguity in goal specification that arises due to our more realistic visual setup where the robot arm is not transparent. Specifically, we modify tasks 4 and 5 in which goals involve images with the cube inside closed drawer. these cases, the agent cannot infer the desired location of the cube since it may be occluded. The goal is instead to place the cube in the drawer when it is opened and locked. Pixel observations are single-view RGB images of dimension 128 128, Action space dimensionality is 5 and includes deltas in gripper 3D position, yaw and opening. A goal is considered reached when all objects are within a threshold distance from the specified position. In Table we detail what each task involves, denoting subtask sequential dependencies with + and non-sequential subtasks with J-
Dataset: Data collection policy is identical to that of OGBench. A noisy scripted policy performs subtasks in random order based on their applicability at the given state (e.g., putting the cube in the drawer will only be selected if the drawer is open). The dataset contains a total of 1M transitions, 1000 episodes each with 1000 transitions.
Evaluation: Maximum evaluation episode length is 1000 timesteps. We evaluate each agent with a checkpoint trained for 1.5M gradient steps on 50 randomly perturbed initial state and goal configu- rations per task.
Differences from Original: see (1)-() in PPP-Cube. (3) We slightly modify task 4 and 5 for reasons described above.
Push-Tetris
Environment: we adapt the Push-T environment introduced in Chi et al. 2023) to multi-object manipulation of Tekris-like blocks. Distinct block types are sampled at random at the beginning of each episode. Each block type has a distinct color which is fixed across episodes. The agent is required to push the blocks to goal configurations including position and orientation in 2D space. Pixel observations are single-view RGB images of dimension 128 x 128. Action space dimensionality is and includes deltas in agent 2D position. A goal is considered reached when all objects reach over 85% state-goal pixel coverage. Obtaining this coverage requires very fine manipulation capabilities. We see that practice the agent able to learn from the sparse reward but achieves low success rates at test-time although end states are visually similar to the goal. We therefore report coverage as the main metric rather than success rate since it is more informative. Maximum evaluation episode length for 3 objects is 1000 timesteps.
Dataset: A dataset is collected using a random policy restricted to a fixed radius around an object, sampled at fixed time intervals, resulting in highly suboptimal behavior.
Evaluation: Maximum evaluation episode length for 3 objects is 1000 timesteps. We evaluate each agent with a checkpoint trained for 1M gradient steps on 100 randomly sampled initial state and goal configurations.
Differences from Original: (1) The original T-block is replaced with 7 different tetris-like blocks. (2) Our dataset contains semi-random interaction with objects in contrast to the original dataset that contains expert demonstrations. (3) The goal specified by a separate image in our setting compared to a shaded region in the same image as the state observation in the original task.
Table 5: Detailed Scene task descriptions. We denote subtask sequential dependencies with and non-sequential subtasks with J.
Task
Subtasks
Total Subuikks
Dependency
open dtane open window uslock draver in close drasgr - lock drawer unlock window close window io lock window open drawer| uuboc window " close window) move cute to side
open drswer - lock drower & place cabe in driwer
unlcc-dlaeer openedisee lockedawerr plux-ube--n--nmee unbock-winkoo + opan-wiulon lnckwwnddow
18

---

Published as a conference paper at ICLR 2026
B.1 ENVIRONMENT DEGREE OF DIFFICULTY
We highlight the complexity of the domains we consider in this work and the efficacy of our method in solving them by comparing concurrent work that requires orders of magnitude more data on similar or more simple variants of the OGBench (Park et al.] 2025a) Cube domains. [ et al. (2025) require 100M transitions in 4-cube environment to learn in single-task RL (i.e., not goal-conditiond) setting while learning from states and semi-sparse rewards (rewarding per-object success) and allowing additional online interaction. We exhibit non-trivial zero-shot generalization performance in manipulating 4 cubes in a fully offline goal-conditioned sparse-reward setting while learning from only 3M transitions containing 3 cubes, both from states and images. Park et al. (025b) show that unstructured RL methods, including hierarchical ones, require up to TB transi- tions in a similar setting to obtain non-trivial performance with more than 2 objects. To the best of our knowledge, based on review of all papers that have cited the OGBench benchmark at the time of writing this paper. there are no other methods that have attained non-trivial performance on the Cube domain with more than 2 objects thus far.
C METHOD IMPLEMENTATION DETAILS
In this section provide extensive implementation details for our method, baselines and image rep- resentations used in our experiments. Table compares methods with respect to various algorithmic aspects.
We implement all RL methods in pytorch based on official implementations and LeanRL: htt pss 7/qithub. com/ metaapyte or archhheenn more efficient version of CleanRL (Huang al. 2022]. The RL agents (i.e., all of the methods excluding EC-Diffuser) are based on goal-conditioned variants of IQL (Kostrikov al. 2022). Low-level policies extracted from IQI. Q-functions with DDPG+BC (Fujimoto & Gul [2221). Value function training goals are sampled uniformly from future states in the same trajectory with 0.8 probability and otherwise taken as the next state. Low-level policy training goals are sampled uniformly from future states in the same trajectory (except for in -peccfi- DDPG+BC
HIQL). We report shared hyperparameters in Table and policy extraction coefficient a in T
Table 6: Shared hyperparameters.
Hyperparameter
Value
Batch size
512
Learning rate
0.0003
Gradient clip norm
20
Discount factor y
0.99
Target smoothing coefficient
0.005
IQL/HIQL expectile
0.9
AWR temperature
3.0
Subgoal Diffuser diffusion steps
10
Subgoal K
50
EIT attention dimension
64
EIT attention heads
8
EIT hidden dimension
256
MI.P layers MI.P hidden dimension
4
512
EC-SGIQL (Ours): we implement our method on top of an ECRL (Haramati et al.) 2024) agent trained with IQL (see following EC-IQL. for details). For the Subgoal Diffuser we adapt EC- Diffuser (Qi et al. 2025) to generate subgoals conditioned on entity-centric states and goals. Specif- ically, we remove action inputs entirely and condition initial state and goal entities as clean inputs which are not denoised. We use learned additive embeddings to encode each individual en- tity's affiliation to either of the 3 sets (i.e., state, goal noisy subgoal) and one of 2 views in the multi-view setting. The Diffuser architecture is an S-layer Transformer which is conditioned on the diffusion timestep via adaptive layer normalization (AdaLN). Fivieonnmnt--peecfiic Diffuser
19

---

Published as a conference paper at ICLR 2026
Table 7: DDPG+BC policy extraction coeflicient
Environment
EC-SGIQL (Ours)
Ours AWR
EC-IQL
HIQL.
IQL
PPP-Cube (State)
0.1
0.1
0.1
0.1
PPP-Cube (Jmage)
0.2
0.2
0.2
0.3
0.2
Stack-Cube (Siate)
0.05
0.05
0.05
0.2
0.05
Scene (Image)
0.3
0.3
0.3
0.4
0.2
Push Push-Tetris (Image)
0.1
0.1
0.1
0.1
0.1
test-time hyperparameters are detailed in Table 8 For our implementation we adapt code from the official EC-Diffuser repository:
Table 8: Subgoal Diffuser test-time hyperparameters.
Environment
Diffusion Samples N
Subgoal Steps Tag
Competence Radius i
PPP-Cube (Siate)
256
25
(Image)
256
25
-30
Stack Cube (State)
256
25
-30
Scene (Image)
64
50
-25
Push-Tet ris (Image)
64
25
-20
EC-SGIQL with AWR: This method is an ablation of our method and is thus identical to it ex- cept for the subgoal policy which is a Transformer based on the Entity Interaction Transformer (EIT) (Haramati et al. 2024) which replaces the final aggregation attention with another self atten- tion block. trained with an AWR objective using the IQL. value function. Since the output and target are unordered entities when learning from DLP particles, we use the Chamfer distance as the loss instead the standard MSE.
corresponds to ECRL (Haramati et al. 2024) agent adapted to the offline setting by integrating the Entity Interaction Transformer (EIT) architecture with IQI.. We slightly modify the EIT architecture for the Q-function by replacing the action entity conditioning with AdaLN condi- tioning. We base our implementation on the official ECRL repository: https:: /ttp:/gggtttb/tt// com/
EC-IQL:
DanHrmt itttt/itiERRII /ECRLL
this baseline corresponds to an agent based on HIQL (Pak et al. 2023). We adapt the official implementation to make it similar to the other methods for fair comparison while keeping the core attributes unchanged. Specifically, we train it with an underlying IQL agent and extract the low-level policy with DDPG+BC. Contrary the other IQL-based methods in this work, in the original HIQL, we train the low-level policy on the subgoal distribution rather than the goal distribution. The high-level policy is trained with AWR using the IQL value function. At test-time, the subgoal is kept for the same duration as the Diffuser subgoals (see Table S All agent components are 4- layer MLPs with hidden dimension of 512. We base our implementation the official OGBench implementation at: https: //t/t//// hub. com/ wwwhongpark. http:/ppppa..kkbbbcch.cccocccccc ogbench,
HIQL:
IQL: this baseline corresponds to a goal-conditioned agent based on IQL (Kostrikov et al. 2022).
All agent components are 4-layer MLPss with a hidden dimension of 512. We base our implemen- tation on the official OGBench implementation at: hittps://gtt thub. com/ haem///eohoggpark ogbench
EC-Diffuser (Qi ct al. 2025); is an entity-centric diffusion-based behavioral cloning method. employs a Transformer-baae based architecture that takes as input the current state and goal state and predicts future states and actions via diffusion. For control, it predicts future states and actions jointly, and executes the first denoised action in an MPC fashion. For all of our tasks, diction horizon of 5 and 10 diffusion steps. We use the reported value for all other hyperparameters. We the official implementation from: httpt::///// /ttpps://gtthbb cwm/ carm//carl-aaa Dtw.mm/aaaaaaaeccdiaaaaa-
pre-
20

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of algorithmic aspects across different methods.
Attribute
EC-SGIQL (Ours)
Ours AWR
EC-IQL
EC-Diffuser
HIQL
IQL
Entity-centric
/
/
x
x
Hierarchical
/
/
Obs. Diffusion
X
X
Image Representations: We provide hyperparameters for DLPv2 (Daniel & Tamar 2024) and VQ-VAE (Van Den Oord et al. 2017) models in Tables T0 and 11 respectively. We use the imple-
mentations provided in: https: httppa://gttttii /ttggtttthhuuu.// wtt.c/mmm/aaanaaa ECRL
Table 10: DLP hyperparameters.
Environment
Prior Particles
Posterior Particles
Visual Feature Dim
BG Particle Dim
PPP-Cube (magee)
32 (per view)
20 (per view)
Scene (Image)
32
24
Push-Tetris (Image)
32
20
Table 11: VQ-VAE hyperparameters.
Hyperparameter
Value
Embedding dimension
16
Dictionary size
2048
Flattened latent dimension
1024
D ADDITIONAL RESULTS AND DISCUSSION
D.1 WORLD MODEL PERSPECTIVE
Our approach has strong connections model-based RL (Sutton 1991) and world modeling (Ha & Schmidhuber 2018). Whereas standard world models model the next-state distribution given the current state and action p (8t+118t, at), our approach models the multi-step next-state distribution given the current state and desired goal (S++ ())),,)) This modeling choice facilitates long- horizon goal-reaching by planning in the temporally-extended state space rather than low-level action space. This in turn allows longer look-ahead planning horizons (K = 50 in our experiments) compared to autoregressive world models (typically around K = 3) which suffer from compounding model errors. Compared to diffusion world models (i.e., diffusers). on accurately capturing single distant states rather than multiple immediate state-action pairs. This enables generation of high quality subgoals which can be fed to the low-level RL policy although it. was never trained on the diffusion- -eenrrrn-eee subgoal states, only on dataset states.
focus the modeling capacity
D.2 THE BENEFITS OF DECOUPLING TRAINING GOAL DISTRIBUTIONS ACROSS THE HIERARCHY
In our goal-conditioned setting, the value function can be viewed as discounted (asymmetric) dis- tance metric between states (Wang et al. (023). The compounding error and discount factor define an "effective radius" of values that provide clear learning signal for policy extraction, which we refer as the policy competence radius, R (see Sec. This quantity is not known apriori because we do not have access to the true value approximation error, and it not clear if it can be recovered or effectively approximated in practice without evaluating an extracted policy. Previous hierarchical methods spanning offline RL (Park et al. 2023) and diffusion planning (Li et al. 2023 Chen al. 2024) fix a single hyperparameter K defining the training distributions for both levels of the hier- archy: the low-level policy is trained state-goal pairs that most K timesteps apart while
21

---

Published as a conference paper at ICLR 2026
the high-level policy is trained produce these stetep states subgoals. Selecting K represents tradeoff between complexity and sample efficiency: small K simplifies the policy goal distribution but makes less of the offline data by limiting the state-goal sotteggoal pairs the policies trained on. po- tentially hindering generalization. Selecting thus requires simultaneously balancing this tradeoff across all levels of the hierarchy.
We opt for an alternative approach: train the low-level policy on the full goal distribution, i.e., the same distribution the value function trained on. This choice decouples the training goal depen- dency between the hierarchies and allows to implicitly select K for the low-level policy by tuning hyperparameter N as test-time constraint (details in Sec. 3.2). which is preferable to tuning K since it does not require re-training. K can then be chosen more flexibly for the subgoal generator to upper-bound the low-level policy's competence R. while balancing the complexity tradeoff.
D.3 LONG-HORIZON OBJECT MANIPULATION
Intermediate success (e.g.. the number of subtasks achieved) reported Table 12 and detailed results Scene tasks in Table 13 EC-IQL performs on par with method most tasks in Scene and is the second leading method in terms of performance overall. We believe this is due to the entity-centric structure which helps learn accurate, internally factored value functions. Scene is the least combinatorially complex which may explain why the hierarchy provides less benefit: the factored value function provides a good learning signal in this domain. EC-Diffuser performs on par with our method on stack
We believe this is mostly due to the nature of the dataset which more goal-directed compared to the others, and the fact that does not rely on a value function and thus does not suffer from the same long-horizon challenges as the RI. methods. HIQL and IQL do not scale well to image- tasks with multiple objects when learning from single-vector representations, which is consistent with previous findings the object-centric pocision-making literature as well as the results on the OGBench benchmark (Park et al.] (2025a). HIQL even under- performs IQL in AWR often results in low quality subgoals (see Sec. and D.6)
cases, which believe is due to the fact that training the subgoal policy with
Table 12: Long-horizon manipulation performance. All values are success fractions except for
Push-Tet ris for which report state-goal pixel Chamfer distance. Best results up standard deviation are highlighted in bold.
Environments
EC-SGIQL
EC-IQL
EC-Diffuser
HIQL
IQL
PPP-Cube (State)
88.0 i 1.4
60.8 4.5
66.5 7.0
62.8 i
46.5 i: 7.0
PPP-Cube (Jmage)
82.3 1: 2.2
60.8 +: 1.5
9.5 113
0.0 1:00
0.0 1: 0.0
Stack Cube (State
58.8 + 1.7
43.0 + 4.6
55.3 6.
6.0 008
38.5 = 1.0
Scene (Image)
90.0 2.2
84.0 2.7
51.5 006
65.3 1.7
77.3 0.5
Push-Tetrie (Image)
19.0 + 3.5
54.7 + 4.9
84.2 4
94.1 0.9
98.8 = 1.8
Table 13: Detailed success rates for Scene tasks. Best results up to a standard deviation are highlighted in bold.
Task
EC-SGIQL
EC-IQL
EC-Diffuser
HIQL
IQL
85.0 1 6.0
83,5 :: 444
14.0 i: 14.3
20.0 5.2.
27.0 :: 4.8
83.5
89.0 = 20
6.0 6.0
15.0 +48
30.5 16.7
52.0 7.1
16.5 13.4
0.0 +
7.0 2.6
15.0 488
43.0 10.5
39.0 1599
0.5 1
0.5+1.
6.0
5
44.0 17.2
37.0 7.8
0.0 0.0
0.0 0.0
10.0 7.1
D.4 GENERALIZATION
Compositional generalization results of the entity-centric methods we compare in this work are presented in Table [4] Maximum evaluation episode length in the generalization experiments with 4, 5, and 7 objects is 1200, 1500, 2000 and 2000 respectively. We additionally increase the number of inference particles feature supported by DL.P M.P.- to 24, 28, 30 and 30 respectively.
22

---

Published as a conference paper at ICLR 2026
Table 14: Zero-shot compositional generalization. All values are success rates except for Push-Tetris ris variants for which we report state-goal pixel coverage, signifies the training variant each segment, which was used to evaluate zero-shot capabilities on the other variants. Best results up standard deviation are highlighted in bold.
Variants
EC-SGIQL
EC-IQL
BC-Diffuser
2 CCubes-Staae (State)
69.8 7000
45.8 # 12.3
87.8 = 54
3 (CCuueeeStttcii (State)
43.5
29.0 299
43,8 9.2
4-Cubes-Stack- (Stae)
28.8 3.9
17.3 2.25
1.5 i 1.3
-1--le (State)
97.0 + 1.8
92.3 8.9
85.0 9.2
PPF-2-Cubes (State)
94.0 3.4
75.0 3.8
76.5 7.6
PPP-3-Cubes (State)
82.5 3.1
51.5 444
44.8 6.7
PPP-4-Cubes (Stare)
65.3 3.5
31.8 4.40
42.8 = 11.4
PPP-5-Cubes (State)
49.0 8.5
23.8 6.7
PPP-6-Cubes (Stam)
25.7 + 1.5
10.5 6.1
8.8 +
PPP-1-Cube (Image)
94,5 2.9
91.5 + 1.3
3.8 + 1.3
PPP-2-Cubes (Image)
77.0 3.36
47.8 6.2
0.0
PPP-3-Cubes" (Image)
64.3 6.0
25.0 557
0.3 0
PPP-4-Cubes (Image)
38.3
11.5
0.0
PPP-5-Cubes (mmuge)
19.3 6.2
4.0 2:00
0.0 1 0.0
PPF-6-Cubes (muge))
9,5 + 1.3
1.5
0.0 0.0
Push-Tetria-110b3 (Image)
8.0 + 1.5
6.6 2.6
5.0 = 1.8
Po jects (Image)
61.4 5.7
40.8 4:22
6.3 1
Push-Tettiis-3-0bjee (Image)
61.4 3.3
31.6 13
7.9 + 0.5
Push-Teetrrs--400beee
(Image)
52.4 3.3.
20.7
6.4
Push-Tetrrss--003 (Image)
41.8 6.0
14.4
4.6 1 0.6
Push-Tetris 6 Objects (Image)
+.80
11.5
4.6
Push Mash-Tetris -0pp--bjjcttt (Image)
21.1 21.1
10.0 +15
4.7 +.8
D.5 SUBGOAL DIFFUSER HYPERPARAMETER SENSITIVITY STUDY
We test the sensitivity of our method the subgoal-related hyperparameters K, T.g and N on PPP-Cube (Imaee) the most challenging task in our suite. Results are presented in Fig. 5 We find that the method is largely robust to these hyperparameters, significantly outperforming the underly- ing RI agent with all of the tested configurations.
K (Fig. 5] left): results align with our intuition that filtering subgoals for reachability makes our method robust to K as long as it upper-bounds the policy competence radius R (see discussion in App. D.2). Performance degrades when
10, hinting that the competence radius larger than
this value.
Tag (Fig. center): Larger values incur faster average action execution time since they require less diffusion generations but may result in decreased decision-timent timestep efficiency if the value is much larger than what the RL policy requires to reach the subgoal. In addition, given that the sampled subgoals not perfect, regenerating subgoals reasonable rate benefit the agent. This is analogous to the re-planning performed in MPC. We see that for larger values, performance begins degrade.
N (Fig. []] right): small sample sizes decrease the probability of sampling "optimal" subgoals which results in high variance in performance. Large sample sizes increase the probability of generating erroneously high-valued subgoals which can degrade performance. Moderate sample sizes between 64 256 result equivalently high success rates.
D.6 AWR SUBGOAL VISUALIZATION
We provide visualizations of subgoals generated by a deterministic entity-centric Transformer trained with AWR DLP representations in Figures 6 and g We observe that across all en- vironments, subgoals often contain duplications of entities, providing ambiguous subgoals for the low-level GCRL policy. This duplication be explained by the redundancy in the DL.P represen-
23

---

Published as a conference paper at ICLR 2026
Sucoess Rate
Figure 5: Hyperparameter sensitivity study. Success rates of our method on PPP-Cube (Image)- A single hyperparameter is varied in each plot while the others are unchanged compared to the ones used for the main results, which we refer to as default. Purple stars default values, error bars standard deviations, dashed lines and shaded regions in green mean and standard deviation of the EC-IQL baseline without subgoals. Left: varying K, the Subgoal Diffuser training distribution timestep difference between state and subgoal. Center: varying Tay. the test-time subgoal rollout timesteps. Right: varying N, the number of test-time diffusion subgoals sampled before filtering.
tation and the AWR objective. As discussed in Sec. 4.5 weighted regression fits weighted average of future observations, which in our "play" datasets, contain many possible futures given initial state and goal. DLP represents a scene with many particles, which due to its unsupervised nature, may represent the same object with multiple particles (see App. A.3) This allows each particle to capture different future subgoal for the same object single subgoal, which is what we observe in these figures. The subgoal averaging phenomenon is not purely artifact of the redundancy in the DI.P representation given that it occurs in the state-based environments as well (see Sec. 4.5 Ta- ble These results highlight the multi-modality in the data and help explain the lower performance of baselines and ablations using AWR to train the subgoal generator. This phenomenon rarely hap- pens with our Subgoal Diffuser (and is much less severe when does), which captures the multiple modes in the data and enables separately sampling from them at test-time. We believe that larger diffusion Transformers may help mitigate this phenomenon entirely.
Goal
Generated Subgoal
State
Figure PPP-Cube AWR subgoals. Subgoals (middle) are reconstructed with the DLP de- coder and were generated conditioned on DLP representations of the state (bottom) and goal (top). Columns are not sequential, i.e., each column represents unrelated subgoals.
D.7 PRETRAINED IMAGE REPRESENTATION RECONSTRUCTION
24

---

Published as a conference paper at ICLR 2026
Goal
Generated Subgoal
hdb
State
Figure 7: Scere AWR subgoals. Subgoals (middle) are reconstructed with the DLP decoder and were generated conditioned on DLP representations of the state (bottom) and goal (top). Columns sequential, i.c., each column represents unrelated subgoals.
Goal
Generated Subgoal
State
7
Figure 8: Push-Tetris AWR subgoals. Subgoals (middle) reconstructed with the DLP decoder and were generated conditioned on DLP representations the state (bottom) and goal (top). Columns are not sequential, i.c., each column represents unrelated subgoals.
25

---

Published as a conference paper at ICLR 2026
Original Image
Partiele Positions
Full Reconstruction
Foreground
Reconstruction
Background
Reconstruction
Figure 9: PPP-Cube DLP decomposition.
Original Image
Particle Positions
Full
Reconstruction
Foreground
Reconstruction
Background
Reconstruction
Figure 10: Scene DLP decomposition.
26

---

Published as a conference paper at ICLR 2026
Original Image
Particle Positions
Full Reconstruction
Foreground
Reconstruction
Background Reconstruction
Figure 11: Teth- DLP decomposition.
Original Image
Reconstruction
Figure 12: PPP-Cube VQ-VAE reconstruction.
Original
Image
Reconstruction
Figure 13: Scene VQ-VAE reconstruction.
Original
Image
Reconstruction
Figure 14: Push-Tetris VQ-VAE reconstruction.
27


Reviews:
[{"review_id": "5v2mBsYGSP", "reviewer": "Reviewer_m4Xk", "summary": "This work proposes a two part hierarchical reinforcement learning framework that targets environments with multiple entities and sparse reward. Their framework (HECRL) is composed of 1. Subgoal diffuser: a high-level diffusion model that generates immediate subgoal, and 2. a value based Goal Conditioned Reinforcement Learning agent that does test-time subgoal selection. \n\nThe work tackles non-stationarity and instability in off-policy hierarchical RL (the issue of reward error propagation in long horizon tasks) by producing immediate subgoal that is reachable and provides clear learning signal to the policy learning agent. This is done by enforcing a value based reachability constraint at test time. \n\nExperiments on several continuous-control benchmarks (Reacher, Pusher, Point Maze, Ant Maze variants, Ant Fall) show improved sample efficiency and final success rates over strong HRL baselines such as HIRO, HRAC, HIGL, SAGA, and HLPS, as well as smaller gaps between generated and actually reached subgoals, and ablations indicate that both diffusion and GP components contribute to the gains", "questions": "1. Lack of a clear high-level framework diagram.\nThe paper could include a high-level illustration of the framework to show how the different components (high-level policy, diffusion model, GP module) interact and how information flows between them. At present, the description is quite text-heavy and the existing figures are not sufficiently clear to understand how the components add together. Having a high level figure of the framework helps reader understand how the framework builds on top of existing frameworks and make the novel parts of the framework more explicit, which is currently difficult to see from the text-heavy description. \n\n2. Insufficiently clear description of ablation variants.\nThe paper could explain more clearly what each ablation variant changes and how it corresponds to the conceptual components of the method (pure diffusion, GP-only, hybrid selection).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Originality: The framework is modular, compatible with various value-based GCRL algorithms. Existing hierarchical diffusers diffuse over global subgoals without explicit entity factorization. EC-Diffuser uses entity-centric diffusion but for behavior cloning rather than subgoal generation. Hence, the proposed combination of conditional diffusion model over entity-factored subgoals with a value-based GCRL agent  is novel. The paper also make it clear that their framework builds directly on two mature lines of work — hierarchical diffusion for subgoals and entity-centric RL, and that their contribution is meaningful but compositional, rather than a completely new paradigm. \n\n2. Significance: The contribution is more than a minor architectural tweak. Using GP prior over the high-level mapping as a separate surrogate distribution provides explicit predictive means and variances over subgoals: this defines a hybrid selection rule that mixes diffusion-sampled subgoals with the GP mean. The paper also shows theoretical justification beyond heuristics. \n\n3. Quality: The benchmarks used includes a reasonably broad suite of continuous-control HRL benchmarks using both sparse and dense rewards, and compared against several strong and closely related HRL baselines (HIRO, HRAC, HIGL, SAGA, HLPS). The learning curves show consistent gains. Additional metrics such as distances between generated and reached subgoals are also reported. \n\n4. Clarity: The related work section clearly states prior subgoal methods and how their framework relate to them (e.g. HIRO, HRAC, HIGL),  making clear that this paper moves from deterministic or adversarial subgoal generators and GP-only models to a more expressive diffusion-GP hybrid with explicit uncertainty-guided selection. Compared to the authors’ previous work, the framework in this paper changes both the generative family (diffusion instead of adversarial or simple parametric models) and the way uncertainty is integrated (GP acting both as regularizer and as a competing proposal). The experimental results demonstrate stronger performance on harder tasks.", "weaknesses": "1. Figure quality and readability:\nThe figures are difficult to read in their current form. In Figure 1, the circles and arrows are thin and low-contrast, and the text labels are very small and faint. As a result, it is hard to discern the structure and details of the illustration. Given that this is the first figure of the paper and is meant to convey the main idea, it should be redesigned with larger fonts, clearer icons, and higher contrast to make the diagram easy to understand for the readers. \n\n2. Insufficient discussion of sensitivity analysis: \nThe framework in this paper is much more complex than most baselines (e.g., HIRO, HRAC, HIGL). There should be a more discussion on the complexity, overhead and practicality measurement such as sensitivity to replay-buffer size and number of inducing points and memory usage to discuss ability/limitation to scale to larger benchmarks. Moreover, the paper could elaborate on the sensitivity analysis of the number of choices HIDI depends on: the HIRO-style relabeling scheme, the GP's kernel form and hyperparameters, number and placement of inducing points, and the mixing probability/schedule between diffusion-sampled subgoals and GP mean. \n\n3. Limitation in the complexity of experiments:\nAll reported experiments seem to be on relatively low-dimensional MuJoCo-style state spaces (Reacher, Pusher, maze-like Ant tasks, Ant Fall). The evaluation could include more pixel-based or complex multi-entity/object-centric environments to demonstrate more clearly whether the additional complexity is strictly necessary and meaningful in those settings, and show how well the GP prior and sparse-GP approximation scale with dimensionality. \n\n4. Limited practical impact of the theoretical guarantees: \nThe proofs are built upon rather ideal assumptions: that the diffusion subgoal policy to be already “near-optimal” and certain regularity conditions holds. Under these assumptions, the paper only proves single-step regret and single-step policy-improvement bounds. However, it does not extend or discuss these guarantees in long-horizon setting, where performance depends on multiple sequential decisions made under function-approximation error, stochastic optimization, and model mismatch. There are no results or discussion on cumulative regret, stability over time, or convergence of the overall learning process. Although the mathematical analysis is helpful to build intuition and to justify the framework’s design, it is insufficient to provide strong guarantees about the behavior of the full end-to-end hierarchical system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2aWpxXLb1P", "reviewer": "Reviewer_q4dk", "summary": "This paper presents Hierarchical Entity-Centric RL (HECRL), composed of a value-based GCRL and a factored subgoal-generating diffusion model to address long-horizon tasks in domains with multiple entities. In addition, the authors introduce a new version of benchmark problems that highlight the challenges of entity-centric tasks.", "questions": "**Questions**\n\nQ. What is the major difference between the proposed model and EC-diffuser?\n\nQ. What is the key difference from the previously proposed method, which utilizes a diffusion model to generate a subgoal? Is it modularity that enables test-time subgoal selection with the value function?\n\nQ. Sensitivity study with respect to $K$ and $T_{sg}$ is missing.\n\nQ. What is the meaning of boldface numbers in Table 2? There are multiple numbers of boldface numbers. The essential explanation seems to be missing or at least not highlighted enough to check. \n\n**Minor Comments**\n\nC. An explicit definition of the entity-centric environment would be helpful.\n\nC. In Figure 1, DLP is presented without a proper definition. Please at least use Deep Latent Particles (DLP) \\cite{xx}.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well written, well-constructed and easy to follow. In particular, its pictorial illustration, such as Figure 2, was very helpful in understanding the overall idea. The proposed method effectively addressed challenging long-horizon multiple-entity tasks.", "weaknesses": "The proposed components are based on existing methodologies, such as entity-factored subgoals and subgoal diffuser, so the novelty of the approach itself may be limited. However, the meaningful combination of these components has yielded strong performance. Please see others in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnG52cMaBM", "reviewer": "Reviewer_K6LA", "summary": "This paper proposes an offline Goal-Conditioned Reinforcement Learning (GCRL) framework to enhance learning efficiency and generalization in long-horizon goal-reaching tasks within complex environments featuring multiple entities. The authors employ a hierarchical structure comprising a diffusion model for generating entity-factored subgoals and a value-based GCRL agent.", "questions": "1. As mentioned in the Weakness section, an ablation study on subgoal filtering using value functions would be helpful. It would be useful to compare scenarios where subgoals generated by diffusion are used directly, or where multiple subgoal candidates are randomly sampled and then filtered, against the current selection method.\n2. I am curious about the impact of hyperparameters K and N, as described in Section 3.2, on performance.\n3. The algorithm indicates that the current subgoal g' is an input, but this is not explicitly addressed in the description. What role does the current subgoal play?\n4. Is it feasible to compare against a baseline that directly applies the EC-IQL method to HIQL?\n5. According to Table 1, HIQL achieves a 0 success rate in PPP-Cube (image) and Stack-Cube (state). What explains this lower performance compared to the cube environment in OGBench? Is it due to the increased number of cubes that need to be manipulated?\n6. Why is only the Stack-Cube (state) environment included, while the Stack-Cube (image) version is omitted?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "While hierarchical structures have been explored in prior work, utilizing diffusion models to generate subgoals that incorporate information from multiple entities stands out as a novel approach. Additionally, modifying the experimental environments to highlight the advantages of the proposed method and comparing it against baselines strengthens the paper's contributions.", "weaknesses": "The paper would benefit from more detailed explanations of model design choices, along with ablation studies to justify them. Although the use of diffusion for subgoal generation is innovative, it remains unclear whether this component is essential for the performance gains. There is a concern that filtering subgoals—generated via the unused value function during diffusion model training—might play a more critical role than the diffusion process itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNmq9K9udl", "reviewer": "Reviewer_pQXN", "summary": "This paper has proposed a novel hierarchical reinforcement learning approach, where the subgoals are entity-centric. In the proposed approach, the policies are implemented with the diffusion networks. The experiment results demonstrate that the proposed approach significantly outperforms the state-of-the-art hierarchical baselines. Beyond that, the entity-centric subgoals improve the explainability of the learned policies.", "questions": "1.\tWhat is the difference between entities and objects?\n2.\tDoes the paper assume that the collection of entities in Line 129 is given before learning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe hierarchical learning problem is of great importance in the RL domain, which is promising for solving the long-horizon problems in real-world applications.\n\n2.\tThe proposed approach is extensively evaluated in the simulated robotic manipulation domain and has demonstrated impressive experimental performance.", "weaknesses": "1.\tFigure 2 is confusing. The goals are represented with small squares. It is unclear what the difference is between the factored squares and the unfactored squares, and even how they influence the learning process.\n\n2.\tThe authors are encouraged to show more about the learned subgoals, and the goal-conditioned policies. It is unclear what goal-conditioned policies are learned which leads to the superior task performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)