3D-aware Disentangled Representation for Compositional Reinforcement Learning
Authors: Sungbin Mun
Younghwan Lee
Cheol-Hui Min
Mineui Hong
Young Min Kim
Venue: ICLR 2026 Poster (2026)
Keywords: object-centric learning
compositional generalization
goal-conditioned reinforcement learning
Primary area: reinforcement learning

Abstract: Vision-based reinforcement learning can benefit from object-centric scene representation, which factorizes the visual observation into individual objects and their attributes, such as color, shape, size, and position.
While such object-centric representations can extract components that generalize well for various multi-object manipulation tasks, they are prone to issues with occlusions and 3D ambiguity of object properties due to their reliance on single-view 2D image features. 
Furthermore, the entanglement between object configurations and camera poses complicates the object-centric disentanglement in 3D, leading to poor 3D reasoning by the agent in vision-based reinforcement learning applications.
To address the lack of 3D awareness and the object-camera entanglement problem, we propose an enhanced 3D object-centric representation that utilizes multi-view 3D features and enforces more explicit 3D-aware disentanglement.
The enhancement is based on the integration of the recent success of multi-view Transformer and the prototypical representation learning among the object-centric representations.
The representation, therefore, can stably identify proxies of 3D positions of individual objects along with their semantic and physical properties, exhibiting excellent interpretability and controllability.
Then, our proposed block transformer policy effectively performs novel tasks by assembling desired properties adaptive to the new goal states, even when provided with unseen viewpoints at test time.
We demonstrate that our 3D-aware block representation is scalable to compose diverse novel scenes and enjoys superior performance in out-of-distribution tasks with multi-object manipulations under both seen and unseen viewpoints compared to existing methods.

Published as a conference paper at ICLR 2026
AWARE DISENTANGLED REPRESENTATION FOR COMPOSITIONAL REINFORCEMENT LEARNING
Sungbin Mun, 'Yonnghaan Lee, CCheul-Hui Min, Minnui Hong, 'Young Min Kim
Scoul National University SSamsnng Electronics *Camnegie Mellon University (brian0429, frredy99, youngmin. kim @ snu. ac. kr
cheolhui minesamsung com mineuih@andrew. cmu.edu
ABSTRACT
Vision-based reinforcement learning can benefit from object-centric scene repre- sentation, which factorizes the visual observation into individual objects and their attributes, such as color, shape, size, and position. While such object-centric rep- resentations extract components that generalize well for various multi-object manipulation tasks, they are prone to issues with occlusions and 3D ambiguity of object properties due to their reliance on single-view 2D image features. Further- more, the entanglement between object configurations and camera poses com- plicates the object- disentanglement in 3D, leading to poor 3D reasoning by the agent in vision-based reinforcement learning applications. To address the lack of 3D awareness and the object-camera entanglement problem, we propose enhanced 3D object-centric representation that utilizes multi-view 3D features and enforces more explicit 3D-aware disentanglement. The enhancement is based the integration of the recent success of multi-view Transformer and the pro- totypccal representation learning among the object-centric representations. The representation, therefore, can stably identify proxies of 3D positions of individ- ual objects along with their semantic and physical properties, exhibiting excellent interpretability and controllability. Then, our proposed block transformer policy effectively performs novel tasks by assembling desired properties adaptive to the new goal states, even when provided with unseen viewpoints at time. We demonstrate that our 3l--awrre block representation is scalable to compose di- verse novel scenes and enjoys superior performance in out-of-distribution tasks with multi-ohject manipulations under both seen and unseen viewpoints compared to existing methods.
INTRODUCTION
Vision-based reinforcement learning utilizes images as input modality, enabling flexible, human-like perception for robotic tasks, compared to specific sensory or motor configurations. However, the unstructured nature of observations further challenges the inherent inefficiency in reinforcement learning (RL.. As the visual observations from various viewpoints result from a complex interplay between color, lighting, geometry and their non-linear projection, the images do not easily map to a coherent composition of meaningful attributes, which challenges their generaliz- ability and controllability. While visual perception module can benefit from recent advancements in computer vision literature, the feature-level abstractions are often high- nigh-dimensional and are no longer interpretable by humans.
Systematic factorization of visual observation can lead to generalizable perceptual reasoning, one of the desired benefits of vision-based RL (Yoon et al. 2023). Object-centric representations promote component- wise analysis, where individual objects are detached from the background. Furthermore, learning factor-level representations like Neural Systematic Binder (Singh et al.) 2022) deduces the
spatial locations and visual attributes, such as colors and shapes, from which humans can interpret without additional training. Thus, there have been studies that exploit object-centric representation state representation for agent control (Driess et al.) 2023] Haramati al.) 2024 Min & Kim 2025). which has been shown to improve both the task solving performance of vision-bass RL agents and the efficiency of the training sample.

---

Published as a conference paper at ICLR 2026
However, there remain two key challenges that prevent the agent from fully utilizing object-centric representations: 1) insufficient aw-awrressss and 2) imprecise object description. First, the insuffi- cient 3D-awarensss stems from missing multi-view attention and 2D UV UV-grid-based decoding, and in practice many slot-attention based methods operate on a single 2D view or feature grid, so they cannot reliably infer depth, occlusion, multi-views consistency, full 3D spatial object poses (Chen ct al., [222]] Biza et al., 2023] Min & Kim 2025). This limitation reduces their applicability in interactive environments with occlusions and viewpoint changes. Second, due the unsupervised learning process, the object variables remain ambiguous in terms of the agent's 's manipulability, be- cause slot representations often correspond to clusters in 2D feature maps (e.g., color, texture, or region) and do not guarantee that each slot encodes distinct physically manipulable object, its pose, its affordances, or its dynamic interactions (Locatello et al. 2020 Kim el. 2023 Kori al., 2024 Rezazadeh et al. 2024).
To address these challenges, we propose a novel end-to-end neural network architecture called 3D block-slot representation, providing 3D-aware and interpretable state estimation to facilitate vision-based RL tasks. Specifically, we adapt the block-slot concept from the attribute-level struc- tural factorization (Singh al., 2022) to facilitate explicit yet unsupervised object-centric decom- position. Further, to achieve 3D-awareness, we lift the decomposed attributes into 3D space using a light-field decoder (Sajjadi et al. 2022a b) Smith et al. 2022 Yu al. 2021 Stelzner al. 2021 Qi et al.] 2023; Luo et al. 2024 Liu et al. 2024), enabling camera view-object 3D position dis- introduce block transformer policy that significantly improves the agent's 's generalization to novel object configurations. By recognizing matching properties between the current and goal states, the policy effectively reduces the search space for goal-cot conl-conditooned planning.
entanglement. Building on our proposed representation,
To sum up. our main contributions are as follows:
We propose novel 3D block-slot attention mechanism that enables the construction of view-independent 3D structured representations by disentangling object attributes such as shape, color, size, and position.
In the 3D block-slot attention, we design a novel slot attention scheme that applies block- slot attention to object slots, while using vanilla slot attention for background and agent slots, enabling consistent and disentangled representations across different scene compo- nents.
We develop a block transformer policy that performs block-wise cross-attention between current and goal states, enabling precise goal conditioning object attributes.
Our experiments on attribute-level object manipulation confirm that the 3D block-slot rep- resentation is effective for structured and generalizable coal-conditioned RL, providing 3D-aware features that make the policy hiew-agnostic and robust to unseen viewpoints.
METHOD
We extract 3D-aware representation that decomposed into interpretable components with blocks, such that it efficiently generalize various goal-driven tasks with dision-basee RL. We first describe the encoder architecture with disentangled components (Section 2.J) followed by the trans- former policy architecture (Section 2.2h. The overall architecture is summarized in Figure]
2.1 3D BLOCK-SLOW REPRESENTATION
BLOCK-SLLTRRRREEETTTTTI
Our latent embedding encodes a consistent, disentangled representation that can match visual obser- vations in arbitrary viewpoint, thus being robust to occlusion or perspective distortion. We build upon the 3D object scene representation transformer (OSRT) (Sajjadi et al. 2022a),, which disentan- gles individual instances and further identifies the active and passive components in fulfilling robotic tasks. The additional block-slot attention module (Singh et al. 2022) builds a strong structure with decomposed factors that associated with various attributes of the objects.
2

---

Published as a conference paper at ICLR 2026
3D Block-Slot Encoder
Query Rays
Current Observations
Goal Observations
Current
Action
Zig
[ACT
Slot-Misee Mixer Docoder
SRT Encoder
Zag
3D Block-Stot Encoder
3D Block-Slol Envoder
Novel View
Block Transformer Policy
Multi-View
Slot BBoook
Slot Attention
Output
Images
Block-Slot Attention
(Action, Q-value)
Stage I: Pre-training block-slot encoder
Stage Policy training with block transformer policy
Figure 1: Overall structure of our method: Our proposed pipeline consists of two steps: represen-
tation learning and policy training. (a) Pre-training 3D block-slot encoder: The object slots further decomposed into blocks attributes. Then, the slot-mixer decoder mixes the object-centric representation to generate images at a query view. (b) Policy training with block transformer policy: We utilize the 3D block-slot encoder to extract structured representation for the current observation and the goal image. The decomposed latent embedding serves as the input and the goal tokens, respectively, for our block transformer of the policy architecture.
2.1.1 PRELIMINARY: 3D OBJECT-CENTRIC LEARNING
We adopt a transformer-based SRT encoder (Ea) (Sajjadi et al. 2022b to aggregate multi-view observations (I, F Hx and generate scene-level latent representation F RLXD.
F Eg ((I.))
The slot attention mechanism (Locatello al. 2020) is applied to the scene feature producing a set of slots {z., E R. _ where each slot represents an individual object. Specifically, the mechanism observes the image latent embedding R Nx Das, where each row E R Pas of the matrix U stores values to update of the slot matrix Nu Our framework augments the slot attention mechanism reflect our additional factorization, which is described in Section 2.1.3
to produce an attention readout matrix U
The decoder employs the Slot Mixer (Sajjadi et al. 2022a). which a 3D-aware light-field decoder that can identify the spatial extents of objects and account for occlusion. Given query ray (o, d). the Slot Mixer generates feature x that aggregates features from the object slots. We compute normalized dot-product similarity between and Z using the learned linear projections Wa and WK, and then use this similarity to compute weighted mean of the slot matrix Z:
softmoo/////oooooooo (attaaxx((W (WW WW ((W,x),
w Z.
From the aggregated slot features Z, MI.P decoder predicts the RGB value C(r) of the query ray r. The entire network trained with an Ln reconstruction loss for novel view synthesis, which matches the synthesized images with the ground-truth images {IT))
Crecnn = arg min E E [[((r) [r(
(3)
2.1.2 DECOMPOSITION OF BACKGROUND, FOREGROUND AND AGENTS
Out of the disentangled 3D entities, we clearly identify the active and passive elements provide the necessary evidence to learn intuitive physics in robotic tasks. Instead of handling the latent vectors { permutationniivvaiiat set, we explicitly assign specific indices the background slots Zby and agent slots Zag- The remaining slots are considered as foreground slots {zo.. N-2 which correspond to active objects, and we maintain permutation invariance among object slots. Our training dataset contains the background mask m and agent mask mäy within image regions from the simulator, which can also be obtained by detecting agents or objects with foundation models. We design auxiliary loss that matches the background and agent slots their corresponding masks.
3

---

Published as a conference paper at ICLR 2026
The loss measures the pixel-wise differences between the attention- regions in the predicted image and the regions in the ground truth mask of the RGB input. Formally, the loss is defined as:
Lig
Σ
(bg v) I(u, v) - m (u, v) I(u, w) 1|
(u,D)R
(4)
Lag
Σ (u.))E
Wag(u, ( mag (u, v) I(u, w)
where I(u, v) E R is the ground-truth RGB pixel v), I(u, u) is the predicted RGB, and Ω is the set of sampled pixel coordinates from the image grid.
Our total loss combines reconstruction and auxiliary mask loss terms
Looall
Lrecon + Abg + >ag AagLug.
(5)
We can therefore visually observe the physical consequences of various robot interactions and distin- guish the actions of robots from other movements. Along with the 3D reasoning, the decomposition can stabilize the training of the robot action policy.
2.1.3 3D BLOCK- SLOT ATTENTION
While the slot-based representation interprets the scene into disentangled entities in 3D, the repre- sentation is trained only to reconstruct the observed scenes and is limited in generalization. Objects exhibit different properties, such as shape, size, position, or color, which can be categorized and interpreted combination of partially shared characteristics. If we define an extensive set of task goals by combining object attributes, the trained policy can generalize to novel manipulation targets, earning even more sample efficiency.
We propose a novel mechanism called 3D) block-slot attention, which further disentangles the object slots Zn E RR Dan into the concatenation of M blocks of different attributes {Zn,m IR Ducck 1. m==1 from latent vector F extracted by SRT encoder, such that Datee M Dilock- Since background and agent do not share concepts (e.g., shape, size, position, or color) in the same compositional manner as objects, we adapt the block-slot attention mechanism (Singh et al.] (2022) only to the object slots and retain the vanilla slot-attention mechanism for the background and agent slots. By fixing the indices of the agent and background as described in Section 2.1.2 we can design the update mechanism that appropriately mixes block-slot attention and slot attention across all N slots Z {Zog Zag to...... -
The block-slot attention learns visual concepts by extracting features from images and embedding them into blocks. It extends the slot-attention mechanism with an additional factor binding step, which operates at the block level. To enable this update, each slot update u, is equally divided into M segments Up RPaaa Each block Zn,m E Ruak is then updated independently using u,
with a GRU (Cho et al. 2014), followed by an MLP
Zn,m GRU (Zn,m; Un,m)
2n. MLP Pm (LN(Zn.m)).
(6)
To make each block retrieve its corresponding representation, we use concept memory C. I which serves as a learnable soft information bottleneck. The concept memory Cm consists of K latent prototype vectors associated with the particular factor m, which allows each block to perform dot-product attention over the concept memory.
After disentanglement, we can further identify the attributes assigned specific blocks with ad- ditional analysis. For example, if we perform K-means clustering on blocks or swap blocks of different objects, and interpret the semantics of factorized attributes. We observe that the individual blocks represent semantic attributes, such as colors, shapes, sizes, and positions of objects, which are utilized to generalize the policy into novel combinations of scenes.
For non-object slots, Zhg and Bag, we apply vanilla slot-attention updates using their corresponding update vectors Ubg, Uag E RDDa, each processed with GRU and MLP independently
Zn GRU,, (Zn'; )
Z, MLP d.,, (LN(Z,,));
where n' E (bg, ag). (7)
4

---

Published as a conference paper at ICLR 2026
2.2 BLOCK TRANSFORMER POLICY
We propose a block transformer policy that utilizes interpretable attributes in the 3D block-slot representations efficiently train an agent for goal-conditioned RL (GCRL) task. We first ex- tract the decomposed slot embeddings using the pre-trained 3D block-slot encoder to represent the current state s S and the goal g E for an RL task. Given the multi-view images for the current state { the pre-trained 3D block-slot encoder extracts the N slots of latent em- bedding Z = { Zag: Zo...... - where the object slots {z } are further decomposed into M blocks. Similarly, the goal images {I provide the latent embedding for the goal state Z gg 2ggg 2% ......... 22y x8x-22 2). The goal-conditioned policy x<" (8,g) maps these structured rep- resentations to actions in A to maximize the reward:
20
E Errr.
where r2 = r(2,, Ct. 9)
where 7 E [0, 1) is the discount factor, r S R is the reward function designed to minimize the discrepancy between the current and the goal state. By leveraging block-level attributes and agent information in the structured latent embedding, the policy can accurately interpret alignment against the desired goal. Consequently, the policy is efficient in training and achieves stable performance compared policies employing permutation armuutaton-invariint OCR (Haramati et al. 2024 Zadaianchuk
et al. 2020
Our block transformer policy employs block-wise cross-attention, instead of object-wise cross- attention, such that the agent focuses on manipulating target objects with correct attributes, as de- scribed in Figure[2] We can interpret and combine factorized attributes within the blocks to scrutinize critical relations for the desired policy, which significantly enhances the performance. Suppose the task to relocate objects with the matched attributes into the goal positions. In that case, we first match objects within the current state and the goal by comparing their semantic attributes using Hungarian matching, while ignoring their positions. Then, we compute block-wise cross-attention between the matched objects. The matched object slot in the current state z E RRMx Dava serves as the query, and the corresponding object latent in the goal state z!_.. RR 2 Drick acts the keys and values. Then we pool the resulting attention features (h Jm1 within the object's blocks:
H CrossAtin(z) 20,,), h = Poo,A))
As the latent structure correctly disentangles components with factorized attributes, the resulting at- tention feature successfully encodes the pairwise relationship between matched objects in the current and goal states.
In addition to the features extracted from matched objects h, the policy incorporates the agent slot features of both the current state zig and goal state zly- and the current action a,- Utilizing the agent representation, which is the entity that produces the action, is expected to improve the sample efficiency of visedn-based RL (Pore et al. [0024]. Additionally, the performance of the policy improves when the attention-ban -baee policy network handles the action input separately (Haramati eet al. 2024). A self-attention module (Vaswani et al. (017) aggregates the information to output P. followed by an MLP, which predicts the final output composed of action for the actor and the Q-values for the critic (Figure
P = SclfAttn(hh (bi,..., by zug: 2gg atl), Output MLP(AttnPoo((P)
(10)
extended architectural description of this policy network is provided in Appendix A.
RELATED WORK
3D object-centric learning. Several recent works have expanded object-centric learning from 2D to 3D representations. They train implicit representations with rendering losses in novel views, inspired by advances in novel view synthesis. OSRT (Sajjadi et al. (222a)) and COLF (Smith et al.) 2022) employ an efficient light-field decoder and can quickly decompose complex static scenes. On the other hand, slot attention module can adapt formulations in neural radiance field, and be trained
to reduce rendering losses Yuu et al. 2021 Stelzner et al. 2021 Qi et al. 2023 Luo et al. 2024 Liu et al. 2024). However, all these methods trained on static scenes without interaction mutual occlusion, and cannot accurately reason about physical context in dynamic scenes. Our approach additionally provides highly structured latent representation, enabling systematic reasoning.
5

---

Published as a conference paper at ICLR 2026
Object-wise Cross-Attention
Current blocks
Block-wise Cross-Attention
{ht
(hn)
h1
Current OCR
Object Matching
lt
Attes-Attnntion
Cross- Attention
Pooili liing
Rp
2
fty
Goal OCR
Goal blocks
Figure 2: Comparison between object-wise cross-attention and block-wise cross-attention: When computing cross-attention between the current and the goal states of the scene, our block transformer policy utilizes a novel structure of decomposed blocks, rather than objects. (a) Object- wise cross-attention is applied between the set of permutation-invarree
slots, without distinguish- ing objects, agents, and background. (b) Block-wise cross-attention operates between blocks, which consistently decomposed attributes within object slots. By observing the attributes di- rectly, we can stably match corresponding objects in the current and goal representations, and effi- ciently apply cross-attention to the blocks of desired objects.
Object-Centric Learning for Structured Representation. Structured representation methods factor object representations into interpretable attributes (e.g., color, shape, position). which can be composed produce different combinations of the scenes. SysBinder (Singh et al. 2022) intro- duces "block-slot" design to learn such factors in an unsupervised manner. Dreamweaver (Baek) al. 2025) discovers both static and dynamic primitives (e.g., motion direction, speed) for com- positional world models in video data. Deep Latent Particles (DLP) (Daniel & Tamar 2022) and
DLPv2 (Daniel Tamar 2023) represent scenes as probabilistic particles that encode position and appearance. Including other approaches that factorize object-centric representations (Kosiorek et al.,
2018 Stanić Schmidhuber 2019. Jiang et al. 2019 Crawford & Pineau 2019), these approaches
often lack explicit 3D reasoning. limiting their robustness to occlusion, viewpoint changes, and real-world interaction.
Object-Centric Reinforcement Learning. Object-centric representations in RL enable policies that generalize across diverse scenes with fewer samples. POCR (Shi et al., 2024) integrates "what" (semantic) and "where" (mask) embeddings, improving zero-shot and imitation learning perfor- mance in cluttered manipulation tasks. Entity-Centric RL. (ECRL) (Haramati et al. 2024) employs DLPv2 perception (Daniel & Tamar 2023) together with an Entity Interaction Transtormer (EIT)-an attention mechanism-base. policy architecture- but struggles to recombine attributes with novel ob- jects. PaLM-E (Driess et 2023) grounds language models in slot-based scene representations for embodied reasoning, though falters on unseen property combinations. Additional works (Qi et al.,
2024
Watters al. 2019 Wu al. [0022] Yoon al.) 2023 Zadaianchuk al. 2020) 2022][Zhor et al. 2022) enhance RI. with structured perception, yet most methods still fail compositional variations shape, size, and color, highlighting the need for richer, disentangled, and 3D-aware representations.
4 EXPERIMENTS
4.1 3D BLOCK-SLOT REPRESENTATION
We first analyze our scene representation, which captures a 3D-aware scene representation with factorized components. For the 3D-awareness, we evaluate the quality of generated novel-view images with PSNR. The object decomposition evaluated with ForeGround Adjusted Rand Index (FG-ARI) Hubert & Arabie 1985 Rand 1971 Sajjadi et al. 2022a). The additional factorization in block-wiss wise attributes evaluated in three perspectives: Disentanglement (D), Completeness (C), and Informativeness (I), proposed in Singh et al. (2022). These values extracted by training a gradient boost tree algorithm (Locatello et al. 2019) to map the ground- truth factor to the learned representation.

---

Published as a conference paper at ICLR 2026
Dataset
Method
PSNR
FG-ARI
D
C
OSRT
31.57
0.365
0.140
0.083
0,452
Clevr3D
Ours
31.11
0.942
0.867
0.789
0.844
OSRT
27.35
0.321
0.403
0.222
0.769
IsaacGym3D
Ours
26.55
0.619
0.659
0.550
0.938
Table 3D awareness with novel-view synthesis and decomposition performance: Our method outperforms OSKT across FG-ARI, disentanglement (D), completeness (C), and informativeness (I), while achieving comparable PSNR. The results indicate that our approach improves object decom- position and effectively disentangles information into latent vectors, while maintaining 3D-aware representation.
Datasets. We evaluate the performance of scene representation using two datasets: Clevr3D and IsaacGym3D. Clevr3D builds upon the CLEVR dataset (Johnson al. 2017 Stelzner al. 2021
Girdhar& & Ramanan 2019), a widely used dataset for object-centric representations. We employ the data generation pipeline (Li et al. 2020: 2021 Lin et al.) 2020) construct Clevr3D, which comprises multi-view observations of multiple objects exhibiting rich combinations of attributes. Each scene consists of white background and a set of randomly positioned objects, with diverse attributes. Object attributes include three types of shapes (cube, cylinder, sphere), two types of sizes (small, large), and eighteen colors combined with materials (matte, glossy). The cameras are placed randomly on the surface of hemisphere, capturing six different viewpoints of each scene.
The Sym3D dataset contains more realistic scenarios, in which a robot manipulates objects with randomized actions a white table. Objects in the scene are randomly assigned one of three types of shapes (cube, cylinder, cone), two types of sizes (small, large), and nine types of colors. They are placed at arbitrary (I, y) positions within the boundaries of the table. The cameras randomly placed on the surface of a hemisphere to capture diverse perspectives of the environment, resulting in challenging setting for interpreting agent behavior.
Results. We compare the performance of our 3D block-slot representation against OSRT (Sajadii jet al. 2022a), which produces a 3D object-centric representation. They incorporate similar en- coder and decoder architecture, but treat individual components with permulation-invariant slots. In contrast, our approach distinguishes between foreground objects, agents, and backgrounds, and separately handles factorized blocks with block-slot attention. The quantitative results are shown in T Our results achieve comparable novel-view synthesis results (PSNR) but clearly outperform in distinguishing different components (FG-ARI). The additional factor-binding step enhances the ability to disentangle different attributes the latent vectors, as indicated by D, C, and I scores, measured using the DCI framework proposed in (Singh al. [022).. On the other hand, the 3D slot produced by OSRT misses such a structure and fails to assign consistent information to dedicated location.
In Appendix B. we further visualize the information embedded in each block by clustering it based on the values contained within the block. After the interpretable properties, such as color, size, shape, and position, are clearly separated into different blocks, we can utilize the structured representation to manipulate attributes and create a novel combination of the 3D scene. Figure demonstrates exemplary results of novel-view synthesis after swapping blocks between two objects. With the 3D information, we can even emulate realistic occlusion effects. As demonstrated tion 4.2, such generative capability highly enhances the performance of solving RL tasks. We further provide additional qualitative visualizations, including novel view synthesis, object decomposition, and disentanglement analysis in Appendix B.I In addition, Appendix |B.4 presents ablation stud- ies on the number of blocks, number of prototypes, the auxiliary mask loss, and mixture structure of the slot-attention module and block-slot attention module. Appendix B.2 further provides the single-view inference test results as well as an analysis of the training tendencies.
Sec-
4.2 GOAL-CONDITIONED REINFORCEMENT LEARNING
We exploit the structured representation with the Block-transforee policy to perform goal- conditioned RI. The agent must push the objects to the desired positions based the given goal

---

Published as a conference paper at ICLR 2026
Input
Input
Input
3D Blocks
Input
Input 2
Input
3D Blocks
View
View
View
View
Novel View Synthesis
Novel View Synthesis
Color Swap
Shape Swap
Un
Size Swap
Position Swap
Figure 3: Novel view synthesis after block manipulation: After obtaining the structured latent representation of the scene (top), nation of properties in the 3D representations, which can be consistently synthesized in novel view images (bottom). The left images show the results after swapping the color and size blocks in Clevr3D scene, while the right images are synthesized after swapping the shape and 3D position blocks in the IsaacGym3D dataset.
manipulate block-wise attributes create novel combi-
images. The ground-truth reward is the mean of the negative L2 distance between each object's achieved goal position and its desired goal position. We define success condition when all N objects are within a threshold distance from their respective desired goal. The average return the immediate ground-truth rewards over 7 timesteps.
We set the RI. environment for tabletop object manipulation using the IsaacGym simula- tor (Makoviychuk et al. (022). We use the same environmental configuration of IsaacGym3D dataset Section where cameras are fixed to capture the front, left, and right views of a Franka Panda robot arm pushing objects on white table (Haramati al.) (022)) (see Figure [ right). In each episode, two objects are randomly respawned on the table with random shapes (cube, cone, cylinder) and colors. We adopt off-policy RL algorithm, TD3 (Fujimoto et al.] 2018) with hind- sight experience replay (Andrychowicz et al., 2017). The policy architecture usable with other on-policy or off-policy algorithms, providing fiexibility for various state properties. Appendix A.3 provides additional information on the RL environment configuration.
representations with two policy architectures, including ours. Before training the policy, we first pre-train the scene representation model using the IsaacGym3D dataset. In addition 3D latent vectors of OSRT in Section 4.] also compare against DLPv2 (Daniel & Tamar 2023), one of the best best-performing 2D object-centric representations in goal-conditioned RL. It provides set of disentangled latent vectors in the form of P foreground particles extracted from single image I R #xxxx X3 The 2D representations are trained with separate dataset composed single-view images. OSRT extracts a set 3D object- centric latent vector set, {,,}}, J. from multi-view images of scene (I, E H x3). Implementation details of the pre-training procedures for the basclines are provided in Appendix A.2
Baselines. We evaluate the performance of three structured
The pre-trained visual encoder extracts structured latent embeddings of the current and goal states, which serve as the input to the policy architecture. In addition to the proposed Block Transformer (BT) policy, adopt the Entity Interaction Transformer (EIT) (Haramati al. 2024 baseline, which is a tfte-of-thhe-s object-centric policy for coal-conditioned RL.
Results. Our agent observes environments in three categories: in-distribution evaluation (ID),
compositional generalization (CG), and out-of-distributie generalization (OOD). The ID refers to the scenes with objects that were observed during training. In contrast, CG and OOD are designed to
8

---

Published as a conference paper at ICLR 2026
Shape
Color
Shape
Color
Unseen Colors
Same Colors
Training
Compositional generalization
Training environment
Compositional
Training environment
dusto--dist
environment
generalization
generalization
(Same color)
Figure Evaluation scenarios for compositional and out-of-distribution generalization: Com- position generalization environments consist of objects with properties during training, but novel in their combinations. Out of such unseen combinations, we separately evaluate cases with objects of the same color when the factorization of attributes is unsuccessful. cittof-dditribution environments use objects with colors that were not present the training set.
Representation w/ Policy
DLPv2 w/ EIT
OSRT w/ EIT
Ours w/ EIT
Ours w/ BT
0.984 0.005
0.980 1 0.011
0.984 0.005
0.967 T 0.017
CG
0.747
0.758
0.021
0.773
0.895 0.011
Success Rate
CG (same color)
0.388 0.064
0,414 = 0.167
0.682
0.118
0.837 + 0.035
OOD
0.422 0.170
0.700 + 0.160
0.582 =: 0.270
0.828 + 0.099
-0.189 0.001
-0.175
0.013
-0.184
0.014
-0,218 +: 0.003
CG
-0.365
-0.372
0.002
-0.382
-0.310
Avg Return
CG (same color)
-0,594 0.018
-0.635 = 0.068
-0.494
0.096
-0.388 + 0.016
OOD
-0.587 0.090
-0.414 + 0.139
-0.591 = 0.306 -0.338 + 0.064
Table 2: Performance of goal-conditioned RL: Our proposed 3D block-slot representation, com- bined with a block transformer (BT) policy, can effectively interpret goal conditions and exhibit superior performance in various scenarios. We compare the performance of goal-conditioned RL tasks employing different object-centric representations and policy architectures four settings: ID (in-distribution evaluation), CG (compositional generalization), CG (same color) (compositional generalization with same colored objects), and OOD (out-of-distribution generalization). Results calculated 400 randomly sampled goals per seed, with all reported metrics averaged over three random seeds.
analyze the generalization performance of the representation in unseen environments, depicted in Figure CG uses novel combinations of shapes and colors the individual factors are encountered during policy training, but in the same combinations within the same object. OOD includes instances of attributes that were not encountered during policy training. From the OOD cases, we check how effectively the agent can adapt zero-shot environments.
Table 2 contains an evaluation of performing goal-conditioned RL tasks with various scene repre- sentations and policy architectures. In our model, the agent generalizes well to novel factor com- positions, achieving performance comparable to that in the in-distribution evaluation. In contrast, baseline models experience significant performance drop compared ID.
Among many failure cases in CG scenarios, we have noticed that sharing the same factor between two objects can severely confuse the instances in the scene. CG (same color) tackles these partic- ularly challenging cases, where the agent must manipulate two objects of the same color. Notably, 3D block-slot representations are highly effective compared to unstructured or 2D representations, they capture rich visual features within blocks and distinguish same-colored objects, even though such configurations were not seen during training. BT further enhances performance by carefully matching the desired instances with the correct attributes, task that cannot be achieved with entity- wise attention EIT. The combination of 3D block-slot representations with the block transformer achieves results comparable those of ID.
For this evaluation of OOD scenarios, we randomly spawned objects with six unseen colors, all of which were absent in the training distribution. The results show that model generalizes better

---

Published as a conference paper at ICLR 2026
Generalization Settings
ID
CG
CG (same color)
OOD
Multi View
0.984 + 0.005
0.747 0.030
0.388 + 0.064
0.422 + 0.170
DLP22 w/ EIT HIT
OOD Multi-View
0.059 0.014
0.056
0.026
0.046 0.023
0.078 + 0.037
ID Multi View
0.967 = 0.017
0.895
0.011
0.837 0.035
0.828 + 0.099
OOD Multi- View
0,948 + 0.005
0.877
0.024
0.818 0.025
0,865 + 0.047
Ours w/ BT
Single- View
0.891 + 0.004
0.705 0.023
0.700 0.038
0.727 + 0.105
OOD Single-View
0.802 + 0.028
0.726 0.039
0.676 0.010
0.758 + 0.021
Table Success rate of view-generalization: Our model, which leverages pre-trained 3D block- slot representation and a block transformer (BT), effectively captures 3D object information in viewpoint-agnostic manner and achieves state-of-the-art performance across diverse generalization settings. We evaluate generalization in goal-conditioned RL tasks under four viewpoints settings: ID Multi-View (in-distribution multi-view), ID Single- View (in-distribution single-view), OOD Multi- single-view) Re-
View cout-of-distribuion multi-view), and OOD Single- View sults are computed over 400 randomly sampled goals per seed, with all reported metrics averaged over three random seeds.
than the other baseline methods in the OOD setting. We report additional RI. evaluation results and analysis in Appendix B.3
that our policy can generalize to unseen viewpoints, which implies that it obtains a 3D-aware representation. After training the policy with the front, left, and right camera viewpoints, the view-invariance of the performance is tested on novel viewpoints across multiple generalization scenarios (ID, CG, CG (same color), and OOD). Our method shows only minimal degradation in contrast, the
performance across challenging scenarios, demonstrating strong generalization. 2D) baseline with EIT relies on per-view positional encoding, which overfits to view-specific 21) features, and its policy fails under unseen viewpoints.
A notable property of our approach is that it remains effective even when the number and config- uration of views at test time differ from training. A policy trained with multi-view observations continues to succeed when provided with only single training view, and even with completely novel single-view that appears during training, while still preserving its generalization behavior. This demonstrates that a single-view can provide sufficient information for the policy to infer the necessary 3D-aware object properties. The modest performance drop in the single-view view setting is primarily due to robot-induced occlusions, which cannot be fully resolved without multi-view infor- mation. Additional results using suboptimal mask are provided in Appendix B.4
5 CONCLUSION
We propose 3D block-slot representation, a 3D-aware structured representation that integrates 3D object-centric learning with attribute factorization. We successfully decompose background, fore- ground, and agent, and can disentangle object factors not only in synthetic static scenes but also in simulated robot environments. The block transformer policy effectively applies structured latent sets to the RL framework, leveraging both the advantages of factor-level input and object-level com- putation. Compared to prior methods using pre-trained object-centric representations, our policy network improves the performance of compositional generalization and diitr---iittiittioon gener- alization. The results indicate that the factorized latent structure can he directly used in the pol- icy network, demonstrating superior generalization. While our framework adopts one-to-one static attribute-based matching between current and goal representations, this assumption may be insuf- ficient in scenarios requiring more flexible correspondence strategies, such as matching multiple objects to single goal. Extending our approach to support more dynamic or many-to-on match- ing schemes an important direction for future work, particularly more complex environments. Furthermore, the 3D block-slot representations, which encode semantics akin to language tokens, have the potential to bridge the gap between 3D perception and Lision-LanguaageAAee Action (VLA) frameworks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by Samsung Research Funding & Incubation Center for Future Technology of Samsung Electronics under Project Number SRFC-IT2402-10.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. Advances in neural information processing systems, 30, 2017.
Junyeob Baek, Yi-Fu Wu, Gautam Singh, and Sungjin Ahn. Dreamweaver: Learning compositional world representations from pixels. arXiv e-prints, pp. arXiv-2501, 2025.
Ondrej Biza, Sjoerd Van Steenkiste, Mehdi SM Sajjad,, Gamaleldin F Elsayed, Aravindh Mahen- dran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. arXiv preprint urXiv:2302.04977. 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. Journal of Machine Learning Research, 22(259):1 2021.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 3412-3420, 2019.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent particles. arXiv preprint arXiv:2205. 150.:221,.111 2022.
Tal Daniel and Aviv Tamar. Ddlp: Unsupervised object-centric video prediction with deep dynamic latent particles. arXiv preprint arXiv:2306.05957 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, al. Palm-e: An embodied mul- timodal language model. 2023.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and tem- poral reasoning. arXiv preprint (xxiv:9110004444, 2019.
Dan Haramati, Tal Daniel, and Aviv Tamar. Entity-centric reinforcement learning for object manip- ulation from pixels. arXiv preprint arXiv:2404.01220, 2024.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification 2(1):193-218, 1985.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv:1910.02338 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
11

---

Published as a conference paper at ICLR 2026
Jinwoo Kim, Janghyuk Choi, Jachyun Kang, Changyeon Lee, Ho-Jin Choi, and Seon Joo Kim. Leveraging image augmentation for object manipulation: Towards interpretable controllability in object-centric learning, arXiv preprint arXiv:2310.0892) 2023.
Avinash Kori, Francesco Locatello, Ainkaran Santhirasekaram, Francesca Toni, Ben Glocker, and Fabio De Sousa Ribeiro. Identifiable object-centric representation learning via probabilistic slot attention. Advances in Neural Information Processing Systems, 37:93300-93335, 2024.
Adam Kosiorek, Hyunjik Kim, Yee Whye Tch, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
Nanbo Li, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi- object scenes from multiple views. Advances in neural information processing systems, 33:5656- 5666, 2020.
Nanbo Li, Muhammad Ahmed Raza, Wenbin Hu, Zhaole Sun, and Robert Fisher. Object-centric representation learning with generative mation processing systems, 14:10722-10783, 2021.
factorization. Advances in neural infor-
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving genera- tive imagination in object-centric world models. In International conference machine learning, pp. 6140-6149. PMLR, 2020.
Yu Liu, Baoxiong Jia, Yixin Chen, and Siyuan Huang. Slotlifter: Slot-guided feature lifting for learning object-centric radiance fields. In European Conference Computer Vision, pp. 270- 288. Springer, 2024.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in neural information processing systems, 33: 1525-11538, 2020.
Rundong Luo, Hoog-XXngg Yu, and Jiajun Wu. Unsupervised discovery of object-centric neural fields. arXiv preprint arXiv:2402.07376, 2024.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
Cheol-Hui Min and Young Min Kim. Asimo: Agent-centric scene representation in multi-object manipulation. The International Journal of Robotics Research, 44(1):22-64, 2025.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07193. 2023.
Ameya Pore, Riccardo Muradore, and Diego Dall' Alba. Dear: Disentangled environment and agent representations for reinforcement learning without reconstruction. In 2024 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 650-655. IEEE, 2024.
Carl Qi, Dan Haramati, Tal Daniel, Aviv Tamar, and Amy Zhang. Ec-diffuser: Multi-object manip- ulation via entity-centric behavior generation. arXiv preprint 2rXiv:24/2.18007 2024.
Di Qi, Tong Yang, and Xiangyu Zhang. Slot-guided volumetric object radiance fields. Advances in Neural Information Processing Systems, 36:65766-65783, 2023.
12

---

Published as a conference paper at ICLR 2026
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselimes:: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1-8, 2021.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the Ameri- can Statistical association, 66(336) 1971.
Alireza Rezazadeh, Athreyi Badithela, Karthik Desingh, and Changhyun Choi. Slotgnn: Unsuper- vised discovery of multi-object representations and visual dynamics. In 2024 IEEF International Conference on Robotics and Automation (ICRA), pp. 17508-17514, IEEE, 2024.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd Van Steenkiste, Filip Pavetic, Mario Lucie, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in neural information processing systems, 35:9512-9524, 2022a.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6229-6238, 2022b.
Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre-trained object-
centric representations for robotics from" what" and" where" foundation models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp- 15424-15432. IEEE, 2024.
Dongseok Shim, Seungjae Lee, and H Jin Kim. Snerl: Semantic-aware neural radiance fields for reinforcement learning. In International Conference on Machine Learning, pp. 31489-31509 PMLR, 2023.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. arXiv preprint arXiv:2211.01177 2022.
Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205.03922 2022.
Aleksandar Stanić and Jürgen Schmidhuber. R-sqair: Relational sequential attend, infer, repeat. arXiv preprint 10X0:0990000233 2019.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint arXiv: 099555 2019.
Ziyi Wu, Nikita Dvomikk Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05800 2022.
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object- centric representations for reinforcement learning. arXiv preprint arXiv:2302.044/1 004.044422449, 2023.
Hlong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprini arXiv:2107.079000 2021.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforcement learning with object-centric representations. arXiv preprint arXiv:2011.14388) 2020.
Andrii Zadaianchuk, Georg Martius, and Fanny Yang. Self-super reinforcement learning with independently controllable subgoals. In Conference on Robot Learning, pp. 384-394. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2026
Allan Zhou, Vikash Kumar, Chelsea Finn, and Aravind Rajeswaran. Policy architectures for com- positional generalization in control. arXiv preprint arXiv:2203.05966 2022.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A IMPLEMENTATION DETAILS
A.1 BLOCK TRANSFORMER POLICY.
Block Transformer Policy
Action [ACT]
Q-val' Action [*]
Alu
Zig
Current Observations
3D Block Stot Encoder
Block- wise
Cross- Attention
Salt Saatemii
Cross-Attention
Celf-ain
MLP
Output (Action, Q-value)
Goal
Observations
Slot
Block
Figure 5: Structure of block transformer policy: The block transformer policy processes 3D block-slot representations using block-wise cross-attention, self-attention, cross-attention with pooling token, followed by an MLP to generate the final output.
In this section, describe the architecture and implementation details of our block transformer policy, including its attention modules and how it integrates information from 3D block-slot repre- sentations.
The block transformer policy consists block-wise cross-attention, self-attention and cross- attention modules, followed by MLP described in 2.2 We use only foreground blocks {zn.m} and agent slots zag. zāg- as background information tends to be noisy in robotic object manipula- tion tasks. The self-attention module takes as input the output block-wise cross attention (h,), linearly projected action information, and the slots extracted from current and goal observations, in order to model the interaction between agent information and object attributes. The output of self-attention module is then processed through a cross-attention mechanism with pooling query token, followed by an MLP, to produce the final policy output.
A.2 PRE-TRAINING REPRESENTATION
In this section, we describe the pre-trained representations used in our experiments-DLPv/ OSRT, and our proposed 3D block-slot representations-ant introduce the Clevr3D and IsaacGym3D datasets used for training and evaluation. We first describe the datasets collected for pre-training, and then provide details the 2D and object-centric model.
Dataset. To investigate the capability of 3D block-slot attention, we use datasets that rep- resent multi-object scenes and robotic environments. First, we consider the CLEVR dataset, which has been widely used to evaluate decomposition factorization quality. Clevr3D extends CLEVR (Johnson et al. 2017) by randomizing camera positions instead of using fixed viewpoints. Each scene contains three randomly placed objects with attributes independently sampled for shape, size. color and position. Object attributes include three types of shapes (cube, cylinder, sphere), two sizes (small, large), and eighteen colors resulting from combinations of nine base colors with two materials (glossy, matte). The dataset includes 20,000 training scenes (each with six views) and 1,000 test scenes.
In contrast, IsaacGym3D restricts the camera to the front side of the hemisphere, with viewpoints ranging from -90" to 90%, because the rear views are not necessary for training in the robot manip- ulation environments. Each scene includes three objects with randomized attributes: shape (cube,
15

---

Published as a conference paper at ICLR 2026
cone, cylinder), size (small, large), and color (identical to Clevr3D). The dataset contains 30,000 training scenes and 200 test scenes. For 2D object-centric learning, we additionally collect two fixed views-front and right views-for each scene.
Baseline: OSRT. We train OSRT (Sajjadi et al., 2022a) the 3D object-centric learning baseline in our experiments. Given multi-view observations {1; ##xWx3 the SRT encoder aggregates features into scene-level latent representation E 1R L Using the the vanilla slot attention module is employed to produce set of slots {z, E 12 Jn=1- The Slot Mixer decoder then predicts JD
RGB values for given query ray, as described Section 2.1 The model is trained with slots of 64 dimensions for Clevr3D, and 6 slots of 64 dimensions for IsaacGym3D. We set hyperparameters listed Table The model size intentionally kept small (64-dimensional slots), as larger models- such those with 1536-dimensional slots-incur significant computational cost and slow inference speed, which is not suitable for vision-based RL.
Clevr3D
IsaacGym3D
Module
Hyperparameter
OSRT
Ours
OSRT
Ours
Batch size
40
40
40
40
General
Training steps
1.5M
1.5M
1.5M
1.5M
Learning rate
0.0001
0.0001
0.0001
0.0001
Number of slots
4
4
4
Slot size
64
64
64
64
Slot attention
Number of blocks
N/A
8
N/A
8
Block-slot attention
Block size
N/A
N/A
8
Number prototypes
N/A
MLP hidden dimension
512
512
512
512
Table 4: Hyperparameters of OSRT and 3D block-slot attention used in our experiments.
Baseline: DLPv2. We train DLPv2 (Daniel Tamar 2023) the 2D object-centric learning baseline our experiments. We follow the modification DLPv2 by ECRL (Haramati et al. 2.24), which sets the background particle features have dimensionality of 1, and discards them during RL training. DLPv2 extracts P particles from a single image H x3 where z = (Zp, Zez Zd: Zt, E R6+m, Each latent component-z, 21 E 18""representt the particle's position, scale, depth, transparency and latent feature of visual appearance, respectively. We use the hyperparameters listed in Table
R 2, E IR2 =d IR, R and
Baseline: SNeRL. SNeRL. (Shim et al. 2023) designed to handle dynamic scenes by aggregat- ing multi-view information, including RGB, semantic labels, and distilled features, into unified representation. However, while this approach is effective for simpler dynamics, our IsaacGym3D setting introduces non-stationary distributions with random object re-sampling, severe clutter, and continuous robot articulation that saturate SNeRL's encoding capacity. Consequently. SNeRL fails to establish stable radiance fields under such high stochasticity shown in Figure, underscoring the necessity of our transformer- basedd architecture to robustly model complex manipulation environ- ments.
16

---

Published as a conference paper at ICLR 2026
Scene 1
Scene 2
View I
View 2
View 3
View
View
View
Input
Reconstruction
Figure 6: Qualitative results of SNeRL encoder training IsaacGym3D: The failure recon- struct the input images, particularly the inability to render the rohot arm and manipulated objects despite background convergence, demonstrates that SNeRL's encoder cannot effectively capture the complex dynamics and non-stationary distribution of our scene.
A.3 REINFORCEMENT LEARNING
In this section, describe the reinforcement learning setup in experiments, including the reward function, evaluation metrics, and implementation details of reinforcement learning.
Hyperparameter
DLP
Epochs
Batch Size
64
learning rate
0.0002
Reconstruction Loss
MSE
Prior Patch Size
Posterior KP
Prior KP Proposals L
32
Glimpse Size S
32
Feature Dim
Background Feature Dim mag
BKL
0.1
Table 5: Hyperparameters of DLPy2 used in our experiments.
Reward. The reward is computed from the ground-truth state of the environment and is referred as the ground-truth reward. It is defined as the mean negative L2 distance between each object and its corresponding goal position on the table:
N Σ 9
9
(11)
N
where each of rtt.t, No. gf and of denotes the immediate ground-truth reward at time f, the number of objects, ith object's desired goal, and ith object's achieved goal, respectively.
Evaluation metrics. We adopt five evaluation metrics to evaluate the performance reinforce- Haramati ment learning, Haramaii et al. (2024). Success indicates whether all objects have reached within given distance threshold from their respective goals. Success fraction measures the pro- portion of individual objects that satisfy the same distance condition. Average return is the mean cumulative reward over the episode. Maximum object distance refers to the furthest distance be- tween any object and its target at the end of episode, and average object distance is the mean of all
17

---

Published as a conference paper at ICLR 2026
such distances across objects. The formal definitions are follows:
N
Success 1l
Σ
5i < R)) - No.
(12)
Success fraction
U( U(1lg -Bill2 R),
(13)
T
Average return
2 Tt
(14)
Maximum object distance
max{]lsf
9 b1
(15)
Average object distance
11
95
(16)
Here, denotes the number of timesteps in an episode, 7r is the reward at timestep t, and the distance threshold used to determine success.
Implementation of RL. Our reinforcement learning implementation is based on the ECRL (Hara- mati et al. 2024), built upon 2018) and HER (Andrychowicz et al., 2017). For exploration, we employ both s-greedy strategies and Gaussian action noise, following the approach of Zhou et al. (2022). All models are optimized using the Adam optimizer.
(Rallin et al., (02I), using TD3 (Fujimtoo et al.,
Hyperparameter
EIT & BT
Batch size
512
Learning rate
5c-4
Y
0.98
0.05
Exploration action noise
0.2
Exploration
0.3
HER ratio
Number of episodes collected per training loop
16
Table 6: Hyperparameters of RI. used our experiments.
B ADDITIONAL RESULTS
B.1 QUALITATIVE ANALYSIS OF 3D BLOCK-SLOW REPRESENTATIONS
In this section, we present additional qualitative results to evaluate the effectiveness our 3D block- slot representation. We include visual comparisons of novel view synthesis and object decomposi- tion between our method and OSRT on the Clevr3D and IsaacGym3D datasets. We also show additional examples of block manipulation through block swapping to demonstrate controllability of the blocks. Furthermore, we analyze factor disentanglement using feature importance matrices and clustering visualizations, which highlight how different blocks encode semantically meaningful components of the scene.
Novel view synthesis and object decomposition. We present qualitative comparisons of novel view synthesis results produced by OSRT and our method on the Clevr3D and IsaacGym3D datasets. By leveraging multi-view observations, our method remains robust under occlusions and accurately reconstructs the scene. In particular, our per-slot reconstructions demonstrate more consistent de- composition of background and individual objects across views. Additionally, the robot agent is reliably assigned to a fixed slot index, effectively capturing the agent's presence and action action-related related information.
18

---

Published as a conference paper at ICLR 2026
3D block-sott representations
OSRT
Inpur images
Reconstruction
Slot
Slot
Sloc
Slot 4
Reconstruction Slat
Slot
Slot 3
Sint
View
View
View
View
View
View
View
View 3
bleck-ilt
reprossnaaiion
Reconstraction
View
View
View
View
View
Figure 7: Novel view synthesis and decomposition: Our method achieves comparable novel view synthesis quality to OSRT while providing significantly better separation of foreground objects, background and agent components on the Clevr3D and IsaacGym3D datasets.
19

---

Published as a conference paper at ICLR 2026
Block manipulation and novel view synthesis. We provide additional visualizations of novel view synthesis through block manipulation, as shown in Figure In each scene, two objects are selected, and the specific blocks corresponding to their attributes are swapped. The model then synthesizes novel views based on the modified slot representations. In both Clevr3D and Isaac- Gym3D, method successfully generates novel views with swapped blocks, while preserving the consistency of the remaining blocks.
Novel view
Novel view 2
Input
Reconstruction
Slot
Slot
Roconstraction
4
Original slots
Soupping
Shape swapping
Original slots
Objecta far Soapping
Color swapping
Original slots
Swapping
Size swapping
Novel view
Recometmation
Slse
Sler
Sket
Shar4
Slot 5
Reconstaationn Siet
Siet 2
Siet
Slot 4
Slee
Original slos
Objices Snopptaz
Shape
Original slos
Ulgect
swapping
Original
Objace for Snopptaz
swapping
Figure 8: Additional samples block manipulation and novel view synthesis: We show addi- tional examples of block swapping, where blue-circled objects indicate the slots selected for attribute exchange.
K-means clustering of blocks. By extending k-means clustering from 2D to 3D block-slot repre- sentations, we identify meaningful clusters blocks. Given multi-view observations {I,) the 3D block-slot attention mechanism produces 3D block representations {zn,m }. Unlike the 2D slot at- tention mechanism, our model aggregates multi-view information into latent vectors and thus cannot directly produce attention maps from the encoder outputs. Therefore, obtain the corresponding
20

---

Published as a conference paper at ICLR 2026
object images (y,) using the slot weights from Slot Mixer decoder applied to novel view. Us- ing 1,050 objects in 350 scenes, each containing three independently randomized objects, we collect set of pairs {(z,,m: y.)). K-means clustering of the blocks (with K 10) reveals the underlying structure of the representation helps identify which blocks encode specific object attributes. In block 6 of the Clevr3D dataset, cluster corresponds cylindrical objects, while cluster 3 corresponds spherical objects. These results suggest that block encodes shape-related attributes.
Block (Shape)
Block (Caler)
Block 2 (Size)
Black (x-pas)
Block (o))
Cuewr1
Chewer
Chater
Clasel
Clitte
Chorr
Chevr
Chowe2
Cheser
Chater
Chaoe
Close
Chsse11
Chuner
Chaser
Figure 9: K-means clustering of 3D block representations: Clustering results indicate that differ- ent blocks capture distinct object attributes. For example, block clustered by shape, block by color, block 2 by size, block 4 by x-position, and block 5 by y-position.
Feature importance matrix. We visualize the feature importance matrices for our method and OSRT. As described in Section 4.1 we train a gradient boost tree to predict each ground-truth object attribute from the learned blocks, in order to evaluate how well the blocks capture and disentangle object attributes. Using the trained model, we extract the feature importance matrix, where each row corresponds to a ground-truth object factor, and each column represents latent unit-either a slot in OSRT block in our method. The results show that our model learns more disentangled represen- tations, with different blocks specializing in distinct object attributes across both the Clevr3D and IsaacGym3D datasets. In contrast, the slot-based representations from OSRT show more entangled feature importance patterns, indicating less structured encoding of object attributes.
Position
Position
position
position
Color
Shape
Shape
3D block-slot attention with 8 blocks in slot in Clevr3D
3D block-slot attention with blocks in a slot in IsaacGym33
Postioo
posiiio
Shape
OSRT with 64 dimensions a slot in Clevr3D
Postiion
position
Color
Shape
OSRT with 64 dimensions in a slot IsaacGym3D
Figure 10: Feature importance matrix: Feature importances for predicting ground-truth object attributes from latent representations. Our method shows clear attribute-specific encoding in each block, while OSRT's 3D slot representations are less disentangled.
21

---

Published as a conference paper at ICLR 2026
B.2 QUANTITATIVE ANALYSIS OF 3D BLOCK-SUE SLOT REPRESENTATIONS
In this section, we provide additional quantitative results to further demonstrate the 3D-awareness of our 3D block-slot representation. Specifically, evaluate how model trained under the multi- view setting on the IsaacGym3D dataset performs when only a single view is provided at inference time.
Single-view inference performance. Table reports the performance of our 3D block-slot repre- sentation, which trained with three input views the multi-view setting but evaluated with only single input view. The results show that the performance with single view is nearly identical to that obtained with three views. The slight decrease in PSNR and DCI is expected, as occlusions are more challenging to resolve in the single-view setting compared to the multi-view setup.
View settings
PNSR
FG-ARI
Multi-View
26.55
0.619
0.659
0.550
0.938
Single-View 25.90
0.623
0.626
0.558
0.915
Table 7: Novel-view synthesis and decomposition performance under multi-view and single- view settings: Our model is trained in the multi-view setting, and the table compares its inference performance when using multiple views versus single view.
Training loss behavior. As shown in Figure 11 our model's 's training objective consists of a re- construction loss and an auxiliary mask loss. The reconstruction loss exhibits two notable drops during training. These correspond to the model first learning to synthesize the agent correctly from novel views using the agent slot, and later learning to reconstruct the objects through the object slots. After these two transitions, both the reconstruction loss and the mask losses decrease smoothly and stabilize over time. Importantly, block-wise separation also emerges shortly after successful object reconstruction, and the blocks become increasingly disentangled training progresses.
Reconstruction loss
(b) BG mask loss
(c) AG mask loss
3.0
3.5
300
206
3.5
1.0
200
(143)
Timee
Train loss of 3D block-slot representation: We show examples of suboptimal masks: background masks derived from DINO attention maps in simulation, and agent masks obtained from hand-eye calibration and robot kinematics in real-world settings.
Figure 11:
B.3 ADDITIONAL RL RESULTS
In this section, we present additional reinforcement learning results to complement the main exper- iments. We include agent rollout visualization, additional evaluation metrics, and learning curves. These results provide a more comprehensive understanding of our policy's performance and gen- eralization capabilities. addition. we report results from non-obiect-centric baselines clearly contrast them with object-centric representations and highlight the necessity of pre-trained object- centric representations.
Rollouts of an agent across evaluation scenarios. We assess the generalization capability of our model across four different environment setups: in-distribution evaluation, compositional general- ization, compositional generalization of same-color objects, and out-of-distribution generalization.
22

---

Published as a conference paper at ICLR 2026
our method demonstrates strong performance across all scenarios. The visu- alized rollouts show that the agent successfully completes the task in all evaluation scenarios, further demonstrating the robustness and generalization ability of our policy.
As reported in Table 2
F=
Success cpisodes
Goal Image
In-distribution
Compositional Generalization
Compositional Generalization (Same color)
Out-of-distribution Gencralization
Figure 12: Agent behavior in evaluation tasks: Visualized rollouts demonstrate that the agent completes the task successfully across all evaluation scenarios, including in-distribtution, composi- tional, and out-of-distribution settings.
Additional evaluation metries. We evaluate the performance of four methods that combine pre- trained representations with reinforcement learning policies. Following (Haramati et al. 2024). we adopt five evaluation metrics: success rate, success fraction, average return, maximum object dis- tance, and average object distance. The results of success rate and average return are reported in Ta while the remaining metrics are shown Table Success fraction measures the proportion of individual objects that reach their respective goal positions. Maximum and average object dis- tances are computed based on the Euclidean distances between each object and its assigned goal. As shown in Table 8 our method consistently outperforms all baselines across the additional metrics as well. Specifically, it achieves higher success fraction and lower object distance errors, indicating more precise and consistent goal-reaching behavior. These results further support the effectiveness of our policy and its strong generalization capability across different evaluation settings. Formal definitions of these metries are provided in Appendix A.3
23

---

Published as a conference paper at ICLR 2026
Representation w/ Policy
DLPv2 w/ EIT
OSRT w/ EIT
Ours w/ EIT
Ours w/ BT
0,991 0.033
0.988 + 0.088
0.990 0.003
0.979 + 0.013
CG
0.843 0.016
0.843 + 0.003
0.833 0.037
0.913 + 0.010
Success Fraction
CG (same color)
0.628 0.038
0.598 = 0.073
0,750 + 0.091
0.866 + 0.027
OOD
0.654 0.113
0.812 1: 0.107
0.720
0.192
0.888 1: 0.061
0.026 0.001
0.021 + 0.003
0.022 0.001
0.030 + 0.003
CG
0.084 0.010
0.083 + 0.005
0.083
0.016
0.049 + 0.005
Max Obj Dist
CG (same color)
0.167 0.012
0.172 + 0.037
0.108 + 0.034
0.066 + 0.004
OOD
0.164 0.034
0.102 1: 0.053
0.164
0.113
0.059 1: 0.019
ID
0.018 0.001
0.016
0.002
0.017
0.023
0.002
CG
0.054 0.005
0.054 1 0.002
0.056
0.009
0.039
0.003
Avg Obj Dist
CG (same color)
0.103 + 0.006
0.114 + 0.017
0.081 + 0.021
0.052 + 0.003
OOD
0.100 0,021
0.064 + 0,031
0.105
0.072
0.042 + 0.013
Table 8: Additional evaluation metries for the IsaacGym push task: We report results across four settings: ID (in-distribution evaluation), CG (composition generalization), CG-same (compo- sitional generalization with same colored objects ), and OOD (out-of-distribution generalization). Each metric is averaged over 400 randomly sampled goals per seed.
Visualization of GCRL training. We employed four methods that combine pre-trained vision models with control policies in our setup. We visualize both the success rate and the average re- ward over training timesteps. Among these methods, the combination of 3D) block-slot represen- tations with an object-centric policy achieves the highest sample efficiency. This suggests that the structured 3D block representations are more effective for learning robotic manipulation tasks than unstructured 3D object-centric representations or 2D object-centric representations. In addition, our block-level transformer, while slightly less cample-efficien demonstrates superior generalization performance in environments where the object-centric policy fails.
Avg Reward
Success Rate
100
-43
Achievement
Reward
-03
OSRT ETT
Ours w/ EIT
Step (1e6)
Step (1e6)
Figure 13: Success rate and average reward during GCRL training: Comparison of four meth- ods using different pre-trained representations and policies. The 3D block-slot representation with object-centric policy achieves the fastest learning, and the block-level transformer shows stronger generalization in long-horizon tasks.
Additional results on non-object-centric baseline. Following the non-object-centri baseline used in ECRL (Haramati et al.) 2024), we evaluate policy that encodes multi-view state and goal images into a single latent vector using a pre-trained autoencoder (Narr et al. 2018), which subsequently processed by standard MI.P policy. As reported in Table 9 this baseline does
24

---

Published as a conference paper at ICLR 2026
no reliably solve multi-objeet manipulation tasks. Compressing the entire scene into unstructured latent representations make it difficult to preserve object identities, spatial relationships, and goal correspondence, resulting in unstable learning behavior. In contrast to object-centric approaches, the unstructured latent bottleneck prevents the agent from explicitly perceiving individual objects and reasoning about their interactions, highlighting fundamental limitation of unstructured visual
representations for multi-object control.
Model
Success Rate
Avg Return
Success Fraction
Max Obj Dist
Avg Obj Dist
VAE w/ MI.
0.042
-1.072 0,040
0.185 +: 0.004
0.273 +: 0.010
0.210
Ours w/ BT
0.967 0017
-0.218 0.003
0.979 + 0.013
0.030 = 0.003
0.023 0.002
Table 9:
Performance of goal-conditioned RL of non-object-centric baseline and ours: We report the performance of non-object-centric baseline using VAE-based visual representation with an MLP policy, and our object-centric method combined with a BT policy.
B.4 ADDITIONAL ABLATION STUDIES
In this section, we present additional ablation experiments. We investigate the effect of the number of blocks and prototypes in the 3D block-slot attention module, as well as the impact of the auxiliary mask loss and its scaling. We also report GCRL results using suboptimal masks, showing that our model maintains strong performance even without ground-truth masks. Finally, we ablate the 3D block-slot attention architecture by comparing vanilla slot attention and block-slot attention, demonstrating the necessity of combining the two mechanisms.
Mixture structure of slot attention. We evaluate the contribution of the slot-attention module and the block-slot attention module used in our model for decomposing the background, agent, and ob- jects. our architecture, the background and agent slots are updated using the vanilla slot-attention module, whereas the object slots are updated using the block-slot attention module, which enables learning object attributes in a block-wise manner. contrast, Figure [4 and Table I0 presents the results obtained when all slots are updated either with the vanilla slot-attention module or with the block-slot attention module.
When all slots are updated solely with vanilla slot-attention, which corresponds to applying auxiliary mask loss to an OSRT, the background and agent can be assigned to consistent slots, but overall performance remains clearly inferior. This variant shows lower PSNR, FG-ARI, and DCI scores compared to our mixed architecture and, importantly, fails to disentangle object attributes into block-wise latent components. As result, cannot provide the structured 3D block-slot represen- tations required by the block transformer policy that enables strong generalization. These findings indicate that vanilla slot-attention, even with auxiliary mask loss, is insufficient, and that our mixed design is necessary for accurate decomposition and attribute-awar reasoning.
Conversely, when all slots are updated using the block-slot attention module, the background and agent-which do not share the same attribute structure as objects-an are forced to learn object-like attributes. This causes the model to fail at both correct object decomposition and block-wise attribute learning. These observations indicate that background and agent slots, which do not share object attributes such as color, shape, size, and position, must follow update pathways. Our combination of the vanilla slot-attention module for background and agent slots, the block-slot attention module for object slots, and the auxiliary mask loss enables successful decomposition of the background, agent, and objects in robotic scenes, while also supporting block-wise attribute learning for each object.
Model
PSNR
FG-ARI
D
C
Ours
26.55
0.619
0.659
0.550
0.938
Ours with only Slot-Attention
25.28
0.520
0.501
0.302
0.786
Comparison of slot-attention and our mixture slot-attention architecture: We eval- uate all slots are updated solely with the vanilla slot-attention module. The results are compared against the proposed mixture slot-attention, which applies vanilla slot-attention to the background and agent slots while updating foreground slots using the block-slot attention module..
Table 10:
25

---

Published as a conference paper at ICLR 2026
Novel View Reconstruction
Novel View 2
Input image
Input image
Input image
Novel GT
Reconstruction
Figure 14: Failure cases when updating all slots with the block-slot attention module: When the block-slot attention module is applied to all slots-including background and agent slots-the model fails learn stable attributes, leading to training collapse and incorrect scene decomposition.
Number of blocks. We evaluate the performance of the 3D block-slot representation under differ- ent numbers of blocks using three metrics: PSNR, FG-ARI, and DCI performance. Each model is trained with 4, 8, 16 blocks for approximately 600k iterations, while keeping the slot size fixed to 64; consequently, the block sizes are 16, 8, and 4, respectively. The results show clear trend: in- creasing the number of blocks leads to noticeably improved disentanglement In other words, as the number of blocks increases and the block size decreases, the model tends to assign more independent properties to individual blocks. This behavior arises because smaller block sizes impose stronger constraints on the representational capacity of each block, while having more blocks provides the model with greater flexibility to distribute and separate different object attributes.
Number of blocks
PSNR
FG-ARI
D
C
4
24.45
0.498
0.322
0.400
0.844
8
23.71
0.465
0.429
0.394
0.836
16
24.19
0.437
0,447
0.368
0.848
Table 11: Effeet of the number of blocks on 3D awareness performance: We provide PSNR, FG-ARI, and DCI performance as function of the number of blocks.
Number of prototypes. We evaluate and analyze the performance of the 3D block-slot representa- tion with varying numbers of prototypes using three metrics: PSNR, FG-ARI, and DCI performance. Each model is trained for approximately 1M iterations, differing only in the number of prototypes, which is set to 8, 16, or 32. When using prototypes, the 3D block-slot representation successfully performs novel-view synthesis and decomposes scenes into per-object slots, but it fails disentan- gle object attributes across blocks. The number of prototypes directly influences the representational capacity each block; when the number of prototypes too small, the model cannot reliably sep- arate the underlying 3D attributes. This trend is consistently observed in the models trained with 16 and prototypes. Models with more prototypes not only achieve higher PSNR but also exhibit sub- stantially improved disentanglement performance. These results indicate that increasing the number of prototypes enhances the expressive power each block, allowing individual blocks to more fully encode distinct attributes.
Numher of prototypes
PSNR
FG-ARI
D
C
8
22.43
0.654
0.023
0.030
0.476
24.86
0.491
0.454
0.439
0.916
32
25.33
0.495
0.480
0.432
0.876
Table 12: Effeet of the number of prototypes on 3D awareness performance: We provide PSNR, FG-ARI, and DCI performance as a function of the number of prototypes.
Impact of block disentanglement. We evaluate the GCRL performance of a 3D block-slot repre- sentation with relatively low DCI scores. For this comparison, disentanglement, where the model can already decompose scenes into background, objects, and agent, but the block-level disentanglement is still incomplete, and compare its success rates across
checkpoint with incomplete
26

---

Published as a conference paper at ICLR 2026
four generalization settings: ID, CG, CG (same color), and OOD. As shown in Table 13 the model with insufficient block separation exhibits consistently lower success rates across all four settings. This degradation arises because the dynamic and static properties are not clearly disentangled, mak- ing the Hungarian aatching--peformenn
by the block transformer policy based on static attributes- less reliable. Furthermore, the block-wise cross-attention receives weaker and less distinct signals, which makes it harder for the policy to learn task-relevant object relations to generalize effectively.
Model
ID
CG
CG (same color)
OOD
High-DCI Model
0.967 i 0.017
0.895 1: 0.011
0.837 1 0.035
0.828 :: 0.099
Low-DCI Model
0.823 # 0.015 0.677 = 0.030
0.599 0.037
0.750 = 0.000
Table 13:
Impact of disentanglement quality on GCRL performance: We compare success rates across four generalization settings (ID, CG, CG (same color), and OOD) between high-DCI model and a low-DCI model.
Auxiliary mask loss scale. We evaluate and analyze the performance of the 3D block-slot repre- sentation under different scales of the auxiliary mask loss introduced in Eq 4 using three metries: PSNR, FG-ARI, and DCI performance. Each model is trained for approximately 700k iterations with the auxiliary mask loss weights Abg and Aag set to 0.1, 0.5, 1.0. Compared to the models trained with Abg and Xag = 0.1 or 0.5, the model using a scale of 1.0 achieves higher FG-ARI but shows lower PSNR, disentanglement, completeness, and informativeness scores. This indicates that when the auxiliary mask loss becomes too dominant relative to the primary reconstruction loss -recon- the key objective responsible for novel-view synthesis and unsupervised disentanglement of object attributes into separate blocks-the model focuses more on accurate decomposition at the expense of view synthesis quality and attribute embedding fidelity. In other words, excessively large Aog and Xag can hinder 3D block disentanglement. Nevertheless, the degradation is modest, and more importantly, the inclusion of the auxiliary mask loss regarddess of the specific scaling enables the model to reliably distinguish background, foreground, and agent regions, which is es- sential for downstream block-slot decomposition.
Augs Aag
PSNR
FG-ARI
Abgs Aag = 0.1
24.66
0.426
0.472
0.446
0.856
Pogs Aag 0.5
24.39
0.425
0.450
0.434
0.858
Aigs Xag 1.0
23.62
0.457
0.407
0.382
0.828
Table 14: Effect of the scale of auxiliary mask loss on 3D awareness performance: We provide
PSNR, FG-ARI, and DCI performance as function of the scale of auxiliary mask loss.
Suboptimal mask. We evaluate the performance of our method trained with suboptimal masks- rather than ground-truth masks-across four generalization settings (ID, CG, CG (same color), and OOD), well as under training views, randomized novel views, and single-view inputs. Because our approach requires separating the agent, background, and remaining objects, natural concern is whether high-quality masks necessary. However, we emphasize that our method can operate effectively without direct access to GT agent or background masks. For the agent mask, given the robot's URDF and joint readings, we reconstruct its current visual appearance, and render the cor- responding agent mask for any desired viewpoint with a rasterizer using the camera pose obtained via standard hand-eye calibration. For the background mask, we acknowledge the inherent ambi- guity in defining what constitutes background. To address this, we obtain coarse foreground masks using DINO (Oquab et al., 2023 Caron et al. 2021) by normalizing the principal component of its feature map; their complements provide coarse background masks. Leveraging DINO's large-scale self superviee pre-training allows us to reliably distinguish foreground from background despite the lack of explicit labels. By combining these components, obtain suboptimal background and agent masks for training. Representative examples of the real-world agent masks and simulator- derived background masks shown in Figure 15
27

---

Published as a conference paper at ICLR 2026
As shown in 13] the GCRL performance of our model trained with suboptimal masks is closely approach that of the model trained with ground truth masks. This holds consistently across all four generalization settings- ID, CG, CG (same color), and OOD and remains true when evaluated under OOD randomized views or when provided with only a single input view. other words, model is capable of decomposing background, foreground, and agent regions using only suboptimal masks, without requiring any access to ground-truth segmentations, while still achieving GCRL performance comparable to the ground-truth-trained counterpart. This demonstrates that our approach offers strong 3Daawrre, view-agnostic attribute-centrr generalization, enabled solely by suboptimal masks that are readily obtainable in both simulation and real-workd scenarios.
Simulator RGB Image Suboptimal BG Mask
Real-world RGB Image
Suboptimal AG Mask (Real-world)
(Simulator)
Figure 15: Examples of suboptimal mask: We show examples of suboptimal masks: background masks derived from DINO attention maps in simulation, and agent masks obtained from hand eye calibration and robot kinematics in real-world settings.
Generalization Settings
ID
CG
CG (same color)
OOD
ID Multi-View
0.967 1 0.017
0.895
0.111
0.837
0.035
0.828 1 0.099
OOD
Multi-
0.948 1: 0.005
0.877
0.024
0.818 0.025
0.865 1: 0.047
GT masks
View
ID Single-Vies View
0.891 + 0.004
0.705
0.023
0.700
0.038
0.727 + 0.105
OOD
Single-
0.802 + 0.028
0.726 0.039
0.676
0.758 + 0.021
View
ID Multi-View
0.974 = 0.023
0.771
0.000
0.880
0.891 +
Ours w/
OOD
Multi-
0.943 1. 0.037
0.870 0.022
0.813 0.029
0.870 1 0.052
suboptimal masks
View
ID Single-Vier View
0.917 + 0.000
0.771
0.088
0.703 0.007
0.766 + 0.022
OOD
Single-
0.854 + 0.044
0.730
0.015
0.682 0.066
0.730 + 0.015
View
Table 15: Success rate of goal-conditioned RL for models with GT masks versus subop- timal masks: Our 3D block-slot representation supports stable slot assignments even when trained with suboptimal masks instead of GT masks. To evaluate this, compare the goal-conditioned RL success rates of our model trained with GT masks and with suboptimal masks across four envi- ronment generalization settings (ID, CG. CG (same color), OOD) and four view settings (ID Multi- View, ID Single- View, OOD Multi View, OOD Single-Viee View)
C LLM USAGE
We large language model (LLM) to detect and correct grammatical errors and awkward phras- ing.
28


Reviews:
[{"review_id": "yxVivVMh4e", "reviewer": "Reviewer_PdX4", "summary": "The paper proposes a 3-D block-slot encoder–decoder architecture that fuses a multi-view Object Scene Representation Transformer (OSRT) with a block-slot decomposition module inspired by SysBinder. Each scene is encoded into a set of latent “blocks,” each corresponding to an object slot that factorises attributes such as shape, colour, size, and position. Two auxiliary supervision losses constrain fixed background and agent slots.\nOn top of this representation, a Block Transformer (BT) policy performs block-wise cross-attention between current and goal states for goal-conditioned manipulation.", "questions": "1. How does performance degrade if only a single camera is available at test time? Can the model lift monocular views to 3-D reliably?\n\n2. Please report inference latency per frame and total compute (encoder + BT) versus DLPv2 and OSRT baselines.\n\n3. If background/agent masks are unavailable, does the model still allocate them automatically or does performance collapse?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper correctly identifies that prior slot-based encoders struggle with 3-D occlusion and pose entanglement, and that OSRT-like models lack attribute factorization. The system architecture is easy to follow, and ablations are reasonably thorough.\n\n2. The combination of OSRT’s volumetric scene reasoning with SysBinder-style factorized slots is a clever, pragmatic design. The explicit background and agent slots make the representation interpretable and stable across timesteps.\n\n3.  On both Clevr3D and IsaacGym3D, the model shows clear quantitative gains in decomposition (FG-ARI and D/C/I) and control success. The Block-Transformer (BT) policy demonstrably improves compositional generalization, especially in out-of-distribution color/shape settings.", "weaknesses": "1. The method’s conceptual innovation is modest. The encoder is largely a direct hybrid of OSRT (multi-view scene transformer for volumetric representation) and SysBinder (slot factorisation via attribute prototypes). The Block Transformer (BT) is a small architectural tweak of EIT, with cross-attention operating at the block level instead of the pixel/feature level. This makes the contribution primarily engineering integration rather than algorithmic advancement. The claimed novelty of “3D-aware block-slot” reduces to combining volumetric rendering with slot decomposition.\n\n2. A major conceptual gap is the reliance on explicit segmentation masks for two dedicated slots: Eq. (2) defines auxiliary losses requiring binary masks for the background and agent (manipulator). Without such supervision, it’s unclear whether the model can autonomously discover these roles. This violates the unsupervised or weakly-supervised assumption common in object-centric learning, where slots are meant to emerge purely from reconstruction signals. In real-world robotics, such masks are rarely available. In unlabeled multi-object scenes, the model might not allocate the correct number of slots or separate the manipulator from movable objects.\n\n3. Several notational and definitional issues appear: In Eq. (1), the attention-weight mask loss compares per-ray weights \n𝑤 with 2D masks 𝑚; the mapping from ray to pixel is unspecified, making the loss potentially ill-posed.\n\nIn Eq. (3), λ_bg and λ_ag are introduced but not analyzed, their effect on disentanglement or convergence is unknown.\nThe model description doesn’t clarify how 3D latent “blocks” correspond to volumetric regions or voxels; thus, the claim of 3D-awareness is more architectural than mathematical.\n\n4. Does not currently release its implementation or code repository.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p1lvnvwL20", "reviewer": "Reviewer_BVmH", "summary": "This paper proposes a 3D block-slot representation for vision-based reinforcement learning (RL), targeting improved compositionality, interpretability, and 3D-awareness in multi-object manipulation domains. The method integrates a multi-view transformer encoder with a novel block-slot attention mechanism, decomposing object representations into attribute-level blocks (e.g., color, shape, position) and separating active (objects), passive (background), and agent entities. The paper introduces a block transformer policy leveraging these structured representations for goal-conditioned RL, aiming to boost generalization in out-of-distribution and compositional scenarios. Experimental results are presented on Clevr3D and IsaacGym3D, highlighting gains in object decomposition, disentanglement, and RL performance compared to strong scene-centric and object-centric baselines.", "questions": "1. See weaknesses section\n2. Could you share insights or figures analyzing the impact of number of slots, blocks and prototypes on both decomposition and RL quality?\n3.  It is unclear whether the permutation of slots between the current and goal images influences the policy’s performance. Does the model explicitly handle slot alignment or permutation, or could mismatched slot ordering negatively impact policy learning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Explicit 3D Disentanglement**: The proposed 3D block-slot attention mechanism, as described in Section 2.1.3, provides structured disentanglement of object attributes into interpretable blocks (e.g., shape, color, size, position), enabling clear alignment with compositionality goals. This is well illustrated in Figure 3 and further analyzed in Figure 8 and Figure 9, which show that certain blocks consistently encode specific semantics (e.g., shape, color).\n2. **Separation of Agents, Foreground, Background**: The method distinguishes between active (object), passive (background), and agent entities, as detailed in Section 2.1.2, with dedicated auxiliary objectives enforcing slot assignment and improving downstream policy stability. Figure 6 demonstrates visually superior decomposition versus OSRT, where the agent and background are reliably isolated.\n3. **Goal-Conditioned Block Transformer Policy**: The block transformer policy (Section 2.2, Figure 2 and Figure 5) leverages block-level cross-attention for precise goal conditioning, explicitly matching object attributes in current and target scenes. Table 2 quantitatively supports the claim that this block-level policy bolsters generalization, especially in compositional and OOD tasks.", "weaknesses": "1. **Limited Diversity of Environments**  \n   The evaluation is restricted to Clevr3D and Isaac Gym 3D environments. Expanding the experiments to include additional benchmarks would provide stronger empirical evidence and reinforce the paper’s claims of effectiveness.\n\n2. **Incomplete Baseline Comparisons**  \n   The work compares performance only against earlier object-centric models such as DLPv2 and OSRT. However, more recent and effective object-centric world models [1], [2] should also be considered. Even though these operate in 2D, their inclusion would clarify whether the 3D extension genuinely enhances object-centric policy quality.  \n   Additionally, comparisons with non-object-centric 3D baselines [3] and visual 2D non-object-centric RL approaches [4], [5] would help assess whether object-centric representations offer advantages for policy learning.\n\n3. **Insufficient Ablation and Sensitivity Analysis**  \n   The paper introduces several architectural and training design choices that are not thoroughly examined.  \n   - The choice to apply block-slot attention only to foreground slots (Section 2.1.3), while using vanilla slot attention for agent and background, is motivated by intuition but lacks empirical validation. A comparison among different configurations—such as applying block-slot attention to all slots, a mixed setup, or using only vanilla slot attention—would strengthen the justification. Similarly, the auxiliary mask losses in Equation 4 are not ablated.  \n   - Key hyperparameters such as the number of slots, blocks (Table 3), and prototypes are chosen without detailed diagnostic analysis. The absence of sensitivity studies makes it difficult to assess reproducibility, robustness, and generalization of the proposed method.\n\n\n[1] Ferraro et al. “FOCUS: Object-Centric World Models for Robotics Manipulation.” 2023. [https://arxiv.org/abs/2307.02427](https://arxiv.org/abs/2307.02427)\n\n\n[2] Mosbach et al. “SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels.” 2024. [https://arxiv.org/abs/2410.08822](https://arxiv.org/abs/2410.08822)\n\n[3] Shim et al. “SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning.” 2023. [https://proceedings.mlr.press/v202/shim23a.html](https://proceedings.mlr.press/v202/shim23a.html)\n\n[4] Hafner et al. “Mastering Diverse Domains through World Models.” 2023. [https://arxiv.org/abs/2301.04104](https://arxiv.org/abs/2301.04104)\n\n[5] Zhou et al. “DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning.” 2024. [https://arxiv.org/abs/2411.04983](https://arxiv.org/abs/2411.04983)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ijb4vMkiGx", "reviewer": "Reviewer_xPHR", "summary": "This work proposes a 3D-aware object-centric representation of scenes for training Reinforcement Learning (RL) agents that exhibits improved generalization capabilities to unseen compositions of object-level attributes in goal-conditioned robotic manipulation tasks. The representation is based on the 3D object scene representation transformer (OSRT) and incorporates the block-slot attention module for further decomposition of slots into object-level factors. The representation is evaluated on Clevr3D and a novel simulated robotic manipulation dataset and improves over OSRT in various metrics. The paper additionally proposes a policy architecture for a goal-conditioned RL task that leverages the object-level factorization of the representation to achieve better zero-shot transfer to unseen compositions/variations of object-level attributes compared to baselines.", "questions": "**Suggestion**\n\nIn my eyes, the main contribution of the paper is the 3D-aware object-centric representation and its applicability for training RL agents that generalize. This view is based on the experimental results, but is not highlighted by the structure and sentiment of the paper (other than the title).\nThe following suggestion is structural and semantic, and is of course left to the discretion of the authors, i.e., it will not affect my recommendation for acceptance/rejection:\n\nCenter the contribution around the 3D block-slot representation, and present the block transformer policy as an *example* of how this representation can facilitate design of task-specific architectures (although I do not think policy architectures should be task-specific). I would move the results in Figure 11 to the main text and discuss how with a fixed policy architecture:\n1. 3D-aware representations can improve in-distribution generalization: evidence to that is the sample efficiency of OSRT and your representation compared to DLP on ID.\n2. 3D-aware representations with factored attribute structure (i.e., yours) can improve attribute-level compositional generalization: evidence to that is the performance of yours compared to OSRT in CG and CG (same color).\n3. Maybe discuss why OSRT performs better on OOD generalization with the same architecture.\n\n**Questions**\n\n- How much does the representation rely on knowing the amount of objects/slots in advance?\n- How stable is training the block slot representation in terms of disentanglement of objects to distinct slots and factors to distinct blocks? Does the desired factorization emerge consistently across training runs?\n- There are no details about the attribute matching mechanism in the block Transformer. Can the authors provide further details on how this is done?\n- Are the authors planning to release code or take other measures for reproducibility purposes?\n- A major benefit of the 3D aware representation in my opinion is that it may be view-agnostic. Have you considered training the policy when randomizing the viewpoints (maybe with some restrictions on their directions such that they will capture sufficiently different angles of the scene)? Demonstrating view-generalization capabilities can have a very large impact on the robotics community since vision-based control methods are largely not robust even to slight variations in camera angles to my impression.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Overview**\n- Novelty and contribution are clear.\n- Self-contained and clear method description.\n- Learned representation facilitates human interpretability and controllability. \n- Good empirical results in terms of transfer to unseen test tasks.\n- Thorough study of the different method components.\n- Detailed and helpful figures.\n- Detailed Appendix.\n\nI am positive about raising my score if at least some of the concerns I have raised in the Weaknesses/Questions sections are addressed.", "weaknesses": "**Overview**\n- Abstract and Intro are not clear and logic is hard to follow.\n- Assumes access to agent and background masks.\n- The Block Transformer relies on human annotation of blocks within the proposed block-slot representation (post representation training but prior to RL policy training).\n- Incorporates multiple assumptions about the specific task in the experiment within the RL Block Transformer architecture, which may limit its wider applicability.\n\n**Abstract**\n\nAfter reading the paper and understanding it, I still find the abstract very confusing. It uses terms which I would not consider as common knowledge such as “shared concept memory” and some misleading phrases such as “The representation, therefore, can stably identify and track 3D trajectories”---reading this I would assume the representation operates on sequences of images (i.e., videos) but it is a multi-view image representation if I am not mistaken. I suggest making the abstract more “abstract” to appeal to the general reader.\n\n**Introduction**\n\nThe two key challenges presented in the introduction for “fully utilizing object-centric representations” (lines 52-53) should be backed-up by citations or by results in this work but these are not referenced. This paragraph reads more like—“these two aspects of our method are what is missing”---but the challenges and their causes are not clearly described.\n\n**Assumptions and Limitations**\n\n*Agent and background masks*: this assumption makes the overall method no longer unsupervised. That in itself is not a problem but could introduce a limitation which stems from reliance on humans to e.g., define what is considered the “background” as opposed to being learned from data. It seems that the method strongly relies on the assumption that these masks can be obtained with high fidelity which may make the overall pipeline brittle when this is not the case. The authors claim that these masks can be obtained using foundation models. If so, why not obtain the entire object-centric representation using foundation models? Is your representation-learning method applicable without the dedicated background and agent slots?\n\n*Identifying the meaning of latent block attributes*: your method seems to produce human-interpretable disentanglement in the latent slot representations of scenes considered in this work, which is impressive and a positive outcome in my opinion. That said, your RL policy architecture relies on the human interpretation of the role of each block, which is another form of supervision that can make scaling difficult and introduce brittleness.\n\n*Strong assumptions about a specific task embedded within the policy architecture*: The proposed Block Transformer is designed for a very specific kind of task which limits its wider applicability. The authors “suppose the task is to relocate objects with the matched attributes into the goal position” (lines 242-243), which is not just treated as an example but determines the processing of state and goal slots. The block-wise cross-attention only allows matching objects to attend to each other which is a strong inductive bias that limits its expressivity. What if the task requires reasoning about the relationships of multiple goal objects? What if the task is to bring all the objects to the location of a single predefined object? Every change in the task/reward incurs a change to the architecture, which is not a desirable property. Judging by the results in Figure 11, this limited expressivity seems to not be beneficial even for the task it was designed for, resulting in lower sample efficiency compared to the EIT with the same representation.\n\nLimitations of the approach should be discussed in the paper, possibly in the Conclusion section.\n\n**Experiments**\n\n*3D Block Slot Representation*: it is not clear how much of the improvement in the metrics you considered stem from the additional supervision versus the additional block structure. Experiments evaluating OSRT with the mask objective would help shed light on this matter.\n\n*Goal-conditioned RL*: As I see it, the strong assumptions about the task embedded within the architecture are the main reason for the improved performance on the unseen test tasks. Since the matching mechanism is not learned, these tasks do not require much generalization in the learning sense, would the authors agree with this statement? Thus, although these findings are interesting, they could be considered narrow. I find the results in Figure 11 much stronger and I suggest they be moved to or at least discussed in the main text. The explicitly 3D-aware representations result in significant gains in sample efficiency compared to the 2D representation when paired with the EIT, which points to the fact that they facilitate better in-distribution generalization of the RL policy. Please see my suggestion under **Questions**.\n\n*Misc*:\n- How many seeds were used for the experiments?\n- How many objects are present in the RL environments? It is not explicitly stated in the paper.\n- Are the OOD colors also unseen during the representation model training or just the RL agent?\n- I suggest highlighting results that are within a STD from the best performing method in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)