Explicitly Disentangled Representations in Object-Centric Learning
Authors: Riccardo Majellaro
Jonathan Collu
Aske Plaat
Thomas M. Moerland
Venue: Accepted by TMLR ()

Abstract: Extracting structured representations from raw visual data is an important and long-standing challenge in machine learning. Recently, techniques for unsupervised learning of object-centric representations have raised growing interest. In this context, enhancing the robustness of the latent features can improve the efficiency and effectiveness of the training of downstream tasks. A promising step in this direction is to disentangle the factors that cause variation in the data. Previously, Invariant Slot Attention disentangled position, scale, and orientation from the remaining features. Extending this approach, we focus on separating the shape and texture components. In particular, we propose a novel architecture that biases object-centric models toward disentangling shape and texture components into two non-overlapping subsets of the latent space dimensions. These subsets are known a priori, hence before the training process. Experiments on a range of object-centric benchmarks reveal that our approach achieves the desired disentanglement while also numerically improving baseline performance in most cases. In addition, we show that our method can generate novel textures for a specific object or transfer textures between objects with distinct shapes.

Published in Transactions on Machine Learning Research (01/2025)
Explicitly Disentangled Representations in Object-Centric Learning
Riccardo Majellaro"
ricardomaijellnn@@maad. com
Jonathan Collu"
jonathancollu1199888maaa com
Aske Plaat
a.plaat@@lacc...dddeeaaaa
Thomas M. Moerland'
L. wwn.marrtannd@iiissseeeddaaann
Leiden Institute of Advanced Computer Science, Leiden Universily
Reviewed on OpenReview: https: openreview. net/ forum? td=r8UPp90100
Abstract
Extracting structured representations from raw visual data is an important and long-standing challenge in machine learning. Recently, techniques for unsupervised learning of object-centric representations have raised growing interest. In this context, enhancing the robustness of the latent features can improve the efficiency and effectiveness of the training of downstream tasks. A promising step in this direction is to disentangle the factors that cause variation in the data. Previously, Invariant Slot Attention disentangled position, scale, and orientation from the remaining features. Extending this approach, we focus on separating the shape and texture components. In particular, we propose a novel architecture that biases object-centric models loward disentangling shape and texture components into two non-overlapping subsets of the latent space dimensions. These subsets are known priori, hence before the training process. Experiments
range of object-centric benchmarks reveal that approach achieves the desired disentanglement while also numerically improving baseline performance in most cases. In addition, show that our method can generate novel textures for specific object or transfer textures between objects with distinct shapes.
1 Introduction
A key challenge in deep learning is to learn data representations that can be leveraged in a multitude of downstream tasks. The promise of exploiting large unlabeled datasets has driven research in unsupervised and self-supervised settings, despite the inherent complexity of this task. In the context of computer vision, recent works focus representing, without supervision, visual scenes composed of multiple objects sets of latent vectors centered around objects (Burgess et al. 2019 Greff et al. 2019 Engelcke et al.) 2019 2021 Locatello et al. 2020h. This object-centric strategy allows for more structured representations compared to single flat vector encodings complete images. With single flat vectors, multiple objects can entangled in components of the same representation or separated into distinct dimensions. However, even when separated, objects still cannot share features of common propertics such their position. Instcad, with object-centric vectors, visual entities are encoded individual vectors within the same space, improving generalization and interpretability entities clearly divided but described by the same features. On top of that, these representations naturally scale to an increasing number of objects. Our work is based on this approach, often referred to as object-centric representation learning.
Disentanglement is considered a central aspect learning robust representations complex and structured visual scenes (Bengio et al. 2013 The term refers to the encoding different factors of variation in the data
* Work done while LJACS, Leiden University.

---

Published in Transactions on Machine Learning Research (01/2025)
Inpul
Texture
Nltonton
Masks
Texture Encoder
Teckino
Teckiite
Staple
Decoder
Teckiny
Sobel Fiter
Recerstruted
Shape Enooder
Insarient
Meek Dooocer
Aitentioon
Shapps
Module
Shappl
Shape Vecters
Fitared Irpal
Predicted Masks
Figure 1: Architecture of DISA. The imput image (top-left) is first fed through Sobel filter to partly remove texture information (bottom-left). This filtered image is then encoded and passed through an Invariant Slot Attention (Biza et al. 2023) module to extract object-centric shape (Shape,), position (P;), and scale (S;) vectors (bottom-miie middle)).. This information is used to decode a mask for each object (bottom-right), The initial input image encoded (top-left) and then combined with the object attention masks from the ISA module to produce a texture vector (Texturee) per object (Lop-middle). These texture representations are combined with their associated shape, position, and scale to decode the textures of the objects (top-right). Note that need to add the shape, location, and scale information as the texture decoding should fit the already predicted masks. Finally, the decoded masks and textures combined into image reconstruction (right).
into separate dimensions of the latent space. Disentanglement can provide different advantages, including (1) enhanced interpretability of the learned features, (2) compositional generalization beyond the training data distribution, and (3) helping the lcarning of relevant yet unknown downstream tasks (c.g., in the context of reinforcement learning). Prior probabilistic work on object-centric representation learning (Burgess et al. 2019 Greff et al.) 2019) exploited VAEs (Kingma & Welling 2013) and spatial broadcast decoders (Watters et al. 2019), which helped the disentanglement of the features. In contrast, the non-probabilistic approach Slot Attention (SA) [Lcateelo et al., 2020) extracts object vectors based on a competitive cross-altention mechanism. However, the latter method lacks clear property-level disentanglement, as shown in (Singh al. (2022). Biza al. (2023) proposed an extension of SA, namely Invariant Slot Attention (ISA), aimed at learning object representations invariant to position, orientation, and scale, allowing for the explicit disentanglement of those three factors. However, to the best of our knowledge, research has been carried out the explicit disentanglement of the texture and shape dimensions in unsupervised object-centric learning, especially with non-probabilise bilistic models. Note that, by the term explicit, we refer to the process of choosing which latent components will be responsible for encoding certain factors, knowing them a priori. contrast., probabilistic models such as VAE 3-VAF (Higgins et al., 2016) cannot enforce which properties disentangle and control where they should be encoded. Possible benefits of explicit disentanglement include easier interpretation and more reliable modeling of structured latent spaces. Moreover, it creates the possibility of designing objectives that exploit this prior knowledge from the beginning, removing the need to analyze and interpret the latent components during or after the training process.
Our work focuses on biasing object-centric models toward explicitly disentangling the groups of features responsible for encoding shape and texture information into two non-overlapping subsets of the latent space dimensions. By building upon ISA, we combine the separation of position and scale with that of texture shape. As these are core properties objects, this result can be crucial for further improving the internal structure of the learned representations, especially in non-probabílistic setting. To disentangle texture
2

---

Published in Transactions on Machine Learning Research (01/2025)
and shape, we design a novel architecture that employs two encoder-decoder pairs. The first pair encodes shape information and decodes object masks; the second pair represents and predicts object textures. In addiiion, filter (e.g. Sobel filter, [Soell& & Feldman (1968) ) is used to partly remove texture information from the input image before feeding it to the shape encoder, helping to prevent texture information from flowing into shape-related latent. components. Furthermore, a regularization lerm is introduced to reduce the variance of the latent features across slots, helping to deter shape information from being included texture-related components and vice versa. We name our approach DIsentangled Slot Attention (DISA). DISA is characterized by latent space divided into four distinct groups known since the beginning of the training process: the first two encoding texture and shape, the third position, and the last one scale (Figure
Our experimental evaluations indicate that, in most cases, DISA introduces the desired disentanglement property, enabling novel generative and compositional capabilities. At the same time, DISA competitive with or outperforms the basclines at scene decomposition on three well-known syuthetic datasets, while achieving significantly improved reconstruction quality. As such, DISA enhances both the interpretability and reconstruction quality (information content) of learned object-centric image representations.
2 Background
2.1
Slot Attention
The Slot Attention (SA) module is architectural component proposed by Locatello et al. (2020) based on the concept of cross-attention (Luong et al. 2015 Vaswani et al. 2017) Its aim is to bind a set of latent vectors called slots to different objects, seen as parts of the input perceptual representations, through multiple iterations of a competitive attention mechanism. The set of N, slots S NNN initialized either as learnable vectors or by independently sampling each from a Gaussian distribution with shared and learnable parameters. The N₂ input representations are obtained by augmenting the feature vectors X E Naxd at the output of a CNN backbone with positional embeddings g(G), where G is flattened grid encoding absolute positions and a linear mapping from the grid coordinates space to dimension d,. These augmented imputs are then passed through an MLP f. normalized using layer normalization (LayerNorm)
BBa et al. 2016) and finally projected as keys and values K, V.Ve E TR Na xd of a common dimension d through the linear transformations k v RRd, - Rd. At each of the iterations, the slots from the previous round are (independently) normalized, mapped as queries Q E N.xd by q: Rdd + Rd, and then refined.
At a given refinement step of the Slot Attention mechanism, the previous slots S are mapped as queries and fed to the competitive cross-attention module, which yields one vector U, for each Q, referred to update. The updates are then passed as inputs to shared Gated Recurrent Unit, (GRU) (Cho et al. 2014), while the associated slot vectors from St are set as hidden states. The outputs, which are updated hidden states, are transformed by a shared MLP with a residual connection (He et al., 2016), resulting in the new slot vectors St. The competition between slots for explaining part of the input is induced by an additional normalization in the cross-attention mechanism, performed by using the softmax function over the queries before linearly normalizing over the keys. Formally, the attention coefficients are computed as
Mu
QK E N.N.N.
úij
where
M
(1)
N. 11=1
Mip
vd
producing a probability distribution over the slots for each input voctor. After that, the update vectors U, are calculated as
N, U Σ 1=1
N
Wi with
Wy
wiy No Σ 0u
such that
> J=1
Wi
It possible to exploit Slot Attention, coupled with a backbone, as structured encoder in an autoencoder architecture. The model is trained end-to-end to represent an input image as set of latent vectors that are decoded back to the original image. A shared spatial broadcast decoder (Watters et al. 2019) is used
3

---

Published in Transactions on Machine Learning Research (01/2025)
to decode the slots individually: each is repeated in H x W grid, augmented with positional embeddings, and passed through series of convolutional layers. The output is an H x W x 4 tensor, where the first three channels correspond to RGB components, while the last one is the alpha mask of the decoded object. The reconstructed image is computed as the sum of the RGB predictions weighted by the alpha masks. At training Lime, the model parameters are optimized minimize the mean squared error (MSE) between the reconstructed and input pixels.
2.2 Slot-Centric Reference Frames
With Invoriant Slot Attention (ISA), Biza al. (2023) presented mechanism for introducing per-object invariances to introduce transformations the latent representations. To achieve this, slot-centric relative grids are produced by translating, scaling, and rotating the absolute positional encodings, with the aim of processing the feature veclors in canonical reference frames. The rotation invariance is not covered in this section as it is not employed in our method.
At the first iteration Slot Attention, translation and scaling factors Sp and Secale associated with the slot S, can be randomly sampled or learned. The relative grid then compuled and its projection summed to the feature vectors X to obtaín per-object keys and values K Vi Noxd
G - Spos Secale
G
and K' f(((X)+
V v²= f((x())+
(3)
K' is then processed by competitive cross-attention (Eguation with the query derived from S; to yield M,, finally used to compute Wy and U, with v (Equation 2. During the next iteration, wy is used to extract the new translation and scaling factors as
No WeyG D
No Ee (wy + c), spoeyz
Spos
and
Secale
(4)
Eyn (wi) +
1= Wij
which are again employed to calculate the relative grid of the i-th object. Note that the grids have two dimensions (horizontal and vertical) with values in the range 1].
During the decoding phase, the relative frames are used again to provide back position and scale information and ensure invariant decoding of the objects. In this case, the last translation and scaling factors are exploited to compute the grids in Equation and added to the broadcasted slots.
3 Related Work
Object-centric representation learning The method propose is aimed at models learning object- centric representations of visual scenes. These approaches, learning in unsupervised or self-supervised manner, are capturing increasingly more attention. A first line of work in this direction is defined by AIR Eslami et al. 2016 AIR performs probabilistic inference employing a recurrent neural network (RNN) that attends and processes one object in a scene at a time. Subsequent work extended this idea in order to solve some of its limitations, such as the SQAIR approach (Kosiorek et al. (0188), SuPAIR (Stelzner et al. 2019). and others (Crawford & Pineau 2019 Jiang al.) 2019 Lin al. 2020). Further advances in object-centric representation learning include Tagger (Greff al. 2016), Multi- Entity Variational Autoencoder (MVAE)
Nash et al. 2017), and Neural Expectation Maximization (N-EM) (Greff et al.) 2017) with its extension R-NEM [Van Steenkiste et al.) 2018). More recently, [Burgoss et al. (2019): Greff et al. (2019); Engeleke et al. 2019 2021) achieved meaningful decompasition of non-trivial scenes with a variable number of objects, e.g.,
in the CLEVR dataset (Johnson et al. 2017). Finally, Slot Attention (Locatello et al. 2020 along with
various extensions (Kipf et al. [2021] Singh et al. 2021 Chang et al. 2022 Jia et al. 2022 Biza al. 2023
Kori et al. 2223), introduces a probbbbbaiiiistic iterative mechanism that competitive with its predecessors while being faster to train and more memory efficient.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Disentanglement in object-centric learning Probabilistic models such as in Burgess et al. 20019 Greff et al. (2019) can obtain a degree of disentanglement due to their foundation on the VAF framework. Differently yet still in a probabilistic setting, Anciukevicius et al. (2020) works toward the explicit disentanglement of position and depth. Mansouri et al. (2022). instead, exploits weak supervision from sparse perturbations and causal representation learning to disentangle object properties. In a non-probabilistic setting, [iinh ct al. (2022) Icarns disentangled representations a non-cxplicit manner, while Biza et al. (2023) introduces invariance to changes in position, scale, and rotation with the use of slot-centrie reference frames, allowing for the explicit disentanglement of those three factors. We exploit the concept of slot-centric reference frames in our work. Furthermore, similarly to AAncuuevviiiu et al. (2020); Biza et al. (2023), we possess a priori knowledge of which of the latent dimensions are associated with a disentangled factor, in contrast with, e.g., Burgess et al. (2019); Greff el. al. (2019); Singh el al. (2022). where this knowledge can only emerge after training. Prabhudesai et al. (2020) explicitly disentangles shape and texture in object-centric learning but relics on ground-truth bounding boxes and multiple viewpoints per static scene (more Appendix C To the best of our knowledge, no research dealt with the explicit disentanglement of texture and shape factors in unsupervised object-centric learning, which is the aim of our work.
Texture and shape disentanglement Outside the scope of object-centric learning, approaches including Shu et al.] (0188); Lorenz et al. (2019); Yan et al. (023) attempt to disentangle shape and texture in the domain of single-object images. Deforming Autoencoders (Shu et al., 2018) employs a pair of decoders, of which one synthesizes appearance in a deformation free coordinate system, while the other estimates a deformation field that warps the texture into the input image. One aspoct that aligns with our work is the adoption of two separale decoders for texture and shape synthesis. Lorenz et al. (2019) aims to disentangle appearance and shape in the representations of multiple parts of a single object class, without supervision. To do so, they introduce into the reconstruction task three invariance and equivariance constraints, by exploiting texture and shape transformations of an imput image (more in Appendix (.. Finally, TSD-GAN (Yan et al., 2023) uses an adversarial framework to learn to both reconstruct the item in an input image and mix it with the item from another sample.
4 Methods
The primary aim of our work is to induce object-centric models to learn reprosentations characterized by the explicit, and thus known priori, separation of the groups of features encoding texture and shape information. To allow for this, we propose an architectural design along with a simple regularization of the latent space. We apply our approach to Invariant Slot Attention (ISA) and call the resulting model DISA, for Disentangled Slot Attention (Figure D. The choice of ISA enables us to further integrate the explicit disentanglement of position and scale our model by leveraging slot-centric reference frames.
4.1 Architectural Design
Consider generic architecture for unsupervised object discovery, encoding image into set of latent representations (slots) and decoding them as pairs of mask and texture reconstructions. We want the slots to have a first subset of their components encoding texture information only (e.g. material and color), a second one encoding only shape information, and the rest of the dimensions representing what is still unexplained, such as position and scale. To store shape information within the shape group, we decode object masks exclusively using those components, as knowledge of an entity's shape essential for accurate masking prediction. Similarly, we want to force texture information inside the texture group, which we achieve through separate decoder that infers object textures from those components. However, as textures need to be decoded according to the associated object shape, shape information would also flow into the texture group. To avoid this, we let the lexture decoder exploit both texture and shape subsets. Nevertheless, the current design (with two decoders) allows the shape components to encode all the necessary information, which would thwart all our efforts. Therefore, we ultimately employ two distinct encoders: one extracts and represents texture information from the input image into the texture group, while the other encodes the shape information inside the shape group starting from filtered imput. The filter should ideally remove all
5

---

Published in Transactions on Machine Learning Research (01/2025)
the texture information from the image, but in practice, it can be sufficient to erase part of it to bias the model as we desire.
As it is, the design does not yet deal with position and scale information, which would thus end up in the texture and shape groups instead two different ones, Moreover, the texture components could still include some shape-related information and vice versa. We address these limitations the next section with the implementation of DISA.
4.2 Disentangled Slot Attention
In order to implement the design described in Section Invariant Slot Attention according to the above principles (Figure I and train with the reconstruction loss along with simple latent space regularization.
4.1 while addressing its stated limitations, we extend
Let X' E [0, Hx33 be RGB input image, & E R4 B#x be X processed with a Sobel filter, and Ptex : [0, 1] xx + IRN xd, Pshape 14 #x!x + xd, the texture and shape CNN backbones. First, the texture and shape input representations are obtained respectively as Xtex (xxx(()) and Xshape aahape (R).. Then, the ISA mechanism (Section 2.2) is applied to Xshape: at each iteration t, the keys and values relative to an object are computed as
Kahape fshape (Kahapo (Xshape) + Ishape and
Vi Vahape fahape (VahapeVaaape (Xshape) + Ishape (G¹))
(5)
where G is calculated using Spne and Stcale from the previous iteration. Shape and V shape are used with Q₁ in Equations 2 to yield uy and U,, necessary to get the new shape-related vector Sp Sthape = ( (S; (Sphape U.)- T,
hahape represents the GRU followed by a residual MLP. At t = T, the final shape vectors, which are invariant to position and scale, are returned and employed in an additional ileration coefficient Wiy position vector spos, and scale vector Speale.
+ to get the final attention
As the objects in the image are alrcady discovered, there is no need to apply an additional ISA mechanism to extract the texture-related vectors. Instead, the last Wi, Sp and Secale can be exploited to combine Xtox (in position and scale invariant manner) according to the final ISA iteration as
N wi
Six = 1itex
Vi tex,
with tex = Vroxx(Uttrs (Vtex(Xtxx) + Jtex (G'))
(6)
In this case, MaX simply an MLP without residual connection and not preceded by a GRU. Note that Stex N, xduex, gshape E R2 22x,N,aappe and gnos Secale N22x22 hence each object represented by slot of dimension diex + dshape 2 2.
Finally, the texture and mask of the i-th object are inferred by two spatial broadcast decoders, where the broadcasted slots are augmented with the relative frame obtained with Spos and Secale, Specifically, the mask decoder input is composed exclusively by the shape vector Sthape while the texture decoder is fed with the concatenation of Stex and Sthape
We train the parameters of DISA with the reconstruction loss along with simple regularization on the variance of the latent features across slots:
1 diea
datape
C = Lree + A
Var(Stoxi)
1
Var(Sehapon
drex
dshape
1m1
where Stex, = (Stex... S Sohape, = (St Sshape), ). When the loss computed over a batch of N₂l
images, N, bocomes N, times the number of images in the batch. The second term introduced with the purpose of removing the remaining shape-related information from gtex and texture-related information from Sshape In fact, the same texture can be encoded slightly differently when belonging to objects with distinct.

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: BG-ARI and FG-ARI (t) averaged over seeds and represented as percentages, in the format mean stddev. FG-ARI excludes the background and only considers the foreground objects, while BG-ARI considers both foreground and background. For ISA (on Tetrominoes and CLEVR6) and SA, we perform 10 evaluations of cach test image to address the stochasticity arising from the sampling (Appendix D.1.
BG-ARI
FG-ARI
Tetrominoes
Multi-dSprites
CLEVR6
CLEVRTex
Tetrominoes
Multi-dSpriites
CLEVR6
CLEVRTex
SA
42.7
43.4 38.5
92.8
16.4
98.4
55.8 26.0
92.1 0.4
64.2 1 14.6
ISA
99.1 0.5
97.2 1 0.9
84.1 1 2.0
23.5 3.6
99.3
1 0.3
71.0 1 5.5
93.9 1 0.4
71.3 1 188
DISA
96.7 1 2.0
98.9 1 0.1
96.3 1 0.1
32.2 1 10.0
94.4 1 3.6
90.5 1 1.0
97.0 1 0.2
79.2 1 0.2
Table 2: MSE (L) averaged over 3 seeds and represented in the format mean + stddev. All the numbers in the table are scaled by 10t. For ISA (on Tetrominoes and CLEVR6) and SA, we perform 10 evaluations of each test image to address the stochasticity arising from the sampling (Appendix D.I.. DISA significantly surpasses the baselines in terms of reconstruction quality in most cases.
Tetrominoes
Multi-dSprites
CLEVRG
CLEVRTex
SA
6.87 # 1.99
22.90 + 19.11
5.90 + 0.50
60.90 # 1.92
ISA
6.30
3.3
4.57
1.88
6.03
0.26
37.70 0.78
DISA
2.67
0.97
2.20
0.08
2.10
41.37
shapes, and conversely, an identical shape can be encoded slightly dissimilarly when belonging to objects with uncqual textures. By bringing set of texture (or shape) represontations close together we try to minimize these undesired differences as much possible. At the same lime, the reconstruction loss prevents vectors belonging to different textures (or shapes) from collapsing to the same values since, in that case, it would not be possible to accurately differentiate them during the decoding process.
5 Experiments
In this section, we evaluate DISA on four well-known multi-objecl synthetic dalasets (abra et al. 2019 Karazija et al. 2021): Tetrominoes, Multi-dSprites, CLEVR, and CLEVRTex (Appendix B). For CLEVR, we train on a filtered version of this dataset called CLEVR6. Each training configuration is run three times with different random seeds to account for the stochastic nature of the experiments. Initially, we compare DISA against the baselines (SA and ISA) to assess its ability reconstruct images and decompose them into objects without supervision. Subsequently, we try to demonstrate that DISA achieves the desired disentanglement property in its latent space. Finally, show (through qualitative experiments) compositional and generative capabilities derived from the disentangled representations. Note that this work seeks to achieve the desired disentanglement within the latent space of DISA rather than focusing obtaining state-of-the-art results in unsupervised object discovery and reconstruction quality. Additional experiments are included in Appendix seatengglldds-d-eeeeeee wlot-attention
F The code is available at /https:// http://gttttt.t/// http://gttuu..ooiiiiooeeeeoootet www.ccoceiiiiie.e
5.1 Object Discovery and Reconstruction
This evaluation investigates whether DISA is competitive with the baselines at unsupervised object discovery and image reconstruction. For object discovery, we compare the predicted object masks with the ground truth (hrough the Adjusted Rand Index (ART) score (Rand 1971 Hubert & Arabie 1985) computed both including (BG-ARI) and excluding (FG-ARI) the background masks. The use of the ARI score is in line with Locatello et al. (2020): Biza et al. (2023). As for the roconstruction quality, we employ the mcan squared error (MSE). Table summarizes the ARI scores, while Table 2 the MSE. We find that on the datascts that were used, our models comparable and, in most cases, even outperform both baselines at decomposing
7

---

Published in Transactions on Machine Learning Research (01/2025)
ghapo
Shape
Blue
Texture,
Blue
Input
Input
(a) Property prediction
(b) Inverse property prediction
Figure 2: Illustration of the two property prediction tasks that propose quantitatively study the degree of texture and shape disentanglement in the representations of DISA. (a) Prediction of an object property based on the associated components. For instance, the color from the texture-related latent features. (a) Inverse property prediction task, where properties are predicted based on the "wrong" part of the object representation, e.g., the color from the shape-related components.
1.0
Accuracy R²
0.8
Shape
Shape (inverse)
Shape (random guess)
Color
0.4
(inverse)
(random guess)
0.2
Material
Material (inverse)
0.0
Material (random guess)
9.2
Tetrominoes
Multi-dSprites
CLEVR6
CLEVRTex
Figure 3: Quantitative results of DISA on the regular and inverse property prediction tasks. The predicted properties shape, color, and material, which all categorical variables except for the color Multi- dSprites. With categorical variables, the prediction accuracy is employed and compared with baseline random guess. With the numerical one, the R² score is shown. We report mean and stddev over seeds. On Totrominocs and Multi-dSprites, DISA correctly encodes texture and shape information into two non- overlapping subsets of its latent space dimensions. On CLEVR6 and CLEVRTex, part of the shape and texture information leaks into incorrect components.
the images into objects. Looking at the MSE, DISA achieves better reconstruction quality than SA and ISA on all the experimented datasets by considerable margin (except on CLEVRTex, where it performs slightly worse than ISA). Overall, despite not being the primary goal of this research, our models are competitive with and, in most cases, even exceed the performance of the baselines.
5.2 Disentanglement Analysis
To verify how effective DISA is in constraining the texture and shape information into the desired components, utilize property prediction (Figure 2): we attempt to predict the ground-truth property of object from the subset of its slot where the related information should be present, and also from subset where the information should not be encoded. To do so, we first train multiple MLPs, each predicting one property of the objects based on the respective part of the slots, such as the shape of a tetromino given its shape vector (Figure 2ah. In this task, we expect excellent results as we infer some property leveraging the part of the representations that should encode it. Instead, we switched the parts and trained another set of MLPs predict, e.g., the color of an object from its shape-related components, would ideally expect to obtain
8

---

Published in Transactions on Machine Learning Research (01/2025)
Input Image
Figure 4: Qualitative results showing the position and scale disentanglement DISA. Given input image (first column), we select two objects and modify their scale/position factors incrementally over 4 steps (second to fifth columns). Texture and shape are consistent across the sleps, while position and scale can be independently varied correctly, suggesting that the models can confine position and scale information in their respective components. The visible limitations are displayed by ISA as well, e.g., the missing parts of an object after moving another object covering it or even the inconsistent masks produced when incrementing the scale certain point. Additionally, on Tetrominoes, scaling object does not yield completely wrong results, which is not trivial the model only sees shapes of fixed size during training.
poor results. An "inverse" property prediction task is carried out to gain insight into this aspect (Figure
2bb. In line with previous literature (Greff al. 2019 Dittadi et al. 2021 Papa et al. 2022), we measure performance using the accuracy for categorical variables and the coefficient of determination (R² score) for numerical variables (i.e. only the color in Multi-dSprites). We report the mean and standard deviation

---

Published in Transactions on Machine Learning Research (01/2025)
of accuracy/ computed over the three seeds on the test set. As baselines, we provide the accuracy of a random guess for each property on every dataset. Moreover, since the objects can be represented in any order, the ground truth labels must be matched with the proper predicted slots. We do so by computing cosine similarities between the target and predicted masks and then coupling each ground truth label with the output of its most similar slot.
The results of these experiments on Tetrominoes, Multi-dSprites, CLEVR6, and CLEVRTex are reported in Figure 8 As expected, DISA reaches near-perfect property prediction scores all datasets (except for CLEVRTex) when exploiting the correct groups of features. When inverting them, notice on Tetrominoes and Multi-dSprites that the accuracy of the categorical predictions drops approximately to that of random guess, while the R² of the numerical becomes negative. The negative RD score indicates that the inverse color predictions are woise than those of a model constantly inferring the mean target value. Therefore, these results show that, on the first two datasets, DISA is able to correctly encode texture and shape information into two con-overlapping subsets of its latent space dimensions, achioving the purpose of this work. In the case of CLEVR6, we find that although the color can be successfully restricted within the texture components, part of the shape and material information leaks respectively into the texture and shape latent features. On CLEVRTex, as both ISA and DISA are unable to preciscly segment and reconstruct the objects, it is likely that the representations do not contain the necessary information to allow near-perfect accuracy even on the regular property prediction task. Even in this case, part of the shape and texture information leaks into incorrect components.
We further present in Figure qualitative results showing that the position and scale disentanglement properties of DISA are preserved. Given an input image (first column); we select two objects and modify their scale or position factors incrementally over 4 steps (second to fifth columns). As visible, texture and shape are consistent across the steps, while position and scale can be independently varied with success. This suggests that the models are able to confine position and scale information in their respective components. Note that the visible limitations are displayed by ISA as well. For example, the missing parts of an object after moving another object covering it, or even the inconsistent masks produced when incrementing the scale over a certain point. Moreover, it is interesting to see that, on Tetrominoes, scaling an object does not yield completely wrong results. This is not trivial as, for a given shape, only a single variation of its scale is present in the dataset, meaning that the model only sees shapes of fixed size during training.
5.3 Compositional and Generative Capabilities
Finally, we also explore the compositional and generative capabilities of DISA derived from the disentanglement of texture and shape through simple qualitative experiments. Precisely, as first investigation, for a given image from the test set we take the representations of the objects, randomly permute their texture parts, and decode the permuted representations. Ideally, the model would output a reconstruction where the shapes, positions, and sizes of the entities are preserved, while their textures are interchanged and adapted to fit the same masks. In the second experiment, we analyze whether this ability extends to the generation of new objects by providing new sampled textures or shapes to the objects in a scene. To do so, we average the texture (or shape) vectors encoded from given image and sample from a Gaussian distribution centered on that mean vector. For additional results, including those on Multi-dSprites and CLEVRTex, please see Appendix F.2 and
Figure 5 reports the compositional results on Tetrominoes and CLEVR6. On Tetrominoes (Figure 5aj, transferring the texture components of one object to another translates into passing the color. As visible, the colors are in fact interchanged between objects without affecting the original shapes and positions, coherently with the quantitative results from Figure 3 On CLEVR6 (Figure 5b), colors and materials are correctly transferred between objects while being shaped, scaled, and positioned according to the shape, position, and scale information. Differently from Tetrominoes, this result only partly supports those from Figure 3 In fact, although we expected the colors to interchange accurately, we would have predicted less precise transferring of shapes and matorials. However, looking at more samples and, as the high standard deviation in the inverse material prediction suggests, different seeds may be necessary for coherence with the quantitative results. Overall, Figure 5 shows strong compositional generalization, enabling reliable transferring of textures between
10

---

Published in Transactions on Machine Learning Research (01/2025)
Irputt Imagee
Musk
Nask
Musk
Musk
Figut Imagee
Mask
Nask
Mask
Masi
Muski5
Pasko
Fermuteed
Prrm Testure
Parm "Boture
Femm Recur
Perm Thature
Permuted
Ferm Teature
Pem Tedure
Aerm
Perm Thabure
Tert.red
(a) Permutation applied to Stext, 4-2-1-3.
(b) Permulation applied to Stext; 6-4-2-1-5-3.
Figure 5: Composit ional capabilities of DISA (a) Tetrominoes and (b) CLEVR6. The central row contains the imput image and the predicted object masks. At the top, the reconstrueted image and object textures are shown (empty slots excluded), while the bottom row presents the reconstruction and object textures after interchanging texture vectors between objects. DISA shows strong compositional generalization, enabling reliable transferring of textures between objects while preserving shape, position, and scale information. Moreover, mask predictions are highly accurate with clear background separation.
MNCOMETNTIIN
INOUEE
Wuturn
Todure
Toters
Tolure
Textune
Todore
(b)
Figure 6: Texture generation capabilities of DISA on (a) Tetrominoes and (b) CLEVR6. The top row contains the reconstructed input image and object texture. At the bottom, the same visualization is presented after replacing the texture vectors with ones sampled from ([tex 0.035), with Fsex being the mean of the texture vectors encoded from the input image. DISA's learned latent space allows us to swap completely new textures for specific objects independently of their shape, location, and scale, demonstrating the generative capabilities of the model.
objects while preserving shape, position, and scale information. Moreover, we see highly accurate mask predictions with clear background separation.
In Figure t we show the texture generation results of DISA on Tetrominoes and CLEVR6. On Tetrominoes (Figure 6a), the decoded textures maintain the original shapes but switch appearance. From the first slot, it is also visible that we can generate coherent out-of-distributiin textures, as the dataset only contains objects with plain colors, while the one we obtain mixes red and yellow. Concerning CLEVR6 (Figure 6b), we find again that DISA is able preciscly adapt the sampled textures to the fixed shapes, positions, and scales of the objects in the image. Figure shows therefore that the learned representations of DISA also allow for generative capabilities: can swap in completely textures for specific objects independently of their shape, location, and scale.
We show in Figure 7 the shape generation results of DISA on Tetrominoes and CLEVR6. Note that this task is quite difficul:: first, the number of unique shapes strongly limited in these synthetic datasets; second, without changing the scale the reference frame, the generated shape is restricted one that fits the original frame. On Tetrominoes (Figure 7a), the sampled shapes are roughly defined but close to those present in the training data, with exceptions such that of the cyan object the bottom left. On CLEVR6 (Figure 7
11

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Sler
Reconstruction
Tetrominoes
(b) CLEVR6
Shape generation capabilities of DISA on (a) Tetrominocs and (b) CLEVR6. In cach of the 2 images, the first pair of rows shows final reconstruction (middle left), object textures (first row), and shapes (second row) obtained after encoding and decoding an input image. The second pair of rows presents final reconstructions (middle left) and individual object textures (first row) and shapes (second row) after sampling shape vectors. On Tetrominocs, the sampled shapes the training data with exceptions such as that of the cyan object the boltom left, while on CLEVR6 the generated shapes are almost identical to those present in the training data. The difficulty of this task is mainly due to the strongly limited number of unique shapes in these synthetic datasets and to the fact that, without changing the scale of the reference frame, the generated shape restricted to one that fits the original frame.
Figure 7:
roughly defined but close to those present in
the generated shapes are almost identical to those present in the training data. A higher standard deviation during the sampling phase can lead larger variety in the generated shapes, while at the same time can strongly degrade their quality.
6 Conclusion
In this work, introduce novel approach named Disentangled Slot Attention (DISA). DISA explicitly disentangles the groups of features responsible for encoding texture, shape, position, and scale in object-centric representations. Our method results in latent space characterived by four non-overlapping subsets of its dimensions, known prior to the start of the training process. To train DISA, we employ the commonly used reconstruction loss along with an additional simple regularization of the latent space, aiming to support the architecture in preventing lexture and shape information from flowing within unrelated components. Our quantitative experiments demonstrate that, most cases, DISA achieves the desired disentanglement of texture and shape components while being competitive with or outperforming the baselines in image dccomposition and reconstruction quality. Additionally, show compositional and generative capabilities derived from the disentanglement through qualitative experiments.
There are several directions for future work. First, as seen on CLEVR6 and CLEVRTTx, more complex textures can cause some leakage of texture information into shape-related components. possible direction to address this problem is to exploit a stronger filter (than the Sobel filter) to further remove texture information from the objects. Vice versa, one may also look into additional strategies prevent shape information from flowing into texture dimensions. Finally, DISA does not tackle the further disentanglement of the features inside the explicitly separated groups, which could also be promising direction for future work.
12

---

Published in Transactions on Machine Learning Research (01/2025)
References
Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiw:2044.00444, 2020.
Jimmy Lei Ba, Jamic Ryan Kiros, and Geoffrey Hinton. (al:1607.0444 2016.
Layer normalization. arXiv preprint
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on poltern analysis and machine intelligence, 35(8):1798-188) 2013.
Ondrej Biza, Sjoerd van Steenkiste, Mehdi SM Sajjadi, Gamaleldin F Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. orXiv preprint arXiv:2302.00 04973, 2023.
Christopher P Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation, orXiv preprint arXiv: 1901.1901.11 1390, 2019.
Michael Chang, Tom Griffiths, and Sergey Levine. Object representations fixed points: Training iterative refinement algorithms with implicit differentiation. Advonces in Newral Information Processing Systems, 35:32694-32707 2022.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Damitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2019.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. arXiv preprint orXiv:2107.000 00667,, 2021.
Martin Engeleke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXio preprint arXiu:1907. 13055, 2019.
Martin Engeleke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object represen- tations without iterative refinement. Aduances in Neural Information Processing Systems, 34:8085-80 -0994, 2021.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in neural information processing systems, 29, 2016.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Aduances in Neural Information Processing Systems, 29, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, 30, 2017.
Klaus Greff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424 2433. PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference computer vision and poltern recognition, pp. 770-77 770-778, 2016.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193 218, 1985.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In The Eleventh International Conference on Learning Representations, 2022.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. orXiv preprint 1r1i: 02084, 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and http://lhuub.commeeeeennnn/ /mlliiobbejeeddllaaessss/ 2019.
Alexander Lerchner. Multi-object datasets.
Laurynas Karazija, Iro Laina, and Christian Rupprocht. Clevrtex: A texture-rich benchmark for unsupervisod multi-object segmentation. arXiv preprint ar vrXim:2111.1066, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprinl arXiv:1412. 6980, 2014.
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint Xiv::2111 125994 2021.
Avinash Kori, Francesco Locatello, Francesca Toni, and Ben Glocker. Unsupervised conditional slot attention for object centric learning. arXie preprint wVXiv:2307.04337 2023.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprinl wwXii:2001.024000 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances én Neural Information Processing Systems, 23:11525-11538, 2020.
Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer. Unsupervised part-based disentangling of object shape and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 10955 10955-10964, 2019.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv: 04025,
0402,0 2015.
Amin Mansouri, Jason Hartford, Kartik Ahuja, and Yoshua Bengio. Object-centric causal representation learning. In NearIPS 2022 Workshop Symmetry and Geometry in Neural Representations, 2022.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://hhwb..ommdeeennn/ee//iiie
2017.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Charlie Nash, SM Ali Eslami, Chris Burgess, Irina Higgins, Daniel Zoran, Theophane Weber, and Peter Battaglia. The multi-entity variational autoencoder. In NTPS Workshops, 2017.
Samuele Papa, Ole Winther, and Andrea Dittadi. Inductive biases for object-centric representations in the presence of complex textures. arXiv preprint arXi::224.08447, 2022.
Mihir Prabhudesai, Shamit Lal, Darshan Patil, Hsiao-Yo Yu Tung, Adam W Harley, and Katerina Fragkiadaki. Disentangling 3d prototypical networks for few-shot concept learning. arXiv preprint orXie:2011.03367,
2020.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846 850, 1971.
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In Proceedingss of the European conference on computer vision (ECCV), pp. 650 0006665,65 2018.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns compose. arXiv preprint arXiv:2110.11405, 2021.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh International Conference on Learning Representations, 2022.
Irwin Sobel and Gary Feldman. A 3x3 isotropic gradient operator for image processing. talk at the Stanford Artificial Project in, pp. 271227111700 1968.
Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster attend-infer-repeatt with tractable probabilistic PMLR, 2019.
models. In International Conference Machine Learning, pp.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint Xiv:18022108533 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. orXio preprint arXiv: 1901.00017 2019.
Han Yan, Haijun Zhang, Jianyang Shi, Jianghong Ma, and Xiaofei Xu. Toward intelligent fashion design: A texture and shape disentangled generative adversarial network. ACM Transactions on Multimedia Compuling, Communications and Applications, 19(3):1 23, 2023.
15

---

Published in Transactions on Machine Learning Research (01/2025)
A Sobel Filter
The Sobel filter (Sobel & Feldman, derivatives of an image:
1968 uses two 3 x 3 filters to approximate the horizontal and vertical
+1
0 -17
+1
+2 +1 0
K
+2
-2
and Ky =
(8)
+1
-1]
2
A 2-dimensional convolution operation with 2g and Ky applied on the input image X, producing two maps of the approximated horizontal and vertical derivatives at each pixel, respectively 92 Krr X and Gy Ky X. Figure Sb shows example of the resulting G2 and Gy of the grayscale image Sa To obtain the final filtered input containing the detected edges, the maps are aggregated as
g = - g +
(9)
The filtered image associated with the previous example is shown in figure Sc
(a)
(b)
(c)
Figure 8: Example of the Sobel filter applied to a simple grayscale image. (a) Input image N. (b) Maps of the approximated horizontal (9z, left) and vertical (gy, right) derivatives of X' at each pixel. (c) Filtered input image 9 computed combining 9. and Gy following Equation
When dealing with RGB images, there are two possible approaches. The first is to simply convert the image to grayscale before applying the Sobel filter. The second, which we employ in our method, consists in applying the filter independently each channel, then averaging the three resulting filtered images.
Figure Example the Sobel filter applied to an image from each of the considered datasets. The top row shows the input images, while the bottom row displays the outputs of the Sobel filter.
16

---

Published in Transactions on Machine Learning Research (01/2025)
B Datasets
Tetrominoes Tetrominoes consists of 35 x 35 RGB images presenting 3 "Tetris"-like" blocks and a black background. The blocks are characterized by one of 6 possible colors (rod, green, blue, yellow, magenta, cyan) and a shape and orientation within 19 possible. Figure shows a few examples extracted from the dataset. the image, but cannot overlap with each other. Furthermore,
The tetrominoes can appear anywhere multiple saapee///rrinttion///eeooo /colors of the same kind can be included in single sample. We employ total of 60K samples for the training set and 320 for the test set, in line with Greff et al. (2019) Locatello et al. (2020).
Multi-dSprites Multi-dSprites is a dataset based on dSprites Matthey et al. 2017), where three types of shapes, namely oval, heart, and square, are represented over a plain background. Both the sprites and the background are colored with randomly sampled RGB values. The number of entitics in the image can vary from 1 4 excluding the background, and partial occlusion can be present. Each image has a resolution of 64 x 64. Examples from the Multi-dSprites dataset are visualized in Figure T0b Again, we employ a total of 60K samples for the training set and 320 for the test set, as in Greff et al. (0019) Locatello et al. (2220).
CLEVR CLEVR, introduced by Johnson et al. (017)) 7,, is a collection of 240 x 320 images of 3D scenes containing a number of objects ranging from 3 to 10 (excluding the background). There are three types of shapes, Lee cube, sphere, and cylinder, two possible sizes (small and large), eight colors (gray, red, blue, green, brown, purple, cyan, and yellow), and two materials (shiny "metal" and matte "rubber'). Some samples from CLEVR are presented in Figure 10c] We also employ a filtered version of this dataset, called CLEVR6, that has maximum of 6 entities in the scenes, background excluded. The number of training samples, in this case, is 70K, while the test samples are 15K. In all our experiments, center crop of 192 x 192 is applied, followed by resize to a resolution of 128 x 128.
CLEVRTex CLEVRTex (Karaziaa et al., 2021 consists of 50K 240 x 320 images (cropped and resized to 128 x 128 as in CLEVR), 40K of which are training samples while the remaining 10K are equally split into validation and testing sets. As in CLEVR, each image contains between 3 and 10 objects plus the background. For each object (background included), the material is sampled from collection of 60 materials (more complex than those in CLEVR). Objects can appear under 4 different shapes (cube, cylinder, sphere, and non-symmetric monkey head) and 3 distinct scales (small, medium, and large). In addition, the dataset presents several variations, such as CAMO and OOD. CAMO is a collection of 20K Lest images where all the objects, including the background, share the same material. OOD consists of 10K test samples with 25 materials and shapes (cone, torus, icosahedron, and a teapot) not contained in the training split. In Figure we show some examples extracted from the dataset.
10d
C
Additional Discussion on Related Work
This section further describes two approaches (mentioned in Section 3) dealing with explicit disentanglement of shape and texture, highlighting the differences with our work.
Disentangling 3D Prototypical Networks For Few-Shot Concept Learning Prabhudesai et al. (0200 proposes an interesting approach (D3DP) to explicitly disentangle shape and texture information in an object-centric way. Despite being developed to address the same challenge as DISA, it presents multiple differences from our work. Similar to our architecture, D3DP uses two different encoders to individually embed shape and texture information. However, it employs single decoder -based on adaptive instance normalization (Huang & Belongie, 2017 to simultaneously reconstruct texture and shape. In our work, using two separate decoders is considered very important: by decoding the masks based solcly on the shape-related part of the slots, we ensure that shape information is present in those features. Another substantial difference between the two approaches how they separate objects into distinct vectors or slots. DISA, along with other Slot Attention-based approaches, carries out this task entirely in an unsupervised manner. D3DP, on the other hand, relies on ground-truth 3D bounding boxes, essential to decompose the scene into objects. Additionally, the training strategy of D3DP assumes the availability of multiple viewpoints for given static
17

---

Published in Transactions on Machine Learning Research (01/2025)
a
(b)
(d)
Figure 10: Samples extracted from both train and test partitions of (a) Tetrominocs, (b) Multi-dSprites,
(c) CLEVR, and (d) CLEVRTex.
scene. This is because the encoder should learn to predict a 2D point of view a scene from a distinct 2D viewpoint, passing through a complete 3D feature map of the scone. Such a strong assumption makes D3DP hard to apply to real-world dynamic scenarios, in which it unlikely that an agent can observe the exact. same scene from more than one viewpoint. On the contrary, DISA (and Slot Attention- based models in general) does not depend this assumption. Finally, the authors of D3DP do not quantitatively evaluate the disentanglement of texture and shape, for example, through regular and inverse property prediction tasks (as in our work).
Unsupervised Part-Based Disentangling of Object Shape and Appearance Lorenz et al. (0019 introduces an unsupervised method to disentangle the shape and texture components in object representations. The first difference with DISA that they work on single object images belonging to a unique class (c.g., people, dogs, or birds). In fact, they extract a fixed number of object parts (e.g., left arm or head), each encoded by a pair of texture and shape vectors. In contrast, DISA extracts disentangled vectors representing individual objects in a multi-object scene, where the number of objects is variable. Although both considered architectures rely two separate encoders to represent shape and texture, there are key differences to be noted. Their method exploits spatial transformations to help prevent. shape information from being included in the texture components. Instead, DISA extracts texture information from the original input image and relies on the presence of the necessary shape information in the shape-related features (enabled by the mask decoder, as mentioned in the previous paragraph). The assumption is that, as we already provide the texture decoder with the shape vector, including shape information in the texture vector would be redundant and thus unnecessary (the variance regularization also helps, but Figure 14 suggests that the architecture plays a approach
more important role). In future work, it could be interesting to study whether augmenting with these spatial transformations would lead to even better results. Another difference is in the appearance transformation. In our work, we remove part of the texture information (mainly color) with Sobel filter to bias the shape encoder loward encoding only shape information. Their strategy is instead to change the brightness, contrast, and hue of the image fed to the shape encoder. Both ways are limited in that they do not remove or change the structure of texture, but only color. further distinction is that the shape encoder of DISA represents the shape information of an object as a single low-dimensional vector, while that employed by Lorenz et. al. (209)) encodes specific subpart of the object as activation map the input.
18

---

Published in Transactions on Machine Learning Research (01/2025)
image (similar to those in the Slot Attention mechanism). Moreover, their approach adopts single decoder to reconstruct the entire input image, while DISA uses two decoders to individually predict the mask and texture of every object. Finally, as D3DP, this work does not present any quantitative analysis on shape and texture disentanglement but only a qualitative study on compositionality.
D Training Setup
D.1 Object Discovery
SA, ISA, and DISA are trained as autoencoders on Tetrominoes, Multi-dSpritts, CLEVR6, and CLEVRTex for an equal number of steps. A batch size of 64 and a learning rate of 4 x 10 (using Adam as optimizer (Kingma Ba 2014p) are employed on all datasets for all the architectures. An initial (lincar) warm-up for 10K steps and an exponential decay schedule (from start to end) are applied to the learning rale. On Tetrominoes, we perform roughly 100K steps (107 epochs x 938 batches), on Multi-dSprites around 250K (267 epochs x 938 batches), on CLEVR6 nearly 150K (274 epochs x 547 batches), and on CLEVRTex exactly 150K (240 epochs x 625 batches). The mean squared error is used as the reconstruction loss function.
The number of iterations in the Slot Attention mechanism fixed to 3, while the number of slots set to 4 on Tetrominoes, on Multi-dSprites, 7 on CLEVR6, and 11 on CLEVRTex. Moreover, for SA the slots are initially sampled from learned distribution, while for ISA and DISA are initialized as learnable vectors. When we set the initial slots learnable vectors, hence on ISA and DISA models, we decide to employ the bi-level optimization from Jia et a.. (2022). We set the dimensionality of the slot vectors to 64 (excluded position and scale factors) in all cases and, in DISA, we define the texture components as the first half and the mask components as the second half (32 each) of representation. With DISA, we always set the initial position and scale vectors as Icarnable embeddings, whilc with ISA only on Multi-dSprites and CLEVRTex. On retrominoes and CLEVR6 we sample those as done in the original paper for the two specific da asets. Furthermore, again in line with the original paper, ISA does not use scale vectors on Tetrominoes.
With ISA and DISA, we clip the norm of gradicnts to 0.05 (as in ISA's original implementation). Finally, Tetrominoes and Multi-dSprites, we apply the variance regularization only on Stex with À = 0.32 since the beginning of the training, while on CLEVR6 and CLEVRTex we apply it after the warm-up on both Stex and Sshape (as in Equation with A = 0.05.
On Tetrominoes and CLEVR, DISA tends to learn masks that, instead of precisely segmenting an object, select areas of arbitrary shape around the entities. Despite maintaining good dccomposition capabilities as the objects get correctly divided into separate slots, this behavior can hinder the shape information from being encoded in the shape components. Consequently, the texture components would have to include shape information, leading to an incorrect dlisentaglemmnnt We can notice that the issue seems to arise when dataset is characterized by fixed background, but not with a dynamic one as in Multi-dSprites and CLEVRTex. The simple introduction of random plain noise added to the input images (Figure 11] can avoid this problem: in fact, the intuition behind the solution is to induce slight changes in the background texture, so that it no longer fixed, while however not heavily altering the foreground objects. We hence adopt this augmentation strategy when training our DISA models on Tetrominoes and CLEVR6. Additional experiments are required to gain more insights this, for instance, to understand whether fixed non-plain backgrounds are as affected as fixed plain ones.
19

---

Published in Transactions on Machine Learning Research (01/2025)
+
Figure 11: Example of the plain noise augmentation applied on an image from Tetrominoes. The augmentation induces slight changes the color of a fixed background, turning it dynamic and helping the model avoid suboptimal solutions.
D.2 Property Prediction
Each MLP used in our property prediction experiments has a single hidden layer of 256 nodes activated with Leaky RcLU. The imput layer has 32 nodes because only texture or shape components are used predict the property of object. On every datasel, we perform about 10k steps with batch size of 64 and a learning rate set to 1 x 10-3 (the optimizer is again Adam). With categorical variables employ the cross-entropy loss, while with numerical the MSE. Morcover, on all datascts, the last 1000 samples are excluded from the training set and utilized as validation set.
E Disentanglement Analysis
E.1 Training and Validation Curves
shope_
shape_Joss_vt
proppred Jer dioo, sud propored proppred Joy
im proppeed betr diea, nund proppred
prospred propored cisa run3 propored run2 Inv prospred disa
proppred tet: prospros to0, dsa
proppred
25
0.5
(a) Shape Training
(b) Shape
Validation
color, loss
proppred, bey dits
desa run In, proppred dila dia
proopred irr_dioonnn mw_pmppend,
propored wir dia,, propored Tey. 26s
proppedd Wer
proppred _iae__n_n22 proppred
inv_poppee__
propped
2<
8k
12
(c) Color Training
(d) Color Validation
Figure 12: Training and validation curves Tetrominoes. Each plot represents the cross-entropy loss (y-axis) a function of the training steps (x-axis).
20

---

Published in Transactions on Machine Learning Research (01/2025)
shopo_Joss
shape, val
proppred_ unde_tttoo plooooooonn proppred mnoa paw. proppeet
propored, indo, diso, rund propprod mnd,s disaurund propored mril,
Inc. proppred más, disa, runa Inv, pww.prooppcccccccooo mds, diso, runz lhf, pwwpppooppddd.oo (nda,
groppred mas.
0.6
02
022
(a) Shape Training
(b) Shape Validation
color,
color, lasrr_oossuuuuu
tww_proppred_ng diw..ussannn
mnddsdss dis..rrassa wwc_proppred ruun2
propored_ntteec
www.propprod_mms_cee
03
0.6
0.25
033
0.5
02
0.35
015
01
01
005
(c) Color
Training
(d) Color Validation
Figure 13: Training and validation curves on Multi-dSprites. The first two plots (a and b) represent the cross-cntropy loss (y-axis) function of the training steps (x-axis). The last two plots (c and d) represent the mean squared error (y-axis) as function of the training steps (x-axis).
shape_Joss Josa
ahape Joes vol
dlett dan proppeed diviii dise nin2 prospred dewil.dla
proppred devifl diss, minn in propppeddd dlvvff disa run2 iyv mooppoppred Beviti disa
propored, clevnd paur propered cword chor
pmw_proporeddcomm clevro run2
propored cavit dise
1
2c
10c
(a) Shape Training
(b) Shape Validation
color loss
color, Jasa, vol
davtt nar prosered clevé. dia run2 slopee__deevtt__ee
iny, propored clavr8 dise run3 Iy propored clevt6 dis run2 Iny www_proppreddddeme devni_dies
devit purd proopred devii diaa run2 proppred proppeo_ddevit_te devitt_doo
liy proppred diviif diss suns lnw propprdddddddc. diviié diae run2 Inc wwwppppdd__eeet_deeeeooo
1.
155
x
13k
2x
15c
10
(c) Color Training
(d) Color Validation
21

---

Published in Transactions on Machine Learning Research (01/2025)
material
propored pwu_seppeed__c disaarand proprred _is___aaaa in__roppret_ proppred
prospeed, proppeed ceteedddddeieeeeeee pawei__leoennaat proppeed
mwppmppedd dien
poppradt_com
proppeed
0.
(e) Material Training
(f) Material
Validation
Figure 14: Training and validation curves on CLEVR6. Each plot represents the cross-entropy loss (y-axis)
as function of the training steps (x-axis).
F Additional Experimental Results
F.1 Ablation: Impact of Variance Regularization and Sobel Filter on Texture and Shape
Disentanglement
1.0
0.8
Shape
Accuracy
Shape (inverse)
0.6
Shape (random guess)
Color
0.4
Color (Inverse)
Color (random guess)
0.2
0.0
Variance reg.
x Variance reg.
Variance reg.
/ Sobel filter
/ Sobel filter
Sobel filter
Figure 15: Quantitative results of DISA on the regular and inverse properly prediction tasks. We report the measurements of DISA with Sobel filter and variance regularization (left, shown in Figure 3 without the
use of the variance regularization (center), and without Sobel filter (right) on Tetrominoes. The predicted properties are shape and color, both categorical variables. The prediction accuracy is employed and compared with a baseline random guess. We report mean and siddev over seeds.
22

---

Published in Transactions on Machine Learning Research (01/2025)
F.2
Compositional Results
Reconstruction
Texture 1
Texture
Texture 3
Texture
Input Image
Mask
Mask
Mask
Mask 4
Permuted Recon.
Perm. Texture
Perm. Texture
Perm. Texture
Perm. Texture 4
Figure 16: Compositionality on Tetrominoes. Permutation applied to Stext; 3-2-4-1.
Reconstruction
Texture
Texture
Texture
Texture
Input Image
Mask
Mask 2
Mask
Mask 4
Permuted Recon.
Perm. Texture 1
Perm. Texture 2
Perm. Texture 3
Perm. Texture 4
Figure 17:
Compositionality Tetrominoes. Permutation applied to Stext, 3-4-1-2.
23

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture 1
Texture
Texture 3
Texture
Input Image
Mask
Mask 2
Mask 3
Mask 4
Permuted Recon.
Perm. Texture
Perm. Texture 2
Perm. Texture
Perm. Texture
Figure 18: Compositionality on Tetrominocs.
Permutation applied to Stext; 3-4-1-2.
Reconstruction
Texture 1
Texture 2
Texture 3
Texture
Input Image
Mask 1
Mask 2
Mask 3
Mask 4
r1
Permuted Recon.
Perm. Texture
Perm. Texture
Perm. Texture
Perm. Texture
Figure 19: Compositionality on Tetrominoes. Permutation applied to Stext, 3-2--11,
24

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture
Texture
Texture
Texture
Texture
Texture
Input Image
Mask
Mask 2
Mask 3
Mask
Mask
Mask 6
Permuted Recon.
Perm. Texture
Perm. Texture 2
Perm. Texture 3
Perm. Texture
Perm. Texture
Perm. Texture 6
Figure 20: Compositionality on Multi-dSprites. Permutation applied Stext, 1-2-3-5-6-4.
Reconstruction
Texture
Texture
Texture
Texture
Input Image
Mask 1
Mask 2
Mask 3
Mask 4
Mask
Mask 6
Permuted Recon.
Perm. Texture
Perm. Texture 2
Perm. Texture 3
Perm. Texture
Perm. Texture
Perm. Texture 6
Figure 21: Compositionality on Multi-dSprites.
Permutation applied Stext, 6-1-5-1-3-4.
25

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture
Texture
Texture
Texture
Texture
Input Image
Mask
Mask
Mask
Mask
Mask
Mask
Permuted Recon.
Perm. Texture
Perm. Texture 2
Perm. Texture
Perm. Texture
Perm. Texture
Perm. Texture 6
Figure 22: Compositionality on Multi-dSprites. Permutation applied Stext; 6-2-3-5-1-4.
Reconstruction
Texture
Texture
Texture
Texture
Texture
Input Image
Mask 1
Mask 2
Mask 3
Mask 4
Mask
Mask 6
Permuted Recon.
Perm. Texture
Perm. Texture 2
Perm. Texture 3
Perm. Texture
Perm. Texture
Perm. Texture 6
Figure 23: Compositionality on Multi-dSprites.
Permutation applied Stext, 5-1-4-3-2-2.
26

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture 1
Texture 2
Texture 3
Texture 4
Texture 5
Texture 6
Texture
Input Image
Mask
Mask
Mask
Mask 4
Mask 5
Mask
Mask
Permuted Recon.
Perm. Texture
Perm. Texture
Perm. Texture
Pemn Texture
Perm. Texture
Perm. Texture
Perm, Texture
Figure 24: Compositionality on CLEVR6. Permutation applied to Stext, 2-5-7-4-6-1-8.
Reconstruction
Texture 1
Texture 2
Texture 3
Texture 4
Texture 5
Texture 6
Texture 7
Input Image
Mask
Permuted Recon.
Perm. Texture
Perm. Texture 2
Perm. Texture 3
Perm. Texture 4
Perm. Texture
Perm. Texture
Perm. Texture
Figure 25: Compositionality on CLEVR6. Permutation applied to gtext, 2-4-6-1-37-55
27

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture 1
Texture 2
Texture 3
Texture 4
Texture 5
Texture 6
Texture 7
Input Image
Mask
Mask
Mask
Mask
Mask
Mask
Permuted Recon.
Perm. Texture 1
Perm. Texture 2
Perm. Texture 3
Pemn. Texture
Perm. Texture
Perm. Texture
Perm. Texture
Figure 26: Compositionality on CLEVR6. Permutation applied to Stext, 2-1-7-3-5-6-4
Reconstruction
Texture
Texture
Texture
Texture
Texture
Texture
Texture
Input Image
Mask
Mask 2
Mask
Mask 4
Mask 5
Mask 6
Mask 7
Permuted Recon.
Perm. Texture
Perm. Texture
Perm. Texture
Perm. Texture
Perm. Texture
Perm, Texture
Perm, Texture
Figure 27: Compositionality CLEVR6. Permutation applied Stext, 1-3-6-7-5-4-2.
Reconstruction
Texture
Texture 2
Texture
Texture
Texture 5
Texture
Tenture
Texture 8
Texture
Teature 10
Texture 11
Nask
Permuted Recon
Perm, Teoture
Perm Teature
Perm Tecture
Perm, Texture
Perm Teature
Perm
Perm Texture
Perm Testure
Perm Testure Perm Testure
Perm Tectre
Figure 28: Compositionality on CLEVRTex. Permutation applied to Stext, 122-9-8-5-6--4499-1011111
28

---

Published in Transactions on Machine Learning Research (01/202)
Decondtaacios
Testur2
Texture
Texture
Tecture
Trotuee
Teature
Inpu imoge
Mask
Mack 2
Masi
Nack
Mask 5
Masi
Nock
Mask
Modk
Mask
Mask 11
Figure 29: Compositionality on CLEVRTex. Permutation applied to Stext; 3-2-1-4-5-6-7-11--9--06
Texture
Input image
Permuttd Recon.
Perm. Texture
Pomm Texture
Porm Texture
Form Texture
Pem. "orture
Porm, Toxtune
Pomm Texture
Porm Tonure
Form, Teature
Porm. Texture Pemm Torture 1I
Figure 30: Compositionality CLEVRTex. Permutation applied to Stext; 9-2-7-11-8--3-5-1--0044
Reconstruction
Texture
Texture
Tonure
Texture
Texture 5
Texture
Texture
Texture 8
Texture
Teofure ID
Texture 11
Perm,
Porm,
Ferm:
Perm,
Perm
Nemm
Figure 31: Compositionality on CLEVRTex. Permutation applied to Stext, 1-8-5-4-9------33-22111
29

---

Published in Transactions on Machine Learning Research (01/2025)
F.3 Generative Results
We include here results for lexture and shape generation.
As anticipated, we sample the new textures from a Gaussian distribution centered on the mean of the texture vectors of the objects in a scene, with a standard deviation of 0.035 (manually tuned). Similarly, we sample new shape vectors from Gaussian distribution centered on the mean of the shape vectors of the foreground objects in a scene, with standard deviation manually tuned. On Tetrominoes and Multi-dSprites, the standard deviation is to 0.1, while CLEVR and CLEVRTex 0.01.
Reconstruction
Texture
Texture 2
Texture 3
Texture 4
Texture
Texture
Figure 32: Texture generation on Multi-dSprites. The first row shows final reconstruction and object textures obtained after encoding and decoding an input image. The second to fifth rows present final reconstructions and individual object textures after sampling new texture vectors.
30

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture 1
Texture 2
Texture
Texture 4
E
Figure 33: Texture generation on Tetrominoes. The first row shows final reconstruction and object textures obtained after encoding and decoding an input image. The second to fifth rows present final reconstructions and individual object textures after sampling new texture vectors.
31

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstruction
Texture 1
Texture 2
Texture 3
Texture 4
Texture 5
Texture 6
Texture 7
Figure 34: Texture generation on CLEVR6. The first row shows final reconstruction and object textures obtained after encoding and decoding an input image. The second to fifth rows present final reconstructions and individual object textures after sampling new texture vectors.
Neconstruccion
Figure 35: Texture generation CLEVRTex. The first shows final reconstruction and object textures obtained after encoding and decoding an input image. The second to fifth rows present final reconstructions and individual object textures after sampling new texture vectors.
32

---

Published in Transactions on Machine Learning Research (01/2025)
Reconstructon ion
Siot
Siot 2
Siet
Sior
(a) Multi-dSprites
Reconstruction
Siot
(b) CLEVRTex
Figure 36: Shape generation results on Multi-dSprites and CLEVRTex. In each of the 2 images, the first pair of rows shows final reconstruction (middle left), object textures (first row), and shapes (second row) obtained after encoding and decoding an input image. The second pair of rows presents final reconstructions (middle left) and individual object textures (first row) and shapes (sccond row) after sampling new shape vectors.
33


Reviews:
[{"review_id": "3bDNYzE6P9", "reviewer": "Reviewer_gmpn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduce a novel method of object-centric disentangled representaion learning, which is interesting. The idea offers an intuitive way of achieving disentanglement and is convincing to the audience.\n2. The writing of the paper is clear and easy to follow.\n\nWeaknesses:\n1. The quantitative comparisons are only conducted with variations of the proposed methods, it is recommanded to include comparision with existing works to show the effectiveness of the proposed methods.\n2. The method requires strong prior knowledges about the data, for example, the edges from sobel filter or the texture. This might limits the potential usage and impact of the method as the generating factors are pre-defined and restricted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduce a method for disentangled representation learning from an object-centric perspective. In the proposed framework, two prior feature extractor and encoder is used to encode the input image into two seperate embeddings that contain the texture and shape information of the original information. To aviod the positioning and scaling issues between two embedding spaces, the authors further use slot attension to align the texture and shape embeddings. Extensive experiments have been conducted to shown the effectiveness of the method in finding the texture and shape generative factors, hence validating the disentanglement of features using the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is suggested that the authors address the insufficientness of the the experiments sections.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "opBSPYYviC", "reviewer": "Reviewer_sYNF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n- DISA offers a unique extension to ISA by explicitly disentangling shape and texture features, enhancing both the interpretability and effectiveness of object-centric representations.\n- The paper evaluates DISA on several challenging benchmarks, such as CLEVR6, and CLEVRTex, and shows improved or competitive performance in terms of reconstruction quality and object discovery.\n- The contribution of this study is clear \n\nWeakness: \n\n- Some datasets, such as CLEVR6 and CLEVRTex, show leakage of information between shape and texture components, indicating the disentanglement is not always perfect.\n- The use of solber filter may have difficulties for more complex data, as discussed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach named Disentangled Slot Attention (DISA) to enhance object-centric learning through explicit disentanglement of object attributes, focusing primarily on texture and shape components. Building upon Invariant Slot Attention (ISA), the proposed architecture ensures non-overlapping subsets of latent space dimensions for these features, thus improving interpretability and performance. DISA demonstrates competitive performance in object discovery and reconstruction across several synthetic datasets, along with strong compositional and generative capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Though not required, I was hoping to see an ablation study on how the model by using stronger/diffferent filters would perform on the given datasets. \n\n- I think adding some summarization of the findings in figures, especially fig 2, would make those a lot clearer. \n\n- Related work might need to be tighten.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cXeTTI7dcp", "reviewer": "Reviewer_kf8a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper introduced backgrounds and related works in a decent manner. I personally find them to be useful references for those who are not familiar with the context\n- - The algorithm is build on top of the invariant slot attention module and explicitly disentangle color and shape. This is a decent extension of the ISA algorithm. \n- The settings of the algorithm makes sense on the high-level, and experimental results on some public benchmarks did demonstrate some effectiveness of the approach on disentangling texture and shape.\n\nWeaknesses:\n- Although the insight of the paper is sensible, I find the approach is only tested on synthetic and less complex datasets. The approach did demonstrate a proof-of-concept of the effectiveness of DISA algorithm, but I wonder if authors could discuss more on the what it might influence future algorithm designs, or real-world applications by demonstrating on more complex datasets. \n- In most of the learned textures in the CLEVR dataset, there are heavy checkerboard artifacts. I wonder if there is an issue of the implementation of the algorithm -- The effect is not showing up on Tetrominoes datasets\n- The experiment demonstrated the effect to reconstruct textures. Since the shape is also disentangled, I wonder if the algorithm could also reconstruct shapes?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced an algorithm that builds on top of (invariant) Slot attention to explicitly disentangle the learned representation of shape and texture of input images. The algorithm works by using two encoder-decoder architecture by encoding object mask and object texture information in an object-centric manner. Using the learned representations, the algorithm was able to provide interpretability of the input image and was able to reconstruct the image using the learned representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "At this stage of the paper, I'd recommend authors to at least address some of my questions in weakness sections before future revisions", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were split in opinion on this paper. Arguments in favor of the paper included thorough experimental validation of the approach and the presence of interesting ideas regarding disentanglement. Arguments against included skepticism about whether the proposed architecture can generalize to more complex datasets and uncertainty about the presence of artifacts in the results and the effect of the image filter. These points are all valid and well-received. However, it should be noted that the acceptance criteria for TMLR are twofold: (1) the claims should supported by clear evidence, and (2) at least some individuals in TMLR's audience should be interested in the findings. The answer to these questions are both "yes", as discussed above. Nevertheless, the authors are strongly encouraged to incorporate relevant feedback from the reviewers into the revision, as the suggested changes would strengthen the paper. In addition, the authors are encouraged to move important material from the appendix into the main body of the paper according to their judgment. Examples may include (but are not limited to) the related work in Appendix C, the visualization of property prediction in Figure 9, and selected qualitative results from Appendix F (Figures 14-35), as appropriate.
Decision: Accept (poster)