Learning object representations through amortized inference over probabilistic programs
Authors: Francisco Silva
Hélder P. Oliveira
Tania Pereira
Venue: Accepted by TMLR ()

Abstract: The recent developments of modern probabilistic programming languages have enabled the combination of pattern recognition engines implemented by neural networks to guide inference over explanatory factors written as symbols in probabilistic programs. We argue that learning to invert fixed generative programs, instead of learned ones, places stronger restrictions on the representations learned by feature extraction networks, which reduces the space of latent hypotheses and enhances training efficiency. To empirically demonstrate this, we investigate a neurosymbolic object-centric representation learning approach that combines a slot-based neural module optimized via inference compilation to invert a prior generative program of scene generation. By amortizing the search over posterior hypotheses, we demonstrate that approximate inference using data-driven sequential Monte Carlo methods achieves competitive results when compared to state-of-the-art fully neural baselines while requiring several times fewer training steps.

Published in Transactions on Machine Learning Research (01/2026)
Learning object representations through amortized inference over probabilistic programs
Francisco Silva
mmncciso..c.ilbuu@ineeee..
INESC TEC- Institute for Systems and Computer Engineering, Technology and Science
Facully of Sciences, University of Porto
Hélder Oliveira
helder.f.olivcirru
INESC TEC- Institute for Systems and Computer Engineering, Technology and Science
Paculty of Sciences, University of Porto
Tania Pereira
lania, Ginestteee..
INESC TEC Institute for Systems and Computer Engineering, Technology and Science Paculty of Engineering, University of Porto
Reviewed on OpenReview:
https: // openreview. net/ forum? id= nUFSr\JaUr
Abstract
The recent developments of modern probabilistic programming languages have enabled the combination of pattern recognition engines implemented by neural networks to guide infer- ence over explanatory factors written as symbols in probabilistic programs. We argue that learning to invert fixed generative programs, instead Icarned oncs, places stronger restric- lions on the representations learned by feature extraction networks, which reduces the space of latent hypotheses and enhances training efficiency. To empirically demonstrate this, we investigate neurosymbolic object-centric representation learning approach that combines slot-based neural module optimized via inference compilation to invert prior genera- live program of scene generation. By amortizing the search over posterior hypotheses, we demonstrate that approximate inference using data-driven sequential Monte Carlo methods achieves competitive results when compared to stato-of-the-art fully neural basclines while requiring several times fewer training sleps.
1 Introduction
Neurosymbolic models have recently re-emerged from the need to extract and incorporate priorí symbolic information while processing high-dimensional and unstructured images, bridging pixel-level transformations modeled by neural networks and symbols easily interpretable to humans (Garcez & Lamb 2023 Chaudhuri et al. 2021). This goes line with the idea that incorporating symbolic knowledge, along with the infor- mation they carry by grounding them world entities, should make the generalization power of machine learning systems
particular neural networks closer what humans achicve (Lake al. 2015 2017 Goyal & Bengio 2022 Garcez & Lamb] 2023 Even though recent developments have been made (Ma al.
2019 Feinman & Lake 2021 Liang et al. 2022), the task of learning object representations is still mostly tackled with no significant high-level symbolic constraints on pattern recognition systems, which makes
poor generalization expected (Goyal & Bengio 2022 Lake et al. 2017 Moreover, the natural existence of
uncertain choices is not. addressed in those systems [Paar] 1988
The capacity to individuate and track objects over time is fundamental human ability that is demonstratod
since early infancy, following intuitive spatiotemporal constraints and progressively learning what could be expected from their physical interactions (Spelke. 1990
Carey 2009
Spelke Kinzler 2007 Baillargeon

---

Published in Transactions on Machine Learning Research (01/2026)
et al. 1983 Lin et al. 2022 Ullman et al. 2017]. Such core knowledge could be represented as rich gen- erative models that, innately given or not, humans use to not only make predictions about possible future states but also to efficiently infer explanations for what is being observed (Lake et al. 2017 Ullman et al, 2017 2018 Ullman & Tenenbaum 2020 Within the view of vision as inverse graphics, inferring explana- tory information from an observed image is seen as inverting the generative model that produced such an image (Stuhlmüller al., 2013 Horn 1977). We then propose a model for inference compilation through slot- attention (ICSA), a neurosymbolic system that approximates symbolic object properties by fast amortized inference in probabilistic program of scene generation. Unlike previous works this line (Kulkarni et al, 2015 Jampani et al., 2015), we combine stronger inductive biases on the way features get extracted from the observed images to improve posterior distribution proposals provided by neural networks. In particular, we adapted the SA-MESH algorithm (Zhang el al. 2023) as the neural encoder responsible for extracting object-centric features that are further used for sequential inference of generative variables (Le et al. 2017] Paige & Wood 2016). Morcover, we do not rely on pro-trained architoctures for object detection, as our main objective consists of implementing system that learns object-centric representations (akin to object files [Kahneman et al., 1992 Green & Shillty-uuni 2021 Stavans et al. 2019) using symbolic priori constraints what kind of features should be captured from pixel-level patterns. We evaluate our model in the main downstream task of set prediction against fully neural baselines. The ICSA generative model and inference networks were implemented using the Pyro probabilistic programming language (Bingham et al. 2019
Contributions The main contributions of our work are as follows: (1) the application a data-driven posterior proposal module with strong object-centric inductive biases for more interpretable representations learning; (2) measuring the advantageous of having richer prior knowledge in the form of fixed generative model for learning object-centric representations in terms of data efficiency, while accounting for posterior uncertainty and (3) an experimental setup that evaluates the behavior of object-centric models in different but plausible out-of-distributic inference regimes.
2 Related work
2.1 Probabilistic programming for computer vision
Probabilistic programs specify priori knowledge about explanatory variables of a generative model, along with the dependencics between them using the sample primitive command (van de Meent et al., 2018; Gordon et al.] 2014). Probabilistic programming languages (PPL) also implement inference engines to infer the probabilities of latent variables given the observed data, using a variety of algorithms such as meth- ods within the family of Markov chain Monte Carlo (MCMC), stochastic variational inference (SVT), and amortized inference. Universal PPL, which allow the construction of stochastic and unbounded generative programs (Goodman Stuhlmüller (2014), are also capable of incorporating neural networks-based opera- tions to increase inference speed (Bingham et al. 2019 Lavin al. [2021), which makes their application in computer vision tasks more accessible. For instance, learning the parameters of an interpretable proba- bilistic program of image generation reduces to learning the bottom-up transformation from pixel data to each random variable intended to be inferred from the scene. Several works have already followed this line, motivated by the idea of making MCMC proposal steps based on data-driven functions (Kulkarni et al. 2015 Jampani et al.] 2015: Mansinghka et al. 2013 Ullman et al.] 2018 Yuille Kersten 2006 Wu et al, These ideas are also convergent with some research lines from neurosymbolic modeling, by exploring
2015
the benefits of placing stronger top-down influences (with interpretable symbolic programs) over the kinds of
features that can be exi racted from images with neural networks [Garcez & Lamb 2023 Chaudhuri et al.
2021
Goyal& & Bengio 2022).
2.2
Inference Compilation
Inference compilation (IC) consists of training specialized neural networks to propose distribution parameters for an arbitrary generative probabílistic program (Paige & Wood 2016 Le et al.] 2017 Harvey et al. 2019 Baydin et al. 2019), combining the advantages of sequential inference methods DDuucet et al. 2001b;
2

---

Published in Transactions on Machine Learning Research (01/2026)
TRAINING
INFERENCE
SA-MESH
Neural Module
Prior Generative
Cony Enander
0g
Program
Generativ
Program
Repcal
times
Naural Module: Leaming posterior propocalss
Figure 1: An overview schome of ICSA, starting with a prior generative model implemented as a probabilistic program, which samples object latent variables from broad uniform prior distributions and generates batches of training scenes. A neural module consisting of convolutional encoder and the SA-MESH attention algorithm is responsible to learn object-centric representations in the format of a set of slots s, which are further used by specialized networks fProp(.) to output the parameters of the posterior distributions of each latent variable. At inference time, test scene is observed and K samples are drawn from the proposed posterior distributions, being then evaluated using importance sampling (ICSA-IS), or an object-wise SMC method referenced as score-resample (ICSA-SR), which scores and resamples objects iteratively until the whole scene gets explained.
2009) and amortized data-driven inference (Gershman Goodman 2014 Jampani al. 2015). Before observing any target dataset where inference intended to be performed, IC works by learning distribution proposals for prior generative model. After training, the inference system observes set of unseen data and reuses the learned proposals to approximate the true posterior in sequential importance sampling strategy. Even though the IC algorithm has already been used to solve vision tasks (e.g. captcha solving (Le et al. 2017)), a strong application has been focused on bridging probabilistic programming and large-scale
scientific simulators, allowing a significant improvement in inference efficiency (Baydin et al. 2019
Munk
et al. 2019 Lavin al. 2021 Munk al. 2022
2.3
Object-centric representations
The task of learning object representations has long history in computer vision. Designing systems capable of individuation objects in scene is challenge transversal to several downstream applications (c.g. object calegorization or pose estimation). However, learning object representations is frequently addressed without
any explicit purpose (e.g. (Locatello et al. 2020
Feinman & Lake 2021 Eslami et al. 2016 Kosiorek et al.
2019 Seitzer et al. 2022 Anciukevicius et al.
2022 hence no supervision is provided, hoping to learn general method was introduced and has
representations. Recently, the slot attention (SA) Locatello et al.
2020
served as the foundation for several other approaches for learning object representations (Sajjadi et al. 2022;
Elsayed et al. 2022 Kipf et al. 2021 Wu et al. 2022 Seitzer al.
2022 Singh et al.) 2022 Zimmermann
et al. 2023 Biza al. 2023
Webb et al.]
2023
Mansouri et al. 2023 Zhang et al.] 2023
Brady et al. 2024
2023
Wiedemer al. 2023
Kori et al. 2024).
SA conceptually emulates system of object files (Kahneaan
ct al.
1992 Carcy & Xul 2001 Xu et al. 1999. where scparated representations (slots) bind to specific
regions of the input image via attention mechanisms, aggregaling information about them. This large body of work has followed the idea that inductive biases injected as architectural constraints should restrict the vast combinatorial possibilitics that scene's latent representation might navigate through (Bengio et al.
2013] Goyal Bengio 2022). opening doors for these structured representations to be used in different downstream tasks.
3

---

Published in Transactions on Machine Learning Research (01/2026)
3 Methods
Our proposed system follows the ideas from the analysis-by-syntees paradigm and the Helmholtz ma- chine (Hinton et al. 1995 Dayan et al. 1995), with posterior hypotheses being computed by fast bottom-up neural modules, and weighted by comparing the corresponding hypothetical scene against some observa- tion. ICSA combines probabilistic program that samples a set of properties associated with an unknown number of objects to generate scene, with an inference module that learns object-centric representations and sequentially proposes parameters for the posterior distribution of each latent variable encountered in the program's 's exccution trace. We further describe in detail each module, starting with the generative program (Sec. 3.1) and then the main components of the implemented inference strategy based on the IC algorithm Paige & Wood 2016 (Sec. 3.2
3.1 Generative model
We formalize our generative model as a latent variable model of ground-trutie latent variables p(z), which generates image samples x in the form of object scenes. The generative model samples object attributes from prior distributions over pre-specified and fixed libraries of possible values. Observed images are rendered through a non-differentiable generator : Z + X, which maps a set of latent variables z, with training support Z, to an image x, i.e. g(z) = x. Since there are no causal dependencies between latent variables, p(z) factorizes in II, p(z:).
The number of objects present at each scene is determined by Bernoulli mask array with success probability PM. In experiments, set PM at 0.5 and generate scenes with up to 10 objects. We assume uniform priors over all discrete and continuous variables, no occlusions, and objects always fully appear within the observation canvas. For instance, shape is distributed between the categories ball and square; size bet wetween small, medium and large, with associated diameters set at 10, 15 and 20px, respectively; and color between red, green and blue.
For simplicity, g was implemented using the Python Pillow imaging library, generating batch of images each exccution trace during the training phase.
3.2 Training
The implemented inference network consists of slot-basod fcature extractor, adapted from (Zhang et al. 2023), optimized by an IC algorithm that sequentially learns data-driven parameter proposals for the poste- rior distribution of latent variables. We further detail these main components over the next subsections.
3.2.1 Learning object-centric representations
We adopted the SA-MESH architecture proposed in (Zhang et al. 2023 which connects the cross-attention mechanism implemented in previous SA works (Locatello al. 2020 Kipf et al.] [021] Biza al.] 2023;
Wang et al. 2023) with optimal transport, by minimizing the entropy of the attention mask that results from applying the Sinkhorn algorithm (Cuturi 2013 Sinkhorn & Knopp) 1967 over the computed distance matrix. The core advantage over the standard SA approach is that, by making the slots + inputs mapping exclusively multiset-equivariant instead of only set-equivariant (as in SA), the attention maps become sparser and slot representations are less prone to collapse when objects are identical. We provide more details on this App. A.I
3.2.2 Fixing symmetries at inference time
Since slots are randomly initialized and proposals for posterior distributions of latent variables are assumed to be permuted by any arbitrary object-wise order during training, there are no guarantees that, at inference time, the same order will be followed for every inference particle (since each one represents an independent run of the inference network). More coneretely, fixing no structure to the slot + object binding would result in an incoherent evaluation of the hypotheses raised for a certain individual object. Following a similar associated with
method proposed by Le et. al. (2017), find permutation n² (1,..., N) -
4

---

Published in Transactions on Machine Learning Research (01/2026)
the k-th proposed traced, such that Equation is guaranteed, being the inferred Euclidean distance of each object to the origin of the canvas. Note that, since each trace orders objects arbitrarily, xt and x are independent permutations Vi,j E ik,i
(1) S (2) S ... S (*)(N)
(1)
This way, proposals for individual objects can be weighted independently and sequentially given the sampled choices made by each trace, up to ambiguities raised by this ordering choice, which will be later discussed.
3.2.3 Objective function
As shown in Fig. 1 specialized neural predictors (.ror(.) are used to estimate the parameters of poste- rior distributions regarding each latent, variable, taking the set of slots s as input. For instance, Shape (s) parameterizes the posterior probabilities for each object's shape.
In IC, the optimization direction is guided by 'distance' between the approximated and the real posterior distributions. In contrast with the KT. term traditionally implemented in variational autocncoders (VAE) (Kingma & Welling, 2013) and similar models, the arguments of this KL divergence are flipped. The loss function used is expressed in Equation 2 Note that the expectation is computed with samples drawn from the generative model p(x, z), and not using the approximated distribution q(z)x; o) for latent variables sampling (as in VAE). This training mode is similar to the "sleep' phase of wake-sleep algorithm (Dayan et al. 1995). thus training the neural artifact only requires a fully specified generative model to sample from (see in Equation 21, and offline training dataset (Le et al. 2017). By inverting the arguments, minimizing the inclusive KL divergence encourages the approximate posterior to distribute its density to cover the entire support of the true posterior, which brings benefits when the variational distribution tries to approximate complex and multimodal true posteriors (for more in-depth information about this behavior and the loss formulation, please refer to (van de Meent et al. 2018)).
Bullback-Leibber (KL) divergence, which measures the
((Ø) Bp(x) [DKL (p(z)x)lla(()x o))]
(2)
Ep(x.2) log (<1xx; + const
3.3 Inference through Sequential Importance Sampling
We employed a sequential Monte Carlo (SMC) (Chopin et al. 2020 Doucet et al., 2001a Le al. 2017;
van de Meent et al. 2018) method to approximate posterior inference over the specified generative program. After aligning proposal traces, data-driven posterior hypotheses [+t.v] are proposed for the set of K particles (i.c. each of the K samples drawn from the proposed posterior), which are evaluated and resampled in an object-wise sequential manner (Gothoskar et al. 2023). More concretely, at the first inference step, only the first object - i.c. its associated latent variables is sampled, scored and resampled. Following this, more objects are iteratively added until the observed scene is fully explained. For each proposed object oy such that j (1,2....,NN) NN, the importance weight for the k-th particle, w6y,,,> is computed as following:
p(2 x)
& w
Woro
(3)
(5% xx)
where boro, comprises the sampled values for the latent variables associated with the set of objects "t.....,jj-1.00 o;}, being and & the observed and generated scenes, respectively.
As shown in Equation 3 the likelihood of the resultant image in the k-th proposed trace, when proposing latent variables for object oj, is evaluated against the full observed image x. This means that, instead of only scoring how well the generated object explains the obscrved one at the same location, we also evaluate the impact of accepting the proposed object while aiming to explain the whole scene. Image likelihoods are computed using continuous observation model modeled by Gaussian distribution with fixed noise
5

---

Published in Transactions on Machine Learning Research (01/2026)
parameter OL: M(x; a + Here, s To represents the scene that is produced when rendering objects to according to the proposed attributes inferred in particle k. We further discuss the impact of this inference strategy (Sec. 6
4 Experiments
We first. validate our approach within the generative environment specified by our model (Sec. 3.1 under the set prediction task (Sec. 1.1 and directly compare its performance and data-efficiency against SA- MESH (Zhang et al., 2023). Then, we further evaluate the application of our model in scaled data complexity environment, using the CLEVR dataset [Johnson et al., 2017] For the first case, we generated a test. dataset of 500 scenes uniformly distributed over the number of sampled objects so that we could measure possible failing modes that become more evident when the number of objects to be predicted grows, The main assessment point is to effectively compare the performance of data-driven inference proposals evaluated through Monte Carlo methods (either with an object-wise sequential procedure or by weighting and resampling full posterior proposals) against posterior predictions learned in discriminative way by fully neural models.
4.1 Set prediction
The set prediction task involves taking the set representations (slots) and computing the target attributes for each object, allowing any arbitrary order among them. The main goal then to find a unique match wetween the predicted and the ground-truth sets of object attributes, validating the learned slot representations.
Baseline We compared ICSA performance against SA-MESH (Zhang et al. 2023). since we adopted its slot-based image encoding approach in our inference procedure. Hence, a direct comparison in terms of performance data efficiency can be made. To train SA-MESH in the same data domain, we employed our prior generative program (Sec.3 3.1 1 to generate an offline dataset with 30K images.
Results Following previous works (Locatello et al. 2020 Zhang al. 2019 2023 we computed the average precision (AP) values over different distance thresholds, using the same hold-out test set, for an increasing number of training steps to assess training efficiency. Fig. 25 shows AP values computed over all scenes with N' E (1,..., 10} objects for SA-MESH and ICSA. We split the ICSA results in two different inference procedures, where ICSA-SR denotes the score-resample procedure and ICSA-IS refers to the appli- cation of the traditional importance sampling algorithm. These different inference procedures within ICSA allow to investigate the trade-off between having more structured method that enhances interpretability but raises some ambiguitics, especially with larger N values (ICSA-SR), and faster but less interpretable method that represents posterior hypotheses as full program traces (ICSA-IS).
The AP curves over increasing training requirements (Fig. 2b show that ICSA models achieve at least the same level of prediction performance as SA-MESH, with R 20 times fewer training stops. Comparing both ICSA inference methods, approximating the posterior with a single-step proposal weighting (ICSA-IS) allowed achieve higher AP values, even though not holding for 0.0625. On the other hand, predictions made through ICSA-SR achieve higher ratio of true positives for lower distance thresholds, which can be explained by the sequential characteristic of its inference process that enables thorough evaluation of the object-wise proposals. However, pre-establishing where objects are not discovered due to ordering ambiguities (see Fig 13) Detailed comparative plots of AP values for each group of scones can be found in App. A.2
order for the set. of objects can raise failing mode
Fig. 2a shows examples of the sequential process that occurs during inference with the seore-resample al- gorithm: for each object, the best proposal is resampled given how well it contributes to explaining the initial observation. This is why object overlaps tend to be avoided. Even though overall AP values obtained in ICSA-IS are higher than by going through the sequential score-resample algorithm, the final predictions may suffer from objects overlap (sce Fig. 2ch. which is something that the ICSA generative model does not
The implemented code and samples available httes//gghhha.cm//rraccaaaaaaaco

---

Published in Transactions on Machine Learning Research (01/2026)
133
IC5A GR.4111
KCDA Afo4f
Irage
Okg 1
Org
Oij 8
Oti
ON
City
Ci2y
KSGA E E.A
KCAA 6.A
16/02
Tor12
Ter04
(b) Overall set prediction performance.
Image
Kprppesis
Photodor
(a) ICSA-SR.
(c) ICSA-IS.
(d) SA-MESH.
Figure 2: (a) Sequential inference with ICSA-SR, where at each step, proposals for an additional object (top row) are weighted and resampled (bottom row). (b) Comparative results for set prediction performance expressed AP (%), for different. training requirements, over 5 random seeds, averaged over all scenes with N E {L..., 10) objects. Red lines correspond to inferences made with SA-MESH, while green and blue lines correspond to the ICSA with importance sampling and score-resample inference procedures, respectively. Darker lines are associated with lower distance thresholds a. (c) Inference with IS by evaluating whole-scene proposals. (d) Predictions obtained by rumning the SA-MESH model and rendering the set of inferred object properties. For all inference alternatives, two random test examples (with 10 and 6 objects, respectively) are shown to illustrate overall results. The model parameters used to compute these examples were loaded after 10K training steps for a fair comparison between SA-MESH and ICSA inference.
generate a priori. Object properties inferred by SA-MESH also suffer from the same issue, which becomes more evident when the number of objects present in the scene grows (see Fig. 2d and Fig. 12ch. Finally, at an image level, SA-MESH results tend look worse than the ones obtained with ICSA inference models due to (1) all models were trained for set prediction (asks and not for object discovery (i.e., with objective function that does not operate pixel-level errors) and (2) SA-MESH does not account for uncertainty in its predictions. This is quantitatively shown in the performance decrease for lower a values (sce Fig. 2b We provide additional inference details and ablation studies in App. A.2
4.2 Model misspecification scenarios
We specified our generative program with broad and uniform priors, being reasonable strategy to avoid not covering possible posterior regions (Le et al. 2017 Paige & Wood 2016 Gothoskar et al. 2021 However, we also investigated the behavior of our inference methods under some out-of-distribution (OOD) situations, aiming to explore what could be expected from a misspecified model that tries to infer object properties in data that it cannot generate priori. Here, we focused on OOD scenes that contain objects with properties never seen in training, cither by adding a new shape to the test library, a new color, or both the same time. As shown in Fig. Baj the test sample in the first row contains green triangles, while the second image shows two objects with new randomly sampled RGB code for instance, (150, 56, 195). For both examples in this figure, we sampled object positions and these remained fixed.
We also explored more challenging OOD samples, generated by sampling object properties over distributions broader than the ones covered by our prior generative program, and composing them to generate objects with unseen shapes and colors at the same time (sec Fig. 3b). More concretely, we added different gcometries the shape library (ellipses with varied levels of eccentricity, non-squared rectangles, and triangles), and all colors sampled from uniform distríbutions over the full range of RGB codes. To facilitate the
7

---

Published in Transactions on Machine Learning Research (01/2026)
0C0
and prod.
prod.
prop. and pred.
SA-MESH pred.
0C0 Image
Dej.
Otj.
Objy 3
Orj
(b) Color and shape OOD scenarios generated at the time.
(a) Color and shape OOD scenarios generated separately.
Figure 3: (a) Out-of-distribution samples represent misspecification behavior at different situations (shape in the first OOD image, and color in the second). Property inference and generated samples are shown for ICSA-SR (left), ICSA-IS (middle) and SA-MESH (right). (b) More complex out-offdistribution samples and inference results obtained with ICSA-SR (left), ICSA-IS (middle) and SA-MESH (right) inference procedures. The traces overlay obtained for each object in ICSA-SR are shown in the same row the observed sample, while the corresponding final sampled trace is shown the row below.
interpretation of these results, we chose to maintain the number of objects on these scenes relatively low, and positions were sampled so that ordering ambiguities could not be an issue a priori.
Results In the most simple cases of Fig. 3a it. seems like both ICSA methods hold results that are more coherent and close to the observed objects, even when not capable of generating them (note that the gencrative program remained the same for all approaches, hence OOD latents could not be inferred neither generated). We would like to highlight the fact. that the number of objects that are predicted corresponds to the reality in ICSA posterior proposals, in contrast with SA-MESH. More, located in positions that could cause ordering ambiguities (which is seen in the traces of the second OOD image), uncertainty is easily observed in the shape misspecification example where slots showed difficulties binding to objects exclusively. This was not the case for the color misspecification scene, which might indicate that pixel-level properties are exploited at different extensions when the attention mechanisms of the neural module are optimized. In both examples, the SA-MESH baseline detected additional objects the scene, and uncertainty not as easily observed and qualified as with ICSA probabilistic inference. In the cases of Fig. 3b conclusions are less trivial to be drawn. However, the predictions made by ICSA (mainly with SR inference) seem to be more consistent when multiple OOD scenarios oceur in the same observation, making more coherent shape inferences for objects with similar shapes even if these don't 't belong to its prior library. In SA-MESH, inferred properties are less stable and some inconsistencies seem to occur frequently (c.g. predicting a different color for objects that are presented with very similar colors, as seen in the middle and bottom examples).
though objects not
4.3 Scaling data complexity
We also tested ICSA in a data environment with increased scene complexity: the CLEVR dataset (Johnson et al. 2017), a synthetic datasct of 3D scenes of up to 10 objects, varying in size, shape, color and material. Considering the different group generative variables required explain CLEVR scenes, we modified our prior generative program to include them. To speed up inference times, only the ICSA-TS inference strategy
8

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
1.0
0.8
0.8
mAP (9)
0.6
AP (%)
0.6
particies
0.4
SAT
0.4
100
SA-MESH!
30
0.2
ICSA
0.2
10
ICSA (best)
0.0
0.5
0.25
0,125 0.0625
0.5 0.25 0.1250.0622
distance threshold
distance threshold
(a)
(b)
Figure 4: (a) Set prediction results measured as mAP in % (values are presented as mean with shaded areas representing associated standard deviations) for different distance thresholds Results obtained with SA and SA-MESH (with t) were copied from the respective papers. (b) Particle ablation results of ICSA, considering K E {100,30, 10, 1}, with best-particle values represented as dots on top of cach bar.
is used to compute weights for posterior proposals and further mAP values. This makes the computational cost much lower since it requires only the generation of K predicted hypotheses for each test scene, instead of K N, being N the number of predicted objects.
We provide these results mainly for a malter of empirical practicality test, demonstrating that ICSA scales for more complex data environments, assuming the generative model is known priorí. Given our focus training efficiency, we constrained training duration to the same 10K steps as in the previous experiments, which we believe already allows us to draw some conclusions compared to neural baselines.
Results We the achieved mAP values for ICSA and baseline models in the CLEVR validation dataset in Fig. 4a As mentioned above, the main purpose of this scaling experiment is to demonstrate that our method is capable of learning to invert more complex generative models, assuming kuowledge about them. However, looking these quantitative metrics, we must also address the main explanations for why ICSA is not as accurate SA-MESH, at least regarding higher distance thresholds. First, to speed up TCSA training, chose to decrease some rendering parameters related to image quality, which makes training and validation images different in terms of general image quality in way that may not be negligible. The second reason is related to the misalignment between pixel-level likelihood metrics (which affect particle weights for rosampling) and latents prediction correctness, mcaning that not always the trace that better predicts the set of object properties will be the resampled one for instance, we illustrate such an example in Sec. 4.4 As defined in Sec. 3.3. scoring each particle involves evaluating the observed scene under a Gaussian distribution with an uncertainty parameter øL. For this uncertainty calibration, we used the effective sample size (ESS) metric to find dz values such that the ratio ESS would lie around 0.1, aiming to mitigate this misalignment issue by not having too many hypotheses equally possible of being resampled. We also plot the mAP values have the best particle - found by comparing mAP values posteriori - been resampled for each validation scene (ICSA (best)), which clearly shows the effect of this misalignment. In fact, we see that if pixel-level likelihood scores were paired with property predictions, ICSA would deliver very competitive results. In Fig. 4b we ablate the number of particles used for inference, showing that better results don't necessarily come from having more hypotheses to choose from. Again, we show that better hypotheses could have been sampled from the proposed posterior, but having single particle seems to yield the best results because it avoids particle scoring issues. Finally, measure training efficiency by comparing the number of batches used to run forward and backward for each approach, well as their sizc. Considering that we were not able to find the number of training steps employed for the set prediction task with CLEVR in SA-MESH, we used the values reported by Locatello et al. (2020) in the original SA paper, which were 150K training steps with batches of 512 scenes which is why we added SA results to Fig. 4a In contrast, our experiments with CLEVR used 10K training steps, with batch size of 64. This represents proportion of 120x less images seen during ICSA training.
9

---

Published in Transactions on Machine Learning Research (01/2026)
4.4 Measuring uncertainty in ICSA
Considering the particle-based nature of inference procedures in ICSA, it is possible to observe and measure prediction uncertainty under challenging conditions, e.g. comparing occlusion and no-occlusion pairs of identical scenes, revealing how the prediction of different visual properties is affected in such situations. Fig. 5 illustrates two examples of an identical occlusion scenario, differing in the similarity the colors of the occluder and the hidden object. In the first. case (Fig. [aa, some uncertainty can be observed all properties, but the fact that each object presents unique color makes it easier to coarsely distinguish shape and materials (first two bar plots). Also, it is interesting to note that even though only "small" object could be occluded in that situation, the correct size was only inferred with low uncertainty once the occlusion event ended. When objects present the same color (Fig. 5bb, higher uncertainty can be observed (e.g. when inferring object materials), but these are also resolved once objects are no longer occluded. In the selected case, the shape of the occluded object remained uncertain, but here occlusion no longer a plausible explanation for such behavior.
Resampling mistakes As mentioned before, possible explanation for incorrect prediction lies on misalignment between the likelihood weights computed for each particle and the probability distributions that actually compose the inferred posterior. For instance, in the shape uncertainty case that is shown in Fig. 5b] the likelihood weights of all inference particles are distributed as shown in Fig. 5c aggregated by the sampled shape class. It is clear that, in some cases, these likelihood weights are not very informative about the choice to be made, since they result in very similar ranges of values for particles of both classes. Moreover, in this case, the choice made from the sampled particle was the wrong one.
Ban
Trace log. ht
R
Brge
cube
cylinder
(a)
(b)
(c)
Figure 5: (a) Occlusion between objects of different colors and (b) between objects of the same color. From top to bottom rows, inferred properties represent object shape, material and size, respectively. (c) the case where uncertainty remains high (image (b)), might be the case that the resampled trace (marked as the red x) selected compute mAP results not the one that better predicts the properties of observed objects.
5 Conclusions
We approached object-centric representation learning with an inference neural module composed by special- ized proposal networks that learn to invert a probabilistic program for scene generation. By combining a state-of-the-e the-art neural architecture with object-centric inductive biases with a symbolic generative program, ICSA exhibits superior data efficiency with competitive property prediction performance while enhancing interpretability. In particular, ICSA shows enhanced robustness under OOD situations, with consistent and more plausible inference results when observing scenes explained by values out of the training support..
10

---

Published in Transactions on Machine Learning Research (01/2026)
Moreover, the particle-based inference nature of ICSA allows to seamlessly observe the uncertainty associ- ated with the posterior distribution proposed for each latent variable. We also show that ICSA can scale to more complex data environments while still holding the same data efficiency benefits over state-of-the fully neural baselines.
Finally, ICSA can represent a foundational framework for future approaches within object-centric represen- tation learning in video data of interacting objects, where an intuitive physics engine can be modeled to emulate the noisy expectations that humans exhibit on object physical behavior during childhood (Ullman
et al. 2018
Ullman & Tenenbaum
2020
Lake et al. 2017
6 Limitations
A limitation inherent to learning inference proposals in a fixed generative model lies in the fact that discrete random variables require pre-specified possible values. Even though continuous relaxations can be employed in some cases, learning new categories as data is observed (Sablé-Meyer et al.) 2022 Mills et al.) 2023 is a much more plausible hypothesis that should be implemented in the future. ICSA proposes a view on object representation learning by initializing its generative model with a certain amount of innate knowledge; for instance, the visual properties of objects. However, less complex primitive knowledge could be the basis for learning perceptual modules of object representations, while at the same time, learning primitive compo- sitions of different shapes or colors to progressively better explain obscrvations. This way, the generative model could also be learned through program induction (Rule et al. 2020 Lew et al. 2023).
We highlight that further applicability to real- world settings is limited by two main assumptions, related to each other. It is quite direct to recognize that prior latent factorization, even though often assumed for disen- langlement, or more formally in non-linear independent component analysis (ICA) literature (Khemakhem) et al. 2020 Hyvärinen et al. 2023 Zheng et al.) 2022), largely limits scalability when it comes to models of image representations. Entities in the world are inherently related to each other (Schölkopf et al. 2021] which naturally drives these models to emulate possible dependencies if true scene understanding is the main goal. TCSA is also limited by this, and having richer priors that include dependencies among generative vari- ables is a natural extension for this work. In this case, these dependencies must be learned such that the model is able to extract causally disentangled representations of each entity in the image (Komanduri et al., 2024).
The second assumption probably the strongest one is related to the prior knowledge of the truc generative model. We captured ICSA's behavior under specific misspecification scenarios (Sec. 4.2 showing that we can still provide robust predictions when the prior generative model is not expressive enough to explain a certain observation. However, even though such scenarios pose an interesting OOD asscssment, these still fall under the same domain as the one used for training, which is not the case when aiming to infer object properties in more realistic scenes. A direct path for scaling TCSA requires modifying the prior generative model and its rendering function (as already done in previous related works (Smith et al., 2019 Kulkarni et al. 2015 Gothoskar et al. 2023) which might come with costs at inference times due to the increased computational burden of rendering each posterior hypothesis for likelihood evaluation. Also, running inference over real-world data would require access to the true generative model, which is not feasible. For fully neural approaches, this is not an issue, since these can flexibly be adapted to different datasets with minor architectural changes. Nevertheless, we believe that new methods for likelihood evaluation - for instance, methods that move from pixel-based hypotheses evaluations to validation based on higher-level information between simulated and real images could significantly enhance the applicability of approaches like ICSA in realistic data environments. Without the explicit requirement for the true generative model, pushing the development of more human-like likelihood evaluation procedures may bring ICSA closer to work within more realistic settings. This possibility is aligned with a shift in the onalysis-by-synthen but we believe that it represents a plausible hypothesis for true scene understanding instead of requiring the precise generation of whatever is observed. In addition, as we motivated in our experiments, these approaches should be implemented in a way that aligns evaluation scores with the sampled choices that compose the posterior, such that, from the set of posterior hypotheses, the most likely ones contain predictions that are closer to what is observed in reality. Unfortunately, pixel-level metrics often miss this alignment.
paradigm,
11

---

Published in Transactions on Machine Learning Research (01/2026)
We finally acknowledge the failing mode that occurs set prediction tasks caused by the need to specify identity order over proposed traces for objects to be evaluated within the SR inference algorithm. Even though we still consider evaluating the posterior of each object sequentially a plausible direction to follow, and show its advantages over ovaluating full traces, the chosen ordering scheme does not completely prevent ambiguities and places a non-negligible impact on inference quality. Therefore, new methods should be explored to break these ambiguities.
Acknowledgments
This work is financed by National Funds through the Portuguese funding agency, FCT Fundação para a Ciência e a Tecnologia, within project LA/PP/006/ /2020 and under the PhD grant 2021.0576..BD (https: //doi org/10. 10. 54499/2021.05777 BD), and by the Euro- pean Union under the Horizon Europe Programme AILLUNGS project (Grant Agreement No. 101080756).
References
Titas Anciukevicius, Patrick Fox-Roberts, Edward Rosten, and Paul Henderson. Unsupervised causal gen- erative understanding of images. Advances in Newral Information Processing Systems, 1-.37077-37054
2022.
Rence Baillargeon, Elizabeth S Spelke, and Stanley Wasserman. Object permanence in five-month-old infants. Cognition, 20(3):191-208, 1985.
Atilim Günes Baydin, Lei Shao, Wahid Bhimji, Lukas Heinrich, Lawrence Mcadows, Jialin Liu, Andreas Munk, Saeid Naderiparizi, Bradley Gram-Hansen, Gilles Louppe, et al. Etalumis: Bringing probabilistic programming to scientific simulators at scale. In Proceedings of the international conference for high performance compuling, networking, storage and analysis, pp. 1 24, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-188) 1828, 2013.
Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):973-977 2019.
Ondrej Biza, Sjoerd van Steenkiste, Mchdi SM Sajjadi, Gamaleldin F Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. arXiv preprint arXiv:2302.04 04973, 2023.
Jack Brady, Roland Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius Von Kügelgen, and Wieland Brendel. Provably learning object-centric representations. In International Conference Machine Learn- ing, pp. 3038 3062. PMLR, 2023.
Jack Brady, Julius von Kügelgen, Sebastien Lachapelle, Simon Buchholz, Thomas Kipf, and Wieland Bren- del. Towards object-centric learning with general purpose architectures. In NewrIPS 2024 Workshop on Compositional Learning: Perspectives, Methods, and Paths Forward, 2024.
Susan Carey. The Origin of Concepts. Oxford University Press, 2009.
Susan Carey and Fei Xu. Infants' knowledge of objects: Beyond object files and object tracking. Cognition, 80(1-2):179- 203, 2001.
Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, Yisong Yue, ct al. Neurosymbolic programming. Foundations and Trends& in Programming Languages, 733):158-243, 2021.
Nicolas Chopin, Omiros Papaspiliopoulos, et al. An introduction to sequential Monte Carlo, volume Springer, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard Zemel. The helmholtz machine. Newral computation, 7(5):889 904, 1995.
Arnaud Doucet, Nando De Freitas, and Neil Gordon. An introduction sequential monte carlo methods. Sequential Monte Carlo methods in practice, pp. 3-14, 2001a.
Arnaud Doucet, Nando De Freitas, Neil James Gordon, et al. Sequential Monte Carlo methods in practice, volume 1. Springer, 2001b.
Arnaud Doucet, Adam M Johansen, al. tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704):3, 2009.
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems, 35:28940- (188940-28944, 2022.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E. Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neurol Information Processing Systems, 29, 2016.
Reuben Feinman and Brenden M Lake. Learning lask-general representations with generative neuro-symbolic modeling. In International Conference on Learning Representations, 2021.
Artur A'vli Garcez and Luis C Lamb. Neurosymbolic ai: The rd wave. Artificial Intelligence Review, pp. 1-20, 2023.
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognilive science sociely, volune 36, 2014.
Noah D Goodman and Andreas Stuhlmüller. The design and implementation of probabilistic programming languages, 2014. URL http://ddppl..rrgg
Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. Probabilistic program- ming. In Future of Software Engineering Proceedings, pp. 167-181. 2014.
Nishad Gothoskar, Marco Cosme,, Ben Zinberg, Matin Ghavamizadch, Falk Pollok, Austin Gar- rett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 3dp3: 3d scene perception via proba- bilistic programming. Advances in Neural Information Processing Systems, 34:9600-966 9612, 2021.
Nishad Gothoskar, Matin Ghavami, Eric Li, Aidan Curtis, Michael Noseworthy, Karen Chung, Brian Patton, William T Freeman, Joshua B Tenenbaum, Mirko Klukas, et al. Bayes3d: fast learning and inference in structured generative models of 3d objects and scenes. arXiv preprint arXiv:2312.087/ 2023.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 202222666) 2022.
Fdwin James Green and Jake Quilty-Dunn. What is an object file? The British Journal for the Philosophy of Science, 2021.
William Harvey Andreas Munk, Atalmm Güneş Baydin, Alexander Bergholm, and Frank Wood. Attention for inference compilation. arXiv preprint arXiv:1910.11 19101119,, 2019.
Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The" wake-sleep' algorithm for unsupervised neural networks. Science, 268 1161, 1995.
Berthold KP Horn. Understanding image intensities. Artificial intelligence, 8(2):201-2)) 1977.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlincar independent component analysis for principled disentanglement. in unsupervised deep learning. Palterns, 4(10), 2023.
Varun Jampani, Sebastian Nowozin, Matthew Loper, and Peter Gehler. The informed sampler: dis- criminative approach to bayesian inference in generative computer vision models. Computer Vision and Image Understanding, 136:32 44, 2015.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object-specific inte- gration of information. Cognitive psychology, 24( 1992.
lyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR, 2020.
Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim. Shepherding slots to objects: Towards stable and robust object-centric learning. In Proceedings of the IEEE/CV /CF Conference on Computer Vision and Pallern Recognition, pp. 19198 19207, 2023.
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference on Learning Representations, 2021.
Aneesh Komanduri, Yongkai Wu, Feng Chen, and Xintao Wu. Learning causally disentangled representations via the principle of independent causal mechanisms. In Proceedings of the Thirty- Third International Joint Conference Artificial Intelligence, pp. 4308 1308-4316, 4316, 2024.
Avinash Kori, Francesco Locatello, Ainkaran Santhirasekaram, Francesca Toni, Ben Glocker, and Fabio De Sousa. Ribeiro. Identifiable object-centric representation learning via probabilistic slot attention. anXiv preprint arXi::2406.071 07141, 2024.
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. Ad- vances in neural information processing systems, 32, 2019.
Tejas Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: probabilistic programming language for scene perception. In Proceedings of the ieee conference on computer vision and pottern recognition, pp. 4390-4399, 2015.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332 1138, 2015.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Alexander Lavin, David Krakauer, Hector Zenil, Justin Gottschlich, Tim Mattson, Johann Brehmer, Anima Anandkumar, Sanjay Choudry, Kamil Rocki, Atilm Güneş Baydin, et al. Simulation intelligence: Towards a new generation of scientific methods. arXiv preprint 2rXiv:2112.0922 2021.
Tuan Anh Le, Atilim Gunes Baydin, and Frank Wood. Inference compilation and universal probabilistic programming. In Artificial Intelligence and Statistics, pp. 1338-1348. PMLR, 2017.
Alexander K Lew, George Matheos, Tan Zhi-Xuan, Matin Ghavamizadch, Nishad Gothoskar, Stuart Russell, and Vikash K Mansinghka. Smep3: Sequential monte carlo with probabilistic program proposals. In International conference on artificial intelligence and statistics, pp. 7061-7088. PMLR, 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yichao Liang, Joshua B Tenenbaum, Tuan Anh Le, and N Siddharth. Drawing out of distribution with neuro-symbolic generative models. arXiv preprinl or Xrx::2206.00 01...018810 01829, 2022.
Yi Lin, Maayan Stavans, and Renée Baillargeon. Infants' physical reasoning and the cognitive architecture that supports it. Cambridge handbook of cognétive development, pp- 168-194, 2022.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525 11538, 2020.
Minh-Thang Luong, Hicu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint Nao:1508.0405,, 2015.
Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. Approximate bayesian image interpretation using generative probabilistic graphics programs. Advances in Newral Information Processing Systems, 26, 2013.
Amin Mansouri, Jason Hartford, Yan Zhang, and Yoshua Bengio. Object-centric architectures enable efficient causal representation learning. ar preprint arXiv:2310. 19054, 2023.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2019.
Tracey Mills, Joshua B Tenenbaum, and Samuel J Cheyette. Human spatiotemporal pattern learning probabilistic program synthesis. In Thirby-senenth Conference Neural Information Processing Systems, 2023.
Andreas Munk, Adam Scibior, Atilm Günes Baydin, Andrew Stewart, Goran Fernlund, Anoush Poursartip, and Frank Wood. Deep probabilistic surrogate networks for universal simulator approximation. ar preprinl arXio:1910. 11950, 25, 2019.
Andreas Munk, Berend Zwartsenberg, Adam Scibior, Atili Güneş G Baydin, Andrew Stewart, Goran Fernlund, Anoush Poursartip, and Frank Wood. Probabilistic surrogate networks for simulators with unbounded randomness. In Uncertainty in Artificial Intelligence, pp. 1423 1433. PMLR, 2022.
Brooks Paige and Frank Wood. Inference networks for sequential monte carlo in graphical models. In International Conference on Machine Learning, pp. 3040 3049. PMLR, 2016.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kauf- mann Publishers Inc., San Francisco, CA., 1988.
Joshua S Rule, Joshua B Tenenbaum, and Steven T Piantadosi. The child as hacker. Trends in cognitive sciences, 24(11):900 915, 2020.
Mathias Mayer,Meyer, Kevin Ellis, Josh Tenenbaum, and Stanislas Dehaene. A language of thought for the mental representation of geometric shapes. Cognitive Psychology, 139:101527, 2022.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Newral Information Processing Systems, 35:9512 9524, 2022.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)) 612-634, 2021.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap real-world object-centric learning. arXio preprint arXiv:2209. 14860, 2022.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXim:220.. 14065, 2022.
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, (1(2):343-3)) 348, 1967.
Kevin Smith, Lingjie Mei, Shunyu Yao, Jiajun Wu, Elizabeth Spelke, Josh Tenenbaum, and Tomer Ull- man. Modeling expectation violation in intuitive physics with coarse probabilistic object representations. Advances in neural informalion processing systems, 32, 2019.
Elizabeth S Spelke. Principles of object perception. Cognilive science, 14(1):29 56, 1990.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1))89-96, 2007.
Maayan Stavans, Yi Lin, Di Wu, and Renée Baillargeon. Catastrophic individuation failures in infancy: A new model and predictions. Psychological review, 126(2):196, 2019.
Andreas Stuhlmüller, Jacob Taylor, and Noah Goodman. Learning stochastic inverses. Advances in neural information processing systems, 26, 2013.
Tomer D Ullman and Joshua B Tenenbaum. Bayesian models of conceptual development: Learning building models of the world. 2020.
Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21(9):649-66) 2017.
Tomer D Ullman, Andreas Stuhlmüller, Noah D Goodman, and Joshua B Tenenbaum. Learning physical parameters from dynamic scenes. Cognétive psychology, 104:57-88 82, 2018.
Jan- Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. An introduction to probabilistic programming. arXiv preprint 1809 10756, 2018.
Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot attention. arXiv preprint Xix:2330 069977, 2023.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning physics simulator from video. Advances in neural information processing systems, 30, 2017.
Taylor W Webb, Shanka Subhra Mondal, and Jonathan D Cohen. Systematic visual reasoning through object-centric relational abstraction. arXiv preprint arXiv:2306. 020000000.0000, 2023.
Thaddäus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. arXiv preprint arXiv:2310.08 053227, 2023.
Jiajun Wu, lkkrr Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galilco: Perceiving physical object propertics by integrating a physics engine with deep learning. Advances in neural information processing systems, 28, 2015.
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipl, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.0568 0.00661, 2022.
Fei Xu, Susan Carey, and Jenny Welch. Infants' ability to use object kind information for object individuation. Cognition, 70(2):137 166, 1999.
Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognâtive sciences, 10(7):301-300 2006.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Information. Processing Systems, 32, 2019.
Bennett. Deep set prediction networks. Advonces in Neural
16

---

Published in Transactions on Machine Learning Research (01/2026)
Yan Zhang, David W Zhang, Simon Lacoste-Julien, Gertjan J Burghouts, and Cees GM Snoek. Unlocking slot attention by changing optimal transport costs. ar preprint 00t-2301113997, 2023.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Aduances in neural information processing systems, 35:16411-16422, 2022.
Roland Zimmermann, Sjoerd van Steenkiste, Mehdi SM Sajjadi, Thomas Kipf, and Klaus Greff. Sensitivity slot-based object-centric models to their number slots. arXiv preprint orXiw:2305.18890 2023.
A Appendix
A.1 Implementation details
Additional details on SA-MESH The SA algorithm proposed (Locatello al. 2020) maps an encoded global representation the input image into set of distributed slots using an attention mechanism. It starts by randomly initializing fixed set of slots sNx,, which linear projection is applied to produce associated queries NDD. The imput representation is also projected into keys DDDD, and values DxD, distributed representations, which are then used to compute the dot-product attention (Luong et al. 2015 Here, N refers to the number of instantiated slots of dimension D., and D the size of the flattened representation of the input. Then, a double normalization procedure encourages not only slots to compete to represent input information exclusively but also all partitions the input to be encoded by the group of slots (Locatello al.) 2020). SA-MESH takes an initial cost matrix Ci = d(Q,, K2) from a distance metric d (e.g. Euclidean distance) between the set of queries and keys. Then, Equations and are repeated for T iterations, returning final attention map A Minimizing the entropy H of the transport map at cach iteration increases the sparsily of slot representations.
(ESH(C) = arg min ((sikhoorn(()) C'EV(C)
(4)
A sinkhor((MESH(())
(5)
Proposal networks As explained in Sec. posterior proposals are obtained through separated networks, consisting softmax-activated linear layers, learning attribule-wise projections from slot representations to the number of attribute classes. Given the convergence into object-wise slots, we empirically found that object attributes were quite simple to approximate without requiring deeper or wider classification modules. Also, since we assume that there is no statistical dependency among objects, these proposals are computed given the representation of single corresponding slot.
Finding object positions Training a neural network to obtain posterior proposals for each location latent variable considered unnecessary challenge. Hence, relaxed this problem by leveraging each slot's 's attention mask. Similarly to previous attempts to relax object location inference (Watters et al. 2017 Kipf
ct al. 2021 Kim et al. 2023 Wu et al. 2022 Biza et al. 2023), we used a 2-channel absolute grid (for z- and y- g-coodiinates) such that inferring the position of all objects only requires computing the dot-product (-) between the attention masks ATT) and the absolute grid, The resulting coordinate point was then used as the mcan of a fixed-noise Gaussian posterior over hypothesized locations.
A.2 Set prediction
SA-MESH training details For the experiments reported in Sec. 4.1
we trained SA-MESH for 10k
training steps, ensuring convergence, with batches of 256 images, using the Adam optimizer with a learning rate of 1 x 10-3 and remaining parameters as default. Slots were randomly initialized, instantiated with a maximum number of objects set to 17 (according to the generated synthetic data used for training), slot iterations T 3 and MESH iterations set at 4.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Scaling the number of objects We ran inference with the ICSA-TS procedure over scenes ranging from 10 to 20 objects, aiming to investigate how the model behaves when increasing the number of objects inside and outside the training support. For this, we trained a model exactly at the same settings as in Sec. 4.1 only setting N 15, instead of 10. From the results in Fig. it is possible to observe that AP values tend to slowly decrease as the number of objects approaches the maximum value N = 15 set for training: if we keep increasing it beyond that, the model's actually responds well to having to instantiate a higher number of slots to encode more than 15 objects, and performance stabilizes. Increasing beyond 20 starts to become infcasible considcring the scene dimensions.
objects
1.0
0.8
14
15
16
0.4
17
18
19
20
0.0
0.1250.0000
distance threshold
Figure 6: AP values obtained by run- ning the ICSA-IS inference procedure over scenes with N {10,..., 20) objects. Dur- ing training, the maximum number of slots was set to 15.
Additional inference results We also show comparative bar plots of each model used for object's property inference (ICSA-IS, ICSA-SR and SA-MESH) for increasing training power: in Fig. 9 all models were trained for 100 training steps, in Fig. 10 for 500 training steps, in Fig. II for 1000 and finally, AP performance of models trained for 10k training steps is shown in Fig. 12
An oo distance threshold denotes no distance criterion for an object to be considered true positive once all predicted properties match the ground-truth. Note that, due to its sequential characteristic of the inference procedure, ICSA-SR property prediction performance holds at relatively the same overall values from higher to lower distance thresholds, while consistently decreasing when the number of objects present the test scene grows. One of the reasons why this happens related to object identity ambiguities in the sequential sampler: the more objects are to be inferred, the harder it becomes to account for the symmetries that result from ordering objects according to their Euclidean distance to the origin (see Fig, 13 in test images #2 and #3 for con- crete examples). Hence, since proposal traces do not order the objects the same way, it is more likely that a certain object never gets explained in the final predicted scene. For instance, in test image #2 with N = 10, it is possible to observe that among all proposal traces, the small red square at the bottom was ranked at 4 different positions according to its predicted location. However, at these steps, different objects were selected to explain the observation according to the score-resample algorithm. In the last step, the SMC sampler only scored proposals for 2 objects, none of them corresponding to the 'forgotten' one.
We took the set of test scenes with N = 5 and investigated how the log-likelihood of the set of particles evolves as the SR procedure goes through all objects to be inferred. Fig. shows that as objects get explained, the averaged log-likelihood of the set of posterior proposals tends to decrease. Although this might seem counterproductive, ie. one would expect that progressively explaining an image would result in increased log-likelihoods of the generated images at each inference step, but we found this is explained by small errors at location inference - propagated through the entire procedure that happen to increase the error of selecting another object over selecting none. Regarding inference times, we computed for single scene with 5 objects, and the relation between the set of inference strategies explored in this work is shown in Table [ Surprisingly, the iterative nature of SR does not increase inference times by a significant amount when compared with IS, and SA-MESH holds the faster inference procedure as expected. This is explained by the fact that, in contrast with the experiments conducted with CLEVR, the image generation procedure in this data enviroument does not represent a computational bottleneck due to the simplicity of the data.
Selecting the number of particles We ablate the impact of different amounts of inference particles, showing the resultant mAP values of the ICSA-IS inference procedure in N scenes, averaged across 5 random scods (Fig. D We found consistent results over the different a values, with the main difference being showcased in the 0.0625 case, where higher number of inference particles seem have a larger positive impact in set prediction performance.
18

---

Published in Transactions on Machine Learning Research (01/2026)
particies weight
-5.0
100
-7.5
Inference
Time
80-
strategy
(x10² ms)
AP
12.55
particles
SR
2.683
40-
-15.0
IS
2.502
20
25
-17.5
50
SA-MESH 2.486
Objects
Figure 7: Log-likelingg ligeliiel weights averaged over the number par- ticles across N test scenes.
Table 1: Inference times of sin- gle sample for each analysed in- ference stralegy.
Figure 8: Ablation over the number particles given by mAP values (%) averaged across random seeds.
12
40
TO
2.5 0.25 5.125 0.0055
05 025 0.25 0.025
05 0.25 5125 00525
(a) ICSA-IS.
(b) ICSA-SR.
(c) SA-MESH.
Figure 9: Comparative plots of AP values al distance thresholds (oc, 1.0, 0.5, 0.25, 0.1 0.0625) averaged over 5 random seeds. Vertical lines top of each bar correspond to standard deviation values. Plots (a) and (b) correspond to the ICSA model with single-step importance sampling and following the score-resample procedure, respectively. Plot (c) shows the predictive performance of the trained SA-MESH model. All inference models were trained for 100 training steps.
95 025 0156 00126 treshold
05 0.26 1110 0.05
0.5 025 0.26 00126 tnnehol
(a) ICSA-IS.
(b) ICSA-SR.
(c) SA-MESH.
Figure 10: Comparative plots of AP values at distance thresholds (oo, 0.5, 0.25, 0.125, 0.0625) averaged over 5 random sceds. Vertical lines top of cach bar correspond to standard deviation values. Plots (a) and (b) correspond to the ICSA model with single-step importance sampling and following the score-resample procedure, respectively. Plot (c) shows the predictive performance of the trained SA-MESH model. All inference models were trained for 500 training steps.
19

---

Published in Transactions on Machine Learning Research (01/2026)
100
15
191
8.2 2.280002 oblarce theatol
ditares trisshold
datanc Brosteld
(a) ICSA-IS.
(b) ICSA-SR.
(c) SA-MESH.
Comparative plots of AP values at distance thresholds {oo, 1.0, 0.5, 0.25, 0. 0., 0.0625] averaged over random soeds. Vertical lines on top of each bar correspond to standard deviation values. Plots (a) and (b) correspond to the ICSA model with single-step importance sampling and following the score-resample procedure, respectively. Plot (c) shows the predictive performance of the trained SA-MESH model. All inference models were trained for 1000 training steps.
Figure 11:
F-chock
35 025 1125 00125
05 0.2
0135 01625
055 0.25 0 125 00135 Encanol
(a) ICSA-IS.
(b) ICSA-SR.
(c) SA-MESH.
Figure 12: Comparative plots of AP values distance thresholds {oo, 1.0, 0.25, 0.125, 0.0625) averaged over 5 random seeds. Vertical lines top of each bar correspond to standard deviation values. Plots (a) and (b) correspond to the ICSA model with single-step importance sampling and following the score-resample procedure, respectively. Plot (c) shows the predictive performance of the trained SA-MESH model. All inference models were trained for 10000 training sleps.
20

---

Published in Transactions on Machine Learning Research (01/2026)
prop. and pred.
SA-MESH pred.
Image
Obj.
Obj 2
Obj.
O
Obj.
Obj.
Obj.
Obj.
Obj.
Obj. 10
Figure 13: Examples showing the generated images resultant from each inference procedure, selected ran- domly among all test samples with E {10, 8, 6). From left to right, the first column shows an observed test scene, followed by the scquential procodure of the score-resample inference ICSA (N columns, each one showing the overlay of proposal traces for a specific object on the top row, and the resampled proposal on the bottom row). Then, the following column shows the overlay of proposals to be weighted in IS inference (top row) and the resampled trace (bottom row). Finally, the last column shows the generated images when running the generative program with the object propertics predicted by SA-MESH. All inference models were trained for 10000 training steps. Generated images with orange border demote examples where predicted locations cause objects to overlap (in ICSA-IS and SA-MESH) or ordering ambiguities cause missing an object (in ICSA-SR).
21

---

Published in Transactions on Machine Learning Research (01/2026)
CLEVR set prediction For the experiments with CLEVR (Johnson et al. 2017 maintained the same neural architecture, setting the batch size at 64 and the learning rate at 0.0004. Note that, since more latent variables are instantiated by the generative program, the neural module also appends the corre- spondent prediction heads (MLP). Also, in contrast with the procedure detailed in App. A.1 since CLEVR object positions camnot be directly inferred from atlention masks, the posterior over their coordinates also computed from specialized prediction networks.
22


Reviews:
[{"review_id": "CUD7II7chu", "reviewer": "Reviewer_oHj4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper concerns representing the data such as images, consisting of multiple objects with a semantic description (latent variables). This scenario can be expressed as a joint probability distribution (generative model) between the data and the latent variables. The paper suggests a method to calculate the posterior distribution of the latent variables, that can then be used in various downstream tasks. Following earlier studies, the paper presents a synthetic scenario consisting of multiple objects with certain shapes and colors and presents results in the set prediction task.   \n\nTo avoid modeling at a pixel level, the paper employs the slot attention mechanism (previous work), which returns a set of tokens corresponding to different objects and instead trains a neural based posterior model conditioned on these output tokens. In the downstream task, an importance sampling mechanism is employed.", "claims_and_evidence_yn": "No", "claims_explanation": "I am not sure what the claims of the paper are. If the goal is to have better posterior distributions, other baseline methods for posterior estimation should be studied. The comparison to slot-attention is somewhat unfair because the latter does not intend to exploit the the generative model, while the proposed method is inherently based on such knowledge.\n\nThe details of the methodology in the paper also remain unclear to me (see below), and it is difficult to say if the paper has a significant contribution in terms of its methodology.", "audience_yn": "Yes", "audience_explanation": "As mentioned by the authors, obtaining models that include symbolic elements is a popular goal, which allows more explainable models and better inductive bias due to the domain knowledge transferred by the symbolic representation. The paper contributes to this line of research.", "requested_changes": "My main concern is that the paper is difficult to read for multiple reasons: First, the paper relies on the reference papers to the extent that it  becomes difficult to understand its contribution without referring to other papers. It is not clearly mentioned which parts of the suggested method are the contributions of the paper. There are terms such as \"posterior proposal\" or \"particles\", the definition of which is not clear to me. Even algorithm 1 has subroutines that are not explained as far as I see. I think that in many places, using a mathematical equation is much clearer than explaining the procedure in plain text. \n\nMy next concern is about the contribution of the paper. I think that there are two parts, which can be considered the contribution. First, f^prop and second the importance sampling scheme. First of all neither of these steps are explained in detail. It is not clear to me how f^prop looks like. This is important because one may expect that the posterior distributions of the objects not to be factorizable (i.e. they are dependent) even if the prior is factorized. I think that the sampling scheme where the objects are added one by one emphasizes this fact. Hence, if each slot is predominantly related to an individual object, I do not expect that f^prop is factorized over slots, otherwise, it should not be so significant. The way that the importance weights are further used is also unclear to me and I am not sure what the actual output of this stage is. Is it a set of samples approximating the posterior, for example, which are updated by the importance coefficients? There seems to be also a contradiction between f^prop and the IS. If I am correct f^prop evaluates the probability density/mass function, while SA will sample it. So, which one is the goal?\n\nFinally, as also mentioned in the limitations part, this work is limited by requiring the knowledge of prior. Although the authors examine some OOD experiments with unseen shapes/colors. In practice, the dependency of the object is perhaps more critical. For example, a crosswalk in a driving scene is perhaps strongly related to existence a pedestrian. In a neuro-symbolic scenario this relation is often only partially known. I am not sure if the proposed method can address such scenarios. Moreover, as the final stage depends on  statistical sampling, it may become highly complex in the presence of many inter-related objects (as an exponentially growing batch of samples will be needed).", "broader_impact_concerns": "I do not have any concerns regarding ethical implications or other broader impact factors.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cqnCVWU1mS", "reviewer": "Reviewer_eZWu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Method \n\n(This is my understanding please do correct me it I am wrong)\n\nICSA proposes a neurosymbolic method for object-centric representation learning. The approach works as follows:\n\n**Training process**: A generative model samples object properties (shape, color, position, etc.) from probability distributions and renders synthetic images. The neural network learns to invert this process - given an image, predict the object properties that likely generated it.\nArchitecture: Images are processed by a CNN encoder to extract spatial features. SA-MESH attention then runs iteratively (typically 3-5 iterations) to refine object-centric slots through competitive binding. In each iteration, slots compete to attend to different spatial regions of the image through a cross-attention mechanism. Queries come from slots, while keys and values come from image features. This iterative refinement allows slots to progressively bind to coherent image regions. Specialized neural networks take each slot as input and predict probability distributions for object properties with separate networks for each property type (one for shape prediction, one for color prediction, one for position, etc.).\n\n**Training:** The system is trained end-to-end using KL divergence between predicted and true property distributions, which simplifies to cross-entropy for categorical variables (shape, color) and Gaussian negative log-likelihood for continuous variables (position).\n\n\n**Inference:** At test time, given a new image, the trained networks extract slots using the same CNN encoder + SA-MESH process. The proposal networks then predict object property distributions for each slot. These predictions can be used directly, or refined using Sequential Monte Carlo methods with two approaches: (1) ICSA-IS propose complete scenes all at once, weight all hypotheses, and select the best; or (2) ICSA-SR sequentially discover objects one-by-one, scoring and resampling particles after adding each object until the scene is fully explained.\n\n\n-----------------------\n\n# Strength \n- Novel neurosymbolic framework: ICSA is quite novel in that it uses SA-MESH as a component in a larger framework that connects neural visual processing with symbolic object representations through a fixed generative model.\n- Training efficiency: Reports ~20x fewer training steps than SA-MESH baseline, demonstrating that the fixed generative model provides strong inductive biases and supervised learning is more data-efficient than unsupervised approaches.\n- Out-of-distribution robustness: The OOD experiments are quite interesting as they show how the method can handle unseen data, maintaining coherent predictions when objects have properties never seen during training (new shapes, colors, or both simultaneously).\n -----------------------\n\n#  Weaknesses:\n- Clarity issues: The paper is difficult to follow and unnecessarily complex in its explanations. For example, the KL divergence discussion could simply state it's cross-entropy for categorical variables. Claims like \"learning the generative model and the inference network at the same time makes this optimization loop highly complex\" are made without empirical validation or testing.\n- Unrealistic methodological assumptions: The approach relies on two major assumptions that severely limit real-world applicability: (A) the generative model is known a priori, which is almost never the case in practice, and (B) the visual world consists of a discrete, predefined set of object properties, which contradicts the infinite complexity of real visual scenes.\n- Limited evaluation scope: The evaluation is restricted to overly simplistic synthetic datasets (basic geometric shapes) and CLEVR. Even on CLEVR, the method doesn't consistently outperform SA-MESH baselines on all metrics. No real-world data evaluation is provided, making it unclear how the approach would handle natural images or practical applications.", "claims_and_evidence_yn": "No", "claims_explanation": "- **Unvalidated theoretical claims:** Statements like \"learning the generative model and the inference network at the same time makes this optimization loop highly complex\" are presented as fact but never empirically tested. The authors don't demonstrate that their approach actually solves this supposed complexity issue.\n\n- **Unfair efficiency comparisons:** The claimed 20x speedup over SA-MESH is misleading because it compares supervised learning (ICSA with ground truth labels) against unsupervised learning (SA-MESH without labels), making it an invalid comparison.\n\n- **Unrealistic assumptions not addressed:** The paper requires known generative models and discrete property spaces, which severely limit real-world applicability, but these fundamental limitations are not adequately discussed.\n\n- **Limited experimental validation:** Despite suggesting ICSA could work in complex settings, experiments are restricted to very simple synthetic datasets, providing insufficient evidence for broader claims about the method's capabilities.RetryTo run code, enable code execution and file creation in Settings > Capabilities.", "audience_yn": "Yes", "audience_explanation": "Despite its significant limitations, this paper would likely interest several segments of TMLR's audience:\n\n- **Neurosymbolic learning researchers:** The combination of SA-MESH with probabilistic programming represents a novel approach that bridges neural and symbolic methods, which is an active area of research.\n\n- **Object-centric learning community:** Anyone working on slot attention, object discovery, or structured representations would find the application of SA-MESH in this context interesting, even if the overall approach has limitations.\n\n- **Inference compilation researchers:** The application of inference compilation techniques to computer vision problems is relatively uncommon and could inspire further work in this direction.", "requested_changes": "- I feel the paper is lacking in terms of experimental results. One way to improve the paper is to conduct experiments on the same tasks as SA-MESH to enable fair comparison. Specifically, evaluate ICSA on Multi-dSprites and ClevrTex unsupervised object discovery tasks, and CLEVRER video experiments with varying object numbers. In addition, test the method on scenes with larger numbers of objects (beyond the current limit of 10) to demonstrate scalability and assess when the approach begins to break down.\n\n- I think it would also be helpful to add a discussion of how the approach could be extended when the generative model is not known a priori, or when the world doesn't consist of discrete, predefined object properties. This is essential given that these assumptions severely limit real-world applicability.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oe0M2UM9Ja", "reviewer": "Reviewer_gXPq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes ICSA, a method that combines inference compilation with slot attention. The idea is that a neural network learns to predict posterior distribution for a fixed generative process (thus, in fact, learning to invert the generative process). This network can then be used at inference time to propose particles that are then fed into the prior generative program.\n\nSTRENGTHS\n- The problem of combining symbolic information with slot attention (object centric learning in general) is very relevant and interesting\n- The work despite being a bit incremental is novel.\n- Experiments are quite narrow in domains, but well executed.\n- Limitations are clearly stated.\n\nWEAKNESSES\n- The paper is written in a confusing and unclear way.\n- The method seems quite ad-hoc for the experiments.\n- Experiments are not very impressive (still insightful though).\n\nFEEDBACK / QUESTIONS:\n- Despite being quite familiar with the topics of the paper, it was very difficult to read and understand. Some points of improvements: avoid long sentences (there are a lot of sentences that are 3/4 lines long), avoid using too many synonyms for the same thing (moves the effort to the reader to connect the dots), use examples / intuitive explanations to explain difficult concepts (e.g. when explaining inference compilation, would be more useful to discuss a brief example instead of trying to explain the entire idea in abstract terms; the description of the latent discrete variables (shape, size, ..) is quite important and centra to the paper, it should be in the main body because it helps comprehension a lot), structure could be improved a bit (e.g in “Method” would be better to follow the same structure of Figure 1, so training and inference; in experiments might help to write at the beginning the experimental questions, so it’s clearer where’s the focus), if terms are used to clarify, they should be well-known or explained (e.g. the method sections starts with “Our proposed system follows the ideas from the analysis-by-synthesis paradigm and the Helmholtz machine (Hinton et al., 1995; Dayan et al., 1995)”, but unless someone is familiar with the terms, it just adds confusion and more questions). I will stop here with the feedback on the writing style, but I believe the text would benefit of a serious re-writing.\n- For the method, a question: I did not understand how the proposals at inference time are then combined to get the final predicted image. Can you explain or point me to where it is discussed?\n- In 3.1, does the generator have to be non-differentiable? It looks like the entire section is just to describe a rendering method, and Algorithm 1 seems to add more confusion than clarifying, unless I’m missing something…?\n- In 3.2.1, most of it seems to be just MESH, shouldn’t it be in the preliminaries instead?\n- 3.2.2 looks like an implementation detail, either explain better why it is fundamental, or it could easily be moved to the appendix.\n- In 3.2.3, is where an example could really clarify things, but from my understanding this check of permutations is quite important. Although, it doesn’t seem very scalable, does it?\n- In equation 5, the fact you assume to have access to that joint distribution at the numerator, seems quite a strong assumption. I guess it is what you also discuss in the limitations, but how do you have insights on how to relax it?\n- In general, it seems that the advantage ICSA gets, is due to the heavy induction bias. This is often the case in neurosymbolic methods, but in this case it looks like the bias is “heavier” than usual. It’s really forcing the network to predict some discrete latent variables that then makes the goal of generating very easy. I am not sure how easy is it to adapt to other domains. It feels very ad-hoc, and I am not 100% convinced that the same techniques would easily translate to more complex scenarios. What are your insights on this?\n- Experiments might benefit from an ablation study to isolate what is contributing most.\n- In the conclusions it is mentioned that interpretability is enhanced. However, neurosymbolic methods are known to suffer from reasoning shortcuts, and I am wondering if it is possible that the network in your model learns to predict latent distributions that actually go against what would be the right interpretation, but that somehow pushes the relevant metric up?", "claims_and_evidence_yn": "Yes", "claims_explanation": "Experiments support the claims made by the authors (even though it is easy to get lost in the text).", "audience_yn": "Yes", "audience_explanation": "As mentioned, the topic is relevant, interesting, and the proposed method is novel enoguh to make it interesting to share the insights provided here.", "requested_changes": "My biggest concern is the clarity of the text. If that can be improved, than all the rest is secondary.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)