Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation
Authors: Michael Chang
Thomas L. Griffiths
Sergey Levine
Venue: NeurIPS 2022 Accept (2022)
Keywords: objects
implicit differentiation
slot attention

Abstract: Current work in object-centric learning has been motivated by developing learning algorithms that infer independent and symmetric entities from the perceptual input. This often requires the use iterative refinement procedures that break symmetries among equally plausible explanations for the data, but most prior works differentiate through the unrolled refinement process, which can make optimization exceptionally challenging. In this work, we observe that such iterative refinement methods can be made differentiable by means of the implicit function theorem, and develop an implicit differentiation approach that improves the stability and tractability of training such models by decoupling the forward and backward passes. This connection enables us to apply recent advances in optimizing implicit layers to not only improve the stability and optimization of the slot attention module in SLATE, a state-of-the-art method for learning entity representations, but do so with constant space and time complexity in backpropagation and only one additional line of code.

Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation
Michael Chang', Thomas L. Griffiths!, Sergey Levine "University of Califormia, Berkeley, "Princeton University mbchang@berkelly..cm edu, tomgéprinceton. edu, wwlevine@eecs.berkeee edu
Abstract
Iterative refinement start with random guess, then iteratively improve the guess is useful paradigm for representation learning because it offers way break symmetries among equally plausible explanations for the data. This property enables the application of such methods infer representations of sets of entities, such as objects in physical scenes, structurally resembling clustering algorithms in latent space. However, most prior works differentiate through the unrolled refinement process, which can make optimization challenging. We observe that such methods can made differentiable by means of the implicit function theorem, and develop implicit differentiation approach that improves the stability and tractability of training by decoupling the forward and backward passes. This connection enables us apply advances in optimizing implicit layers to not only improve the optimization the slot attention module in SLATE, a state-of-the-art method for learning entity representations, but do with constant space and time complexity backpropagation and only one additional line of code.
Introduction
Conventionally, neural network models implement feedforward computation, transforming the input into the output through a fixed series of operations corresponding to distinct layers as (x))..)) However, a range of more sophisticated models implement iterative computation with the network, typically formulated and motivated some sort of optimization procedure where the correct answer is the fixed point of an iterative refinement = fz)) of an initial guess 2g- This includes diffusion models 30
52
58, 59] energy-based models 14 3r
7 iterative amortized inference procedures (0.42), neural ordinary differential equations [τлЛ meta- meta-learning algorithms [5 I 23 and object-centric models [Io]
deep equilibrium models
Such iterative refinement procedures may have a number of advantages over direct feedforward computation: they serve to simplify the learned function (e.g., in the way that recursive program might be much simpler than an equivalent program implemented without recursion), introduce an inductive bias into the model that improves generalization, and break symmetries.
35
61 52
In this work, we consider the case of iterative refinement applied representation learning of latent sets, which has primarily been applied to learning representations of objects from pixels 124-
26 35
62 70.. The particular challenge of this setting is that invariance of set elements to permutation means there many latent sets that serve equally plausible explanations for the data. Iterative refinement especially useful in this context because breaks the symmetry among these explanations with the randomness of the initial guess 20 rather than encoding the symmetry brraainy- mechanism in the weights the network, as do conventional methods that learn direct feedforward
Project website: https://sites: httsssiitssgggglllee.eeee hww.g//e..omm/teee///miliittt--. stttssssootttttttto http:://wwppppcllaallaaaaattttot
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a)
K, ATTENTION:
legleii gualor will fritardee Nouvare aypretration is trended lalppee
Li cote
Implerr (cens)
SLOTS COMPETE FOR INPUI KEYS
1.5
toitiing
POSIIION
Irrpher
veaalla
1=1
[=
100
1.5
polnt
KIP
Figure 1: Overview. We address efficient training of iterative refinement methods for learning representations of latent sets, such
slot attention model (a), whose illustration adapted from Locatello et al. [9
Vanilla slot attention backpropagates gradients through the unrolled iterative refinement procedure, which leads to training instabilities as shown by the growing Jacobian spectral norm (b). Implicit slot attention uses the first-order Neumann approximation of the implicit gradient, which simply truncates the backpropagation, leading substantially more effective training, as shown in (c).
mapping from observation to representation. The state-of-the-art of these iterative object-centric methods is the slot attention module from Locatello et 39 serving the focus of our paper.
Unfortunately, these methods have been notoriously difficult to train. Their nature as unsupervised representation learning methods means that we do not have ground-trutt targets to supervise the outputs of each iteration of f. as in other settings (e.g. diffusion models). As a result, prior works train these methods by differentiating through the unrolled iterations of Differentiating through this recurrence contributes to various training instabilities, as we can see by the growing spectral norm of in Fig. Db. Such instabilities result sensitivity hyperparameter choices (e.g., number of refinement steps) and have motivated adding optimization tricks such as gradient clipping, learning rate warm-up, and learning rate decay, all of which make such models more complex and harder to use, restrict the model from optimizing its learning objective fully, and only temporarily delay instabilities that still emerge in later stages training.
To approach this problem, we observe that previous iterative refinement methods like slot attention have not taken full advantage of the fact that f can be viewed fixed point operation. Thus, f can be trained with implicit differentiation applied at the fixed point, without backpropagating gradients through the unrolled iterations
g [5]. This paper investigates how advances in implicit differentiation in neural models can be applied to improve the training of iterative refinement methods.
Our primary contribution is to propose implicit differentiation for training iterative refinement procedures, specifically slot attention, for learning representations latent sets. First, we show that slot attention can be cast fixed point procedure that can be trained with implicit differentiation, resulting in what we call implicit slot attention. Second, we show on the latest state-of state-oftthe-ae t-thhertt method of this kind, SLATE (57), that using the first-order Neumann approximation of the implicit gradient for the slot attention module yields substantial improvement in optimization. Third, we show across three datasets that, compared to SLATE, validation loss in training, as well as lower Fréchet inception distance (FID) 2 and mean squared error (MSE) in image reconstruction. Fourth, our method also removes the need for gradient clipping, learning rate warmup, or tuning the number of iterations, while achieving lower space and time complexity the backward pass, all with just one additional line of code. Fifth, when integrated with the original slot attention encoders and decoders from Locatello et al. (39] implicit differentiation substantially improves object property prediction and continues to predict intuitive segmentation masks as the vanilla slot attention.
method for training achieves much lower
2
Related Work
Much early work in artificial intelligence followed a paradigm of using an iterative learning procedure during exccution time, whether it be a form of search, inference, optimization [5]]. These include

---

Hopfield networks [BT], Boltzmann machines and associative memory [6.. The rise of modern deep networks in the last decade shifted the method for computing solutions during execution time away from iterative procedures and towards producing the solution directly with a feedforward pass of network. Recent works have started to shift the paradigm of execution back to combining the best of function approximation and iterative search, with neural networks parameterizing initializations [17], update rules [3] 24 40 search heurstics 34 56). and evaluation functions 14] Our work concerns
early work on variational inference 12 19 and energy- based models 14 37 65 68
the optimization of neural networks as update rules for these iterative refinement procedures.
Our methodological novelty is the adaptation of implicit differentiation techniques for training iterative refinement procedures for representing of latent sets, where our key insight that the iterative procedure used for symmetry- -beakkinggb reaches fixed point, thus allowing implicit differentiation to be used. We are not aware that this has been done before, as current methods that perform iterative refinement for representing latent sets, often referred to as object-centric learning [24- 7O,, all differentiate through the unrolled dynamics of the fixed point procedure, which we show makes them difficult to train.
26 55
57
22
Our work draws upon innovations in implicit differentiation that have been applied in various other applications besides object-centric learning, such as embedded optimization layers 2 4. neural ordinary differential equations
I. meta-learning [ . implicit neural representations 62]
declarative layers (22], and transformers 77]. Although we evaluate various techniques for implicit differentiation, our results highlight the benefits of the simplest of these, which is that of using a
truncated Neumann approximation 20 21 22 55 While this technique was used in Zoran et al. (70]
without theoretical explanation, we propose implicit differentiation as an explanation for why this technique is beneficial. The closest works to ours is the concurrent work of Zhang et al. (9) which applies implicit differentiation to predicting properties of set elements and finds similar benefits. Our approach generalizes their results to the unsupervised setting and scales to high dimensional outputs (e.g., images), taking a crucial step toward improving representation learning of latent sets.
3 Background
Our work builds on prior works on iterative refinement and implicit differentiation with deep networks.
Iterative refinement for inferring representations of latent sets Current work on inferring representations of latent sets is motivated by learning to represent objects which we consider independent and symmetric entities from perceptual input. Because mixture models are also defined to have a priori independent and symmetric mixture components, they have been the model of choice for representing entities: thus these iterative methods model each datapoint x" (e.g. image) as set of independent sensor measurements r.' (e.g. pixels) which are posited as having been generated from a mixture model whose components represent the entities. Under a clustering lens, the problem reduces to finding the K groups of cluster parameters 0 := {@a.A)
} K Tkmi and cluster assignments dnmm := (d 1= that were responsible for the measurements x. of the datapoint zz"
K
A network f breaks symmetry among components by alternately updating 0 and 8=m starting from independent randomly initialized @r.k's. The slot attention module c.g., computes 0 + (0,, ),), where zz is updated as an intermediate step inside f. The 0r, called slots, serve as input to a downstream objective, e.g. image reconstruction, whose gradients are backpropagated through the unrolling of f. Earlier works applied this approach to binary images 24 and videos (0
Implicit differentiation Implicit differentiation is a technique for computing the gradients of the input and output. For example,
function defined in terms of satisfying a joint condition fixed point operation f is defined to satisfy "find 2 such that x = f(2, x)" (or written as 2 = fx)) rather than through an explicit parameterization of f. This fixed point 2, can be computed by simply repeatedly applying f or by using a black-box root- root-finding solver. Letting fw be parameterized by
weights w, with input and fixed point 2., the implicit function theorem [ enables us to directly compute the gradient of the loss l with respect to w, using only the output 2,:
dt 06 Ow
0 (Z., x)
(I J (2x))
(1)
Oz.
Ow
TT
3

---

where J/w (2+) is the Jacobian matrix of fw evaluated at 2,. Compared to backpropagating through the unrolled iteration of f. which just one of many choices of the solver, implicit differentiation via Eq. | removes the cost of storing any intermediate results from the unrolled iteration. Deep equilibrium models (DEQ) 7 represent the class of functions parameterized by neural networks, which have successfully been trained with implicit differentiation and empirically produce stable fixed points even though their convergence properties remains theoretically not well understood.
Much effort has been put into approximating the inverse-Jacobian term J_ (z+)) which has (n)) complexity to compute. Using notation from Bai al. [ Pineda [48 and Almeida
propose to approximate the u' term in Eq. [ as the fixed point of the linear system:
dl
=u=u Jjn (z.) Oz."
which can be solved with any black-box solver. However, in the context of applying implicit
differentiation neural networks, this approach has in practice required some expensive regularization
to maintain stability []. so Fung et al. [20]. Geng et al. (2I). Huang et al. 32). Shaban et al. [55]
propose instead to approximate (I - Jw (zx)) with its Neumann series expansion, yielding
Oe = lim dw T-000 Oz.
de
T
①s (2,,
Σ 20
/v (2.)²
Ow
The first-order approximation (T' = 1) amounts to applying f once to the fixed point 2, and differentiating through the resulting computation graph. This is not only cheap to compute and easy to implement, but has also been shown empirically [2]] to have a regularizing effect on the spectral norm of J.w without sacrificing performance.
4 Implicit Iterative Refinement
Our main contribution is to propose treating iterative refinement algorithms used for inferring latent sets exed-point procedures, thereby motivating the use of implicit differentiation to train them.
4.1
Motivation
Our approach inspired by the structural resemblance between iterative refinement for latent sets and the
algorithm [K] which also has been pointed out by Greff et al. [24] and Locatello et al. 39 If f were to update the components 0" and assignments ".." a way that
monotonically improves the evidence lower bound (ELBO) L the log-likehood of each image x"
with respect to 0" and dhn.m, then we know from Neal and Hinton 4 and Wu (4 that such an approach is fixed point operation whose fixed point locally maximizes L" With this interpretation, such iterative refinement algorithmss for inferring latent sets can be viewed as performing nested optimization with three levels, where the weights of the slot attention module optimized across images æ", the components 0 are optimized per-image x: but across measurements r,mm, and the assignments n. are optimized per-measurement "...
How these iterative refinement methods are implemented differs from optimizing the per-datapoint ELBO described above, however. Indeed, both Greff et al. [44] and Locatello et al. (3 implemented ablations to their methods that do implement a variant of aopectation-Maxinizaan mixture model, but found that replacing the update rule with a recurrent neural network empirically performs better at extracting representations.
on a well-defined
4.2 Iterative refinement as a fixed point procedure
While in general we still lack the theory to understand whether and how the slots of slot attention optimize well-defined objective like the ELBO, and thus have no theoretical guarantees that slot attention does converge to a fixed point, we empirically observe from its stable forward relative residuals (Fig. 3aj that it does appear to approximately converge to a fixed point. This suggests that slot attention can be understood as an instance of a DEQ that uses naive forward iteration to find fixed point and backpropagates through the iteration to compute the gradient. The novel implication of this to inferring latent sets is that any root-finding solver and implicit gradient estimator can in

---

scop1stots,
Ieoraze X, 10N range (5en]) f(x)
arvignennss
project project_ginore..t
K 1s1ot_stte ex (-9.53)
reture
atte F soff wwwpp//ooc.eeeiscco.. bedkbbgd- k. a), diew:1) bid-2ood
attn attn torch.sumtacee epstion, din--2, Keepdisetrue spdate slote assigoents
terpardtirputs slots)
inputs (innpuus).
slots grurupearos, alots)
slots istrrtt 2: seople. K. 9), slots, noe (ttrattenii slots stepeslats ots, detachill v0
sints sluts nnpppoon, (xlnts)) return store
reture
stors
Figure 2: Code. The first order Neumann approximation to the implicit gradient adds only one additional line of Pytorch code [55] to the original forward function of slot attention, but yields substantial improvement of optimization. attn and slots correspond to Φ and 0 in the text respectively.
principle be used to train slot attention, and we find that many combinations do train slot attention effectively in Fig. 3b This insight is one of our main contributions, but in the next section we descrihe particular combination that empirically found effective.
4.3 Implicit slot attention
We propose implicit slot attention: a method for train- ing the state-of-the-ar ahrt slot attention module [Р] with the simplest and most effective method that we have empiri- cally found for approximating the implicit gradient, which its first-order Neumann approximation (Eq. 3). It can be implemented by simply differentiating the computation graph of applying the slot attention update once to the fixed point 0% where 0. is computed by simply iterating the slot attention module forward as usual, but withow the gradient tape, amounting to truncating the backpropagation. The time and space complexity of backpropaga- tion for implicit slot attention compared to vanilla slot attention as a function of the number of slot
Table 1: Complexity
vanilla
implicit
time (forward)
O n
(
space (forward)
O n
0(1)
time (backward)
0(1)
space (backward)
(
(().
attention iterations n, is shown in Table I While other methods for implicit differentiation are more
complex to implement, this method requires only one additional line of code (Fig. 2
5 Experiments
Our main hypothesis that iterative refinement methods for latent sets, specifically slot attention, can be trained as DEQs, and that consequently implicit differentiation would improve their optimization. We test this premise by replacing the backward pass of the slot attention module in the state-of-the-art SLATE 57 with the first-order Neumann approximation of the implicit gradient. This requires only one additional line of code to the original slot attention implementation (Fig 2 and in one instance improves reconstruction mean-squared error by almost 7x.
5.1 Experimental setup
We summarize the SLATE architecture, datasets we considered, and relevant implementation details.
SLATE SLATE uses a discrete VAB 500 to compress an input image into grid of discrete tokens. These tokens index into codebook of latent code-vectors, which, after applying a learned position encoding, serve as the input to the slot attention module. An Image GPT decoder [[O] is trained with a cross-entropy loss to autoregressively reconstruct the latent code-vectors, using the outputted slots from slot attention as queries and the latent code-vectors as keys/values. Gradients are blocked from flowing between the discrete VAE and the rest of the network (i.e. the slot attention module and the Image GPT decoder), but the entire system is trained simultaneously.
Data We consider three datasets: CLevo-Mirror [57]. Shapestacks 22 and COCO-2017 G8
the former two of which were used in the original SLATE paper. We obtained CLEVR-Mires directly from the SLATE authors and used a 70-15-15 split for training, validation, and testing. We pooled all the data variants of Shapestacks together as Singh et al. [57] did and used the original urain-validation-tes, splits. The COCO-2017 dataset was downloaded from FiftyOne and used the original sain-validation-tes splits.
5

---

1.0
Forowedi resthaal
0.8
toain eutropy
109
0.6
0.4-
0.2
0.0
0.00
0.25 0.50 0.75 1,00 1.25 1.50 1.75 2.00
0.00 0.75
0.50 0.75
1.00
1,25
1.50
1.75
2.00
training iterstions
X10*
training ilstaaions
X10
(=,))) (a)
(b) Different solvers
(1,
Figure 3:
Can slot attention by trained as a fixed point operation with implicit differentiation?
3a The
relative residual of the forward iteration of slot attention is close to zero, which motivates its treatment fixed point operation. (5 The forward and backward computations of vanilla slot attention can be swapped out with different solvers, most of which result in the same improved optimization performance. Here we compare with IB, BB, IN, and BN variants of implicit differentiation across 4 seeds. Table 2 defines these acronyms.
Implementation details In all of our experiments we used the same model and training hyper- parameters as those used for the StapeStacks experiment from Singh et al. [57 Table 6]. run on an A100 GPU. The only difference is the image resolution (and consequently number of image tokens) because the largest image size we could train with was 96x96 due to computing constraints, rather than the 128x128 used for their CLEVR experiment. However, as we show for resolutions of both 96x96 and 64x64, the improvement in optimization appears to be hold across image resolutions.
For clean comparison, simply integrated the official SLATE implementatior with the solvers and backward gradient hook from the official implementation deep equilibrium model Besides implementing logging and evaluation code, as well as the first-order Neumann approximation (Eqq. 3 we did not change anything in the above two official implementations otherwise. The SLATE implementation serves the baseline in all experiments.
5.2 Training slot attention with implicit differentiation
The first test to conduct is to see whether slot attention can be trained with implicit differentiation at all. In summarr: Test: Swap the forward pass with different black-box solver, and train with implicil gradients computed via various gradient estimation methods for Eq. / Hypothesis: There exists (forward solver, implicit gradient estimator) that optimizes cross entropy no worse than backpropagating through unrolled inference. Result: Yes, such a pair exists, and in fact it enables significantly better optimization.
To conduct this test, we compare vanilla slot attention with four vari- ants trained with implicit differentia- tion, shown in Table 2 labeled IB, BB, IN, and BN. B stands for the Broyden solver, used by Bai et al. IN
Table 2: Different instantiations of implicit differentiation
abbrv.
forward computation
gradient estimation Broyden
IB
iteration
BB
Broyden
Broyden
iteration
Neumann approximation
77 We test configurations where the Broyden solver was both used find the fixed point of slot attention and used to estimate the implicit gradient with Eq. stands for the first-order Neumann approximation for computing the explicit gradient (Eq. Our hypothesis would be tentatively refuted if the cross-entropy learning curves of none of these variants converges efficiently the baseline. We train these methods model the CLEVR-Mirrors dataset with an image resolution of 64x64. The slot attention baseline unrolls the slot attention cell for seven iterations and we also limit the maximum number of Broyden iterations seven.
BN
Broyden
Neumann approximation
Results and analysis Figure 3 shows that not only is it possible to train slot attention with implicit differentiation, but three of four of the implicit differentiation configurations from Table
http:://gtchab..ccammmggguuuuaap http:://ggicuub.cc///uuuusaaa/

---

COCO-2017
CLEVR-Mirrons
ShapeStacks
Figure 4: Qualitative results. Across three datasets, optimizing SLATE with implicit differentiation leads improved image reconstructions through the slot bottleneck. Black borders indicate the ground truth image. blue border indicate our method, masks. In the LLEVR-Mirrors dataset. whereas vanilla SLATE misses objects, changes their size, or changes their color (indicated by the cireles), implicit SLATE reconstructs the ground truth more faithfully.
borders indicate vanilla SLATE, rest panels visualize attention
optimize significantly better than vanilla slot attention. Whereas vanilla slot attention plateaus at cross entropy of around 130, the BB, IN, and BN variants all achieve 10x lower cross entropy loss. Moreover, BB, IN, and BN all follow the same learning curve, suggesting that the optimization improvement largely due to whether of how we implement the implicit differentiation. The variant is less stable than the others, and we hypothesize that this can be explained by the tendency for the fixed point of Eq. 2 to become increasingly hard to estimate, an issue discussed in Bai et al.
use implicit differentiation not, rather than the choice
5.3 Does this improvement generalize across datasets?
We now test whether this improved optimization holds across different datasets. If the improvement was simply due to implicit differentiation being serendipitously useful for 64x64 images CLEVR- Mirror, then we should expect implicit differentiation to not help for other datasets. We focus attention on IN variant to conduct this test because it minimal modification to the baseline slot attention for gaining the benefits of implicit differentiation, it requires adding only line of code on top of the baseline slot attention implementation and does not require implementing any black- box solvers We henceforth refer to the IN variant implicit slot attention (and correspondingly implicit SLATE), as described in $4.3] In summary: Test: Apply implicit slot attention to CLEVR- Mirror, ShapeStacks, and COCO with 96x96 image resolution. Hypothesis: Implicit slot attention significantly improves optimization across all three datasets. Result: Yes it does.
Results and quantitative analysis Using the primary metrics used in Singh et al. (57], images generated by implicit SLATE achieve both lower pixel-wise mean-squared error and FID score 29
Table 3: Quantitative metrics for image recon- struction through the slot bottleneck.
The FID score was computed with the PyTorch-
Data
Implicit
Vanilla
Ignite [[]] library using the inception network from the PoTch port of the FID official implementation.
CLEVR (FID)
22.19
25,89
CLEVR (MSE)
10.66
67.04
All methods were trained for 250k gradient steps. B compares the FID and MSE scores of the images that result from compressing the SLATE encoder's set of discrete tokens through the slot attention bottleneck, using Image-GPT to autore- gressively re-generate these image tokens one by one, and using the discrete VAE decoder to render the generated image tokens. Implicit differentiation significantly improves the quantitative image
COCO (FID)
127.79
147.48
COCO (MSE)
1659.15
1821.75
ShapeStacks (FID)
34.2
34.76
ShapeStacks (MSE)
108.67
312.14

---

lesrning dersyy
go graclient elipping
No louming souming
107
10P
109
esteopy
vildaaion coee extcrpy
00P
00'
30
0.0
012
0.4
0.0
13.5
1.0
1.2
1.4
0.0
0.2
0.4
1.0
12
14
0.2
0.4
1,0
1.2
seatning therottonn
X100
X10
tratting
XL0
vabiaatin locilan 10030
10
valskai lashiin patm
10'
vaiiaali lacalio pamm
10-
6x
xx
6x
1x 100
X 10P
x 11P
8x
8
x
2x 100
2 1/
2 >1
101 0.01
0.2
104
0.6
0.5
1.0
122
1.4 X10"
10
0.0
0.2
1.0
1.2
14
10a
0.0
0.8
2.0
tealning Die attioes
Berotions
X10
temottoos
X
mmplent (ouns) no dreey
implisit (suns) aoe tlip grad
impliei (sun)) ao sormup
impleett (ouns)
implicit (oum))
implisit (oum)
vanilla dreay vezilla
bauilla grad
warmup
swnills
vanilo
Figure 5: Does implicit differentiation remove the need for various optimization tricks? We ablate three heuristically-motivaae optimization tricks from both vanilla SLATE and our method, and show that for two out of the three, removing the optimization trick quantitatively hurts the vanilla model but not the implicit model. Whereas removing gradient clipping and lcarning rate warmup causes vanilla SLATE's training to become unstable, as indicated by the growth of the Jacobian norm of the slot attention cell, our method trains significantly more stably and can take advantage of the larger gradient steps.
145
(a) No need for gradient clipping.
(b) No sensitivity to hyperparameters. (c) c.f. 1 iteration
Figure 6: (a) Without gradient clipping, our implicit differentiation technique keeps gradients small while backpropagating through the unrolled iterations causes gradients to explode. (b) Training with implicit differen- tiation also is not sensitive the number of iterations with which iterate the slot attention cell. (c) Using iteration for vanilla slot attention trains as stably, but reconstructs more poorly, than implicit slate.
reconstruction metrics of SLATE across the test sets of CLEVR-Mirross Shapestacks, and COCO. In the case of MSE for CLEVR, this is almost 7x improvement.
Qualitative analysis The higher quantitiatve metrics also translate into better quality reconstruc- tions on the test set, as shown in Figure 4 For CLEVR-Mirross, vanilla SLATE sometimes drops or changes the appearance of objects, even in simple scenes with three objects. In contrast, the generations produced from implicit SLATE match the ground truth very closely. For Shapestacks, implicit SLATE consistently segments the scene into constituent objects. This is sometimes the case with vanilla SLATE on the training and validation set as well, but we observed for both of the seeds we ran for the final evaluation that vanilla SLATE produced degenerated attention maps where one slot captures the entire foreground, and the background divided among the other slots. The visual complexity of the COCO dataset is much higher than either CLEVR-Mirron and Shapestacks, and the reconstructions on the COCO dataset are quite poor, for both SLATE's discrete VAE and consequently for the reconstruction through the slot bottleneck. This highlights the gap that still exists between using the state-of-the-a may want these methods to do. The attention masks for both vanilla and implicit SLATE furthermore do not appear to correspond consistently to coherent objects in COCO but rather patches on the image that do not immediately seem to match with our human intuition of what constitutes a visual entity.
object-centric learning out-of-the-box and what the community
8

---

Aneegy Poriica (dfivrst thedladii
Thinkot
Figure 7: We outperform vanilla slot attention on object property prediction (see Locatello et al.
39
Implicit
Vanilla
Figure 8: Implicit differentiation preserves the quality of predicted segmentations from Locatello et al.
5.4 Can we simplify the need for optimization tricks?
To further understand the benefits of implicit differentiation, then ask whether it stabilizes the training of slot attention without the need for optimization tricks like learning rate decay, gradient clipping, and learning warmup. In summary: Test: Remove learning rate decay, gradient clipping, and learning warmup each from vanilla SLATE and our method. Hypothesis: Our method should not require these tricks to optimize stably. Result: No, it generally does not require these tricks.
Fig. shows that the ease and stability of training correlates with the spectral norm of the Jacobian of the slot attention cell. Decaying the learning rate regularizes the Jacobian norm from exploding, but it also hurts optimization performance for both our method and vanilla SLATE, as expected. When we remove gradient clipping the Jacobian norm of vanilla SLATE explodes, do its gradients (Fig. 6a). whereas both stay stable for our method. This translates into a qualitative drop in performance for vanilla SLATE (Fig. 3 but not for implicit SLATE (Fig. 12 Lastly, removing learning rate warmup also consistently makes vanilla SLATE's training unstable, whereas it only affects the stability of our method for one of three seeds. Finally, Fig. Gh shows that implicit slot attention is sensitive the number of iterations with which to iterate the slot attention cell, whereas vanilla slot attention is, with more iterations being harder train.
5.5 Does this mean that iterating slot attention for one iteration is enough?
One might be tempted to interpret Fig. 6b to suggest that fewer iterations of vanilla slot attention is sufficient to improve performance. This not necessarily the case: although vanilla slot attention with one iteration trains more stably than its seven iteration counterpart, it still achieves 2x worse reconstruction MSE than implicit slot attention with seven iterations CLEVR (Fig. (c).
Futhermore, Fig. shows that using only single iteration is not enough to improve optimization of vanilla slot attention for the object property prediction task used by Locatello et al. 39 We directly modified the released code from Locatello al. (39, which used three iterations, to use implicit differentiation. Implicit slot attention can instead scale to many forward iterations needed.

---

5.6 Does implicit slot attention still produce intuitive masks with different architecture?
We sought to check whether implicit differentiation still preserves the quality of the segmentation masks produced by the original slot attention architecture by Locatello et al. [39, which uses spatial broadcast decoder (63] rather than a transformer decoder as SLATE does. It indeed does (Fig. B
suggesting that our findings are not specific to SLATE but apply to slot attention more broadly.
6
Discussion
The connection we made in this paper between slot attention and deep equilibrium models also highlights various other properties about iterative refinement procedures for inferring latent sets that suggest connections to other areas of research that are worth theoretically developing in the future. interpreting slots as parameters that are modified during the inner optimization during execution time may give us novel formulation for how to represent and update fast weight memories. Second is the connection to the literature on 60]: interpreting slots as solutions to an inner optimization problem during
First is the connection the literature on fast weights [ 33 54]:
meta-learning S execution time may give us a novel perspective on perception as itself a learning process. Third is the connection to causality 46 47 interpreting the independently generated and symmetrically processed slots as parameterizing independent causal mechanisms (28] may give us a novel approach for learning to represent causal models within neural scaffolding. Fourth is the connection to dynamical systems :4]: : interpreting slots as a set of attractor basins may offer a novel theory of how the error-correcting properties of discrete representations emerge from continuous ones. These different fields have their own conceptual and implementation tools that could potentially improve understanding of how to build better iterative refinement algorithms and inform how objects could potentially be represented in the mind.
Conclusion We have proposed implicit differentiation for training iterative refinement procedures for inferring representations of latent sets. Our results show clear signal that implicit differentiation can offer significant optimization improvement over backpropagating through the unrolled iteration of slot attention, and potentially any other iterative refinement algorithm, with lower space and time complexity and only one additional line of code. Because it is so simple apply implicit differentiation to any fixed point algorithm, we hope our work inspires future work to leverage tools developed for implicit differentiation for improving learning representations of latent sets and iterative refinement methods more broadly.
Acknowledgements
This work was supported ARL,, 1911NF2110099 and ONR grant number 800014-18-1-2877 Part of this work was completed while MC was an intern Meta AI. Computing support came from Google Cloud Platform and Meta. We would like to thank Shaojie Bai for help on implicit differentiation, Gautam Singh for help on SLATE, Alban Desmaison for help on PyTorch. We would also like to thank Yan Zhang, David Zhang, Thomas Kipf, and Klaus Greff for insightful discussions and Jianwen Xie for pointing out previously missing references.
References
[1] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for Boltzmann machines. Cognitive science, 9():147-169, 1985.
[2] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. arXiv preprint
2019.
[3] Luis B Almeida. learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In Artificial neural networks: concept learning, pages 102-111. 1990.
[4] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[5] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by
10

---

gradient descent. In Advances in Neural Information Processing Systems, pages 3981-3989, 2016.
[6] Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances in Neural Information Processing Systems, 29, 2016.
[7] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. arXiv preprint arXiv:1909.0133 2019.
[8] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Stabilizing equilibrium models by Jacobian regularization. arXiv preprint arXiv:2106. 14342, 2021.
[9] Augustin-Louis Cauchy. Résumé d' un mémoir sur la mécanique céleste et sur un nouveau calcul appelé caleul des limites. Oeuvres Complétes d' d'Augustun Cauchy, 12(48- (148--122:33 12):3, 1831.
[10] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[1]] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint (eti::1806.0777 2018.
[12] Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The Helmholtz machine. Neural computation, 7(5):889-904, 1995.
[13] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
[14] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 1903.086899 2019.
[15) David Duvenaud, J. Zico Kolter, and Matthew Johnson. Deep implicit layers tutorial neural ODEs, deep equilibirum models, and beyond. Neural Information Processing Systems Tutorial, 2020.
[16] Gamaleldin F Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprint arXiv:2206.07764 2022.
[17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-118 1135. PMLR, 2017.
[18] V. Fomin, J. Anmol, S. Desroziers, J. Kriss, and A. Tejani. High-level library to help with training neural networks in pytorch. httpp:://ithi: con/pyytocchhignite 2020.
[19] Karl Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience,
1273)127 2010.
[20] Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Fixed point networks: Implicit depth models with Jacobian-free backprop. arXiv preprint
arXiv:2103.12803, 2021.
[21] Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Advances in Neural Information Processing Systems, 34, 2021.
[22] Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks: A new hope. arXiv preprint arXiv: (00./99999990 9909004666, 2019.
[23] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting baadient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.089:3 2018.
[24] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. arXiv preprint arXiv: 1708. 03498, 2017.
[25] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loie Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433. PMLR, 2019.

---

[26]
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
[27] Oliver Groth, Fabian B Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision-based physical intuition for generalised object stacking. In Proceedings of the European Conference on Computer Vision (ECCV), pages 702-717, 2018.
[28] Siyuan Guo, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár. Causal de Finetti: On the identification of invariant causal structure in exchangeable data. arXiv preprint MrXiv:2203.157565 2022.
[29] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[30] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 (1880-68511, 2020.
[31] John J Hopfield. Neural networks and physical systems with emergent collective computational
abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2558 1982.
[32] Zhichun Huang, Shaojie Bai, and J Zico Kolter. Implicit": Implicit layers for implicit represen- tations. Advances in Neural Information Processing Systems, 34, 2021.
[33] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond lin- car transformers with recurrent fast weight programmers. Advances in Neural Information Processing Systems, 34, 2021.
[34] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 34: 1273- 1286, 2021
[35] Thomas Kipf. Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594 2021.
[36] Teuvo Kohonen. Self-organization and associative memory, volume 8. Springer Science & Business Media, 2012.
[37] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[38] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[39] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiv:2006. 1505,, 2020.
[40] Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In International Conference Machine Learning, pages
PMLR, 2018.
[41] Joseph Marino, Milan Cvitkovic, and Yisong Yue. general method for amortizing variational filtering, arXiv preprint 1rXiv:/881.0530 2018
[42] Joseph Marino, Alexandre Piché, Alessandro Davide Ialongo, and Yisong Yue. Iterative amortized policy optimization. arXiv preprint arXiv:2010. 1000:0000.00670, 2020.
[43] Radford M Neal and Geoffrey Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355-368. Springer, 1998.
[44] Helena E Nusse, James A Yorke, and Eric J Kostelich. Basins of attraction. In Dynamics: Numerical Explorations, pages 269-314. Springer, 1994.
[45] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-800 8037, 2019.
[46] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4))669-688, 1995.
12

---

[47] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- tions and learning algorithms. The MIT Press, 2017.
[48] Fernando Pineda. Generalization of back propagation to recurrent and higher order neural networks. In Neural information processing systems, pages 602-611, 1987.
[49] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. 2019.
[50] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv: 10..:2202..2202 2021.
[51] Stuart Russell. Artificial intelligence modem approach. Pearson Education, Inc., 2010.
[52] Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord.
Step-unrolled denoising autoencoders for text generation. arXiv preprint 1 2021.
[53] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
[54] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139; 1992.
[55] Amiireza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1723-1732. PMLR, 2019.
[56] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[57] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E learns to compose. arXiv preprint arXiv:2110. 1405, 2021.
[58] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathhan and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256 2265. PMLR, 2015.
[59] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, Advances Neural Information Processing Systems, 32, 2019.
[60] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
[61] Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv: 1802. 10353, 2018.
[62] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pages 1456. PMLR, 2020.
[63] Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerehner. Spatial broadcast decoder: simple architecture for learning disentangled representations in VAEs. arXiv preprint / 1901.07017, 2019.
[64] C. F. Jeff Wu. On the Convergence Properties of the EM Algorithm. The Annals of Statistics, 11 (1):95 - 103, 1983. doi: /etl12////////17734660000 URL https:///dttt http::///ooooo/1///// org/10. 1214/aos/ 1176346060
[65] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pages 2635-2644, PMLR, 2016.
[66] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45, 2018.
[67] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto- encoder as amortized sampler. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10441-10451, 2021.
13

---

[68] Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: Cooperative learn- ing of langevin flow and normalizing flow toward energy-bascd model. arXiv preprint arXiv:2205.006 06924,, 2022.
[69] Yan Zhang, David W Zhang, Simon Lacoste-Julien, Gertjan Burghouts, and Cees GM Snoek. Multriset-equivariant set prediction with approximate implicit differentiation. arXiv preprini arXiv:2111.12199, 2021.
[70] Daniel Zoran, Rishabh Kabra, Alexander Lerehner, and Danilo Rezende. Parts: Unsupervised segmentation with slots, attention and independence maximization. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 10439-10447, 2021.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section X.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] See Fig. 2
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "X6KOrtmnkvi", "reviewer": "Reviewer_KLwZ", "summary": "The paper proposes to treat object-centric models with iterative refinement procedures as fixed point operations and optimize them using implicit differentiation. In their experiments, the authors show that the training of SLATE becomes more stable when truncating the gradients after the last refinement step instead of propagating it through the unrolled refinement procedure and show performance increases on three datasets (CLEVR, ShapeStack, COCO).\n", "questions": "Can this methods we applied to other object-centric models and if so, do the claims hold there?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly structured, and the experiments nicely build on each other to support the claims of the paper.  Object-centric models are known to be brittle in training and with regard to hyperparameters. The authors show this problem and propose a solution. They convincingly show that slot attention can be trained using implicit differentiation while retaining/ improving its performance on synthetic datasets. As the authors rightfully state, object-centric model currently do not scale to natural datasets and are therefore of limited use at the moment. Unfortunately, the changed optimization procedure proposed here does not help with scaling to more natural data but fails to learn to identify objects in COCO. \nFurthermore, I don’t see evidence that the optimization procedure helps with getting rid of optimization tricks. While implicit SLATE does show a more well-behaved Jacobian norm with and without learning rate decay, gradient clipping and learning rate warmup, it does not show a qualitatively different behavior when removing these tricks compared to vanilla SLATE. The results of the experiments rather seem to suggest that these tricks did not have a significant effect in the first place and could be in general removed (except maybe lr warmup).\n\nStrength:\n- Clear motivation of the work\n- Well written and clearly structured\n- Experiments on several datasets to support their claims including a natural image dataset\n\n\nWeaknesses:\n- The idea of using implicit differentiation to optimize neural networks is not new and the technical contribution limited.\n- The paper would be stronger if the authors would show that their claims hold for more than one model. \n- The model is still of very limited use since it only works on synthetic datasets. \n- Experiments do not support claims regarding optimization tricks.\n- Segmentation performance should be quantified in order to make a claim about it (Sec. 5.6)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RytjmUjWqmz", "reviewer": "Reviewer_pdiR", "summary": "Summary:\n* The paper combines existing techniques from two groups of works. First, unsupervised object-centric learning with sets representation. Second, implicit differentiation techniques. ", "questions": "* How do you expect the proposed approach to perform with more complex visual scenes / videos? \n* Do you have any intuition of how changing Neumann series expansion affect he model? Will higher order approxmiation benefits the model?", "limitations": "I don't believe there is a limitation section explicitly in the main paper. But there are some in the supplementary.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrength:\n* The paper is very nicely written overall. The logic flow take us through the author's thinking and reasoning clearly. Relevant sources are all cited at the correct places, making understanding the paper an enjoyable journey. The experiments and results are clearly presented with some helpful color-coding to aid the process. I really enjoy reading the paper. \n* Though similar idea has been empirically explored in prior work [44], this paper does a much more fine-grained investigation, analysis and evaluation.\n* Impressive empirical results are achieved by the techniques. The authors do a great job evaluating the performance quantitatively and qualitatively.\n\n\nWeakness:\n* While I do think the paper shows very interesting results, I don't think it's reasonable to motivate them with \"Despite their conceptual elegance, it has been difficult to scale iterative refinement methods **beyond modeling simple static scenes or short video sequences** because differentiating through the unrolled forward iteration makes training unstable.\" The paper do not showcase beyond modeling simple static scenes or short video sequences (which is fine), but promising it in the intro makes it an empty promise. \n* The techniques in implicit differentiation are by themselves not novel. The authors are applying existing techniques to a recurrent problem that has yet to see these techniques being applied. \n* I personally think that mentioning that \"in one instance improves reconstruction mean-squared error by almost 7x\" feels like an oversale.\n* If I'm not mistaken \"BB\" variant is basically applying Bai et al.'s DEQ [6] to the current setting. And from what I can tell from the training curve Fig 3b, they perform similarly. I think it might be interesting to show how previous technique performs quantitatively. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Et2zNog6ocX", "reviewer": "Reviewer_qYDF", "summary": "This paper notes that the a number of state-of-the-art object-centric learning models rely on an iterative refinement step that can be understood as a fixed point algorithm. It then uses implicit differentiation to avoid a costly and unstable unrollment to increase the performance of SLATE, a state-of-the-art object-centric learning method, on a range of metrics and datasets with minimal changes to the code.", "questions": "- On Coco 2017 (Fig. 4), the solutions with iterative refinement seem to settle on a very different minimum: while standard SLATE finds pixel blobs in each slot (but doesn't reach good reconstruction), implicit SLATE seems to find stripes of small blobs. Do you have an explanation for this behaviour?\n- In Figure 5, what dataset are you using here?", "limitations": "I don't see potential negative impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: At its core, the technical innovation itself is limited to transferring existing methods from implicit differentiation to object-centric learning.\n- Quality: The range of experiments is excellent, with carefully chosen modifications from the original code of previous methods to ensure reliable and fair comparisons.\n- Clarity: The manuscript was one of the most enjoyable reads in a while, and I complement the authors for their careful, structured and clear presentation of their methods, the background and the experiments.\n- Significance: While technically simple, the insight yields how performance gains that make it likely that this method will become a standard in the further development of the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8hpdV3PYUq1", "reviewer": "Reviewer_STEd", "summary": "\nThe paper deal with the problem of objections (or object centric) learning by using differentiable iterative refinement to improve the stability and tractability during training. By that, the method seems to improve the stability of a vanilla slot attention architecture such as SLATE. The paper evaluates the proposed extension on the CLEVR-Mirror, Shapestacks, and COCO-2017 datasets.", "questions": "\nI'm not an expert in the field, so I would be one of the readers who would highly profit from a broader introduction of the topic and a bit more context with respect to related works in the application area and comparison to respective benchmarks.\n\nI get and like the idea of implicit differentiation, but I#m wondering if objectness in slot attention is really the only use case here.", "limitations": "The method seems to be very generic. Therefore, I don't see any direct limitations and potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses a not so well explored topic, but nevertheless important and interesting topic that would justify more research.\n\n- The idea of extending the vanilla SLATE by a differentiable component during training seems interesting on a high level, but I'm not an expert, so it would be ideal to consider feedback from someone with hands-on experience in this domain.\n\n- The paper provides an extensive evaluation of the different aspects of the proposed method.\n\nWeaknesses:\n\n- No coverage of related work in object discovery or unsupervised object detection:\nThere seems to be a decent body of work on this topic, that's not covered in the paper (e.g. for some older work: Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videos, Bo Xiong et al. TPAMI 2018). There is also no comparison to any of those methods. \n\n- No comparison to related work:\nIt would be good to see and understand the general state-of-the-art in the field and how the proposed method compares to this on some benchmarks. \n\n- Limited scope?\nOnly framing this paper as a valid SLATE extension would be a bit narrow and might serve only a very small niche. If Slot attention architecture has a wider application that this idea might benefit from, it would be good to mention it and show the impact on respective experiments.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept