Learning What and Where: Disentangling Location and Identity Tracking Without Supervision
Authors: Manuel Traub
Sebastian Otte
Tobias Menge
Matthias Karlbauer
Jannik Thuemmel
Martin V. Butz
Venue: ICLR 2023 poster (2023)
Keywords: object permanence
CATER
unsupervised learning
binding problem

Abstract: Our brain can almost effortlessly decompose visual data streams into background and salient objects. Moreover, it can anticipate object motion and interactions, which are crucial abilities for conceptual planning and reasoning. Recent object reasoning datasets, such as CATER, have revealed fundamental shortcomings of current vision-based AI systems, particularly when targeting explicit object representations, object permanence, and object reasoning. Here we introduce a self-supervised LOCation and Identity tracking system (Loci), which excels on the CATER tracking challenge. Inspired by the dorsal and ventral pathways in the brain, Loci tackles the binding problem by processing separate, slot-wise encodings of 'what' and 'where'. Loci's predictive coding-like processing encourages active error minimization, such that individual slots tend to encode individual objects. Interactions between objects and object dynamics are processed in the disentangled latent space. Truncated backpropagation through time combined with forward eligibility accumulation significantly speeds up learning and improves memory efficiency. Besides exhibiting superior performance in current benchmarks, Loci effectively extracts objects from video streams and separates them into location and Gestalt components. We believe that this separation offers a representation that will facilitate effective planning and reasoning on conceptual levels.

Published as a conference paper at ICLR 2023
LEARNING WHAT AND WHERE:
DISENTANGLING LOCATION AND IDENTITY TRACKING WITHOUT SUPERVISION
Manuel Traub
Sebastian Otte
Tobias Menge
Neuro-Cognitive Modeling
Neuro-Cognitive Modeling University of Tübingen
Neuro-Cognitive Modeling University of Tübingen
University of Tübingen
Matthias Karlbauer
Jannik Thümmel
Martin V. Butz
Neuro-Cognitive Modeling University of Tübingen
ML in Climate Science University of Tübingen
Yeuro-Cognitive Modeling University of Tübingen
ABSTRACT
Our brain can almost effortlessly decompose visual data streams into background and salient objects. Moreover, it can anticipate object motion and interactions, which are crucial abilities for conceptual planning and reasoning. Recent object reasoning datasets, such as CATER, have revealed fundamental shortcomings of current vision-based AI systems, particularly when targeting explicit object representations, object permanence, and object reasoning. Here we introduce a self-supervised LOCation and Identity tracking system (Loci), which excels on the CATER tracking challenge. Inspired by the dorsal and ventral pathways in the brain, Loci tackles the binding problem by processing separate, slot-wise encodings of 'what' and 'where'. Loci's s predictive coding-like processing encourages active error minimization, such that individual slots tend encode individual objects. Interactions between objects and object dynamics are processed in the disentangled latent space. Truncated backpropagation through time combined with forward eligibility accumulation significantly speeds up learning and improves memory efficiency. Besides exhibiting superior performance current benchmarks, Loci effectively extracts objects from video streams and separates them into location and Gestalt components. We believe that this separation offers a representation that will facilitate effective planning and reasoning on conceptual levels.
INTRODUCTION
Human perception characterized by segmenting scenes into individual entities and their interactions
This ability poses non-trivial challenge for computational models of cognition [37 84]
the binding problem [79]. Visual features need to be selectively bound into single objects, segregated from the background, and encoded by means of compressed stable neural attractors
46 73
Recent years have seen revolutionary progress in the ability of connectionist models to operate on complex natural images and videos 50 65). Yet, neural network models still do not fully solve the binding problem [37]. Indeed, recent work on synthetic bdseo-based reasoning datasets, like
CLEVR, CLEVRER, or CATER
43 93 suggests that state-of-the systems 16
89 92 still
struggle to model fundamental physical object properties, such as hollowness, blockage, or object permanence concepts that children learn to master the first few months of their lives 2 63
In comprehensive review on the binding problem in the context of neural networks, Greff et al. [37 define three main challenges for solving the problem: representation, segregation, and composition. Representation refers to the challenge to effectively represent the essential properties of an object, including its appearance and potential dynamics. We will refer to these properties as the 'Gestalt' of
an object 48 86 87 Moreover, the individual objects' locations and motions dynamics should be
Source Code: http://s///.tttlccc/////ttttttte inw.:/eeeeeeeeeeineeeeeee ig/l.oci

---

Published as a conference paper at ICLR 2023
disentangled from their Gestalt to enable compositional recombinations. Meanwhile, the representa- tions should share a common format to enable general purpose reasoning. Segregation describes the challenge extract particular objects from a perceived scene. This extraction should be done context- and dese-dependently to identify the currently relevant entities. As a result, a good segregation should enable effective dynamic predictions of the whole, rather than only the parts. Finally, composition characterizes the challenge to develop object representations that enable meaningful re-combinations of object properties- particularly those that facilitate the prediction of object interaction dynamics. As result, compositional representations enable conceptual reasoning about object properties as well as relations and interactions between objects.
We introduce a novel Location and Identity tracking system. While observing videos, Loci dis- entangles object identities ( ('what') from their spatial properties ('where') in a fully unsupervised, autoregressive manner. It is motivated by our brain's 's ventral and dorsal processing pathways [67] 80) Loci's key contribution lies in how object-specific information disentangled and recombined:
(i) Loci fosters slot-respective object persistence over time via novel combination of slot- specific input channels, temporal slot-interactive predictions via sell-attention (83) followed by GateLORD-RNY [39] and object permanence-oriesn oriented loss regularization.
(ii) Our slot-decoding strategy combines object-specific Gestalt codes with parameterized Gaussians in a, to the best of our knowledge, novel manner. This combination fosters the emergent explication of an object's size, its position, and current occlusions.
(iii) We improve sample and memory efficiency by training Loci's recurrent modules by means of tome-local backpropagation combined with forward propagation of eligibility traces.
As a main result, we observe superior performance on the CATER benchmark: Loci outperforms previous methods by large margin with an order of magnitude fewer parameters. Additional evaluations on moving MNIST, an aquarium video footage, and on the CLEVRER benchmark underline Loci's contribution towards the self-organized disentangled identification and localization of objects as well as the effective processing of object interaction dynamics from vidco data.
RELATED WORK
Previous work by (59] has emphasized that, in general, unsupervised object representation learning is impossible because infinitely many variable models are consistent with the data. Inductive biases are thus necessary to ensure the effective learning of a system that segregates a visual stream of information into effective, compositional representations (37). Accordingly, review related work in the light of the binding problem and their relation to the proposed Loci system.
Representation A powerful choice of an encoding format is the formulation of 'slots', which share the encoding module but keep the codes of individual objects separate from one another. To ensure a common format between the slot-wise encodings, typically, slot-respective encoder modules share their weights [8] 60; 83 To assign individual objects to individual slots, though, the system
needs to break slot symmetry. Recurrent neural networks have been used to disentangle encodings or assignments [IT] 28:
35 60). Other mechanisms explicitly separate spatial slot locations 20 42 [4]. which we also do in Loci. However, instead of treating every spatial location as a potential object, each slot has spotlight, which is designed approximate the object's s center. To further foster a compositional object representation, Loci enforces disentanglement of 'what' from "where' by separating an object's Gestalt its location, size (visual extent), and priority (current visibility with respect to other objects). This stronger disentanglement and more complex 'where" representation is related work that models selective visual attention, realizing partially size-invariant tracking of one particular entity, such as a pedestrian 22 45 72). Advancing this work, Loci tracks multiple objects in parallel, imposes interactive, object-specifie spot-lights, and enables more compressed, object-specific appearance representations due its novel way of combining 'what" and 'where" for decoding.
29
representing shape and surface pattern-from
Segregation Segregating object instances from images is traditionally solved via bounding box detection (38) 74], where more advanced techniques extract additional masks for instance segmen-
tation [55 21 40 Through slot-attention mechanisms, recent unsupervised approaches partition
2

---

Published as a conference paper at ICLR 2023
To segregate objects from images, Burgess et al. [10] and von Kügelgen et al. [85] combine a soft attention (mask) approach with encoder, recurrent attention, and decoder modules, where slots compete for encoding individual objects. Loci pursues a similar approach but segregates objects even further to encourage object- respective 'what' and 'where' representations, where the latter additionally disentangle location, size, and approximate depth. Slot-respective masks compete via softmax attention to actively minimize the prediction error of the visual content. Moreover, pre-trained background model separates potentially interesting from uninteresting regions. While we keep the background modeling rather simple in this work, more advanced techniques may certainly be applied in the near future [26 68 69 (2).
image regions into separate slots to represent distinct objects [6: 61
Composition Compositional reasoning in our model builds on two modules, which process object- to-object interactions and object dynamics. Object-to-object interactions are modelled using Multi- Head Self-Attention (MHSA) [ in close relation to [27 47
Others have employed message passing neural networks to simulate object interactions 66 18 Another promising approach uses Neural Production Systems to dynamically select a learnable rule to model object interactions [23]. Object dynamics are modelled using a recurrent GateLoRDD module [39] GateLORD is designed to apply latent state updates sparsely, which encourages the maintenance of stable object encodings over long periods of time. Previous approaches have also employed recurrent structures to propagate slot- wise encodings forward in time [42] 49 Although some previous works have combined recurrent structures with attention B3 34 recent slot-attention frameworks tend to adopt fully auto-regressive
designs based on transformers without explicit internal state maintenance 27
Tracking models While our primary goal is to separate object location and Gestalt representations, extracted object locations likely facilitate object tracking: a wide area of research on its own [9]]. Statc-of-the-a the-tr-rtttttttt object tracking methods rely on features extracted via attention modules, which are typically applied autoregressively on individual frames 19 64]. Again, these approaches do not explicitly foster separation of "what' from 'where', which may limit their applicability and accuracy in distractive environments, where humans still maintain high tracking skills, as shown in [55]. Loci maintains separate object encodings and thus copes with the presence of distractors more readily. The tracking model proposed in (55) is similar in spirit to Loci, extracting object encodings and propagating them with a competition mechanism and recurrent module. However, to model complex objects their model relies on additional supervision. Disentangling movement and appearance is a common principle video models with a notion of optical flow [57) But optical flow based methods are not designed to deal with occlusions as they only represent parts of the scene that are currently visible, In contrast, Loci is able to maintain the encoding of an object even under occlusion.
The CATER benchmark We evaluate Loci mainly the CATER challenge [32. Previous SOTA methods on this challenge, like Multi-Hopper or OpNet 96). have reused well-established neural network building blocks. Others have attempted to build their system on top of a supervised pre-trained bounding-box-bae reported using a video-level TFCNet (95), which makes efficient use of spatio-temporal convolutions on long videos. In contrast, Loci effectively combines an entity-oriented representational format with several kinds of interactive neural processing modules. While Loci disentangles location from identity when trained in fully self euf--uuper manner, it can be further fine-tuned with supervision to minmmize location- -speifi- prediction error. Due to the employed inductive biases, Loci yields superior performance with a fraction of the number of parameters used in other models.
object detector [15] 97). Most recently, current best results were
3 METHODS
The core idea of Loci involves two major aspects: First, distributing the visual input across recruitable mooulmo, which try to explain the ongoing scene dynamics in a compositional, mutually exclusive manner. A similar principle was shown to be powerful inductive bias for emergent time series decomposition [70). Second, splitting the encoding of a moving entity within a slot-module into its "what' and 'where" components. These components are represented as a latent Gestalt vector ('what'), which can intuitively be understood as a symbolic encoding of an object's 's appearance, including its shape, color, size, texture, and other kinds of visual properties, and latent spatial code ('where') ) which explicates location, size, and relative priority [67 [8). Moreover, we use the Siamese network approach implemented in slot-attention 660, sharing weights across slots. In fact, Loci employs the
3

---

Published as a conference paper at ICLR 2023
same encoding and same decoding network for every slot starting from the raw image. Each encoder slot k generates output Gestalt vectors Gj: and position codes Pq. Entity dynamics and interactions then processed based on these pairs of latent Gestalt and location encodings. We predict the future states of latent encodings in a transition module and allow cross-slot interactions via multi-head self-attention, Figure shows the main components of Loci. Information processing in one iteration unfolds from left to right. A detailed algorithmic description is provided in Appendix B Here, we provide overview and then detail the loss computation and training procedure.
3.1 LOCI OVERVIEW
The slot-wise encoder is based on conventional ResNet architecture. The input consists of eight main features, which encode the image frame I' current iteration t, the prediction error map E a common background mask M as well as, specifically for every slot k, a content mask Mi a position map Qi and maps for both the other masks M and the other positions 0% (cf. Figure [ The computation path of Gestalt and position for an individual slot is tree-shaped, starting with a shared ResNet-trunk after which the path is split into four pathways, which compute Gestalt, location, size, and priority. This encoder design encourages that separately moving entities are encoded in individual ResNet activity dynamics as well as that each slot encodes 'what' (Gestalt) and 'where' (location, size, and priority) of the slot-encoded moving entity separately but conjointly. Thereby, the error map Ex encourages active error minimization, following the principle of predictive coding fostering attractor dynamics [4 73 Appendix B specifies further details.
The transition module then predicts location P and Gestalt dynamics G as well as interactions between slots. The module is loosely based on the architecture of transformer-like encoder [8] where several multi-head attention layers are stacked together with residual feed forward layers in
1-1
Background module
Poot prooecaiing (daso fasion)
Slot module
R!!!
Ri Q1"
Slot modille
Ay
Ag Sigg 4%
Slot-wiee ancoder madale
Slot-wrien transition module
Slot-wise decoder module
stutt--hood Self belf-attuntion
pot
GateLoRD LRD
Priority actention
RowNet
ReeNat
Gestalt code Of
L. Lines
Figure 1: Loci's main processing loop has three components: Individual entities & are processed by slot-wise encoder module to generate the disentangled latent Gestalt code G2 and position code Pk- The transition module consists of L alternating blocks of multi-head self-attention (modeling slot-to- slot interactions) and GateLoRD (predicting within-slot dynamics). The resulting Gestalt and position codes finally combined by the slot-wise decoder into aatity-respectiv RGB reconstructions R and masks Mg. Tensors colored in red (M.. Q% are calculated as complements from Mt; and Q
4

---

Published as a conference paper at ICLR 2023
between. We replace the residual feed forward layers with residual GateLORD [39 layers. GateLORD is a recent gated recurrent neural network module, which is very well-suited to learn and distinctively encode interaction events, which have been described as "structured, describable, memorable units of experience" " [ cf. [94]. In accordance with our entity-focused processing approach, we apply a version, operating on the individual slot. level while receiving information form other slots via the attention layers. At the end of the transition module, the Gestalt code is pushed through a binarization layer, inspired by the principle of vector quantization [T]] This layer enforces an information bottleneck and thus contributes to the development of disentangled entity codes.
Siamese Gatel
The slot-wise decoder recombines the 'what' and 'where' output from the transition module, To do so, first the potential influence of each Gestalt code G₁ is computed over the output range by means of 2d isotropic Gaussian parameterized by the position code Pk. yielding density maps Qr- Next, a wriorit---bbased attention is applied to account for the fact that only one slot-object can be visible at any location (transparent objects are left for future work, see Algorithm-Zjin Appendix [). As result, when two slots cover the same location, the one with the lower priority will have its feature maps set to zero. The rest of the decoder is based on a conventional ResNet, which increases the resolution back to the video size. Like the encoder, the decoder shares weights across all slots. Eventually, the decoder outputs predictions of RGB entity reconstructions RI-i and individual mask predictions M,+1 for each slot k. The masks from all slots and from the background are then combined to construct the final output image prediction R2++1 and competitive mask predictions M To fully reconstruct the image, Loci uses its simple background module, which generates background image estimates Rog and a background mask Mbg. In case of a static background, a Gaussian mixture model over the whole training set used and flat background mask set to the bias value 0bg. For more dynamic backgrounds, we employ a simple auto encoder.
3.2 TRAINING
Loci is trained using a binary cross-entropy loss (LBCE) pixel-wise on the frame prediction applying rectified Adam optimization [RAdam, ef. (56). Several regularizing losses are added to foster object permanence. Additionally, to speed-up learning, we use truncated backpropagation trough time and enhance the gradients with an e-prop-like accumulation of previous neural activities (7].
Empirical evaluations showed that backpropagating the gradients through the networks' inputs creates instabilities in the training process. Thus, detach the gradients for the latent states. As result, the only part of the network that needs to backpropagate gradients trough time are the Gatel layers. Here, we found that using the described combination of e-prop and backpropagation is not only comparable in terms of network accuracy, it also greatly decreases training wall-clock time, as it allows the use of truncated backpropagation with length 1, effectively updating the weights after each input frame (see supplementary material for details).
Another important aspect for successful training is the use of warmup phase, where we mask the target of the network with foreground mask computed with threshold 7:
M = < Mean ((r-
avis ='rbb'
(1)
where Rog is the background model estimate detailed above. black background is used instead of Rog to construct the next frame prediction during this phase. The foreground mask together with the zeroed background enforces the network to focus on reconstructing the foreground objects only. This encourages the usage of the position-constraine encoding and decoding mechanism. After about 30 000 updates- the network has sufficiently learned to use the position encodings-we switch from the masked foreground reconstruction to the full reconstruction.
To encourage object permanence, an additional loss is computed based on Equation 2] which favors slot that keeps encoding the same object, even if the object is temporarily occluded and thus invisible:
Object Permanence Loss
L. = Σ ((u((e) -
- (K(,, )²))²
(2)
C - max (MH) Gi (ME),
(3)
5

---

Published as a conference paper at ICLR 2023
where Ps: and G₁ denote location and Gestalt encoding in slot k. D₂ refers the RGB part of the decoder network, while Gf. denotes the Gestalt code averaged around the last time step in which the entity was visible, and Mr denotes the mask of slot & at time step t. As result, Lo is only applied when the object becomes invisible, which is the case when Mt is approaching zero.
Time Persistence Loss A second mechanism to enforce object permanence and to also regularize the network towards temporal consistent slot encodings, a time persistent regularization loss:
L = 0.1 Σ (Pa(po, t) (popo G+))
where again Da refers to the RGB part of the decoder network and po is the center position in the image. L, essentially penalizes large visual changes in the decoded object between two consecutive time steps.
Position Change Loss In order to encourage the network to predict small slot-position changes over time, a simple L2 regularization loss is based on the position change between two time steps:
Lp 0.01 Σ P 2
Supervision Loss Finally, for the experiments that using supervised target object location signal for fine-tuning, detailed in Eqqation 6) we added gating network & that operates on the latent Gestalt codes G before binarization and predicts single softmax probability, which is used to decide which entity corresponds to the Snitch in the CATER dataset. The location of the selected entity then used in an L2-loss foster regression to the target location provided in the dataset.
2
Lg = Me EEP PA - PPsnttch
The final loss for the network results from adding the individual loss components, denoting the unsupervised and supervised losses, respectively:
Lunsup L LBCE + Lo Lt + Lips
Leup LBCE + Lo + Lt + Lp + Lg.
(8)
EXPERIMENTS & RESULTS
We performed unsupervised training in all experiments. Only for the CATER challenge we addition- ally evaluated a version where we fine-tuned the network via supervision using Equation 6
4.1 CATER-TRACKING CHALLENGE
In the CATER tracking challenge (cf. Figure 2), the task is to locate a unique yellow object, called Snitch, at the end of video sequence. During the video, different geometric objects rotate, move,
contain
30
Figure 2: The CATER Snitch localization challenge: The task is to locate the yellow spherical object called Snitch within the last frame. The challenge is that the Snitch might be contained and moved by a cone. So its location has to be inferred by recognizing and remembering containment event and then tracking the position the container. Image adapted from (2

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative results of the CATER Snitch localization challenge. Referenced results
from [96] and from the other system-respective papers.
Method
Parameters (M)
Top
Top 5
L1 (grid)
Random
2.8
13.8
3.9
Transformer
13.7
39.9
3.53
SINet 62
138.69
21.1
47.1
3.14
TSN (RGB) + LSTM 88
25.6
67.2
2.6
DaSiamRPN 97
33.9
40.8
2.4
1 LSTM
60.2
81.8
Hopper-transformer 99
15.01
1.42
TSM-50 63]
64.0
85.7
0.93
TPN-101 ?2
65.3
83.0
1.09
Hopper-sinet 96
139.22
69.1
91.8
1.02
Inferno 17
88.9
96
6.39
73.2
93.8
0.85
Aloc 24
74.0
94.0
0.44
OPNet 76
74.8
0.54
TFC V3D Depthwise
95
24.64
79.7
95.5
0.47
Loci supervised (ours)
2.96
0.14
Loci unsupervised (ours)
4.14
92.0
0.45
0.44
frame 191
frame 194
frame 287
frame 291
Figure Object permanence shown on a CATER tracking example. While the network (after completely unsupervised training) struggles to keep the shape of the Snitch when contained for longer time span, color and, importantly, the position of the Snitch are preserved during containment. Top row: tracked objects visualized trough colored masks. Bottom row: RGB representation of the Snitch.
and hop scripted random way. By doing so, cone objects can engulf other objects and move them to another location before releasing them again, which can lead to situations where the Snitch remains hidden in the last frame of the video. Therefore, the challenge events, but also to track the specific cone that contains the Snitch. For classification purposes, the 3D Snitch position is partitioned into a 6 grid, resulting in total of 36 classes. In order account for the fact that small location enror could lead to miss-classification, common reported metric is the I. grid distance. We additionally report the continuous T.2 distance with grid length 1, such that the distance is comparable to, but more informative than the Lı grid distance.
only to recognize containment
As described in Girdhar Ramanan [32] we split the dataset with a ratio 70:30 into a training and test set and further put aside 20% of the training set validation set, leaving 56% of the original data for training. To blend-in the supervision loss, we first set the supervision factor μ4 = 0.01 for the first 4 epochs, fostering mostly unsupervised training, and then set μ. 0.1 for the duration of the training process and to μe = 0.3 during the last epochs, to give the Snitch location a weak pull towards the target location.

---

Published as a conference paper at ICLR 2023
Figure 4: Two challenging CATER tracking examples with several co-occurring containments / occlu- sions: Video frames overlaid with the predicted Snitch location (network trained with supervision).
In order to produce labels from the purely unsupervisedly trained network, we train a separate small classifier with around 17k parameters, which only operates on the latent states of the trained Loci network with a correction and gating network. The correction network computes residual for the location of each entity. The gating network computes softmax probabilities, which are used to select the location belonging to the Snitch object similar to Equation 6 To prevent supervised gradient flow, the classifier network module trained once the unsupervised training has finished. Eventually, the location and Gestalt codes are computed for the whole dataset and extracted into a separate data file, from which the classifier module is trained, again using a 70:30 train/test split.
As shown in Table Loci not only surpasses all previous methods by large margin segginn-ahheviing top 1 accuracy of 90.7% and a top 5 accuracy of 98.5% with an L1 distance 0.14 it also surpasses most of the existing work when the main architecture is trained purely unsupervised. Loci learns object permanence merely by means inductive bias and regularization. This is shown in FFiuure 3) where Loci keeps the representation of the contained Snitch in memory. While the shape details blur over time, importantly, the locating is correctly tracked to reconstruct the Snitch once is revealed. Figure 4|also demonstrates the tracking performance in case of several co-occurring occlusions and double contained situations, where two cones are covering the Snitch. Here, the location output of the supervised gating network is marked the video frames.
In additional studies (detailed in Appendix A.3.5), we furthermore investigate Loci's object per- manence and also demonstrate the effective disentanglement position, Gestalt, size, and priority codes, which underlines Loci's ability to separate position and Gestalt of objects from scene without supervision.
4.2 MOVING MNIST
The moving MNIST (MMNIST) challenge is dataset for video prediction [77]. The task to predict the motion of two moving digits, with maximum size of 28 x 28 pixels, moving independently inside 64 x 64 pixel window and bouncing off the walls in a predictable manner. While the dataset usually implemented by simply bouncing the entire 28 28 MNIST sub-image within the 64 64 window, we first crop each digit to its actual appearance in order to obtain a realistic bounding box of the digit and thus generate more realistic bouncing effects (once the actual number touches the border, instead of its superficial bounding box). This removes the undesired bias in the dataset to remember each digit's individual bounding box, individually, which then the only way to correctly predict bounce off a wall. Instead, now the network can predict the bounce based on the pixel information alone.
While the task was originally formulated to predict the next 10 frames after receiving the first 10 frames, we also evaluate the ability to generate temporal closed-loop predictions up to 100 frames after being only trained to predict 10 frames. We compare Loci to the state-of-the-e approach PhyDNet, also designed to disentangle 'what' from 'where'; more specifically: Physical dynamics from unknown factors such the digit's Gestalt code. While using the original code provided by the authors, we did not reach the same reported performance of PhyDNet using our unbiased MMNIST dataloader. Nevertheless, PhyDNet reaches high accuracy for the 10 frame prediction. which is only slightly topped by Loci. For extended temporal predictions, though, PhyDNet quickly dissolves the digits, while Loci preserves the Gestalt of each digit over the 100 predicted frames, as
8

---

Published as a conference paper at ICLR 2023
Table 2: Moving MNIST prediction accuracy PhyDNet Loci. Both networks where trained using the same dataloader to predict the next 90 frames given an input sequence of 10 frames.
T
10
T
30
50
T
100
Method
MSE
SSIM
MSE
SSIM
MSE
SSIM
MSE
SSIM
MSE
SSIM
PhyDnet
38
35.6
0.913
58.6
0.830
80.7
0.739
113.9
0.594
155.1 221.2
0.484
Loci
30.7
0.923
52.5
0.881
98.3
0.814
165.1
0.720
0.639
I=
2=12
I=18
I=20
I=23
t=29
I:33
1:37
1=42
t=
I=78
2
2
1/2
2
.11
22 1
1.3
2
J
9
2
2
2
2
1
f
5
2
2
2 12
Figure 5: Comparison between ground truth top row, PhyDNet center row and Loci bottom for a prediction of up to 90 frames. Both PhyDNet and Loci were trained on 10 frame prediction. While in PhyINett the appearance of the digits dissolves after few frames beyond the initial training distribution, Loci manages to keep the Gestalt code and the location accurate until the fourth collision around frame 42. The Gestalt codes remain stable until the end of the considered 100 time steps.
shown Figure 5 Table 2 shows a qualitative comparison between Loci and PhyDNet: While Loci consistently outperforms PhyDNet for the structural similarity index measure (SSIM, (90), PhyDNet has lower MSE than Loci after frames. This might due the blurring of PhyDNet, which accounts for the uncertainty in the position estimate at the cost of the digits shape.
4.3 OTHER DATASETS & SUPPLEMENTARY MATERIAT.
Apart from a video footage for the main experiments and further algorithmic details on Loci, we provide additional insights and tests in the supplementary material: We evaluate the real world tracking performance of Loci on ten hour aquarium footage found on YouTube Additionally, we examine Gestalt preservation and indicators of intuitive physics in closed loop predictions on the CLEVRER dataset [93]. Furthermore, several ablation studies are provided.
CONCLUSION AND FUTURE WORK
We presented Loci a novel location and identity disentangling artificial neural network, which learns process object-distinct position and Gestalt encodings. Loci significantly exceeds current state-of-the-arr architectures on the CATER challenge, while requiring fewer network parameters and less supervision. Loci thus offers a critical step towards solving the binding problem [37] 79 84). We particularly hope that the mechanisms proposed in this work bear potential to enrich the field of object representation learning and highlight the importance of well-chosen inductive biases.
Currently, Loci operates on static background estimates and cameras, which we denote as its main limitation. We intend to extend the background model incorporate rich and varying backgrounds more flexibly in future work. Potential avenues for this extension may include an explicit, separate encoding of ego-motion and depth (22 Furthermore, image reconstructions may be fused with subsequent image inputs in information-drii manner. We also expect to create even more compact object-interacte conceptions and models in computational neuroscience 22
encodings following theories event-predictive cognition and related 30 78 Moreover, are excited to
[4
explore the potential of Loci to address problems of intuitive physics and causality [66 71 75), seeing that Loci offers suitable compositional structure 51] enable symbolic reasoning. Finally, we hope that Loci will also be well-combinable with reinforcement learning and model-predicee planning in system control setups pursue active, gial-directed environmental interactions.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENT
This work received funding from the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC number 2064/1 Project number 390727645 as well as from the Cyber Valley in Tübingen, CyVy-RF-2020-15.. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Manuel Traub and Matthias Karlbauer, and the Alexander von Humboldt Foundation for supporting Martin Butz and Sebastian Otte. We also thank Simon Frank for the Gestalt Code analysis in Figure 16
REFERENCES
[1] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352-1361. PMLR, 2021.
[2] Renée Baillargeon, Elizabeth Spelke, and Stanley Wasserman. Object permanence in five- month-old infants. Cognition, 20(3):191-200 1985. doi: 10.10160011000077655)000666
[3] Dare Ann Baldwin and Jessica E. Kosie. How does the mind render streaming experience as events? Topics in Cognitive Science, 13:79-105, 2021. doi: 10.111 1/tops. 12502.
[4] Lawrence W. Barsalou. Perceptual symbol systems. Behavioral and Brain Sciences, 22: 577-600, 1999.
[5] Lawrence W Barsalou. Abstraction in perceptual symbol systems. Philos Trans R Soc Lond B Biol Sci, 358(1435): 177-1187, 177-141111777111877 Jul 2003.
[6] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[7] Guillaume Bellec, Franz Scherr, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. Biologically inspired alternatives backpropagation through time for learning in recurrent neural nets. arXiv preprint arXiv:1901 .09049, 2019.
[8] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. European conference on computer vision, pp. 850-865. Springer, 2016.
[9] Gary Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
[10] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerehner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: / 13900 2019.
[11] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposi- tion and representation. CoRR, abs/1901. 2019.
[12] Martin V. Butz. Towards a unified sub-symbolic computational theory of cognition. Frontiers in Psychology, 7(925), 2016. ISSN 1664- 1078. doi: 103389/fpsyy.2016600955..
[13] Martin V. Butz and Esther F. Kutter. How the Mind Comes Into Being: Introducing Cognitive Science from a Functional and Computational Perspective. Oxford University Press, Oxford, UK, 2017.
[14] Martin V. Butz, Asya Achimova, David Bilkey, and Alistair Knott. Event-predictive cognition: root for conceptual human thought. Topics in Cognitive Science, 13:10-24, 2021. doi: 10.1111/tops.11 12522.
[15] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229, Springer, 2020.
10

---

Published as a conference paper at ICLR 2023
[16] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
[17] Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Inferno: Inferring object-centric 3d scene representations without supervision. ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[18] Michael Chang, Tomer Ullman, Antonio Torralba, and Joshua Tenenbaum. compositional object-based approach to learning physical dynamies. arXiv preprint arXfv:1612.00341, 2016.
[19] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. CoRR, abs/2103.
2021.
[20] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the Thirty-Third AAAI Conference on Ar- rificial Intelligence and Thirty-Firss Innovative Applications of Artificial Intelligence Con- ference and Ninth AAAI Symposium on Educational Advances Artificial Intelligence, (AAT19ЛAATT19AAM v33101.33013412.
AAAI Press, 2019. ISBN 978-1-57735-800 doi: 10. 10.1609/aai..
[21] Jifeng Dai, Kaiming He, and Jian Sun. Instance- semantic segmentation via multi-task network cascades. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3150-3158, 2016.
[22] Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural Computation, 24(8):2151-2184, 2012. ISSN 08899-7667. doi: 10.1 D0.166//EECO_. a_00312.
[23] Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 14:25673-25663 2021.
[24] David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Object-based attention for spatio- temporal reasoning: Outperforming neubo-symbolic models with flexible distributed architec- tures. arXiv e-prints, pp. arXiv-2012. 2020.
[25] John Duncan. Selective attention and the organization of visual information. Journal of Experimental Psychology: General, 113:501-517, 1984. doi: 000077009963444111..0
[26] Sébastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engeleke, Ingmar Posner, Niloy Mitra, and Andrea Vedaldi. Relate: Physically plausible multi-object scene synthesis using structured latent spaces. Advances in Neural Information Processing Systems, 33:11202-11215, 2020.
[27] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world vidcos, 2022.
(28) Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: generative scene inference and sampling with object-centric latent representations. CoRR, abs/1907.13052. 2019.
[29] S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. CoRR, abs/1 103.066030855 2016.
[30] Nicholas T. Franklin, Kenneth A. Norman, Charan Ranganath, Jeffrey M. Zacks, and Samuel J. Gershman. Structured event memory: A neuro-symbolic model of event cognition. Psycho- logical Review, 127(3):327-361. 2020. ISSN 1939-1477 (oelletrommie.00332255XXXX doi: 10.10377/e00000177.
11

---

Published as a conference paper at ICLR 2023
[31] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convo- lutional neural networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414-2423, 2016.
[32] Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for compositional actions and temporal reasoning. CoRR, abs/1910.04744, 2019.
[33] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909.
2019.
[34] Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006. 2en::2006..2225, 2020.
[35] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zo- ran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. CoRR, abs/1903.00050, 2019.
[36] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loie Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2423-2433. PMLR, 2019.
[37] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05206 2020.
[38] Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11474-11484, 2020.
[39] Christian Gumbsch, Martin V Butz, and Georg Martius. Sparsely changing latent states for prediction and planning in partially observable domains. Advances in Neural Information Processing Systems, 34, 2021.
[40] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
[41] Michael Janner, Sergey Levine, William T Freeman, Joshua Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint arXiv: 1812. 10972, 2018
[42] Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. CoRR, abs/2010.12152. 2020.
[43] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2911 2017.
[44] Michael I. Jordan. Attractor dynamics and parallelism a connectionist sequential machine. In Proceedings of the Eighth Annual Meeting of the Cognitive Science Society, pp. 531-546. Erlbaum, Hillsdale, NJ, 1986.
[45] Samira Ebrahimi Kahou, Vincent Michalski, Roland Memisevic, Christopher Pal, and Pascal Vincent. Ratm: Recurrent attentive tracking model. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1613-1622, 2017. doi: 10.1 109/ CVPRW.2017.206
[46] Franziska Kaltenberger, Sebastian Otte, and Martin V. Butz. Binding dancers into attractors. In
12th IEEE International Conference Development and Learning (ICDI.), 2022.
12

---

Published as a conference paper at ICLR 2023
[47] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. CoRR, abs/2111.12599 2021.
[48] Kurt Koffka. Principles of Gestalt psychology. Routledge, Abingdon, UK, 2013, first published in 1935.
[49] Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. CoRR, abs/1806.01794, 2018.
[50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., Red Hook, NY, 2012.
[51] Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 2017. doi: 10.1017/ $0140525X1600 6001837.
[52] Jianfeng Li, Junqiao Zhao, Shuangfu Song, and Tiantian Feng. Unsupervised joint learning of depth, optical flow, ego-motion from video. arXiv, 2021. doi: 10.48550/ARXIV21005.5 14520.
[53] Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7083-7093, 2019.
[54] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. CoRR, abs/2001 .02407,7 2020.
[55] Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi Narasimhan Govindarajan, Ennio Mingolla, and Thomas Serre. Tracking without re-recognition in humans and machines. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19473-19486. Curran Associates, Inc., 2021.
(56) Liyun Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint 1908::9900.02660 2019.
[57] Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 13137-13152. Curran Associates, Inc., 2021.
Shang-Yang [58] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 11--3..prrnngee 2016.
[59] Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. CORR, abs/1811 12359, 2018
[60] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 1525-11538, 2020.
[61] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 1525-112111111 11538, 2020.
[62] Chih-Yar Ma. Asim Kaday, lain Melvin, Zsolt Kira, Ghassan AlRegib, and Hans Peter Graf. Attend and interact: Higher-order object interactions for video understanding. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 6790-6800, 2018
13

---

Published as a conference paper at ICLR 2023
[63] Jean M. Mandler. Thought before language. Trends in Cognitive Sciences, 8(11):508 513, 200..00.000
2004. ISSN 1364-6613. doi:
[64] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. CoRR, abs/2101.02701 2021.
[65] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8844-8854, June 2022.
[66] Albert Michotte. The Perception of Causality. Louvain, Ed. de I'Institut Supérieur de Philoso- phie, 1946. translated into English by C. A. Mace, 1962.
[67] Mortimer Mishkin, Leslie G. Ungerleider, and Kathleen A. Macko. Object vision and spatial vision: Two cortical pathways. Trends in Neurosciences, 6:414 417, 1983, ISSN 0166-2236. doi: (.100660016--2236663))))000
[68] Thu Nguyen- Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Block- gan: Learning 3d object-aware scene representations from unlabelled images. Advances in Neural Information Processing Systems, 33:6767- 2020.
[69] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11453-11464, 2021.
[70] Sebastian Otte, Patricia Rubisch, and Martin V. Butz. Gradient-based learning of compositional dynamics with modular rnns. In Artificial Neural Networks and Machine Learning - ICANN 2019, number 11727 in Lecture Notes in Computer Science, pp. 484 496. Springer International Publishing, September 2019. ISBN 978-3-030-30487-4,
[71] Judea Pearl. The Book of Why The new science of cause and effect. Basic Books, New York, NY, 2018
[72] Marc Aurelio Ranzato. On learning where to look. arXiv, abs/1405.54833 2014.
[73] Rajesh P. Rao and Dana H. Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, January 1999. doi: 10.
((1:))99 2(1):79-87,
[74] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 779-788, 2016.
[75] Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbren- ner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. CORR, abs/2102.1 1107, 2021.
[76] Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. Learning object permanence from video. In European Conference on Computer Vision, pp. 35-50. Springer, 2020.
[77] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of vidco representations using Istms. In International conference on machine learning, pp. 843-852. PMLR, 2015.
[78] David Stawarezyk, Matthew A. Bezdek, and Jeffrey M. Zacks Event representations and predictive processing: The role of the midline default network core. Topics in Cognitive Science, 13:164-186, 2021.
doi: 10 1/tops. 12450.
[79] Anne Treisman. The binding problem. Current Opinion in Neurobiology, 6(2):171-178, 1996. ISSN 0959-4388. doi: 040016680005-438889888807700
[80] Leslie G. Ungerleider and James V. Haxby. "What" and "where" in the human brain. Current Opinion in Neurobiology, 4(2):157-65, 1994.
14

---

Published as a conference paper at ICLR 2023
[81] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[82] Sjoerd van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-322 2020.
[83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[84] Christoph von der Malsburg. Binding in models of perception and brain function. Curr Opin Neurobiol, 5()): 5260 Aug 1995.
[85] Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004. /2906 2020.
[86] Johan Wagemans, James H. Elder, Michael Kubovy, Stephen E. Palmer, Mary A. Peterson, Manish Singh, and Rüdiger von der Heydt. A century of gestalt psychology in visual perception: I. perceptual grouping and figure-ground organization. Psychological bulletin, 138 6:1172-217, 2012.
[87] Johan Wagemans, Jacob Feldman, Sergei Gepshtein, Ruth Kimchi, James R. Pomerantz, Peter A van der Helm, and Cees van Leeuwen. A century of gestalt psychology in visual perception: li. conceptual and theoretical foundations. Psychological bulletin, 138 6:1218-52 2012.
[88] Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer. 2016.
[89] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
[90] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4): 600-612, 2004.
[91] Marissa A Weis, Kashyap Chitta, Yash Sharma, Wicland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Benchmarking unsupervised object representations for video sequences. J. Mach. Learn. Res., 22: 12:183-1, 2021.
[92] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 591-600, 2020.
[93] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.0144 2019.
[94] Jeffrey M. Zacks, Nicole K. Speer, Khena M. Swallow, Todd S. Braver, and Jeremy R. Reynolds. Event perception: mind-brain perspective. Psychological Bulletin, 133(2):273-2) 2007.
[95] Shiwen Zhang. Tfenet: Temporal fully connected networks for static unbiased temporal reasoning, arXiv preprint arXiv:2203.05922 2022.
[96] Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizit, Martin Renqiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning, arXiv preprint arXiv:2103. 10574. 2021.
[97] Zheng Zhu, Qiang Wang. Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. Proceedings of the European conference on computer vision (ECCV), pp. 101-117, 2018.
15

---

Published as a conference paper at ICLR 2023
A APPENDIX
As additional content, first provide details further evaluations conducted with Loci. In order to evaluate the tracking performance of Loci in a real world example, we trained it on 10 hour aquarium footage found on YouTube. The task poses the additional challenge to cope with backgrounds that are not fully stationary. The results are shown in Figure and demonstrate that Loci is able track 15+ objects in complex real world environment.
The Gestalt preserving performance Loci for closed loop predictions are also demonstrated exemplary on the CLEVRER dataset in [niiuuuee 7| Here, the effects of collisions of different geometric objects are predicted into the future. While the location deviates visibly over time, Loci able to preserve the Gestalt code also for more complex objects closed loop setup, which considered specifically challenging for RNNs.
A.1 CATER EVALUATION DETAILS
In order produce the results from Table we trained five networks with different initial seeds and and then evaluated each network five times with different initial seeds. In [Tale I we reported the mean results over the five evaluation runs from our best performing network (Network from Table 3).
A.2 BACKGOUND MODEL.
In order to compute an background model for datasets with simple static background like CATER, CLEVR or CLEVRER we use a simple Gaussian Mixture Model. The specific function used is PrautBacctrrundSduraacoooMCC from OpenCV [9] which we with learningrate of 0.00001 to compute an background image based on the training set.
For more complex backgrounds like in the aquarium example where the camera is still static, use an autoencoder ResNet that encodes the input image into a latent vector with the same size as the Gestalt code. This latent background code is then through residual Gatel LOrd in order capture temporal dynamics, like the chancing water level, and then run through an ResNet decoder. The background autoencoder is pretrained using LI reconstruction loss in order to focus on the most dominant features of the background and to not care so much about foreground objects.
Figure Fully unsupervised real world tracking example trained on hour aquarium footage.
16

---

Published as a conference paper at ICLR 2023
t 10
t 20
t= 30
t= 40
50
t 60
t 70
:90
.99
.99
.90
9
:
Figure 7: Three examples predicting object dynamics from the CLEVRER 93 benchmarks. Loci runs 70 time-steps in closed loop after 50 times-steps of teacher forcing. Top show ground truth wile bottom rows show Loci's closed loop predictions.
A.3 ABLATIONS
Using the binary cross-entropy with non binary targets, while producing valid gradients, gives little insights into the network's actual performance. In order to better compare different designs and to also take the objects into focus, we use a modified Lo-loss for our ablation studies that is computed based on Equuttio 9.
Σ (10
L2object
I++1 bbcchhwo
Σ c=1
[1t+1 B(+1)²
(9)
Here the MSE is masked with the error between input and the background. As result, we get much higher error when the network produces a tracking error compared to a background reconstruction error. The T.2object loss takes into account instances, where, for example, the network overlooks object makes mistake in the prediction of the movement of object. It thus offers itself as a good metric for comparing the performance of different architectures during training.
A.3.1 GATELORD vs LSTM
As shown in Figure using GateLORD within the predictor network significantly increases the L2 -Loss. Thus, it appears that GateLORD's piece-wise constant latent state regularization mechanism indeed suitably biases the network towards assuming object permanence.
A.3.2 INPUT CHANNELS
In Figure 10 we show the networks performance in terms of L2 when we zero out different input channels.
17

---

Published as a conference paper at ICLR 2023
Table 3: Evaluations the CATER Snitch challenge for supervised training. Table shows five networks trained with different initial seeds, each evaluated five times with different initial seeds.
Network
Top
Top 5
L1 (grid)
Lu
91.7
99.0
0.118
0.122
90.8
98.8
0.129
0.126
98.8
0,119
0.121
91.4
98.9
0.122
0,123
91.3
98.8
0,123
0,126
2
91.1
98.6
0.129
0.133
90.4
98.3
0.144
0.141
89.8
98.3
0,151
0,144
90.0
98.5
0.149
0.140
89.5
98.2
0.164
0.151
3
89.6
97.8
0.173
0.170
90.1
98.0
0.159
0.157
89.5
97.9
0.174
0.164
88.9
97.5
0.174
0.167
89.8
98.1
0.162
0.157
4
91.5
99.1
0.112
0.118
90.4
98.9
0.133
0.128
91.0
98.7
0.525
0,129
90.5
0,525
0.124
90.8
98.8
0.127
0.126
5
91.7
98.2
0.130
0.139
91.4
98.4
0.135
0.139
98.0
0,146
0.150
98.3
0.130
0.140
91.3
98.1
0.150
0.151
Table 4: Evaluations the CATER Snitch challenge for supervised training, statistics of the 25 evaluations presented in Table 3
Metric
min
std
Top1
88.9
90.7
0.8
91.8
Top5
97.5
98.5
0.4
99.1
Li (grid)
0.112
0,140
0.019
0.174
L2
0.118
0.139
0,015
0.170
A.3.3 REGULARIZATION LOSSES
In Figure II
show the networks performance in terms of L2object without certain regularization
losses.
A.3.4 E-PROP VS
THRROUHH TIME
In Figure we compared the L2object loss for truncated back- bockkpropagation through time with different sequence lengths against a version where we used back-propagatton together with e-prop. In all experiments the networks are trained on the full Cater sequence with 300 frames, which are fed into the network sequentially. We then updated and afterwards detached the gradients using truncated pook-propagttion through time with different time intervals. Using e-prop informed gradients not only drastically sped up training time, but it was also more sample-efficient while achieving the same performance as truncated back- proepprrpagationn through time with an interval three time steps. Especially interesting is the experiment where tested truncated back- proc-propagatton with sequence length one, without using e-prop's 's informed gradients. this case, the performance is significantly worse than when using e-prop's informed gradients.
18

---

Published as a conference paper at ICLR 2023
GateLOrd LSTM
10
L/2object
10-6
10a
10a
10
Updates
Figure 8: Comparison between GateLORD and LSTM modules within the predictor part of network.
backprop sime step backprop 3 time step
backprop stepe
backprop 30 time steps
10
e-prop backprop time
122bject
103
104
10
106
Seconde
Figure 9: Comparison between different sequence lengths for truncated back-propagation trough time vs pockppropagation and e-prop.
A.3.5 LESION STUDIES
In order to quantify the effect of the Gestalt and position codes, we conducted ablation by adding normally distributed noise with standard deviations E [0, 0..,...... 1.0] to the Gestalt and position codes provided by the encoder-right before they were forwarded the transition module and calculated the resulting accuracies and errors, which are reported in Figure 12 and Figure 13 for the CATER and moving MNIST benchmarks, respectively. Furthermore, exemplary images of according position, priority, size, and Gestalt code manipulations are presented in Figiure 14 and 5) again for the CATER and moving MNIST benchmarks. Effectively, manipulating the according codes result in changes in position, priority, size, or Gestalt code, demonstrating the disentangled encoding of object features in the respective latent codes. particular, while the Gestalt-code manipulation results in changes of the actual numbers in the moving MNIST experiment under conservation of positions (first vs. third row of Figure 5), the position-code manipulation alters the position of the numbers in space without modifying the object shape (first vs. fourth row of Figure 5).
19

---

Published as a conference paper at ICLR 2023
No Ma
No Rs
MC and Qi
10-4
No Qk paper-baseline
12obfect
10
10a
10
10c
Updates
Figure 10: Importance of different input channels.
No Lo No Lp
No lat paper-baseline
10-4
12object
10
10a
10a
10c
Updates
Figure 11: Importance of regularization losses.
20

---

Published as a conference paper at ICLR 2023
Average distance
Average 12 distance
Average object loss
Average top accuracy
Average top 5 accuracy
3.0-
0.8
3.0
0.0
0.8
Gesta
54
DA
0.2
02
02
Position
1200
Figure 12: Gestalt-positinn lesion study on the CATER benchmark. Top: Metrics resulting from different noise intensities added the Gestalt and position codes. Bottom: Explicit visualization of the bottom-most row (manipulating position only, blue), left-most column (manipulating Gestalt only, orange), and diagonal from bottom left to top right in the plots the top (manipulating both position and Gestalt, purple).
SSIM
0.6
0.6
28
0.2
0.2
12
64
10
0.2
caa
1b
Noba
Fostion
Postion
Figure 13: Same analysis as Figure 12
applied to the moving MMNIST dataset.
ground truth
position + noise
priority noise
size noise
Gestalt noise
Figure 14: To study the disentanglement of the different parts of our learned codes, we perform lesion study. The principle of the study follows: For each component code (position, priority, size, and Gestalt) we add gaussian noise to the code before applying the decoder while keeping all other codes as predicted by the model.
4
9
4
9
4
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
9
4
4
9
O
8
0
9
6
8
6
4
6
9
4
9
4
4
7
4
4
2
94
94
4
4
Figure 15: Lesion study on the moving mnist closed loop predictions (frames t 11 to 20). First row: undisturbed prediction. Second row: 0.25 scalled gaussian noise added to the position (size) code, which is then cliped to [0, 1]. Third row: gaussian noise added to the Gestalt code witch then rounded and clipped to [0, 1]. Fourth row 0.25 scalled gaussian noise added to the position (x.y) code which is then clipped to -0.99 0.9
21

---

Published as a conference paper at ICLR 2023
Figure 16: Traversing the Gestalt code manifold: Using Gaussian Mixture together with t-SNE find distinct clusters in the latent Gestalt code. Traversing the main axis variance given by a PCA we found color axis (top rob) and a shape axis (middle row). Since these axes are disentangled they can be used in an additive manner to simultaneously change the color and shape (bottom row)
100
1.00
64
64
2.01
221
pestalard
Figure 17: Comparison between unsupervised (left) and supervised (right) tracking, while the Snitch is contained progressively longer. The x-axis depicts the number of frames the Snitch has been contained. The left y-axis shows the L2 distance to the target position, while the right y- axis shows the number of times Snitch area). The thick line represents the median Lg distance, while the thin line represents the mean L2 distance. The weaker colored area shows the 90- 90-10-quantile, while the stronger corresponds the 75-25--ua
contained that long within the test-set (green shaded
A.4 GESTALT CODE INVESTIGATION
Here we investigate the latent landscape of the Gestalt codes learned by Loci when trained on the CATER challenge.
this, we first selected a fixed amount (450) Gestalt codes created by Loci using the CATER dataset. We then clustered these Gestalt codes using Gaussian Mixture model and then performed dimensionality reduction using t-SNE. In some of the found clusters mainly the color of the objects different, while in others the color and the shape varied.
The images from Figure 16 where created by using a Principal component analysis (PCA) to calculate the main axes of variance in different clusters. A random Gestalt code was chosen from clusters and was modified by subtracting or adding a portion of the axes or in other words, the Gestalt code used starting point and then new Gestalt codes were sampled walking down and up the selected axes. These new Gestalt codes are not binary anymore but were still clipped at and 1, to create meaningful inputs the decoder. We were able to identify of the objects, while the shape stays the same, and one axis, which only varies the shape, while the color stays the same. Since these axes disentangled they can be used in additive manner to simultaneously change the color and shape of objects.
axis, that only varies the color
A.5 OBJECT PERMANENCE EVALUATION
In order to analysed Loci's object permanence abilities we evaluated how the number of time steps the Snitch was hidden affects Loci's 's ability to correctly locate As shown in Figure 17] where the tracking accuracy is plotted for progressively extended periods of Snitch containment. For both the supervised and unsupervised tracking, the location enror remains constant after an initial increase, over extended time spans.
22

---

Published as a conference paper at ICLR 2023
A.6 E-PROP FOR GATELORD
GoteLORD is defined in Equation 10 10 Equation 15 following Gumbsch et al. [39 with the gating network g. the candidate network r, and the cell state c. Next, detail how applied e-prop in G.teLORD.
A.6.1 GATELORD MODEL
03 = A
gren
Σ
oi 11.+ + +N(0.
(gating network state)
(10)
Σ
Opeir
Σ
(candidate network state)
(11)
of of
+ (1 0/)c0 1-1 + :1o
(consequent cell state)
(12)
pj = 0
Σ
@ree.p
Js
& + Σ gima xt +b,
(output factor p)
(13)
0f = a
Σ
gren.o
J.1
c + Σ gi x + 69
(output factor o)
(14)
h1 = o/p
(resulting prediction at time t)
(15)
A.6.2 E-PROP UPDATES:
core derivative and eligiblity determination:
N' 0A(x)
[0
if S 0 otherwise
(pseudo-derivative of gate)
(16)
Or ((x))))²)
dct 1-1
dct
(1 --8)) 85) + dct
1/,
+
(cell-specific eligibility eligibbiityy value)
(17)
de²
80,,
003,:
gating network forward eligibility propagation:
OE
0E dcz j/ + xx7((0))
Og
(actual update signal)
(18)
00
80
r.c.p.4
(1 df) + +c (r) - -
(recurrent weights-specific elig.)
(19)
j (1 1f) + (ra -c
in.g.e g.t
(input weights-specific elig.)
(20)
b.g.t 3.
(1 - 077 + +A'
- 1)-1)
(bias weight-specific elig.)
(21)
candidate network forward eligibility propagation:
0E
0E Or Sj.
(actual update signal)
(22)
00,,
5
rec.r, 0 1577 + +(1 (4)))//11 '9
(recurrent erights-specift elig.)
(23)
in,r.
- 0/) + +(1 (r:) )o
(input weights- elig.)
(24)
b.t.t
(1- 2f77 + (1 (1)) )
(bias weight-specific elig.) (25)
Note that the partial derivatives 8E in Equation 18 and Equation 22 address the current time dg 50 step only. Also note that dct and backprogated through the feedforward connections of the architecture. In contrast, original e-prop uses local approximations of the learning signal only, which does not allow to stack multiple layers without losing the exact local gradient.
är carry true gradient information of the current time step,
23

---

Published as a conference paper at ICLR 2023
Table 5: Evaluations of Loci's foreground segmentation masks. Trained networks from the Cater challenge are evaluated on CLEVR with mask by rumning the single CLEVR images 30 iterations through the network and then comparing the masks using the Intersecion over Union (IoU) metric. We compare mean per mask accuracy (mask avg) and size weighted average that represents how many pixels where segmented correctly (pixel avg).
Network
mask avg (%)
pixel avg (%)
1
83.3
96,8
2
84.6
97.0
3
84.3
97.1
4
84.1
96.7
5
85.6
97.2
B LOCI ALGORITHM
Loci processes sequence of RGB video-encoding images I RTxHxWx3, Processing is mostly done slot-wise, whereby the system initialized with variable number of K processing slots. Its main components consist of slot-wise encoder, transition module, and slot-wise decoder. Moreover, a background processing module is implemented. The slot-wise encoder implemented by tree-structured ResNet-bos based processing encoder (see Figure 18). The transition module processes slot-wise temporal dynamics and between-slot interaction dynamics (see Figure 19). The slot-wise decoder is again implemented by ResNet (see Figure 20p. For simple backgrounds,
Gaussian Mixture Model to obtain default background estimate Räg. which is used for the whole training set. For more complex backgrounds we use an additional Auto-Encoder Module.
In the remainder, denote scalar values by lower-case letters, tensors by upper-case letters, and vectors by bold letters. Moreover, we denote slot-specific activities with a subscript & E 1,.., K and time by the superscript We drop & for temporary values.
We now first define data and neural encoding sizes and types used throughout Loci's processing pipeline. We then specify neural activity initialization. Finally, we detail the unfolding overall processing loop.
B.1
SLOT WISE ENCODER
Inputs
The encoder inputs at each time step t consist of:
RGB input image I' Hxxx3
MSE map E Hxxxx (pixel-wise mean squared error between / and At),
Slot-specific RGB image reconstructions R "xxx33
Slot-specific mask predictions Mr #xx11
Slot-specific mask complements Mr.* Σ Eve (I... Mr.
Slot-specific isotropic Gaussian position map predictions Q% "xW
Slot-specific Gaussian position map complements Q E Qi
Background mask Mi, E R HxW, which is equivalent to 1 - Exe(,...)) M
Outputs Based on these inputs, the slot-wise encoder network generates latent codes, which are forwarded to the transition module:
Slot-specific Gestalt codes G E DR.
Slot-specific position codes Pf R encode isotropic Gaussian (Mi, of) and slot- priority code ={{
where Dy denotes the size of the Gestalt code and uh R22
24

---

Published as a conference paper at ICLR 2023
B.2 TRANSITION MODULE
transition module is used process interaction dynamics within and between these slot-respective codes and creates prediction for the next state, which is fed into the decoder. The input to the transition module equals G³, pt It is processed across slots and per slot in the respective layers: Multi-Head Attention predicts slot interactions (across slots), while GateLORD predicts slot-specific dynamics (per slot). In our main CATER implementation we use two attention layers with two heads each with GoteLORD layers in between.
Outputs The outputs of the transition module G++1 and P have the same size its inputs. Ad- ditionally, recurrent, seot-respective hidden states H are maintained in the time-recurrent GateLORD layers:
Slot-specific position codes Pt E IR,,
Slot-specific Gestalt codes G% E IR. Ps
Slot-specific GaeeLoRD-laeerresspeccie hidden states Hg E D.,
where Dj denotes the size of the recurrent latent states.
B.3 SLOT-WISE DECODER
Inputs The outputs of all slots from the transition module pt+11 and Gt-1 then act the input to the decoder.
Outputs
The output of the decoder includes the slot-respective masks and RGB reconstructions:
Slot-specific mask outputs M RxWw
Slot-specific RGB image reconstructions Att "xxx33
which are then used as part of the input at the next iteration as specified above.
We generate the combined reconstructed image R++1 by summing all slot estimates R R4+1 and the background estimate Rag weighted with their corresponding masks M and M, as specified further in Algorithm
B.4 SEQUENCE INITIALIZATION
At time step = determine the network's inputs based randomly generated, fictive posi- tion and Gestalt estimates for each slot k. which are then fed through the Loci decoder module. Initial position and Gestalt codes P! and G are sampled from an isotropic Gaussian distribution N Posision (Mp, σp) and f'actorieed Gaussian distribution NGessaat (,,,or)) with learnable parame- ters M4 E R33 Tp E IR and My Ds, og E RR,,, respectively. The third position code value of P) that is, the Gaussian standard deviation 0 is set to 1/width, where w h denotes the number of pixels in row, effectively setting df to one pixel distance. The fourth position code value, that is, the priority value 2/. is set to its index value 2/ x- k. inducing ordered priority, which biases initial random slot assignments and thus bootstraps initial alot-assignment progress.
Based on the initial codes, we generate estimates of the output mask M reconstruction R/ and Gaussian positions Q) by calling the slot-wise decoder (see Algorithm line 19 and following): M. Rt. Q +- SlotWist ise Decoder Pt G.). We finally determine the first RGB image reconstruc- tion R!.
The hidden states of the recurrent neural network GateLORD are initialized zero, that is, H <- O.
25

---

Published as a conference paper at ICLR 2023
B.5 MAIN PROCESSING LOOP
Loci first runs the main processing loop for time steps with the first video image of particular image sequence. It thereby bootstraps the objects into individual slots, somewhat similar to previous slot-attention approaches 660] After the ten initial time steps, Loci keeps re-initializing positions Pk of slot to random values (as specified above) given that mar M2)) has been smaller than 0.5 for all time points until i. This induces initial active search process. For longer video sequences, such as the Aquarium footage, this search process was also used for invisible slots, which increased the number of tracked objects, but negatively influenced object permanence.
Note that the prediction error is calculated across the three RGB channels. It determines the MSE between the input / and the static background Reg. multiplied per Hadamard-product with the forth square root MSE between the input I' and the predicted input R'. The fourth square root emphasizes small differences, encouraging accurate encodings of individual entities.
In the transition module we apply a single trainable parametric bias neuron alpha, as proposed in Bachlechner al. [ instead of noyealiormalization. Alpha is initialized to zero. Its current
Algorithm Loci -oriiiA-looo www.lllgrrrtt (main processing loop)
1: Inputs: Input video E RRTxxxWxx3 static background Roy E IR. x3
2: Network parameters: eencoder. Otransition deccoder
3: Additional parameters: initialization parameters Oinit' background threshold Obg. which is
encoded uniform offset mask Mbg Obg: number of slots K; processing steps T
4: Initialize H1. R1, R Mi. Q1 see Section for details
5: for = 1... do
6: # Pre-processing:
Rbg )² aais 'rgb') y Mean ((r- ftt) ,aris 'rgb'
7:
E Mean
(I'' Rbg
Mr. Qr
8:
compute
9: # Slot-wise encoder:
10: St + Encoderrrank (T, E Rig Mg. M Oi O'" Mag
11:
G Encoder (St)
12:
P < Encoder
EncoderPosition Position (S)
13:
P concat (n4, o concat [Encoder, y (PH), Encoder, (P!), Encoder, (PP))
# Transition module:
15:
G++1 pt+1 H++ TTransitionModuleeCC H))
16:
# Gestalt binarization:
17:
GI+1 + Sigiid (
18:
G G + G++ ((0,)
# Slot-wise decoder:
20:
# encodes all pixel positions normalized to [-1,1], width the number of pixels in row
21:
Q + exp max minne
11y2
decodex Priority Ba sed Attention At+1 Q++1 2(11
22:
23:
Rett M +- Decooeerrea (decodex)
24:
# Post-processing:
|M M .M softmax(concat|M"" ..... Mith Mogl aris 'K')
26: RI+1 + sum(concat p1+1 .... ..., Roo) o M
G1+1
axis = 'K')
end for 28: return (
26

---

Published as a conference paper at ICLR 2023
Algorithm 2 Priooritt--basee-ttteetiii
1: Inputs: Gestalt: G2 E IR Gaussian 2d position: Qu E TR. xX'x x1, priority: z E R.K
2: Additional parameters: values the learnable O E R are initially set to 25, while 0 {0,1,.... (K })} induces default slot-order bias.
R
3: x' (z-K+N 0.1) 0 # Scale priorities and add Noise
4: # Subtract Gaussian attention from other slots, scaled by priority (a denotes the sigmoid) 5: Q% f max(0, Qx K)), (24 2 Q:)
6: combine, XGx combines E RR H2'' xD,
7: return combiner
value is multiplied with the output vector before the residual parts of the transition module, These alpha-residuall enforce the predictor initially compute the identity function. The mechanism bootstraps Loci's learning progress by initially focusing it on developing decoder-suitable Gestalt encodings.
27

---

Published as a conference paper at ICLR 2023
1ix11 conw, 4 striee 12 24
RELJ
3x3 cony, 24
1ix11 conw, 4 stride 24 48
3x3
RELU
3x3 conu,
5X5 cony. 3 5tr de, 3 pad. 48
BFUU
RETU
strice,
3x3 cony, 48
RELLI
REL
coru, 40
1x1 conw, 48
3x3 conv,
RELU
softmaxifeattree map0) x. sumimapo grid)
Sigmoidtfeaaurc mapi 10: sigma sumimap1) alpha
TaHiffaature map2)
sumimap2i
1x] conw, 24
Figure 18: Encoder module diagram (identical for all slots k)
28

---

Published as a conference paper at ICLR 2023
CAT k. Pk) S
RELU
linear, 100 -> 200
RELU
linear, 200 200
+
X
@
MHSA(..... S S St 1 ...)
x2
GateLOrd
RELU
linear, 200 -> 100
RELU
linear, 100 -> 100
+
X
SPLIT (St)
Gt k:
Pt
Figure 19: Transition module diagram (identical for all slots
29

---

Published as a conference paper at ICLR 2023
2x3 cone 45
MBLU
MLU
RELU
FeLU
353 coTV 43
243 caRV. 43
6R.11
FELU 353 cotH 43 [E.
FELu
343 c2nv. 43
TEU
3x3 care
KR
Figure 20: Decoder module diagram (identical for all slots k)
30

---

Published as a conference paper at ICLR 2023
C POTENTIAL NEGATIVE SOCIETAL IMPACTS
Here we take potential negative social impacts of Loci into consideration. Orientated on the NeurIPS Ethies Guidelines we identified two topics where Loci could potentially be used in an harmful way:
Could Loci directly facilitate injury to living beings? At its current stage it unlikely that Loci could directly be used in any weapons systems. However, when developing Loci further, its unsupervised nature allows training on huge amounts of unlabeled data and then fine tune a weapons systems to identify and track specific target.
Could Loci help develop or extend harmful forms of surveillance? Potentially yes; Loci already works well with static cameras and backgrounds. While we did not evaluate Loci's performance on pedestrians, it is likely that Loci could be trained to track pedestrians in a surveillance setting. Supervised tin-ttniig could then potentially be used to identify a specific target group of people.
While these potential misuses of Loci are concerning, they are unavoidable byproduct of advancing the field of (unsupervised) object tracking as whole. Seeing that current object tracking systems are mostly black boxes, particularly when it comes to deciphering how and why they track certain entities, Loci may enable better control over what is tracked and where exactly tracking may be applied. Hopefully, Loci can thus be used mitigate unwanted tracking biases, at least, to facilitate the identification such tracking biases.
31


Reviews:
[{"review_id": "vEUNy2g_EB", "reviewer": "Reviewer_BwtU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, inspired by the “what” and “where” parallel pathways of the visual system, the authors present a novel architecture for object segmentation and tracking with disentangled representations of objects and their position. The proposed slot-based object representation architecture excels on the position tracking benchmark CATER which is a complex task requiring detecting and tracking multiple objects and understanding their interactions in dynamic scenes. \n", "summary_of_the_review": "$\\textbf{Questions}$:\n\n1- How do we know whether the Gestalt and the position encoding variables in the trained model represent the intended information, e.g. object identity and position, respectively? The Gestalt and position lesion results shown in figures 14 and 15 imply a more significant role for the position than the Gestalt. Especially, in the example in figure 15, the Gestalt lesion doesn’t seem to damage shape or object information much in the reconstructed images. How would the lesions affect the moving MNIST task? It should be simpler to see the effects of lesions in the moving MNIST task, e.g. Gestalt lesions should damage digit shapes (or misrepresent digits) while position lesions should interfere with the digits’ movements. \n\n2- Do different slots learn to represent different objects in a scene? Showing a few examples would help. \n\n3- The architecture of the transition module isn’t clear. Unlike the other parts of the model, the transition module's equations are not included in the appendix either. I found it difficult to follow the inner functioning of the transition module. For example, what does “interaction between slot-encoded entities” mean?\n\n4- Although explained in some details in the main text, the results with more complex backgrounds are only limited to one example (the aquarium) in the appendix. Can the authors give more examples of segmentation/training tasks with complex, naturalistic backgrounds? Although the performance of the model is impressive in the case of the given example, the average performance of the model and its cases of failure cannot be understood from the given example. \n", "strength_and_weaknesses": "$\\textbf{Strengths}$: \n\nThe main idea underlying the proposed method is an interesting, algorithmic implementation of the parallel visual processing in the brain. The results, as presented in the paper, strongly support the efficacy of the proposed method. The paper is very well written, and except for one part of the architecture, the rest of the paper is straightforward and easy to understand. Thorough ablation experiments perfectly demonstrate the contribution of each design choice in the performance of the model. Overall, I believe the manuscript is ready for publication as is, and can make a significant contribution to both machine learning and cognitive neuroscience.\n\n$\\textbf{Weaknesses}$:\n\nAs mentioned above, I believe this paper doesn’t have a significant weakness that needs to be addressed before acceptance. However, there are two parts of the paper that could benefit from further explanation and exploration: the transition module and the information represented by the Gestalt and the position variables. I’ll elaborate in my questions and comments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z-PLkOfO6J", "reviewer": "Reviewer_fFzL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Loci, a self-supervised system for object-centric location and identity tracking in video. The motivation is to build a hybrid symbolic + neural system that addresses challenges pertaining to the binding problem. Emphasis is placed on disentangling “what” from “where” in each object representation. The main experimental results compare Loci against competitive baselines on CATER and Moving MNIST.\n", "summary_of_the_review": "I believe the paper, in its current state, is probably not ready for publication. I encourage the authors to spend time improving the presentation and clarity of the work to be able to identify and highlight novel insights that advance our understanding of how to address aspects of the binding problem. This may involve simplifying the proposed Loci system.\n\n---\nUpdate: After reading the author response, I have increased my score from 5 to a 6. Justification is provided in the comments.", "strength_and_weaknesses": "Strengths\n=======\n- Identifying how to use self-supervised learning to solve binding-problem-respective challenges for object-centric perception is an important and relevant problem.\n\n- Good empirical results on CATER and MovingMNIST. The results of the self-supervised version are  mostly comparable to baselines while the supervised version shows a stronger improvement.\n\nWeaknesses\n=======\n- I believe that the idea of disentangling “what” from “where in object-centric representation learning has been explored repeatedly in the past by previous systems (as the authors also indicate in the paper), and so I view this aspect of the paper to be of limited novelty.\n\n- It is unclear to me what other aspects of the Loci architecture are novel. I struggle to see how Loci advances our understanding of how to solve binding-problem-respective challenges. Rather, Loci as a whole appears to be an agglomeration of various ideas from prior work into a single, complex system. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "DBTsb1l6wX", "reviewer": "Reviewer_Nkjj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method along slot based, generative multi-object tracking following the approach of representation, segregation and composition (Greff et al. 2020). It claims the idea to introduce two encoding streams one for appearance and one for position. The paper is well written, however some illustrations and the way how the algorithms are described make the paper challenging to understand.\n", "summary_of_the_review": "There is a huge effort in the paper showing appealing results on the CATER dataset. The reason the authors claim is the new idea of disentangling appearance from position encoding in their slot network. It seems that this idea is new to this line of research, however it is not new in ML. The paper needs improvement in the illustrations and in the presentation of the algorithms.", "strength_and_weaknesses": "The paper follows a line of research with the new idea to disentangle appearance encoding from position encoding. The idea is motivated by the human visual system which also operates as dorsal and ventral streams. The work shows appealing results on the standard datasets (Table 1). There are two shortcomings: 1) the description of the algorithms is somehow difficult to read and understand. My suggestions would be instead of showing pseudo code (page 25) and narrow, nearly unreadable network illustrations (page 3, page 26, 27) to save page space by neglecting the pseudo code and referring to the implementation code. By that the architectural illustration could be made readable which I think is essential to understand the paper. 2) You state on page 3: \"But optical flow based methods notoriously struggle with occlusions as they can only represent parts of the scene that are currently visible.\". Optical flow by definition is the vector field of apparent motion. So there is no direct intention to represent flow vectors caused by hidden objects in the vector field. I agree OF struggles with occlusion but because they usually do not consider the concept of occlusion. There are exceptions (see e.g. Detecting Occlusions as an Inverse Problem, Estellers & Soatto, J Math Imaging Vis, 2016).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ah532ks6GN", "reviewer": "Reviewer_vHnv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Here, the authors desribe a variant of Siamese networks that performs objecting identification and tracking, Loci. A key feature of Loci is that it treats the “Gestalt” of objects separately from their position. Loci is benchmarked using CATER and Moving MNIST, and it generally shows impressive performance gains relative to other techniques, especially in CATER.", "summary_of_the_review": "This work presents an advance over the current state of the art in object identification and tracking. Moreover, it is clearly presented and well-argued. Overall I think this would be a good contribution to ICLR.", "strength_and_weaknesses": "**Strengths**\n\n\n1. For the most part the paper is well-written and straightforward to follow.\n2. While Loci is fairly complex, I found most of the modeling decisions to be reasonably well-justified.\n3. The results, especiall on CATER, are impressive.\n\n**Weakness**\n\n\n1. I found parts of the model to be under-described, e.g. what exactly is a Gestalt code. This should be presented clearly in the main text since it is central to the model.\n2. The total loss function appears to be specifically tuned to CATER. For instance, would the position change loss need to be altered to accomodate different object movement speeds?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster