SPECTRA: Sparse Entity-centric Transitions
Authors: Rim Assouel
Yoshua Bengio
Venue:  (2020)
Keywords: representation learning
slot-structured representations
sparse slot-structured transitions
entity-centric representation
unsupervised learning
object-centric

Abstract: Learning an agent that interacts with objects is ubiquituous in many RL tasks. In most of them the  agent's actions have sparse effects : only a small subset of objects in the visual scene will be affected by the action taken. We introduce SPECTRA, a model for learning slot-structured transitions from raw visual observations that embodies this sparsity assumption. Our model is composed of a perception module that decomposes the visual scene into a set of  latent objects representations (i.e. slot-structured) and a transition module that predicts the next latent set slot-wise and in a sparse way. We show that learning a perception module jointly with a sparse slot-structured transition model not only  biases the model towards more entity-centric perceptual groupings  but also enables intrinsic exploration strategy that aims at maximizing the number of objects changed in the agent’s trajectory.

Under review as a conference paper at ICLR 2020
SPECTRA: SPARSE ENTITY-CENTRIC TRANSITIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning agent that interacts with objects is ubiquituous in many RL tasks. In most of them the agent's actions have sparse effects : only a small subset objects in the visual scene will be affected by the action taken. We introduce SPECTRA, a model for learning slot-structured transitions from raw visual ob- servations that embodies this sparsity assumption. Our model is composed of perception module that decomposes the visual scene into a set of latent objects representations (i.e. slot-structured) and transition module that predicts the next latent set slot-wise and in a sparse way. We show that learning perception module jointly with sparse slot-structured transition model not only biases the model towards more entity-centric perceptual groupings but also enables intrinsic exploration strategy that aims at maximizing the number of objects changed in the agents trajectory.
INTRODUCTION
Recent model-free deep reinforcement learning (DRL) approaches have achieved human-level per- formance in a wide range of tasks such as games (Mnih et al. 2015). A critical known drawback of these approaches is the vast amount of experience required to achieve good performance. The promise of model-based DRL is to improve sample-efficiency and generalization capacity across tasks. However model-based algorithms pose strong requirements about the models used. They have to make accurate predictions about the future states which can be very hard when dealing with high dimensional inputs such as images. Thus one of the core challenge in model-based DRL is learning accurate and computationally efficient transition models through interacting with the envi- ronment. [Buesing et al. (2018) developed state-space models techniques to reduce computational complexity by making predictions at higher level of abstraction, rather than at the level of raw pixel observations. However these methods focused on learning a state-space model that doesn't capture the compositional nature of observations: the visual scene is represented by a single latent vector and thus cannot be expected to generalize well to different objects layouts.
Extensive work in cognitive science (Baillargeon al. 1985 Spelke 2013) indeed show that hu- man perception is structured around objects. Object-oriented MDPs (Diuk et al.) 2008) show the benefit of using object-oriented representations for structured exploration although the framework as it is presented requires hand-crafted symbolic representations. [Bengio (2017) proposed as a prior (the conseiousness prior) that the dependency between high-level variables (such those describing actions, states and their changes) be represented by sparse factor graph, i.e., with few high-level variables at time interacting closely, and inference performed sequentially using attention mecha- nisms to select a few relevant variables at each step.
Besides. a recent line of work (Greff et al. 2017 van Steenkiste al. 2018 Eslami et al. 2016
Kosiorek et al.) 2018 Greff et al. 2019 Burgess et al. 2019) has focused on unsupervised ways to decompose raw visual scene terms objects. They rely on a slot-structuree representation (see Figure of the scene where the latent space is set of vectors and each vector of the set is supposed to represent an "object" (which we refer to as "entity") of the scene. However, to the best of our knowledge, Watters et al. (2019) is the only work that investigates the usefulness of slot-structured representations for RL. They introduced method to learn transition model that is applied to all the slots of their latent scene representation. Extending their work, go further and posit that slot-wise transformations should be sparse and that the perception module should be learned jointly with the transition model.

---

Under review as a conference paper at ICLR 2020
Input image
predicted next frame
Scene
Encoder
3.
Decoder
slot
Mer me
# slots
Potx)e)
E,miN(s,:ufo))
Figure I: A: SPECTRA. Illustration an cetity-centric transition model. B: Naive Perception mod- ule with CNN-based encoder and slot-wise decoder. Hyperparameters description in Appendix A.
We introduce Sparse Entity Centric Transitions (SPECTRA), entity-centric action-conditioned transition model that embodies the fact that the agents actions have sparse effects: that means that each action will change only few slots the latent set and let the remaining ones unchanged. This motivated by the physical consideration that agent interventions are localized in time and space. Our contribution is motivated by three advantages:
Sparse transitions enable transferable model learning. The intuition here is that the spar- sity of the transitions will bias the model towards learning primitive transformations (e.g. how pushing box affects the state of box being pushed etc) rather than configuration- dependent transformations, the former being more directly transferable to environments with increased combinatorial complexity.
Sparse transitions enable a perception module (when trained jointly) to be biased towards more meaningful perceptual groupings, thus giving potentially better representations that can be used for downstream tasks, compared to representations learned from static data.
Sparse transitions enable an exploration strategy that learns to predict actions that will change the state of as many entities as possible in the environment without relying pixels error loss.
2 RELATED WORK
Unsupervised visual decomposition. Learning good representations of complex visual scenes is a challenging problem for AI models that is far from solved. Recent work (Greff et al. 2017 van Steenkiste al.) 2018 Eslami et al. 2016; Kosiorek et al. 2018 Greff al.) 2019 Burgess et al.] 2019) has focused on learning models that discover objects in the visual scene. Gr- eff et al. (2019) further advocates for the importance of learning to segment and represent objects jointly. Like us they approach the problem from spatial mixture perspective. van Steenkiste et al.) (2018) and Kosiorek al. (2018) 8) build upon Greff et al. (2017) and Eslami et al. (2016) respectively by incorporating next-step prediction part of the training objective in order to guide the network to lear about essential properties objects. As specified in van Steenkiste et al. (2019) we also be- lieve that objects task- meskdeepnndent and that learning slot-based representations along with sparse transitions bias the perception module towards entity-centrie perceptual groupings and that those structured representations could be better suited for RI downstream tasks.
2

---

Under review as a conference paper at ICLR 2020
Slot-based representation for RL. Recent advances in deep reinforcement learning in part driven by a capacity to learn good representations that can be used by an agent to update its policy. Zambaldi et al. (2018) showed the importance of having structured representations and computation when it comes to tasks that explicitly targets relational reasoning. Watters et al. (019)) also show the importance of learning representations of the world in terms of objects in a simple model-based setting. Zambaldi et al. (2018) focuses on task-dependent structured computation. They use a self- attention mechanism (Vaswani al., 2017) model an actor-critic based agent where vectors in the set are supposed to represent entities in the current observation. Like Watters et al. (2019) we take a modell-asee approach: our aim is learn fask-independent slot-based representations that can be further used in downstream tasks. We leave the RL part for future work and focus on how learning those representations jointly with sparse transition model may help learn a better transition model.
SPECTRA
Our model composed of two main components: perception module and a transition module (section 3.1). The way we formulated the transition implicitly defines an exploration policy (section 3.3) that aims at changing the states of as many entities as possible.
Choice of Environment. Here are interested in environments containing entities agent can interact with and where actions only affect a few of them. Sokoban is thus a good testbed for our model. It consists of difficult puzzle domain requiring locations. Irreversible wrong moves can make the puzzle unsolvable. Each room is composed of walls, boxes, targets, floor and the agent avatar. The agent can take 9 different actions (no-op. types of push and 4 types of move).
agent to push set of boxes onto goal
Fully Observed vs Learned Entities. The whole point is to work with slot-based representations learned from raw pixels input. There is no guarantee that those learned slots will effectively correspond to entities in the image. We thus distinguish two versions of the environment (that correspond to two different levels of abstraction):
Fully observed entities: the input is structured. Each entity corresponds to spatial loca- tion in the grid. Entities are thus represented by their one-hot label and indexed by their x-y coordinate. This will be referred to as the fully observed setting. There is no need for perception module in this setting.
Raw pixels input: the input is unstructured. We need infer the latent entities represen- tations. This will be referred to as the latent setting.
3.1 MODEL OVERVIEW
The idea to learn an action-conditioned model of the world where at each time step the following take place:
Pairwise Interactions: Each slot in the set gathers relevant information about the slots conditioned on the action taken
Active entity selection Select slots that will be modified by the action taken
Update: Update the selected slots and let the other ones remain unchanged.
Ideally, slots would correspond to unsupervisedly learned entity-centric representations of a raw visual input like it is done by Burgess et al. (2019): Greff al. (2019). We show that learning such perception modules jointly with the sparse transition biases the perceptual groupings to be entity-centric.
Perception module. The perception module is composed of an encoder fene and decoder faee- The encoder maps the input image x to a set of latent entities such that at time-step t we have fxne(x') s¹ E k xp. It thus outputs slot-based representation of the scene where each slot is represented in the same way and is supposed to capture properties of one entity of the scene. Like (Burgess al. 2019 Greff al. 2019) we model the input image x with spatial Gaussian
3

---

Under review as a conference paper at ICLR 2020
Mixture Model. Each slot sf. is decoded by the same decoder fdee into a pixel-wise mean P4kk and pixel-wise assignment mi inon-negative and summing to 1 over k). Assuming that the pixels i are independent conditioned on s', the conditional likelihood thus becomes:
(x s')
II W p o2) with plaee mfa = fdee(sh))
As our main goal to investigate how sparse transitions bias the groupings of entities, in our exper- iments very simple perception module represented in Figure I to incorporate more sophisticated perception modules.
We leave it for future work
Pairwise interactions. In order to estimate the transition dynamics, we want to select relevant entities (represented at time by the set s E KK) that will be affected by the action taken, so we model the fact that each entity needs to gather useful information from entities interacting with the agent i.e. the agent close the agent blocked by wall box etc..). To that end propose to use a self- attention mechanism (Vaswani et al. 2017). From the k-th entity representation si at time t, we extract row-vector key K row-vector query Q and row-vector value Vi conditioned on the action taken such that (aggregating the rows into corresponding matrices and ignoring the temporal indices):
KQT Vd
= =0fflmar
)V
where the softmax is applied separately on each row. In practice we concatenate the results of several attention heads to use it as input to the entity selection phase.
Entity selection. Once the entities are informed w.r.t possible pairwise interactions the model needs to select which of these entities will be affected by the action taken a". Selection of the entities are regulated by a selection gate (Hochreiter & Schmidhuber 1997 Cho al. 2014) computed slot- wise as:
f (MLP(((((:
where f can be interpreted as the probability for an entity to selected.
Update. Finally, each selected entity is updated conditioned on its state sy at time-step and the action taken a'. We thus simply have:
$ flst, e²1) + fi)st
fo a learned action-conditioned transformation that is applied slot-wise. We posit that enforcing the transitions to be slot-wise and implicitly sparse will bias the model towards learning more prim- itive transformations. We verify this assumption in next subsection in the simpler case where the entities are fully observed (and not inferred with a perception module).
EXPERIMENTS
In this work we demonstrate three advantages of entity-centric representations learned by SPEC- TRA:
Implicitly imposing the transitions to be sparse will enable us to learn transition models that will transfer better to environments with increased combinatorial complexity. Section 4.1.
Learning slot-based representations jointly with sparse transition model will bias the per- ceptual groupings to entity-centric. Section 4.2.
Finally we investigate the usefulness of the implicit exploration scheme induced by SPEC- TRA when learning the model jointly. Section 4.3.
4

---

Under review as a conference paper at ICLR 2020
Action taken move s_rrihh
Envrorment at Time 2000
Envirommnn 20e 14
mele) Pes athect 2 Dence L.
box
Floce
monee
needs selected by the model
True mapsings that need to be leernal
903300
Figure 2: left: Full and sparse settings are trained on environment containing one box and evaluated out-of-distribution on two boxes. We plotted the validation losses of both settings during training. The full connectivity architecture unable achieve out-of-distribution generalization to envi- ronment with two boxes. right: Illustration of what the model has to learn in the fully observed setting: be correct the model needs map any concatenation of [agent,move] sition floor and to select only the right entities to be changed. The learned mappings are general rules that are directly transferable to settings with more boxes.
vacated po-
4.1 LEARNED PRIMITIVE TRANSFORMATIONS
In this section we show that sparse selection the transitions yields learned slot-wise transforma- tions that transferable
dutto--distribution settings with increased combinatorial complexity. We restrict ourselves to the fully observed setting. Like (Zambaldi et al.] (2018) the entities corre- spond spatial location in the 7 grid. Each entity 8 thus described in terms of its label to which we append its x-y coordinate. The results in Figure are intuitive; to learn the right transitions with our formulation, the model forced to:
select only the relevant entities to be updated.
learn the right primitive transformation (e.g. the agent slot is selected to be modified by any of the move actions, then its position vacated, so the model should map any concatenation of [agent, move] to the floor label etc...). See Figure 2 right.
Here entity representations are not learned and thus correspond to their labels. We thus train the model with simple cross-entropy loss. We are interested in comparing two settings:
Sparse setting:
transformation is still done slot-wise to selected entities only. Each slot contains the label and x-y coordinate of the entity only. The transformation is applied to concatenation of the entity label and the action [label,action].
Full setting: transformation still done slot-wise but this time each slot &, potentially contains information about all the other slots in the set. The transformation is applied to a concatenation of the entity representation 5 and the action [tildes [.ttion), Thus we hypothesize that the transformation module will learn ceafiguraaionndepeene if an agent is close to a box and a wall, and 3 steps ahead there a target to be reached, and it takes move action to do so) that will not be easily transferable environments with increased complexity and a wider variety of contexts.
rules (e.g.
Both settings are illustrated in Figure 7 of the Appendix. In Figure 2 we reported the evolution of training and evaluation losses of both the full and the sparse settings when the models are trained in a 7x7 environment with one box and evaluated in 7x7 environment with two boxes.
4.2 STRUCTURED REPRESENTATION LEARNING
In this section we demonstrate how learning a perception module along with sparse transitions will bias this module towards learning entity-centrie perceptual groupings of the pixel input. In

---

Under review as a conference paper at ICLR 2020
S
Ground Truth
H+1
[+1
Sior kl masbed
Sitt-iie transiiin
Slet ki mosked decoding
Slat-oise teanstion
Sko k2 mesked decnd
franshion
lelo natias about aggn
Total trursition
5
Infoeation about toc apoit
SPECTRA
Total transition
5
5
SFECTRX
Joint training
Separate training
Figure 3: Comparison of slot-wise masked decodings when the perception module is trained sep- arately jointly with the sparse transitions. We show the reconstruction associated with the slots that contain information about the agent. When the perception module is trained jointly, slots in the learned latent set are biased to be entity-centric (here agent-centric).
order to verify this intuition we compare in Figure the reconstructions from the perception module when trained separately vs jointly with the sparse transition module. In this experiment the input is not structured anymore but just raw 12x 12x3 pixel image. We used a simple perception module as described in Figure
We thus distinguish two losses, reconstruction loss
Speroep
log Σ
and transition loss
D Σ log
Livans
i+1 mikk MottN((x) 1+1 24+1 ,0²)
Nx :1:+++111))))
with Mik mia = fdee(s()) st (x'). p/ mis ++1 the future state predicted by the transition function.
faee(Stt)) )i. and 8 = Strans (s(c is
fdees fenc and ferons are respectively the decoder, the encoder and the transition modules. For the joint training (resp. separate training) setting, gradients from Lirans are back-propagated through parameters of fene and ftrans (resp. ftrans only). In both settings, gradients from Lpercep are back propa-aadgattd through parameters of fene and fdec-
In Figure 3 we put particular attention on the masked reconstructions from slots containing visual information about the agent. We can directly notice that the perceptual groupings done by the encoder, when it is trained jointly with the transition module, are agent-centric: the information about the agent is contained in one slot only (whereas it is often contained several slots in separate training settings). Moreover, in Figure we see the joint training setting leading to better transition model: we hypothesize that the transformations are easier to learn specifically because they have to focus dependent variables at a time rather than more global but more specific configurations involving all the variables in the state, suggested by Bengio (2017)
the effects of the actions taken entities, i.e., involving few strongly

---

Under review as a conference paper at ICLR 2020
2000 10020
150E 1940 78:00 10000 11500 20000
25000 5000 1000 R00I 21000 1903) 10W00 20/00
Figure 4: Loss vs training updates, with training is done in pixel space, transitions are sampled randomly and results averaged over runs. left: Validation perception loss Lpercep of joint and separate training right: Validation transition loss Ltrans of joint and separate training. Separate training better in terms of perception loss but joint training gives better transition model. We posit that this is because the slots are biased to be entity-centric and transformations involving only relevant entities are easier to learn.
f==0
fe=0.1
fu=0.22
fu=0.33
fu=0.4
fe=0.55
f.=0.6
f.=0.7
fu=0.8
Sot
mosked
Joint training
f--0.1
f.-0.2
0.-0.3
f.-0.4
fe-0.5
f--0.6
f.-0.7
fu-0.8
Sioe
musked
mg
Separate training
Figure 5: Reconstructions of a slot for different levels of its activation gate (Bq. [ Transformations learned when the perception module trained jointly vs separately from the sparse transitions. When the training done end-to-end, transformations seem to be more interpretable. We posit that this is because the slots are biased to be entity-centric as shown in Figure 3 It is interesting to notice that enough changes appear for activation gate value of 0.1. We may want to explore an explicit sparsity constraint for the selection mechanism.

---

Under review as a conference paper at ICLR 2020
Fully obowerd No poncoption mocule
Pbots input With Porcostic mocs
2010
1200
Nuntbrr t0 Bhe55466
5000
4033
5000
8000
3000
0000
Nambor
2200
9030
1000
1230
40000
80003
80330
00003 120330
40300
60003
10000
100030
120000
Upteces
Upseetes
Comparison is done against randomly sampled transitions. left: Number of entities changed in the 1-step buffer during training. As expected, the number of transitions with 2 spa- tial locations changed in the grid increases whereas the ones with no location changed decreases. We also notice a slight increase in the number transitions with 3 spatial locations changed (corre- sponding to the agent moving a box!). Training is done in the fully observed setting. right: Training done in pixel space. Again here, the number of transitions with two spatial locations changed in the grid increases whereas the ones with no location changed decreases. However the number of of transitions with the agent that moves a box did not increase.
Figure 6:
We also visualized the transformations learned by both settings. To do so, we manually increased the value of the update gate fk for few slots k. An example is given in Figure 5 and additional ones are given in section B of the Appendix.
4.3 INTRINSIC EXPLORATION STRATEGY
In many environments a uniformly random policy is insufficient to produce action and observation sequences representative enough to be useful for downstream tasks. In this paper we suggest to leam an exploration policy jointly with the model, based on an intrinsic reward that depends on the transition model itself and exploits its entity-centric structure to quantify the diversity of aspects of the environment modified by exploratory behavior. Our model learns first select entities that will be changed and then learns how to transform the selected entities. Similar to the empowerment intrinsic objectives (Klyubin et al. 2005 Kumar 2018). natural exploration strategy in settings like Sokoban would be to follow trajectories that overall have as many entities being selected as possible. If the agent indeed never pushes a box on target when learning its transition model, it will not be able to transfer its knowledge to a task where it has to push all the boxes on all the targets. We thus suggest to learn a policy that maximizes the number of entities selected, as predicted by the current model. We alternate between policy update and model update.
We used a 10-step DQN for the exploration policy and have the DQN and the model share the same I-step replay buffer. The DQN policy e-greedy with e decaying from I to 0.3. In order to train the DQN we used the following intrinsic 1-step reward:
r(s ax) E10EEM)
with h a chosen threshold for the update gate value. We expect this training strategy to promote trajectories with as many entities that will have their state changed as possible. We thus expect the agent learn not to get stuck, aim for the boxes, push them etc... In order to validate that intuition, we first conduct experiments in the fully observed setting. In this setting we consider the following types of moves:
valid move: Whenever the agent takes move action in valid direction, two entities will have their state changed: the initial location of the agent and the next one.
valid push Whenever the agent takes push action and box is available to be pushed in the chosen directions, three entities will have their state changed: the initial location of the agent, the initial location of the box and the next location of the box.
8

---

Under review as a conference paper at ICLR 2020
Whenever the agent takes push action when there is no box to push in the chosen direction, nothing happens.
blocked push
blocked move: Whenever the agent takes a move action in a non-valid direction (against wall, box etc...,, nothing happens.
With our suggested training strategy we expect the agent to promote trajectories with more tran- sitions of type valid move than blocked move and blocked push and hopefully with the number of valid_push transitions increased as well. During training, we thus monitor the true number of entities changed in the transitions stored in the shared 1-step buffer. We also performed the same experiment in the raw input pixels setting and monitored the true number of entities changed in the 1-step buffer during training. Results are reported in Figure 6 and confirm our hypothesis: the agent learns to avoid actions that will result in no changes in the environment (blocked push and blocked move. Details of the hyperparameters are given Appendix.
CONCLUSION
We have introduced SPECTRA, novel model learn sparre slot-structured transition model. We provided evidence to show that sparsity in the transitions yields models that learns more primitive transformations (rather than confguration-deeeenden) and thus transfer better to ditttr-disttiuutionn environments with increased combinatorial complexity. We also demonstrated that the implicit spar- sity of the transitions enables an exploration strategy that aims at maximizing the number of entities that be will be modified on the agent's trajectory. In Figure we we showed that with this simple explo- ration strategy the agent leans to avoid actions that will not change the environment (blocked move and blocked -uss). Preliminary results in pixel space show that SPECTRA biases even simple per- ception module towards perceptual groupings that are entity-centric. In Figure 5 we also showed the benefit of jointly training the perception (encoder) module and the transition module. We anticipate that our model could be improved by incorporating a more sophisticated perception module. In the future we aim use SPECTRA investigate possible uses in model-based reinforcement learning.
REFERENCES
Rene Baillargeon, Elizabeth Spelke, and Stan Wasserman. Object permanence in five-month-old infants. Cognition, 20:191-208, 09 1985. doi: 10. 10.016600000227/1559000333
Yoshua Bengio. The consciousness prior. arXiv preprint 1eXXi:1709.089000 2017.
Lars Buesing, Theophane Weber, Sebastien Racaniere, S. M. Ali Eslami, Danilo Rezende, David P. Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, and Daan Wierstra. Learn- ing and querying fast generative models for reinforcement learning, 2018.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation, 2019.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, abs/1406.1078, 2014. URL http:://aaxivvorr/abbbb/
1078
Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for effi- cient reinforcement learning. Proceedings of the 25th International Conference on Machine Learning. ICML '08, pp. 240-244 New York, NY, USA, 2008. ACM. ISBN 978-1-605558-05 URL http: /ooi acm. wtt.aaa.............. wtt.:oo.g/........ 1ttp///...11/1191111111
4. doi: 10.1145/13901.6.11 1391199999911111118777 1390187. 1390187
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jrgen Schmidhuber. Neural expectation maximization, 2017.

---

Under review as a conference paper at ICLR 2020
Klaus Greff, Raphal Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98)):1335
1780, November 1997. ISSN 0899-7667. doi: 10.1 10.1162/neco.19/ 1997.988.173 URL http:://xx doi doi. www.1166cccoooo 9997.9.8.773
Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. All clse being equal be empow- ered. In European Conference on Artificial Life, pp. 144-553. Springer, 2005.
Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Tch. Sequential attend, infer, repeat: Generative modelling of moving objects, 2018.
Navneet Madhu Kumar. Empowerment-driven exploration using mutual information estimation.
arXiv preprint 10X.058/10005533 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-53. February 2015. ISSN 00280836. URI www.orgg1110 10.
http:://xx.. wtw.sddw...oooo.oooooooooooo
14336
Adam Santoro, David Raposo. David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Timothy P. Lillicrap. simple neural network module for relational reasoning. CoRR, abs/1 706.011277 2017. URL
org/ abs/ 11111776..0011 01427
Elizabeth S. Spelke. Where perceiving ends and thinking begins: The apprehension of objects in infancy. 2013.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jrgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions, 2018.
Sjoerd van Steenkiste, Klaus Greff, and Jürgen Schmidhuber. A perspective on objects and systematic generalization model-based RL. CoRR, abs/1906.01035, 2019. URL http:
//axxiv..rg//ab//333.. 01003
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration, 2019.
Thophane Weber, Sbastien Racanire, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomnech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination-augment for deep reinforcement learning, 2017.
agents
Vinicius Zambaldi, David Raposo, Adam Santoro, Vietor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Relational deep reinforcement learning, 2018.
A ARCHITECTURE AND HYPERPARAMETERS
A.1 FULLY OBSERVED SETTING
In the fully observed setting the input at time is set o' N0x7 corresponding to one-hot labels (that can be agent (off and on target), box ( off and on target), wall, target and floor). of each entity in 7 7 grid (N 49). We also append their normalized y coordinates so that the
10

---

Under review as a conference paper at ICLR 2020
Transformation
5'
Linear
Linear
f
Transformtionn
5443
Q
K
Multi Head DPA
Selector
Multi Head DPA
of
Sparse secting
Full setting
Figure Transition model with and without selection phase.
final input to the transition model is set s [0,1] Like detailed previously in Figure 7 the transition model is composed of two modules: the selection module and the transformation module.
In section 4.1 we also distinguished between the sparse and the full setting and they are described in 7 In the full setting, there is no more selection bottleneck and the transition module is a simple transformer-lic architecture.
Selection module. The selection module is transformer-like architecture. It takes input at time step ł the concatenation c [s I'] of the set sf and the action The selection module is then composed of 2 attention heads where head is stack of 3 attention blocks (Vaswani et al. 2017 Zambaldi et al. 2018). The 3 blocks are I-layer MLP that output key, query and value vectors of channels size 32, 64, 64 respectively. The first two blocks are followed by RELU non linearities and the last one doesn't 't have any. The output of the attention phase is thus the concatenation of values obtained from the attentions heads E R xxx112 To obtain the selection binary selection variables then simply apply slot-wise single layer MLP the concatenation ē [s' a') followed by logSoftmax non-linearity in order to compute the log-probabilities each entity to modified by the action taken. The output of the selection module thus of log-proo oog-robabillites! I' Nx2
Transformation module. The transformation module is simple shared 2-layers MLP that is applied slot-wise to the the concatenation e = [s', I'] of the input set s' E (0. N and the action taken. It outputs channels of sizes 16, respectively. The first layer is followed by RELU non-linearity and the last one by logSoftmax non-linearity in order to compute the log-probabilities of the label of each predicted entity.
Full setting. In the full setting, we don't have a selection bottleneck anymore. The transformation module is thus directly applied to the output of the attention phase e' = [5*,a"], It consits this time of simple shared 3-layers MLP that applied slot-wise and outputs channels sizes 64, 32, 7 respectively. The first two layers are followed by RELU non- con-linearity and the last one by logSofuaax non-linearity
A.2 LATENT SETTING
In the latent setting the input at time t is a raw pixels (RGB) image of R112x112X3, In the latent setting, the transition model is composed of a perception module, selection module and transformation module.
Perception module. When dealing with unstructured input we first need a way to extract entities latent representations. For this work used very simple and naive perception module, with an
11

---

Under review as a conference paper at ICLR 2020
encoder similar to what is done by Zambaldi et al. (2018); Santoro et al. (2017). Like detailed in Figure [ we use CNN to parse pixel imputs into feature maps of size n x n, where & is the number of output channels of the CNN. We choose arbitrarily n 4 and didn't 't perform any hyperparameter search for the CNN architecture. We then concatenate x and coordinates to each k-dimensional pixel feature-vector to indicate the pixels position in the map. We treat the resulting pixel-feature vectors as the set of entities s' E IRA xk where here N = n² 16. We denote as Soood IRN x the entities set to which we have appended the x-y position in the map.
As our loss a pixel loss we also need decoder that decodes each entity s kocorrd of the s' back to its corresponding mean 2 and mask mi. The CNN the encoder outputs channels of size (16, 32, 32, 32, 32). All layers (except the last one) are followed by RELU non-linearities. Kernel sizes are (3, 3, 4, 3) and strides (2, 2, 2, 2, 1). The decoder is composed of 2-layers MLP followed by stack of transposed convolutions. The MLP outputs channels of sizes (7 34, 7 x 7 x 34) with RELU non-linearity between the 2 layers. The output is then resized to x 7 x 34 map that will be fed the convolution part. For the convolution part, it outputs maps of channel sizes (4, 4, 4, 4) with RELU non non-linearities between each layer. The kernel sizes are (3, 3, 4)
Selection and Tranformation modules. The selection and transformation module are very sim- ilar to the fully observed setting, except that they operate the latent space, we do not apply LogSofmax non-linearities for the transformation part. The input the selection module soord and the input to the transformation module is s'. The selection module is composed of 2 attention heads where is head is stack of attention blocks (Vaswani et al. 2017 Zambaldi et al. 2018). The 3 blocks I-layer MLP that output key, query and value vectors of channels size 34, 16, 16 spectively. The first two blocks are followed by RELU non linearities and the last one doesn't 't have any. The output of the attention phase is thus the concatenation of values obtained from the 2 at- tentions heads S E R x32 To obtain the selection binary selection variables we then simply apply slot-wise 3-layers MLP of channels sizes 16, 32, 32 respectively to the concatenation è' = - a²) followed by Softmax non-linearity in order to compute the probabilities of each entity to be modi- fied by the action taken. The output of the selection module is thus a set of probabilities p' e R The transformation module is simple 2-layers MI. of channels sizes 32,32 respectively with RELU non-linearity between the two layers.
B ADDITIONAL VISUALISATIONS
In this section we reported additional visualizations similar 3 where we monitor:
Differences in slot-wise masked decodings of the perception module when it is trained jointly and separately from the sparse transitions.
Differences the slot-wise transformations earned by the transition model when it is trained separately and jointly with the perception module.
We notice that joint training enables to learn stot-structured representation that are entity-centric and thus enable to learn better transition models. The transformations learned are especially visually more interpretable.
12

---

Under review as a conference paper at ICLR 2020
t+1
Ground Truth
t+1
t+1
Slot masked decodlnng
Slot kl masked accoding
Slet wise transition
Slot-wise transition
Slot-wise transition
masked decoding
decoding
Information the agent
Information the agent
Total transition
Total transition
SPECTRA
SPECTRA
Joint training
Separate training
Figure 8: Additional visualisations masked decodings from joint and separate training settings.
13

---

Under review as a conference paper at ICLR 2020
t+1
Ground Truth
t+1
t+1
Slot masked docading
Slet decoding
Slot-wise transition
Slot-wise
transition
Slor k? masked docoding
Slot-wise transition
Slet masked decoding ng
Slot-wiis trans siliioon
Information
about agont
agant.
Total transition
Total transition
SPECTRA
SPECTRA
Joint training
Separate training
Figure 9: Additional visualisations of masked decodings from joint and separate training settings.
14

---

Under review as a conference paper at ICLR 2020
t+1
Ground Truth
t+1
t+1
kl masked decoding
Slot-wise transition
Slot masked decoding
Slot-wise transition
Slat-wise transition
Slat masked decading
S Slot-wise transition
masked decoding
Information about
Information the agann
Total transition
Total transition
SPECTRA
SPECTRA
Joint training
Separate training
Figure 10: Additional visualisations of masked decodings from joint and separate training settings.
15

---

Under review as a conference paper at ICLR 2020
f==0
f==0.1
f==0.2
f==0.3
f==0.4
fx=0.5
f==0.6
f==0.7
f==0.8
2190
Slot k masked
decoding
Separate training
f=0
f==0.1
f==0.2
f==03
f==0.4
6=0.5
ft=0.6
f1=0.7
f==0.8
40-03
-::0
Slot k
40:23
macked
decoding
Joint training
Figure 11: Additional visualisations of the transformations learned when the perception module is trained jointly and separately from the transitions. Joint training yield visually interpretable and localized transformations of the slots.
16


Reviews:
[{"review_id": "SkeIrpvpKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model that learns to disentangle visual scene into objects (slots), and simultaneously learns a dynamics model to capture how these objects interact with each other.  The authors demonstrated that the proposed model has the potentials to discover objects without supervision, and also enables an exploration strategy that maximizes the number of objects changed in the agents' trajectories.\n\nThis paper is for sure studying an important problem. The approach presented in the manuscript makes intuitive sense. The experimental results are reasonable but can be strengthened. However, I was shocked that the authors seem to be unaware of the abundant related work in this area (see below). I'd like to see the authors' responses regarding the missing related work. As of now, my recommendation is a clear reject.\n\nAs the authors mentioned, objects play a key role in human perception, and the problem of discovering objects from visual input is for sure an important problem. I commend the authors for pursuing research in this direction.\n\nThe experimental evaluations are however a little limited: it's restricted to games, where the visual appearance of objects is almost identical. In those cases, it'd be hard to access how the model may generalize to real-world data, where object appearances and texture can be complex. There are also no comparisons with published, SOTA methods.\n\nThe major problem of this manuscript, to me, is its ignorance of related work and, therefore, overclaiming at a few places. Most importantly, I suggest the authors cite, discuss, and ideally compare with many related works from Josh Tenenbaum's group and Sergey Levine's group. A few papers listed below, in particular [A] and [B], also formulated the problem in a similar fashion, where they decompose the scene into object-centric representations and learn the interactions among objects. [B] and [C] also explored how the model can be used in an RL setting. Further, [D] studied learning an object-oriented dynamics predictor in a similar context as presented in this paper. \n\nThe sparse effects have also been explored by Xia et al. [E]. Similarly, the work has built upon object-centric representations. I, therefore, wonder how the proposed method compares with all those published papers, both at the conceptual level and at the experimental level.\n\n[A] Wu et al. Learning to See Physics via Visual De-animation. NeurIPS 2017\n[B] Janner et al. Reasoning About Physical Interactions with Object-Oriented Prediction and Planning. ICLR 2019\n[C] Co-Reyes et al. Discovering, Predicting, and Planning with Objects. ICML 2019 Workshop, https://sites.google.com/view/dpppo/\n[D] Zhu et al. Object-Oriented Dynamics Predictor. NeurIPS 2018\n[E] Xia et al. Learning sparse relational transition models. ICLR 2019\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgQLsmz5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a ‘slot-based’ (factored) representation of a ‘scene’ s.t. a forward model learned over some observed transitions only requires sparse updates to the current representation. The results show that jointly learning the forward model and the scene representation encourages meaningful ‘entities’ to emerge in each slot. Additionally, the paper argues that this representation allows for better generalization and can also guide exploration by rewarding actions that change multiple entities\n\nI really like the overall idea i.e. jointly learning the scene representation and the transition model, while enforcing sparse transitions. The experiments in Sec 4.2 and the visual results in Fig 3 do clearly highlight the benefits of joint learning, and show that the emergent representations are more meaningful compared to learning representations independently.\n\nHowever, while the overall approach is intuitive and seems to yield desirable results, I have concerns regarding the experiments, comparisons to prior work, and the exact contributions of this work. Specifically:\n\n1) It is unclear what this paper is claiming to improve over prior work: is the goal to a) learn a good forward model, or b) show that emergent entities allow better downstream tasks. If it is ‘a’ , then there are several other ways of learning good forward models e.g. convolution flow-based [1], and simply showing comparisons to a naive baseline is not sufficient. Similarly, the only downstream task examined is exploration, and again the only comparison is against a variant of the method. Therefore, while the results regarding the emergent representation are good compared to a variant without joint training, they are not shown to be useful in context of any task when compared to the approaches in literature.\n\n2) Despite the motivation in the introduction regarding applications to RL, the paper essentially learns a specific form of factored forward model. There have been several prior works which also pursue a similar approach (though mainly in context of video prediction) e.g. [2], and I don’t think this paper’s approach is novel in context of these. In any case, some comparisons should be made to these form of models which also show emergent entities with a graph-structured transition model.\n\n3) While the experiments in Sec 4.2 clearly demonstrate the benefits of the approach, the ones in Sec 4.1 and 4.3 are less convincing:\n3a) Sec 4.1 shows that the slot based transition model generalizes better, but this is only in comparison to a naive fully-connected baseline. I feel any prediction model with some structure e.g. graph-based forward model, convolutional forward model etc. would also similarly generalize.\n3b) Sec 4.2 argues that this slot-based representation can help exploration, but this is in fact a chicken-and-egg problem, as one needs to have collected interesting transition samples for a good representation to emerge. In fact, the results shown in Fig 6b) show that actions affecting 3 blocks were not explored, perhaps because the random transitions did not sample these to begin with.\n\nOverall, the approach proposed is desirable, but there are closely related alternatives that exist in literature, and there need to be more concrete comparisons against these for either prediction quality or success in downstream task.\n\n---\nReferences:\n[1] Self-Supervised Visual Planning with Temporal Skip Connections, Ebert et. al, CORL 17\n[2] Learning to decompose and disentangle representations for video prediction, Hsieh et. al., NeurIPS 18", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeZ0fDAFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a model that learns a slot-based representation, along with a transition model to predict the evolution of these representations in a sparse fashion, all in a fully unsupervised way. This is done by leveraging a self-attention mechanism to decide which slots should be updated in a given transition, leaving the others untouched. The model learns to encode in a slot-wise and is trained on single step transitions.\n\nThis work tackles an important problem, and is very well motivated and presented in a very clear fashion. It reuses some known ideas and components, but combines them in a nice way. I especially liked the use of attention to select what to update, which is a good prior to have.\n\nHowever, the results presented unfortunately seem to fall a bit short in this current version, and some decisions might have had too much of an effect on some of these shortcomings. Given some improvements, this work might become quite promising, but for the time being I am leaning against publication.\n\n1.\tMost modeling decisions are clear and well-motivated, however the choice to make the transition model f_trans always be applied only “slot-wise” might be too restrictive. Indeed, for a given action, this means that 2 slots have to independently learn the effect of that action (e.g. as shown in the example in Figure 2. right), and that some interactions are ~impossible to learn (e.g. in Sokoban, pushing a box requires knowing about the location of both the agent and the box). This could have been alleviated if the transition had access to “interactions outcomes” (e.g. if using a GraphNet, or in your model, if \\tilde{s} was provided to the transition function f_theta). Other works (including Zambaldi et al 2018, which is cited several times), handle this appropriately. Did you try to provide \\tilde{s} to the transformation operator (e.g. in Figure 7 left)?\n2.\tAdding a direct comparison to pure non-slotted versions of the model/baselines would have been quite useful, as currently it is unclear why certain things are failing.\n3.\tSimilarly, finding what was the output of the CNN encoder for pixel inputs was a bit too difficult. It is explained in the Appendix that one maps into 4x4 feature maps, but that might be too large for the current environments? Indeed for Sokoban, this means that any grid is partially supported by several “slots”, which may hurt the results more than they should (especially combined with the slot-wise transition constraints expressed above).\n4.\tThe early state of the current results are quite visible in all examples of the “Separate training” model predictions (Figure 3, 5, 8, 9, 10 and 11). None of these actually show this model performing a “correct” prediction for t+1? They only predict no changes, or nonsensical interpolations… This is not sufficient to try to make an argument about the “joint training” helping, and most discussions about “what information they contain” is strenuous at best.\n5.\tThe paper keeps mentioning that it “implicitly imposes transitions to be sparse”, however it is never explained how that would come about? I understand that the softmax in the self-attention may tend to become “peaky” and hence only affect a few slots (and the results do seem to confirm this observation), but I was expecting an explicit loss to enforce this fact. The current emphasis seems a bit ill-funded, so I would present more evidence to it or downplay it.\n6.\tSome of the results shown seem hard to interpret or provide only weak evidence for the proposed model:\n\ta.\tFigure 2. Left does seem to indicate a benefit in using the self-attention module, but it is hard to know how much of an effect the gap between the orange and red curves actually imply. This figure is overall a bit too small to interpret, and it might be better to split the 2 conditions into sub-plots. The names of the curves in the legend do not correspond to anything described in the text/caption (but I could understand them…). I was expecting more discussion of the results in Figure 2, for example at the end of Section 4.1.\n\tb.\tAs explained above, Figure 3 only shows that the Joint Training can perform a 1-step prediction, which is ok but is the bare minimum. Does it handle multi-step rollouts?\n\tc.\tFigure 4 does not seem to provide any significant results or insights. I would interpret them as showing no significant difference between the curves, and they are too small to extract any information out of them. I would remove this figure fully.\n\td.\tFigure 5 is unclear about what f_k=0 really is, and once again just shows that Separate training does nothing. I expected it to be exactly reconstructing s_t (given that the others are trying to predict s_t+1)? But this is only the case for the joint training, and without knowing what x_t actually was, it is hard to trust. The fact that f_k changes what it does as it is being increased makes the whole point hard to interpret.\n7.\tFigure 6 was quite interesting, and I feel like this could be pushed forward in a quite interesting manner. The choice of “maximizing the number of entities selected” was fair as a first try, but I could imagine it failing to generalize to more complex environments, or to be easily exploited if one ever decides to pass gradients back into the representation from the policy. It was unfortunate that the legends do not correspond to anything expressed in the main text or caption (e.g. why do you not reuse the “valid_move”, …, ”blocked_push” names that you thoroughly introduce?). What is “random_XX”? Could you comment on why the curves seem to have a large increase in variance along the 80000-100000 updates region?\n\n\nDetails:\n8.\tFigure 2 (right) was good to explain how the model worked, but I would actually change its location and try to move it into Figure 1. Similarly, Figure 7 in the Appendix seemed rather necessary to understand the model, and belongs in the main text in my opinion.\n9.\tWhen presenting the self-attention block, it would be good to directly state that these are MLPs receiving [s_t, a_t] (as done in the Appendix).\n10.\tIs sigma^2 in the decoder fixed? To which value?\n11.\tWhen presenting the “sparse” and “full” settings, having access to Figure 7 and the rest of the Appendix might be beneficial, it would be good to point forward to it.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a model that learns a slot-based representation and its transition model to predict the representation changes over time. While all the reviewers agree that this paper is focusing on an important problem, they expressed multiple concerns regarding the novelty of the approach as well as lacking experiments. It certainly is missing multiple important relevant works, thereby overclaiming at a few places. The authors provided a short general response to compare their approach with some of the previous works and conduct stronger experiments for a future submission. We believe this paper is not at the stage to be published at this point.
Decision: Reject