Factorizing Declarative and Procedural Knowledge in Structured, Dynamical Environments
Authors: Anirudh Goyal
Alex Lamb
Phanideep Gampa
Philippe Beaudoin
Charles Blundell
Sergey Levine
Yoshua Bengio
Michael Curtis Mozer
Venue: ICLR 2021 Poster (2021)
Keywords: procedural knowledge
declarative knowledge
Systematicity

Abstract: Modeling a structured, dynamic environment like a video game requires keeping track of the objects and their states (declarative knowledge) as well as predicting how objects behave (procedural knowledge). Black-box models with a monolithic hidden state often fail to apply procedural knowledge consistently and uniformly, i.e., they lack systematicity. For example, in a video game, correct prediction of one enemy's trajectory does not ensure correct prediction of another's. We address this issue via an architecture that factorizes declarative and procedural knowledge and that imposes modularity within each form of knowledge. The architecture consists of active modules called object files that maintain the state of a single object and invoke passive external knowledge sources called schemata that prescribe state updates. To use a video game as an illustration, two enemies of the same type will share schemata but will have separate object files to encode their distinct state (e.g., health, position). We propose to use attention to determine which object files to update, the selection of schemata, and the propagation of information between object files. The resulting architecture is a drop-in replacement conforming to the same input-output interface as normal recurrent networks (e.g., LSTM, GRU) yet achieves substantially better generalization on environments that have multiple object tokens of the same type, including a challenging intuitive physics benchmark.


Published as a conference paper at ICLR 2021
FACTORIZING DECLARATIVE AND PROCEDURAL KNOWLEDGE IN STRUCTURED, DYNAMICAL ENVI- RONMENTS
Anirudh Goyal', Alex Lamb ', Phanideep Gampa 1.2 Philippe Beaudoin 9, Sergey Levine
Charles Blundell 5, Yoshua Bengio Michael Mozer
ABSTRACT
Modeling structured, dynamic environment like video game requires keeping track of the objects and their states (declarative knowledge) as well as predicting how objects behave (procedural knowledge). Black-box models with monolithic hidden state often fail to apply procedural knowledge consistently and uniformly, i.e., they lack systematicity. For example, in video game, correct prediction of one enemy's ss trajectory does not ensure correct prediction of another's. s. We address this issue via an architecture that factorizes declarative and procedural knowledge and that imposes modularity within each form of knowledge. The architecture consists of active modules called object files that maintain the state of a single object and invoke passive external knowledge sources called schemata that prescribe state updates. To use a video game as an illustration, two enemies of the same type will share schemata but will have separate object files to encode their distinct state (e.g., health, position). We propose to use attention to determine which object files update, the selection of schemata, and the propagation of information between object files. The resulting architecture is drop-in replacement conforming to the same input-output interface normal recurrent networks (e.g., LSTM, GRU) yet achieves substantially better generalization on environments that have multiple object tokens of the same type, including challenging intuitive physics benchmark.
INTRODUCTION
An intelligent agent that interacts with its world must not only perceive objects but must also remember its past experience with these objects. The wicker chair in one's living room is not just chair, it is the chair which has an unsteady leg and easily tips. Your keys may not be visible, but you recall placing them on the ledge by the door. The annoying fly buzzing in your left ear is the same fly you saw earlier which landed on the table.
Visual cognition requires short-term memory that keeps track of object's location, properties, and history. In the cognitive science literature, this particular form of state memory is often referred to as an object file (Kahneman et al. 1992), which we'll abbreviate as OF, An OF serves as a temporally persistent reference to an external object, permitting object constancy and permanence as the object and the viewer move in the world.
Complementary to information in the OF is abstract knowledge about the dynamics and behavior of object. We refer to this latter type of knowledge schema (plural schemata), another term borrowed from the cognitive-science literature. The combination of OFs and schemata is sufficient to predict future states of otject-structured environments, critical for planning and goal-seeking behavior.
To model complex, structured visual environment, multiple OFs must be maintained in parallel. Consider scenes like PacMan video-game screen in which the ghosts chase the PacMan, public
Mila, University of Montreal, IIT BHU, Varanasi, Waverly, UC Berkeley," Deepmind, Google Research, Brain Team, Corresponding author: ani rudhgoya 199199gmaa 98gma 1 c.c

---

Published as a conference paper at ICLR 2021
Figure Two successive frames of PacMan, illustrat- ing the factorization of knowl- edge. Each ghost is rep- resented by a persistent OF (maintaining its location and velocity), but all ghosts op- erate according two schemata, depending on whether the ghost is in nor- mal or scared state.
Orject
file
PadMin
Narrel
Scand
Lei: Frane
Righ Troms
one of
square or sports field in which people interact with one another, or a pool table with rolling and colliding balls. each of these environments, multiple instances of the same object class are present; all operate according to fundamentally similar dynamics. To ensure systematic modeling of the environment, the same dynamics must be applied to multiple object instances. Toward this goal, we propose method of separately representing the state of an individual object-via OF OF-and how its state evolves over time-via a schema.
Object-oriented programming (OOP) provides metaphor for thinking about the relationship between OFs and schemata. In OOP, each object an instantiation of an object class and it has a self-contained collection of variables whose values specific to that object and methods that operate all instances of the same class. The relation between objects and methods mirrors the relationship between OFS and schemata. In both OOP and view of visual cognition, key principle is the encapsulation of knowledge: internal details of objects (OFs) are hidden from other objects (OFs), and methods (schemata) accessible all and only objects (OFs) which they applicable.
The modularity of knowledge in OOP supports human programmers in writing code that is readily de- bugged, extended, and reused. We conjecture that the corresponding modularity of OFs and schemata will lead to neural-netwer models with more efficient learning and more robust generalization, thanks to appropriate disentangling and separation concerns.
Modularity is the guiding principle of the model we propose, which we call SCOFF, an acronym for schema object-file factorization. Like other neural net models with external memory (e.g.. Mozer and Das 1993 Graves et al.) 2016) Sukhbaatar et al. 2015). SCOFF includes a set of slots which are each designed to contain an OF (Figure 2). In contrast to most previous external memory models, the slots are not passive contents waiting to be read or written by active process, but are dynamic, modular elements that seek information in the environment that relevant the object they represent, and when critical information is observed, they update their states, possibly via information provided by other OFs. Event-based OOP is a good metaphor for this active process, where external events can trigger the action of objects.
As Figure suggests, there a factorization of declarative knowledge the location, properties, and history of an object, contained in the OFs-and procedural knowledge- the rules of object hehavior, as contained in the schemata. Whereas declarative knowledge can change rapidly, procedural knowledge is more stable over time. This factorization allows any schema be applied to any OF as
Figure 2: Proposed SCOFF model. Schemata are sets of parameters that specifyy the dynamics of ob- jects. Object files (OPs) are active modules that maintain the time- varying state of an object, seek information from the input, and select schemata for updating, and transmit information to other ob- ject files. Through spatial atten- tion, OFS compete and select dif- ferent regions of the input.
SCOFF
Scheemo
Fredi ctidcction
Objject file
Object Fic
Object file
Object
Spatial attention
CNN
2

---

Published as a conference paper at ICLR 2021
deemed appropriate. The model design ensures systematicity in the operation of a schema, regardless of the slot to which an OF is assigned. Similarly, an OF can access any applicable schema regardless of which slot it sits in. Furthermore, schema can be applied to multiple OFs at once, and multiple schemata could he applied to an OF (e.g., Figure In OOP, systematicity is similarly achieved by virtue of the fact that the same method can be applied to any object instantiation and that multiple methods exist which can be applied to an object of the appropriate type.
Our key contribution (the location, properties, and history of an object) and procedural knowledge (the way objects behave). This factorization enforces not only an important form of systematicity, but also of exchangeability: the model behaves exactly the same regardless of the assignment of schemata to schemata-slot and the assignment of objects to OF-slots, i.e., the neural network operates on a ser of objects and a set of schemata, With this factorization, we find improved accuracy of next-state prediction models and improved interpretability of learned parameters.
demonstrate the feasibility and benefit of factorizing declarative knowledge
2
THE SCHEMATA / OBJECT-FILE FACTORIZATION (SCOFF) MODEL
SCOFF, shown in Figure 2 is architectural backbone that supports the separation of procedural and declarative knowledge about dynamical entities (objects) in an input sequence. The input sequence {ar1..... .t.....; Dr), indexed by time step is processed by neural encoder (e.g., a fully convolutional net for images) to obtain a deep embedding, {21, žt, 2r), which then drives a network with n, OFs and n, schemata.
Ofs are active processing components that maintain and update their internal state. Essentially, an OF is layer of GRU (Chung et al., (014)) or LSTM (Hochreiter and Schmidhuber 1997) units with three additional bits of machinery, which we now describe.
1. Our earlier metaphor identifying OFs in SCOFF with objects in OOP is apropos in the sense that OFS are event driven. OFs operate in temporal loop, continuously awaiting relevant input signals. Relevance is determined by an attention-base soft competition among OFs: the current input serves a key that is matched to query generated from the state of each OF; based on the goodness of match, each OF is provided with a value extracted from the input.
2. Each OF performs one-step update of its state layer of GRU or LSTM units, conditioned on the input signal received. The weight parameters needed to perform this update, which we will denote generically as 9, are not as in standard GRU or LSTM internalaanl to the layer but rather are provided externally. Each schema j is nothing more than a set of parameters fj which can be plugged into this layer. SCOFF uses key-value attention mechanism to perform Gumbel-basee hard selection of the appropriate schema (parameters).
3. OFs may pass information to other OFs (analogous to arguments being passed to method in OOP), again using a soft attention mechanism by which each OF queries all other OFs. Keys provided by the other OFs are matched to the query, and soft selection of the best matching OFs determines the weighting of values transmitted by the other OFs.
This operation cycle ensures that OFs can update their state response to both the external input and the internal state comprised of all the OFs' contents. This updating is an extra wrapper around the ordinary update that takes place in a GRU or LSTM layer. It provides additional flexibility in that (1) The external input is routed to OFs contingent on their internal state, (2) OFS switch their dynamics from one time step to the next conditioned on their internal state, (3) OFS are modular in that they do not communicate with another except via state-dependent selective message passing.
OFS are placeholders in that particular OF has no weight parameter specific to that OF. Instead, the parameters are provided from two sources: either the schemata by the y OFs. This generic parameter pool is used to implement key-value attention over the input, the schemata, and communication among OFs. The sharing of schemata ensures systematicity; the sharing of the generic parameter pool ensures exchangeablity model behavior is unaffected by the assignment of object instances to OF slots.
pool of generic parameters shared
2.1
SCOFF SPECIFICS
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 SCOFF model
Input: Current sequence element, æ; and previous OF state, (A,-1IA 1al & nn))
Step Process image by position p with fully convolutional Cp [C:N
dp Cp epl
(concatenate encoding of position to CNN output)
Step 2: Soft competition among select regions of the input to process
q 1,k
sk.p softmaxx
((NN)) where Hip = ( where up W. V (1,...,,4}
Z; -E, Sp
Step 3: OFs pick the most relevant schema and update
hu.k. GRU@, 0, (zx, 11))..) wk E "ll...,,n)..... (I,...,n.)
q W
argmax, (कt Rx,, +7), where Rag (hea.) WoyT and Gumbel(0,
h hi.k.tt
Step 4: Soft competition among OFS to transmit relevant information to each OF
q h. W4
Vkell....,ny
Sk. softmaxx
GER
where Reg (Iu,, WOnT V K (I....,
vdg
he. +- he.k + Ev
Ow where Uy h1 W Wk E (l....:m4}
Algorithm provides precise specification of SCOFF broken into four steps. Step 1 is external to SCOFF and involves processing an image input to obtain a deep embedding. The processing is performed by fully Convolutional Neural Net (CNN) that preserves positional information (typically 64 x 64 in our simulations), and for each position encodes the processed input Cp and concatenates learned encoding of position, ep to produce a pusitioonsspccinn hidden state. (SCOFF can also handle non-visual inputs, such as vectors discrete tokens; in this case the CNN is replaced by an MLP.) The subsequent core steps are as follows.
Step 2: Soft competition among OFs to select regions of the input to process. The state of each OF k. h is used to form a query, qu. which determines the input positions that will attend to. The query is matched to set of position-specific input keys, Kk,p for position p. producing competition among the OFs results the
match score, Sk.p- Soft OFs selecting complementary image regions attend to. The contents of the attended positions combined yielding an OF-specific input encoding, z):-
Step 3: OFs pick the most relevant schema and update. OF l picks one schema via attention as follows. OF binds to each schema j. and then performs a hypothetical update, yielding h,, In experiments below, the OF state is maintained by a GRU layer, and schema j is parameterization of the GRU, denoted GRU 0, which determines the update. The previous state of the OF, h,- serves as a query in key-value attention against a key derived from the hypothetical updated state, he.k.j- The schema ig corresponding the best query-key match for OF i is used to update OF itss state. Selection is based on the traight-throup
Gumbel-softman method (Jang et al. (2166), which makes a hard choice during forward propagation and during backward propagation, considers a softened version of the output to permit gradient propagation to non-selected schemata.
Step 4: Soft competition among OFs to transmit relevant information to each OF. This step allows for interactions among OFs. Each OF & queries other OFs for information relevant to its update as follows. The OF's previous state, h is used to form query, qR. in an attention mechanism against key derived from the new state of each other OF K., Rg,, and softmax selection (Sk,k) is used to obtain weighted information from other OFs, V into OF kts state.
Number of Parameters. SCOFF can be used as drop-in replacement for LSTM/GRU layer. There a subtlety that must be considered for successful integration. If the total size of the hidden state is kept the same, integrating SCOFF dramatically reduces the total number of recurrent parameters in
4

---

Published as a conference paper at ICLR 2021
the model because of its block-sparse structure. The majority of SCOFF parameters are the schemata, {0, j E [L....,n.J) n. The remaining parameters are those of query (W W W key (W5, W W",, and value W W" functions. Note that these linear functions could be replaced by nonlinear functions. Its also interesting to note that SCOFF actually has far fewer parameters than other modular architectures (e.g., RIMs, Recurrent Entity Networks, Neural Module Networks) when ns < z (which holds true for most of our experiments), because of the potential one-to-many mapping between schemata and OFs. Optionally, at Step 2, during training instead of activating all the OFs, we can use a sparse attention, to only activate a subset of OFs that are relevant at that time step L, and only update the state of the activated OFs. We note that this is not specific to the proposed method, but the performance of the method can be improved by only selectively activating the relevant OFs. We also note that the different schemata in the proposed method are only used to update the state of OFsand hence is agnostic as to how one obtains set of OFs. Any other method for entity extraction (Burgess al., 2019 Greff al. 2019) can be used for obtaining a set of OFs. Similarly, other methods for decoding which preserves equivariance (Watters et al. 2019) among different OFscan be used for decoding the contents of different OFs.
3 RELATED WORK
CNNs. SCOFF applies the same knowledge (schemata) to multiple OFs, yielding systematicity. Similarly, a CNN is a highly restrictive instantiation of this same notion, where knowledge- the form of a convolutional filter-is applied uniformly to every location in an image, yielding equivariance. SCOFF is a more flexible architecture than a CNN in that OFs are defined by abstract notion of objects not a physical patch of an image, and schemata are applied dynamically and flexibly over time, not in a fixed, rigid manner the filters in CNN.
Memory Networks. A variety existing models leverage an external slot-based, aontent-addressible memory (e.g.. Graves et al. 2016 Sukhbaatar et al. 2013). The memory slots are passive elements that are processed by a differentiable neural controller such as an LSTM. Whereas traditional memory networks have many dumb memory cells and one smart controller. SCOFF has many smart memory cells-the OFş- whsch also act as local controllers. However, SCOFF shares the notion with memory networks that the same knowledge is applied systematically every cell.
Relational RNN (RMC). The RMC (Santoro et al. 2018) has a multi-head attention mechanism which allows it to share information between multiple memory locations. RMC is like Memory Networks in that dynamics are driven by a central controller: memory is used to condition the dynamics of RNN.
Recurrent Entity Networks. Henaff et al, 2016) describe a collection of recurrent modules that update independently and in parallel in response each input in sequence. The module outputs are integrated to form a response. It shares modular architecture with SCOFF, but the modules have a fixed function and do not directly communicate with one another. This earlier work focused on language not images.
Recurrent Independent Mechanisms (RIMs). RIMs (Goyal et al., 2019) are a key inspiration for our work. RIMs are a modular neural architecture consisting of an ensemble of dynamical components which have sparse interactions through the bottleneck of attention. Each RIM module has at its core an LSTM layer (Hochreiter and Schmidhuber 1997). A RIM module is much like our OF. Both are meant to be dynamical entities with a time-evolving ving state. However, in contrast to Ors in SCOFF, RIM modules operate according fixed dynamics and each RIM module is specialized for a particular computation. RIM modules are thus not interchangeable.
Neural Module Networks. A modular module network (Jacobs et al. 1991 Bottou and Gallinari 1991 Ronco et al. 1997 Reed and De Freitas
et al. 2016 Rosenbaum et al. [017]] Fernando et al.] 2017] Shazeer al. 2017 Kirsch al.] 2018 Rosenbaum et al.) (019)) has an architecture which is composed dynamically from several neural modules, where each module is meant to perform a distinct function. Each module has fixed function, unlike our Ors, and modules are applied one at a time, in contrast to our OFS, which can update and be used for prediction in parallel.
5

---

Published as a conference paper at ICLR 2021
4 METHODOLOGY
SCOFF is drop-in replacement for standard LSTM or GRU layer, conforming to the same input- output interface. Because of their interchangeabiliyy we compare SCOFF to LSTM and GRUs. We also compare SCOFF to two alternative modular architectures: RMC, memory based relational recurrent model with attention between memory elements and hidden states (Santoro et al. 2018), and Recurrent Independent Mechanisms (RIMs), a modular memory based single layered recurrent model with attention modulated input and communication between modules (Goyya et al. 2019).
In all simulations present, the input is video sequence, each frame of which is preprocessed by a CNN backbone. We consider two task types: video prediction and reinforcement learning (RL). For video prediction, the simulation output is prediction of the next frame in the sequence. For RI, the output is an action choice. In both cases, SCOFF's internal state is mapped to an output in the following manner. The state of each OF is remapped by siamese network that transforms the state. (By siamese network, we mean that every OF is remapped by the same function.) The transformed state is concatenated and used by an attention-basee mechanism operating over the OF slots to ensure exchangeability of the OFs. In the case of video prediction, a deconvolutional backbone yields an image as output; in the case of reinforcement learning, the output is a distribution over actions.
The heart of the model is single SCOFF layer, consisting of n OFs and n, schemata. Most simulation details are contained in section A of the Appendix, but we summarize some key points here. Unless otherwise indicated, optimizer with a learning rate of 0.0001 and momentum of 0.9. As a default, we use ny = 6 and
always train in end-to-end fashing using the Adam (Kingma and Ba 2014)
n, = 4, except where are specifically exploring the effects of manipulating these hyperparameters. We include more experimental results in the Appendix, and we will release the code.
5 EXPERIMENTS
Our experiments address the following questions. (1) Does SCOFF successfully factorize knowledge into OFs and schemata? (2) Do the learned schemata have semantically meaningful interpretations? (3) Is the factorization of knowledge into object files and schemata helpful in downstream tasks? (4) Does SCOFF outperform state-of-the-art approaches, both modular and non-modular, which lack SCOFF's strong inductive bias toward systematicity and knowledge factorization?
We begin with a series of experiments involving greyscale synthetic video sequences consisting of a single ball-like object moving over time according switchable dynamies. We model this scenario with SCOFF having a single (nz = 1) OF and the number of schemata matching the number of different types of dynamics.
Single object with fixed dynamics. To demonstrate that the proposed model able to factorize knowledge, consider video scenes in which a single object, starting in random location, has dynamics that cause it to either (a) accelerate in a particular direction, (b) move at a constant velocity in particular direction, (c) take random walk with a constant velocity. Following training, SCOFF can predict trajectories after being shown the first few frames. It does so by activating a schema that has one-to-one correspondence with the three types of dynamics (Figure 3, left panel), leading
T-o Twd
TH5 Tu? T
Burin T-O
T-7
T= T-a T-oT=7
1.0
Timesteps along trajectory
Schemeta
Schermada
0.50
Acceleration
Constant Velocity
Random walk
0.0
T-0
T-5
T-10
T:15
Figure 3: [left panel) Single object sequences with three possible dynamics of motion. After about five video frames (burn-in period), SCOFF locks into the type of motion and activates a corresponding schema to predict future states. Relative activation of the three schemata indicated by the color bar. The selected schema indicated by the faint blue border. [right panel] RL Maze task. The agent learns to navigate a maze of randomly interconnected rooms to find key. Array indicates schema activation over sequence of steps along a particular trajectory.

---

Published as a conference paper at ICLR 2021
Horizontal dynamics (correlates with right visual marker)
Two mar markers
Figure 4: Switching dynamics task. Sequence of steps showing horizontal (top row) and vertical (bottom row) dynamics of a ball. Frames contain visual markers that indicate the cur- rent dynamics.
Vertical dynamics (correlates with left visual marker)
to interpretable semantics and a clean factorization of knowledge. For additional details, refer to Appendix C
Single object with switching dynamics. In this experiment, video scenes contain single object, starting in a random location, operating according to one of two dynamics: vertical and horizontal motion. In contrast the previous experiment, the object can switch dynamics in the course of a sequence. The current dynamics are indicated by markers in the image (see Figure 4 which SCOFF must learn detect in order select the corresponding schema. In this experiment, we find that SCOFF is able to learn a one-to-one correspondence with two types of dynamics, yielding nigh-precision prediction of the next frame, even on the first frame after switch.
Single object with switching dynamics in an RL paradigm. In this experiment, we use the partially- observable GotoObjMazze environment from Chevalier- Boisvert et al. (2018), in which an agent must learn to navigate to navigate a 2D multi-room grid world to locate distinct objects such as a key. The world consists of 6 x 6 square rooms that are randomly connected through doors to form 3 x 3 maze of rooms. The agent's 's view of the world is an egocentric 5 5 patch. Our experiment involves nl = 4 schemata whose activation pattern over time steps is shown the right panel of Figure 3 The schema activation pattern is interpretable; for example. schema 4 is triggered when the 'key is in the agent's ' field of view (see Appendix, Figure [3 and Section D). To obtain quantitative evaluations, we test transfer by increasing the room size to IO x 10 during testing. SCOFF is able to successfully reach its goal on 82% of trials, whereas a GRU baseline succeeds only 56% of trials. Even overparameterized GRU, which matches SCOFF in number of free parameters, still succeeds less often-on 74% of trials, All models are trained for an equal number of time steps. To summarize, the factorization of knowledge in SCOFF leads not only better next-step prediction but also improve performance on downstream control task.
Multiple objects with multiple dynamics. We now turn from single-object sequences to sequences involving multiple objects which operate according complex dynamics. We consider a bouncing-be balls environment in which multiple balls move with billiard-ball dynamies (Van Steenkiste et al.) 2018). The dataset consists of 50,000 training examples and 10,000 test examples showing ~50 frames of either solid same-color balls bouncing in a confined square geometry with different masses corresponding their radii (4Balls), 6-8 same-color balls bouncing in confined geometry (678Balls), 3 same-color balls bouncing in a confined geometry with a central occluder (Curtain), or balls of four different colors (Colored 678Balls). Although each ball has distinct state (position, velocity, and possibly color), they share the same underlying dynamics of motion and collision. We expected that SCOFF would learn to dissect these dynamics, e.g., by treating collisions, straight-line motion, interactions with the walls, and moving behind an occluder as distinct schemata. We use the encoder and decoder architecture of (Van Steenkiste al., 2018) for all models. SCOFF has ny 4 for 4Balls and Curtain, and n = 8 for 678Balls and Coloured678Balle and n. 4 for all simulations. All models are trained for 100 epochs. As shown in Figure 5 SCOFF achieves dramatic improvements in successfully predicting 10- and 30-step iterated predictions relative to GRUs, and RIMs. (For further details, see Appendix Section C) We omitted RMC (Santoro et al. 2018 from FiguresBl-d because RIMs performs strictly better than RMC in these environments (Goyal et al. 2019), but have included RMC in the Appendix.
Increasing the number of schemata. Our previous experiment used 4 for all simulations. In order to study what happens if SCOFF has access to a large number of schemata, we perform an experiment with nt, = 10. For the curtain task, we found that only three schemata were being used, and performance was about the same as when training with n, = 3 or n, = 4. For 678Balls, we found that by the end of training only four schemas were being used, and performance is about the same as when training with n, = 4. Thus, providing SCOFF with excess resources leads to some not being used which is waste of computation and memory but does not lead suboptimal performance.
7

---

Published as a conference paper at ICLR 2021
steps stops
stops
Stops steps
steps steps
1.0
BCE (Relatve GRU baselin
0.0
0.8
0.6
0.6
0.2
5.22
0.0
GRU
RIMs
SCOFF
GAU
RIM
SCDFF
0.0
GRU
RIN
SCOFF
GRU
RIMs
SCOFF
(a) 4Balls
(b) Curtain
staps 10 stops
steps 30 steps
(Relative to GRU baraiiee
CRU baceline))
1.0
0.6
0.4
0.4
0.2
0..2
RIMs
SCOFF
GRU
GRU
RIMs
SCOFF
(c) 678Balls
(d) Coloured678Balls
Figure 5: Bouncing ball motion. Error relative to GRU baseline on 10- and 30-frame video prediction of multiple-ohject video sequences Predictions based on 15 frames of ground truth. The advantage of SCOFF is amplified as the number of balls increases (4Balls versus 678Balls) and as predictions require greater look ahead (10 versus 30 frames).
Block O1 (Disappear)
Block 02 (Shape Change)
Block 03 (Teleport)
RMC (Santoro et al. 2018
0.43 0.05
0.39 + 0.03
0.46 + 0.02
Intphys (Riochel al. 2019
0.52
0.52
0.51
SCOFF (ours)
0.35
0.42 + 0.02
Table 1: Results on the IntPhys benchmark. Relative classification error of unrealistic physical phenomena (Riochet et al., 2019) for three models, demonstrating benefits of SCOFF scenes with significant occlusions ("Occluded"). This particular benchmark has three subsets, and for our experiments we evaluate the proposed model on the "occlusion" subset the task. The three columns correspond to different types of occlusions. Lower is better. Average taken over 3 random seeds.
For example, one might have been concerned about overfitting with too many resources. However, with n, 3 schemata, difficulty using the resources we provide to it. That is, there are no 'dead' schemata that fail to be trained due to local optima.
observe unused schemata, suggesting that the model does not have
Modeling physical laws in a multi-ohject environment. Modeling physical system, such as objects in world obeying laws of gravity and momentum, requires factorization of state (object position, velocity) and dynamics. All objects must obey the same laws while each object must maintain its distinct state. We used the Intuitive Physics Benchmark (Riochet al.) (219)) which balls roll behind brick wall such that they are brieily occluded. The training data is constructed such that the examples all physically realistic. The test set consists of both sequences that behave according to the laws used to synthesize the training data and sequences that follow unrealistic physical laws. test three forms of unrealistic physics: balls disappearing behind the wall (O1 task), balls having their shape change for reason (02 task), and balls teleporting (03 task). The Benchmark has three subsets of experiments, and we chose the challenging subset with significant occlusions. We trained models to perform next state prediction on the training and jet al. 2019) to discriminate between realistic and unnatural test sequences. Further details are in Appendix As Table indicates, SCOFF significantly outperforms competitors all three tasks.
model likelihood (Riochet
8

---

Published as a conference paper at ICLR 2021
6 CONCLUSIONS
Understanding the visual world requires interpreting images in terms distinct independent physical entities. These entities have persistent intrinsic properties, such as color or velocity, and they have dynamics that transform the properties. We explored a mechanism that is able to factorize declarative knowledge (the properties) and procedural knowledge (the dynamics). Using attention, our SCOFF model learns this factorization into representations of entities OFFis and representations of how they transform over time-scheman By applying the same schemata to multiple OFs, SCOFF achieves systematicity of prediction, resulting in significantly improved generalization performance over state-of-the-ie methods. also addresses a fundamental issue in and cognitive science: the distinction between types and tokens. SCOFF is also interpretable, in that can identify the binding between schemata and entity behavior. The factorization declarative and procedural knowledge has broad applicability to wide variety of challenging deep learning prediction tasks.
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48, 2016.
Léon Bottou and Patrick Gallinari. framework for the cooperation of learning algorithms. In Advances in neural information processing systems, pages 781-788, 1991.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: /901. 13300, 2019.
Maxime Bhovaller-Boisverr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with a human in the loop. arXiv preprint arxiv:/810.08222,, 2018.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv:1412.3555. 2014.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 1701008734 2017.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprinl arXiv:1909. 10893, 2019.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 478771)))441,, 2016.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference Machine Learning, pages 2424-2433. PMLR, 2019.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. arXiv preprint arXiv:1612.03966 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, Geoffrey E Hinton, et al. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 161 I. 1.01 144, 2016.

---

Published as a conference paper at ICLR 2021
D. Kahneman, A. Treisman, and B. Gibbs. The reviewing of object files: object-specific integration of information. Cognitive psychology, 24(2):175-219, 1992.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 6980, 2014.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning decompose neural computation. In Advances in Neural Information Processing Systems, pages 2408-200222 2008-241,, 2018.
Michael C Mozer and Sreerupa Das. A connectionist symbol manipulator that discovers the structure of context-free languages. In Hanson, D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages 863 870. Morgan-Kaufmani 1993.
Scott Reed and Nando De Freitas. Neural programmerr-nneeppeeen.. arXiv preprint 1rXiv:1511.0622 2015.
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A benchmark for visual intuitive physics reasoning. 2019.
Eric Ronco, Henrik Gollee, and Peter Gawthrop. Modular neural networks and self-decomposition. Technical Report CSC-96012, 1997.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint 1711v.1711.012399 2017.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation, arXiv preprint arXiv: /2774, 2019.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals. Razvan Pascanu, and Timothy P. Lillicrap. Relational recurrent neural networks. CoRR, 0sss/1806.01222, 2018. URLhttp: arxiv. wwtg:////////////ee/ 01822
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.065333 2017.
Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2440-2448. Curran Associates, Inc., 2015. URL htttp://papes
pddf
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv: 2018.
Nicholas Watters, Loie Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.00017, 2019.
10

---

Published as a conference paper at ICLR 2021
Appendix
Table of Contents
Implementation Details and Hyperparameters
11
B Adding Task
11
Bouncing Ball
D BabyAl: Reinforcement Learning
12
E Intuitive Physics
13
F
Switching Dynamics
13
Experiment Setup
f22 The factorisation of dynamics into schemata
15
G Pseudocode for SCOFF Algorithm
24
A IMPLEMENTATION DETAILS AND HYPERPARAMETERS
The model setup consists of three main components: an encoder, the process of interaction between object files and schemata followed by decoder. The images are first processed by an encoder, which is parameterized as a CNN. For experiments containing multiple entities, we use spatial attention (i.e., output of the CNN preserves the spatial information) such that different OFs can attend to different spatial regions.
Resources Used. It takes about days train the proposed model on bouncing ball task for 100 epochs on V100 (32G). We did not do any hyper-parameter search specific to a particular dataset (i.e 4Balls or 678Balls or Curtain Task). We ran the proposed model for different number of schemata (i.e 2/4/6). Similarly, it takes about 3 days run for 20M steps for the Reinforcement learning task.
B ADDING TASK
We analyzed the proposed method the adding task. This is standard task for investigating recurrent models (Hochreiter and Schmidhuber) 1997). The input consists of two co-occuring sequences: 1) N numbers (ag aN--av111- -1) sampled independently from U 1], index to in the first half of the sequence, and an index i in the second half the sequence together encoder as a one hot sequences. The target output is a, a,, As shown in figure (a), factorisation of procedural knowledge into two schemata effectively, one schema is triggered when operand is encountered and the other when non-operand encountered.
clearly observe the
For demonstrating the generalization capability of SCOFF, we consider a scenario where the models trained to add mixture of two and four numbers from sequences of length 50. They are evaluated on adding variable number (2-10) numbers on 200 length sequences. As shown in Table 1, note better generalization when using SCOFF. The dataset consists of 50,000 training sequences and 20,000 testing sequences for each different number numbers to add.
Generalization Result:
All the models trained for 100 epochs with learning rate of 0.001 using the Adam optimizer. We use 300 as the hidden dimension for both the LSTM bascline and the LSTM's in RIMS, sCOFF. Table 2 lists the different hyperparameters used for training SCOFF.
11

---

Published as a conference paper at ICLR 2021
Table 2: Hyperparameters for the adding generalization task
Parameter
Value
Number of object files (12)
Number of schemata (n,)
Optimizer
Adam Kingma and Ba
2014
learning rate
10
batch size
64
Inp keys
64
Inp Values
60
Inp Heads
Inp Dropout
0.1
Comm keys
Comm Values
Comm heads
Comm Dropout
0.1
Number of Values
LSTM
RIMS
SCOFF
2
0.8731
0.0007
0.0005
3
1.3017
0.0009
0.0007
4
1.6789
0.0014
0.0013
2.0334
0.0045
0.0030
8
4.8872
0.0555
0.0191
9
7.3730
0.1958
0,0379
10
11.3595
0.8904
0.0539
Table 3: Adding Task: Mean test set error on 200 length sequences with number of numbers to add varying among {2, 3, 4, 5, 8, 9, 10). The models are trained add mixture of two and four numbers from sequences of length 50.
C BOUNCING BALL
The dataset consists of 50,000 training examples and 10,000 test examples showing ~50 frames of either solid balls bouncing a confined square geometry (4Balls), 6-8 balls bouncing in a confined geometry (678Balls). 3 balls bouncing in confined geometry with occluded region (Curtain), or balls of different colors (Colored 678Balls), The balls have distinct states (and hence distinct object files) but share underlying procedures (schema), which aim capture using SCOFF.
We trained baselines as well as proposed model for about 100 epochs. We also provide the rollouts predicted by the models in the Figures 8 I0
I2
As part future work, We also provide scenario where instead of activating all the OF only a subset of the OF that are most relevant the input based attention scores activated. We denote the original model as SCOFF-ab, and the ablation of only activating a subset of the OF as SCOFF.
Comparison to the RMC baseline is shown in fig. 7
D BABYAI: REINFORCEMENT LEARNING
We use the GotoObjMaze environment (i.e ManiGrid-GotoObjMaaaa ctt al., 2018). Here, the agent has to navigate to an object, the object may be in another room. We exactly the same RL setup as in Seovallir-Boosved et al., 2018) except extend the setup in BabyAl to only apply RGB images of the world rather than symbolic state representations, and hence making the task much more difficult.
from (Chevalier-Boisve) Boisvert
for this task are listed in Tab. 5
In this environment, the agent is expected to navigate a 3x3 maze of 6x6 rooms, randomly inter- connected by doors to find an object like "key". Here use only object file, but different number
12

---

Published as a conference paper at ICLR 2021
1.0
Apadender
9.5
Fondor Walk
2.20
Schernsis (iven)) Objec (-aés)
Indies File
Schemata (X-axis) vs Object Files (Y-axis)
(a)
(b)
Figure 6: (a) OF (ny 4) Schemata (n, 2) activation for example of length 8 of the adding task. "Null" refers to the elements other than the operands on which the addition is to be performed. use a particular schema. Each row corresponds
The figure shows the affinity of each particular OF, and column represents particular schema (dark color shows high affinity of an OF toward particular schema). As shown in the figure, the active OFs trigger Schema when an operand is encountered, and Schema 2 when "Null" element is encountered. (b) Here, we have single OF, and that can follow three different dynamics. We found that our method able to learn these different modes once it's passed an initial phase of uncertainty.
steps 10 steps
steps 30 steps
BCE (Relative to GRU baseline)
1.0
0.8
0.6
0.2
0.0
GRU
RMC
RIMs
SCOFF
GRU
RMC
RIMs
SCOFF
(a) Coloured678Balls
Figure Bouncing ball motion. Error relative to GRU baseline on 10- and 30-frame video prediction of multiple-object video sequences Predictions are based on 15 frames of ground truth.
of schemas in this example). If look the object files (vs) schemata, schemata 4 is being triggered when the "key" is in agent's view as shown in fig. 13
E INTUITIVE PHYSICS
We the similar training setup (Riochet al. 2019). Hyper-parameters related to the proposed method are listed Tab. G
SWITCHING DYNAMICS
We consider a scenario where a ball follows of the two dynamics, horizontal and vertical oscillations at a given time. We limit the number of switches between the dynamics followed to be another experiment, where
one. An example ground truth trajectory is given in figure [4] Here we have more number of OFs as compared to number of entities or objects, and then we investigate still able have factorization of different dynamics in different schemata.
13

---

Published as a conference paper at ICLR 2021
Table 4: Hyperparameters for the bouncing balls task
Parameter
Value
Number of object files (nz)
4
Number schemata (n,))
2/4/6
Size Hidden state of object file Optimizer
100
Adam (ingma and Ba
2014)
learning rate
I0
batch size
64
Inp keys
64
Inp Values
100
Inp Heads
1
Inp Dropout
0.1
Comm keys
Comm Values
32
Comm heads
4
Comm Dropout
0.1
Table 5: Hyperparameters for BabyAl
Parameter
Value
Number object files (nz)
Number schemata (n.)
2/4/6
Size of Hidden state of object file
510
Optimizer
Adam/Kingmi and Ba
2014
Learning
3
10
Inp keys
64
Inp Values
256
Inp Heads
4
Inp Dropout
0.1
Comm keys
16
Comm Values
32
Comm heads
4
Comm Dropout
0.1
F.1 EXPERIMENT SETUP
The dataset consists of 10,000 trajectories of 51 length with the switching between dynamics happening at the middle of the trajectory.
Table Hyperparameters for IntPhys benchmark
Parameter
Value
Number of object files (ny) Number schemata (n,)
4/6
Optimizer
Adam Adam(Kingma and Ba
2014)
learning rate
batch size
64
Inp keys
64
Inp Values
Inp Heads
Inp Dropout
0.1
Comm keys
Comm Values
32
Comm heads
4
Comm Dropout
0.1
14

---

Published as a conference paper at ICLR 2021
We use the same architecture for encoder well decoder as in (Van Steenkiste et al. 2018 At each time step, we give the last five frames stacked across the channels as input to the encoder.
Table 7: Hyperparameters for the switching dynamics task
Parameter
Value
Number object files (ny) Number of schemata (n,)
Optimizer
Adam Kingma and Ba, 2014)
learning rate
10
batch size
50
Inp keys
Inp Values
Inp Heads
Inp Dropout
0.1
Comm keys
32
Comm Values
32
Comm heads
4
Comm Dropout
0.1
F.2 THE FACTORISATION OF DYNAMICS INTO SCHEMATA
As shown in figure 13 SCOFF is effective in factorising the different dynamics into different schemata, even if the order of dynamics followed is different.
15

---

Published as a conference paper at ICLR 2021
Figure Rollout for 4Balls. In all cases, the first frames of ground truth are fed in (last shown) and then the system is rolled out for the next 30 time steps. In the predictions, the transparent blue shows the ground truth, overlaid to help guide the eye.
16

---

Published as a conference paper at ICLR 2021
Figure 9: Rollout for Curtain. cases, the first 10 frames of ground truth are fed in (last shown) and then the system is rolled out for the next 30 time steps. In the predictions, the transparent blue shows the ground truth, overlaid to help guide the eye.
17

---

Published as a conference paper at ICLR 2021
Figure 10: Rollout for 678Balls. In all cases, the first 10 frames of ground truth are fed in (last shown) and then the system is rolled out for the next 30 time steps. In the predictions, the transparent blue shows the ground truth, overlaid to help guide the eye.
18

---

Published as a conference paper at ICLR 2021
111 11A
200
JOTH md
Figure I1: Rollout for Colored 4Balls. In all cases, the first 10 frames of ground truth are fed in (last shown) and then the system is rolled out for the next 25 time steps.
19

---

Published as a conference paper at ICLR 2021
200
Figure 12: Rollout for Colored 678Balls. In all cases, the first 10 frames of ground truth are fed in (last 6 shown) and then the system is rolled out for the next 25 time steps.
20

---

Published as a conference paper at ICLR 2021
Timesteps along trajectory
Schemnaa
T=0
T=5
T=10
(a) Object Files (ny (vs) Schemata (n, 4) Activation
twi
t-5
6
tw7
1-8
t-10
[-11
t-12
t-13
tw14
t=15
t-16
(b) Agent view in environment
Figure BabyAllGGooObjaazze Trajectory In environment, the agent is expected to navigate 3x3 maze of 6x6 rooms, randomly inter-connected by doors to find object like "key". Here use only one object file, but different number of schemata (4 this example). we look at the object files (vs) schemata affinity, schema is activated while close to opening doors while schema is triggered when the "key" is the agent's view.
21

---

Published as a conference paper at ICLR 2021
T=1-10
T=10-20
T*20-30
T=30-40
T=40-50
T=50-60
Figure 14: Switching dynamics task. An example ground truth trajectory, where the ball oscillates horizontally and switches to vertical oscillations after few steps indicated by the red box. The bottom left has two bulb indicators corresponding to the two dynamics.
22

---

Published as a conference paper at ICLR 2021
1.00
0.75
0.50
0.25
0.00
(a) Object Files (ny = 6) (vs) Schemata (n.,
for trajectory one.
1.00
0.75
0.50
0.25
0.00
(b) Object Files (ny 6) (vs) Schemata (n, for trajectory two.
Figure 15: Switching dynamics task. 0 denotes horizontal oscillations and / denotes vertical oscillations. We can clearly observe that the dynamics are being factorised into separate schemata. Schemata one is being used for vertical oscillations and schemata for horizontal oscillations.
23

---

Published as a conference paper at ICLR 2021
G PSEUDOCODE FOR SCOFF ALGORITHM
#Assume we have defined: in (number of positions in the incoming input), nun_OF_out (number of object files on the current layer), of_dim (the number of hidden units in each object file), baz (the batch size).
#Step 1: Process image with fuily convoluticnal net.
#Step 2: Soft competition among OFs to select regions of the input
#Step 3: OFs pick the most relevent schema and update
#Step 4: Soft competition among OFs to transmit relevant information def scofi_core (inp, h)
inp self. (in()
att_inp - object le_ iesttsp__seleeeee settieitloonn (inp, h)
senettschem__iiiiii - seleemattaselettieee___eeee update (att_inp, h)
nent h_next + cetertfile_commmiicatiiin nexnext, schema indicee)
return h_next
def bnstectfile_inpuu..... input_selection (inp, h)
inp inp reshape ( (baz, pos_in, CF (F_dim))
at at-_inp,- self. inp_attn (h. (h. reshape (bsz, nun_oF_out, OF OF_dim)), inp, inp) att_inp act_inp reshape (bs7, of of_c@menun_oF_out)
return att_inp
def sechemttaaseeeeciiee..eeeedd
h)
h_all - torch. barchh2eros. zeros (bsz*nur num_schemata, OF_dim))
#We batch the rest of these operations over the object files. inp_to_objfile
hap_to__bjfilee ressaaeeersshppee reshape (bsz*num_Of, OF ) reshapp (bsz*num__F, CF_dim))
for range (num (num_schemataa)
b_all[:,d] seif.gru (inp_to_objfile h)
keyl:, 11 - self. (h_all [:,]])
q query (h)
h sel,, ind soaebel_ssfttaxttttteo
(key, q. unsqueeze (1)))
return h_sel. (baz, num -1), ind, reshape (bzs, (um__FF,-1)
def cejjecffileeccmmnicaaiinnn (h,
h = b.reshape (bsz, num_OF_out, of_dim))
g self.comm colf.coom_query (h) [;, schemaanndicces)]
felf.coon_key (h) [;, schema_indi ces]
v sell..com_vaalie value (h) [: schema
indiindiies schema_indi
h_ttt,_ = self. comm.commmcc..c wwwooo_ttttttooooooe (q, k, v)
hLatt = hLatt. reshape (bsz, num_oF_oute +of_dim)
return h_att
Figure 16: The core SCOFF module for single recurrent step using GRU independent dynamics. We show how the attention scores from attention are used re-weight the input the different object files (Step 2 in the box). Then different object files select particular schemata (Step in algorithm), and then different object files transmit relevant information (Step algorithm).
24


Reviews:
[{"review_id": "wGPf00QSi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Brief summary of your review:\n\nI like the main idea of having an “active memory” where each slot can choose which operation to perform. It is similar to the notion of variables and functions, which should be advantageous for systematic generalization, which is one of the most significant issues of current neural networks.\n\nThe paper focuses on the visual domain and is motivated by extracting objects and their behavior. However, the model does not use attention over the input. The only way that the proposed model can use different object files for different instances of the same object type is to have separate schemata (because all other weights are shared), which defeats the purpose.\n\nThus, sadly, I must reject the paper.\n\nReview:\n\nThe authors focus on a very important question of current neural networks: systematic generalization. Their method is interesting: they factorize modeling the scene of objects into object files (memory slots) and schemata (different learned operations). This is related to but different from memory networks, which use a single controller (instead of the multiple schemata), and also to routing networks, which use a single state (instead of multiple object files) and multiple sets of weights describing the different operations.\n\nThe authors introduce their method as a way of modeling multiple objects, some of which may share behavior. This can be seen clearly from Figure 1, but also the tone of the whole paper is organized around this goal. However, the way the input image is handled makes this very implausible. Specifically, the problem is in step 2 of Algorithm 1:\n\nThe object files compete for the input. In terms of the Pacman example, this implies that multiple moving ghosts, each with its own OF, would have to compete with each other to see the input, which makes no sense: all of them should see the change in their respective object.\n\nThe other, perhaps even more significant issue regarding input handling is that there is no attention over the input image. I cannot see how this architecture can focus on a different subset of the input corresponding to different objects. To focus on objects separately, they necessarily have to have different schemata, because the rest of the weights are shared between OF. That is the only way of having a different transformation of the input (assuming the same initial hidden state). This defeats the purpose of schemata, which is to reuse computation when possible. The current setup would make more sense in other input domains, for example for textual input, where different “objects” are not represented by different subsets of a single input z_t.\n\nIn Figure 3 the authors analyze which schema the model uses for a single object slot. The result is very nice and it shows that the model learns to use different schema for different types of motion/rooms. However, especially because of the aforementioned issues, it is unclear how the model uses the object files. I would like to see a plot similar to Figure 3, just showing which OF is used. The experiment could be to add/remove objects, and see the number of object files, or to corrupt specific files and see which object becomes unpredictable, thus identifying which OF corresponds to which object. \n\nI still consider the paper interesting. But the issue above must be fixed, e.g., by changing the tone of the paper and shifting away from focusing on objects or switching the attention in stage 2 to attend to image regions and not over the object files.\n\nMore minor issues/questions:\n\nIn step 3, the attention is based on comparing the updated state to the old one. Why does this make sense? Is there an intuition behind this? Wouldn’t it make more sense to have fixed keys identifying the schemata?\n\nIn step 4, why do the queries use state from the previous time step as opposed to the current?\n\nOn page 4, step 2, $\\kappa_k$ is mentioned, however, it should be $\\kappa_t$, as $\\kappa$ is independent of the object file.\n\nOn page 4, saying that Gumbel softmax “softens the output” is misleading. The goal is to have a hard, but differentiable output instead of a soft one. \n\nIn the related work section, you compare to CNNs. The same argument of weight sharing can also work for RNNs. However, differently from SCOFF, CNNs can actually “attend” to different parts of the input image, thus in theory focus on different objects differently.\n\nRelated work: Routing networks are [1,2,3] are also related. Page 2 and 4: GRU is a variant of vanilla LSTM with forget gates (Gers et al, 2000).\n\nOn page 5, the last paragraph, it is described how the output is produced from the OFs. However, it is not clear how this attention works. What is the query? Is it a transformed state? If the transformed stats are concatenated to form the query, then it is not symmetric anymore, so it will not “ensure exchangeability of OFs”. It can just focus on a single OF, based on its index.\n\nIn the experiments section, page 6, for the “Single object with switching dynamics” why does the network need markers? Can’t it infer the schema to use just from the motion itself?\n\nPage 6, last paragraph, it should refer to Figure 3 instead of Figure 4.\n\nOn page 7, “Multiple objects with multiple dynamics”, the authors use the dataset proposed by “Van Steenkiste et al.”, and claim that they outperform their baselines. But they do not compare to the original method proposed in the same work. How does it compare?\n\nCould the 1.3 lines long appendix E be a footnote instead?\n\n[1] Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning\n\n[2] Kirsch et al: Modular Networks: Learning to Decompose Neural Computation\n\n[3] Chang et al: Automatically Composing Representation Transformations as a Means for Generalization\n\n******************\n\nAfter rebuttal:\n\nGlad to see this was just an error in the decription of the algorithm! Score increased from 4 to 7. We'd even increase our score to 8 if the authors added an analysis similar to the one of Figure 3, just showing which OF is used. As stated in the original review: The experiment could be to add/remove objects, and observe the number of object files, or to corrupt specific files and see which object becomes unpredictable, thus identifying which OF corresponds to which object.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "elLyEavOhV9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose SCOFF, a novel architectural motif, one with memory, which, as they describe, can serve as a drop-in for an LSTM or GRU within any architecture. It is inspired by the notion that when modeling a structured, dynamic environment (such as one with objects moving around), one must keep track of both declarative knowledge and procedural knowledge. They propose that these two types of knowledge be factored, creating an architecture consisting of \"object files\" (OF) whose evolution is governed by input, all objects, and  \"schemata\" which can be selectively applied to each OF.\n\nThey evaluate SCOFF along several axes:\n(1) Does SCOFF successfully factorize knowledge into OFs and schemata? \n(2) Do the learned schemata have semantically meaningful interpretations?\n(3) Is the factorization of knowledge into object files and schemata helpful in downstream tasks?\n(4) Does SCOFF outperform state-of-the-art approaches?\nTo do this, they use several video prediction tasks as well as an RL task.\n\nThis is a very interesting paper, with a natural, novel motif. It is well-written -- the motif has several important attributes, and one can quickly come to understand them (though perhaps a more involved diagram, like Figure 2 but showing what parameters come into play where, might be useful). The experiments appear to be carefully done, with considerable effort, and they put forth interesting evidence towards an affirmative in each of the above four questions.\n\nWhile the evidence put forth is useful, I do think there is considerable follow-up work needed to really demonstrate the efficacy of this system. In the realm of video prediction, the tasks considered are fairly simple, and certainly what is of true interest is video prediction much closer to the real world. With this, there are a wide variety of techniques and benchmarks (https://arxiv.org/abs/1804.01523 and its follow-ons come to mind as useful to quickly try). Much recent work has deferred the task of obtaining a reasonable encoding from/decoding to real-world images and assumed it in order to make progress on predicting the future within a fixed encoding (https://arxiv.org/abs/1612.00222, https://arxiv.org/abs/2002.09405, https://arxiv.org/abs/1806.08047) and have with them benchmarks to which the authors' method could be adapted. Certainly for some of these, SCOFF could be a subcomponent that augments these methods. I think these more complex future predictive tasks could better stress-test the structure of how information passes through SCOFF. Related, many of the experiments rely on the encoding/decoding provided by [Van Steenkiste 2018], which showed considerable success on the environments used, and it would be interesting to see how crucial that is.\n\nI recommend acceptance. The paper is interesting, well-written, and the experiments are useful. While I look forward to more definitive demonstrations of the utility of this approach, I do think that the amount done is considerable and warrants publication, and these important follow-ups would take a great deal more effort and are for future work.\n\nA more minor comment, I think more details could be given for the RL task, both in model implementation and in exactly how the test task is specified -- apologies if I missed but I only see train environment details in the appendix.\n\nMinor, wrong \"two\"/\"to\" in \"Single object with switching dynamics\" experiments description.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YTn7URCdM0g", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new type of recurrent neural network architecture called schema / object-file factorization (SCOFF). This model contains multiple weight-sharing GRU cells. The input information is fed into each GRU cells through an attention layer. The output information is fetched from these GRU cells and mixed with another attention layer. The model is tested on several intuitive physics benchmarks and basic reinforcement learning environment. This model demonstrates superior performance than other modular RNN architectures such as RIM on specific tasks.\n \n+ves: \n\n+ Overall, the paper is well written. Section 2 clearly explains the proposed model. Section 3 systematically compared the proposed model against other RNN architectures.\n\n+ All experiments results covered in detail including hyperparamters and experimental setting. \n\n \nConcerns: \n\n- This paper uses large amount of neuroscience terminology and vague concepts, which are just renaming of existing concepts. This is not novelty or contribution. And it is unnecessary and inappropriate for the conference publication.\n\n \n- The model is simply doing attention + weight sharing RNNs + attention. The only difference from Recurrent Independent Mechanisms (RIMs) is that the modules has shared weights - an inductive bias is be useful for specific tasks. The novelty is weak.\n\n \n- The proposed model on intuitive physics experiments shows slightly better performance than RIM. However, all these experiments have extreme setting, which is for the model's inductive bias that all objects follow exact same rules and contain full input information. It's obvious that the proposed model will perform worse than RIM when not all objects sharing same rules. \n\n\n- For these module based RNN models, it is necessary to show that the attention layer is functioning as expected. Therefore, I would suggest that the paper add visualization of the attention layers.\n\n=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for the response. All my concerns are addressed. I will increase my score to 6.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KA7OlaY4_q_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation and the proposal for splitting the schema from the procedural (representational) block makes sense. This is a good idea. A the authors build on top of RIMs, which have shown reasonable ways to model dynamical systems. However the paper itself needs to be improved and we need to evaluate the model more before publication. \n\nFirstly, the proposal that SCOFF is a direct alternative for LSTM or GRU and showing that it beats them is not entirely correct. SCOFF comprises of a GRU with a sequence of CNNs operations i.e., its doing more than what a GRU does? What exactly is that? And so when proposing evaluations it is expected that compared to GRU, SCOFF does better (Fig 5). A more valid comparison would be GRU+some standard CNN-style feature learning vs. SCOFF. Its not entirely clear how to do this -- and needs thinking. Secondly, while on Fig 5, the errors cannot be reported as a ratio with respect to GRU because this would miss the true error values; and we cannot know if there is significant difference here (especially with RIMs.vs SCOFF). \nNext, how is question 3 evaluated here? What downstream task is being considered? Am I missing something? \nNext, how doe we interpret the error change here i.e., what does 0.1 change in error mean here for better understanding where things are changing? \n\nPresentation of the paper needs a lot of improvement: \nAlgorithm 1 in the Table needs better clarity. Firstly, a bulk of the notation in the SCOFF presentation is very confusing. Its hard to parse what is going on in each stage. The description in the steps is helpful but the motivation and sequence of operations within each step needs better explanation. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a modular RNN architecture called SCOFF. The work was inspired by cognitive science(object file and schema) and was built upon previous work RIMs. The method is validated on tasks having multiple objects of the same type.

Pros:
- It addresses an important problem in DNN -- systematic generalization.
- The proposal makes sense and is more flexible than RIM.
- Experimental results outperform baselines.

Cons before rebuttal:
- The presentation of the algorithm is not very clear due to some confusing notations and missing details of algorithm steps.
- The comparison with baselines might not be fair due to extra parameters.
- The novelty is limited, because the only difference from RIM is weight sharing.

The reviewers raised concerns listed in Cons. The authors successfully addressed concerns: they indicated that the comparison was fair with the same input to both; SCOFF is more flexible than RIM, and there is spatial attention to input.
The authors added the missing details in the revised version.

All reviewers agree that the problem is important and the idea is interesting.  Since the authors' rebuttal was very helpful in clarifying the questions raised, I recommend accept.

Decision: Accept (Poster)