Neural Production Systems
Authors: Anirudh Goyal
Aniket Rajiv Didolkar
Nan Rosemary Ke
Charles Blundell
Philippe Beaudoin
Nicolas Heess
Michael Curtis Mozer
Yoshua Bengio
Venue: NeurIPS 2021 Poster (2021)
Keywords: production systems
attention-based sparse entity interactions
knowledge factorization

Abstract: Visual environments are structured, consisting of distinct  objects or entities. These entities have properties---visible or latent---that determine the manner in which they interact with one another. To partition images into entities, deep-learning researchers have proposed structural inductive biases such as slot-based architectures. To model interactions among entities, equivariant graph neural nets (GNNs) are used, but these are not particularly well suited to the task for two reasons. First, GNNs do not predispose interactions to be sparse, as relationships among independent entities are likely to be.  Second, GNNs do not factorize knowledge about  interactions in an entity-conditional manner. As an alternative, we take inspiration from cognitive science and resurrect a classic approach, production systems, which consist of a set of rule templates that are applied by binding placeholder  variables in the rules to specific entities. Rules are scored on their match to entities, and the best fitting rules are applied to update entity properties. In a series of experiments, we demonstrate that this architecture achieves a flexible, dynamic flow of control and serves to factorize entity-specific and rule-based information. This disentangling of knowledge achieves robust future-state prediction in rich visual environments, outperforming state-of-the-art methods using GNNs, and allows for the extrapolation from simple (few object) environments to more complex environments.


Neural Production Systems
Aniket Didolkar' Anirudh Goyal ",, Nan Rosemary Ke ? Charles Blundell 2,
Philippe Beaudoin 3 Nicolas Heess" 2, Michael Mozer 4. Yoshua Bengio 1,
Abstract
Visual environments are structured, consisting of distinct objects or entities. These entities have properties-vin they interact with one another. To partition images into entities, deep-learning re- searchers have proposed structural inductive biases such as slot-based architectures. To model interactions among entities, equívariant graph neural nets (GNNs) are used, but these are not particularly well suited to the task for two reasons. First, GNNs do not predispose interactions to be sparse, as relationships among inde- pendent entities are likely to be. Second, GNNs do not factorize knowledge about interactions in antit-condditioall manner. As alternative, take inspiration from cognitive science and resurrect classic approach, production systems, which consist of set of rule templates that are applied by binding placeholder variables in the rules to specific entities. Rules are scored on their match to entities, and the best fitting rules applied to update entity properties. In series of experiments, we demonstrate that this architecture achieves a flexible, dynamic flow of control and serves to factorize entity-specific and rule-based information. This disentangling of knowledge achieves robust future-state prediction in rich visual environments, out- performing state-of-the-ar methods using GNNs, and allows for the extrapolation from simple (few object) environments to more complex environments.
or latent- that determine the manner which
Introduction
Despite never having taken a physics course, every child beyond a young age appreciates that pushing plate off the dining table will cause the plate to break. The laws of physics accurately characterize the dynamics our natural world, and although explicit knowledge of these laws is not necessary to reason, can reason explicitly about objects interacting through these laws. Humans knowledge in propositional expressions such as "If a plate drops from table height, it will break," and "If a video-game opponent approaches from behind and they carrying weapon, they likely to attack you." Expressing propositional knowledge is not a strength of current deep learning methods for several reasons. First, propositions are discrete and independent from one another. Second, propositions must be quantified in the manner of first-order logic; for example, the video-game proposition applies any X for which X an opponent and has a weapon. Incorporating the ability to express and reason about propositions should improve generalization in deep learning methods because this knowledge is modular propositions can be formulated independently of each other and can therefore be acquired incrementally. Propositions can also be composed with each other and applied consistently to all entities that match, yielding a powerful form of systematic generalization.
verbalize
The classical Al literature from the 1980s can offer deep learning researchers a valuable perspective. In this era, reasoning, planning, and prediction were handled by architectures that performed proposi- tional inference on symbolic knowledge representations. A simple example of such an architecture is
Equal Contribution, Equal Advising Mila, University of Montreal, Google Deepmind, Waverly, Google Research, Brain Team. Corresponding authors: amirudhgoyall1188@mal.. com. adidelkar123@gmai.
com.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

NP5
Puppi
Creir bellast
Dot sopreseeiinng
Figure 1:
In this figure we show visual comparison between NPS and dense architectures like GNNs. In NPS, rule is only applied when an interaction takes place and it is applied only to the slots affected by the interaction. NPS also uses different rules for different kinds of interactions, while in GNN a common rule is applied to all slots irrespective of whether an interaction takes place or not (because of parameter sharing). Note the dynamic nature of the interaction graph in NPS, while in GNN, the graph is static.
the production system (Laird et al. 1986 Anderson 1987). which expresses knowledge by condition- action rules. The rules operate working memory: rule conditions are matched to entities in working memory inspired by cognitive science, and such a match can trigger computational actions that update working memory or external actions that operate the outside world.
Production systems were typically used to model high-level cognition, e.g., mathematical problem solving or procedure following; perception was not the focus of these models. It was assumed that the results of perception were placed into working memory symbolic form that could be operated on with the rules. In this article, we revisit production systems but from a deep learning perspective which naturally integrates perceptual processing and subsequent inference for visual reasoning problems. We describe an end-to-end deep learning model that constructs object-centric representations of entities videos, and then operates on these entities with differentiable-anne thus caamable-producttinn rules. The essence of these rules, carried over from traditional symbolic system, is that they operate on variables that are bound, or linked, to the entities in the world. deep learning implementation. each production rule is represented by distinct MLP with query-ket attention mechanisms to specify the rule-entity binding and to determine when the rule should be triggered for given entity. We are not the first propose neural instantiation of production system architecture. Touretzky & Hinton 1988) gave a proof of principle that neural net hardware could be hardwired to implement production system for symbolic reasoning; our work fundamentally differs from theirs that (1) we focus on perceptual inference problems and (2) we use the architecture as an inductive bias for learning.
1.1 Variables and entities
What makes a rule arbitrary values or or-the term prefer in this article-ntiitis This notion of binding is familiar in functional programming languages, where these variables are called arguments. Analogously, the use of variables in the production rules we describe enable model to reason about any set of entities that satisfy the selection criteria of the rule.
that it incorporates placeholder variables that can be bound to
Consider simple function in C like int add (int a, int b). This function binds its two integer operands to variables a and b. The function does not apply if the operands are, say, character strings. The use of variables enables a programmer reuse the same function to add any two integer values
In order for rules operate entities, these entities must be represented explicitly. That is, the visual world needs to be parsed in a task-relevant manner, e.g., distinguishing the sprites in a video game the vehicles and pedestrians approaching an autonomous vehicle. Only the past few years have deep learning vision researchers developed methods for object-centric representation (Le Roux al. 201T, Eslami al. 20066] Greff al. 2016 Raposo al.
2017 Van Steenkiste al. 20188 Greff et al.
Kosiorek et al. 2018 Engelcke et al. 2019 Burgess et al. 2019
et al. 2020; 2020;|Raaama Rahaman et al.) [020
2020al 2020al Ahmed et al., 2020; Goyal et al.]|2019| 2019 Zablotskaia
al.
These methods differ in details but fixed number slots (see Figure for example), also known as object files, each encapsulating information about single object. Importantly, the slots are interchangeable, meaning
et al.] 22200) Ding et al.. [2020] Goyal et al. 2020) Ke et al. 2021).
share the notion
that it doesn't 't matter a scene with an apple and an orange encodes the apple slot and orange in
slot 2 or vice-versa.

---

A model of visual reasoning must not only be able to represent entities but must also express knowledge about entity dynamics and interactions. To ensure systematic predictions, model must be capable of applying knowledge to an entity regardless of the slot it is in and must be capable of applying the same knowledge to multiple instances of an entity. Several distinct approaches exist in the literature. The predominant approach uses graph neural networks to model slot-to-slot interactions (Scarselli et al., 2008;; Bronstein et al. 2017; Watters et al. 2017 Van Steenkiste et al., 2018 Kipf et al.] 2018 8: Battaglia et al. 2018 Tacchetti et al. 2018). To ensure systematicity, the GNN must share parameters among the edges. In recent article, Goyal et al. (2020) developed a more general framework in which parameters are shared but slots can dynamically select which parameters to use in a state-dependent manner. Each set of parameters referred to as a schema, and slots use a query-key attention mechanism to select which schema to apply at each time step. Multiple slots can select the same schema. In both GNNs and SCOFF, modeling dynamics involves each slot interacting with each other slot. In the work describe in this article, we replace the direct slot-to-slot interactions with rules, which mediate sparse interactions among slots (See arrows in Figure
Thus our main contribution is that we introduce NPS, which offers way to model dynamic and sparse interactions among the variables in graph and also allows dynamic sharing of multiple sets of parameters among these interactions. Most architectures used for modelling interactions in the current literature use statically instantiated graph which model all possible interactions for a given variable at each step i.e. dense interactions. Also such dense architectures share single set of parameters across all interactions which maybe quite restrictive in terms of representational capacity. A visual comparison between these two kinds of architectures is shown in Figure I Through our experiments we show the advantage of modeling interactions in the proposed manner using NPS in visually rich physical environments. We also show that method results in intuitive factorization of rules and entities.
2 Production System
Formally, our notion of production system consists of a of entities and a of rules, along with a mechanism for selecting rules to apply on subsets of the entities. Implicit in a rule is a specification of the properties of relevant entities, c.g.. a rule might apply to one type of sprite in a video game but not another. The control flow of a production system dynamically selects rules as well as bindings between rules and entities, allowing different rules to be chosen and different entities to be manipulated at each point in time.
The neural production system we describe shares essential properties with traditional production system, particularly with regard to the compositionality and generality of the knowledge they embody. Lovett & Anderson (2005) describe four desirable properties commonly attributed to symbolic systems that apply to our work as well.
Production rules are modular. Each production rule represents unit of knowledge and are atomic such that any production rule can be intervened (added, modified or deleted) independently of other production rules in the system.
Production rules are abstract. Production rules allow for generalization because their conditions may be represented as high-level abstract knowledge that match to wide range of patterns. These conditions specify the attributes of relationship(s) between entities without specifying the entities themselves. The ability to represent abstract knowledge allows for the transfer of learning across different environments as long as they fit within the conditions of the given production rule.
Production rules are sparse. In order that production rules have broad applicability, they involve only a subset of entities. This assumption imposes strong prior that dependencies among entities are sparse. In the context of visual reasoning, we conjecture that this prior is superior to what has often been assumed in the past, particularly in the disentanglement secrature-indepenndee entities Higgins et al. (2016): Chen et al. (2018).
among
Production rules represent causal knowledge and thus asymmetric. Each rule can be decomposed into a (condition, action) ] pair, where the action reflects a state change that is a causal consequence of the conditions being met.
These four properties are sufficient conditions for knowledge to be expressed in production rule form. These properties specify how knowledge is represented, but not what knowledge is represented. The
3

---

latter is inferred by learning mechanisms under the inductive bias provided by the form of production rules.
3 Neural Production System: Slots and Sparse Rules
The Neural Production System (NPS), illustrated in Figure 2 provides an architectural backbone that supports the detection and inference of entity (object) representations in an input sequence, and the underlying rules which govern the interactions between these entities in time and space. The input sequence indexed by time step I, 'x'...... ......, x²") for instance the frames a video, are processed by neural encoder (Burgess et al., 2019; Greff et al. 2019) Goyal et al.) 2019 2020) applied to each x', to obtain a set of M entity representations {VI (V......... Vūr J. one for each of the M slots. These representations describe entity and are updated based on both the previous state, Vt-1 and the current input, x².
NPS consists of N separately encoded rules, (R1, R2,... R Each rule consists of two compo- nents, R (R. .L.,,, where R, is learned rule embedding vector, which can be thought of as a template defining the condition for when a rule applies; and M L.P, which determines the action taken by a rule. Both R. and the parameters of M LP; are learned along with the other parameters of the model using back-propagation on an objective optimized end-to-end.
In the general form of the model, each slot se- lects rule that will be applied to it to change its state. This can potentially be performed several times, with possibly different rules applied at each step. Rule selection is done using an atten- tion mechanism described in detail below. Each rule specifies conditions and actions on pair of slots. Therefore, while modifying the state of a slot using a rule, it can take the state of an- other slot into account. The slot which is being modified is called the primary slot and other is called the contextual slot. The contextual slot is also selected using an attention mechanism described in detail below.
Figure 2: Rule and slot combinatories.
Condition- rules specify how entities inter- act. Slots maintain the time-varying state of an entity. Every rule is matched to every pair of slots. Through key-value attention, a goodness of match is determined, and rule is selected along with its binding to slots.
3.1 Computational Steps in NPS
In this section, we give detailed description of the rule selection and application procedure for the slots. First, we will formalize the definitions of a few terms that we will use to explain our method. We use the term primary slot to refer to slot V, whose state gets modified by rule R,. We use the term contextual slot to refer to the slot V. that the rule R, takes into account while modifying the state of the primary slot Vp-
Notation. We consider a set of N rules {RI, Ra...., Ry} and a set of T input frames (x+ x² ...., ,z,....,""' following discussion, we omit the index over t for simplicity.
Each frame xf is encoded into set of M slots (Vi, Vi...., M). ). In the
Step I. is external to NPS and involves parsing an input image, x", into blot-based entities conditioned on the previous state of the blotd entities. Any of the methods proposed in the literature obtain a slot representation of entities can be used (Burgess et al.) 2019: Greff et al. 2019 Goyal et al. 2019 2020). The next three steps constitute the rule selection and application procedure.
Step 2. For each primary slot Vp. we attend to a rule R, to be applied. Here, the queries come from N). The rule is selected using a straight-througe Gumbel softmax (Jang et al. 2016) to achieve Icarnable hard decision: version of rule matching and prioritization in traditional production systems.
the primary slot: qp Vp " and the keys come from the rules: k, R, W. Vi E
argmax, (q + 7), where ~ Gumbel 1). This competition is noisy
Step 3. For a given primary slot Vp and selected rule R,, contextual slot V. is selected using another attention mechanism. this case the query comes from the primary slot: Φ, = V,W², and the keys from all the slots: ky- = V,W" Vj E [1...., M). The selection takes place using straight- through Gumbel softmax similar to step 2: = argmax, ( + 7): where 7 ~ Gumbel(O, 1). Note that each rule application is sparse since it takes into account only contextual slot for modifying

---

primary slot, while other methods like GNNs take into account all slots for modifying primary slot.
Step 4. Rule Application: the selected rule R. is applied to the primary slot V based on the rule and the current contents of the primary and contextual slots. The rule-specific M.P,,, takes as input the concatenated representation of the state of the primary and contextual slots, Vp and Ve, and produces an oulput, which is then used to change the state of the primary slot V, by residual addition.
3.2 Rule Application: Sequential vs Parallel Rule Application
In the previous section, we have described how each rule application only considers another contextual slot for the given primary slot i.c., contextual sparsity. We can also consider application sparsity, wherein we use the rules to update the states of only subset of the slots. In this scenario, only the selected slots would be primary slots. This setting will be helpful when there is an entity in an environment that is stationary, or it is following its own default dynamics unaffected by other entities. Therefore, it does not need to consider other entities to update its state. We explore two scenarios for enabling application sparsity.
Parallel Rule Application. Each of the M slots selects a rule to potentially change its state. To enable sparse changes, we provide an extra Null Rule in addition to the available N rules. If slot picks the null rule in step 2 of the above procedure, we do not update its state.
State Change
No State chonge
Sequantial
Siete
Sequential Rule Application. In this setting, only one slot gets updated in each rule applica- tion step. Therefore, only one slot selected as the primary slot. This can be facilitated by modifying step 2 above to select one (primary slot, rule) pair among NM (rule, slot) pairs. The queries come from each slot: qj = V,W Vj e
Figure 3: This figure demonstrates the sequential and parallel rule application.
(1...., M). the keys come from the rules: k; = R& W Vi E (I,..,, Nj. The straight-through Gumbel softmax selects one (primary slot, rule) pair: p,r = argmax,,, (g (q,kk; + 7), where y ~
Gumbel(0, 1). In the sequential regime, we allow the rule application procedure (step 2, 3, above) to be performed multiple times iteratively in K rule application stages for each time-step t.
A pictorial demonstration of both rule application regimes can be found in Figure 3 We provide detailed algorithms for the sequential and parallel regimes in Appendix.
Experiments
We demonstrate the effectiveness of NPS on multiple tasks and compare to comprehensive set of baselines. To show that NPS can learn intuitive rules from the data generating distribution, we design a couple of simple toy experiments with well-defined discrete operations. Results show that NPS can accurately recover each operation defined by the data and learn to represent each operation using a separate rule. We then move to a much more complicated and visually rich setting with abstract physical rules and show that factorization of knowledge into rules as offered by NPS does scale up to such settings. We study and compare the parallel and sequential rule application procedures and try to understand the settings which favour each. We then evaluate the benefits of reusable, dynamic and sparse interactions as offered by NPS in a wide variety of physical environments by comparing it against various baselines. We conduct ablation studies to assess the contribution of different components of NPS. Here we briefly outline the tasks considered and direct the reader to the Appendix for full details each task and details on hyperparameter settings.
Table This table shows the segrega- tion of rules for the MNIST Transforma- tion task. Each cell indicates the number of times the corresponding rule is used for the given operation. We can see that NPS automatically and perfectly learns a separate rule for each operation.
Rule 1 Rule
|Rule Rule 4
Translate Down
5039
0
0
Translate Up
4950
0
Rotate Right
0
0
5030
0
Rotate
Discussion of baselines. NPS is interaction network, therefore we use other widely used interac- tion networks such as multihead attention and graph neural networks (Goyal et al. (2019). Goyal et al.) (2000), Veerapaneni et al. (2019). Kipf et al. (2019) ) for comparison. Goyal et al. (2019) and Goyal

---

et al. (020) use an attention based interaction network to capture interactions between the slots, while Veerapanemi et al. (2019) and Kipf al. (2019) use GNN based interaction network. We also consider the recently introduced convolutional interaction network (CIN) (Qi et al. (02T) which captures dense pairwise interactions like GNN but uses a convolutional network instead of MI.ss to better utilize spatial information. The proposed method, similar to other interaction networks, is agnostic to the encoder backbone used to encode the input image into slots, therefore we compare NPS to other interaction networks across wide-variety of encoder backbones.
4.1 Learning intuitive rules with NPS: Toy Simulations
We designed couple of simple tasks with well-defined discrete rules to show that NPS can learn intuitive and interpretable rules. We also show the efficiency and effectiveness of the selection procedure (step 2 and step in section 3.I by comparing against baseline with many more parameters. Both tasks require a single modification of only one of the available entities, therefore the use of sequential or parallel rule application would not make difference here since parallel rule application in which all-but-one slots select the null rule is similar to sequential rule application with 1 rule application step. To simplify the presentation, we describe the setup for both tasks using the sequential rule application procedure.
MNIST Transformation. We test whether NPS can learn simple rules for performing transforma- tions on MNIST digits. We generate data with four transformations: (Traassltte Up. Translate Down, Rotate Right, Rotate Left]. We feed the input image (X) and the transformation (o) to be performed as a one-hot vector to the model. The detailed setup is described in Appendix. For this task, we evaluate whether NPS can learn to use unique rule for each transformation.
We use 4 rules corresponding to the 4 transformations with the hope that the correct transforma- tions are recovered. Indeed, we observe that NPS successfully learns to represent each trans- formation using a separate rule as shown in Table Our model achieves an MSE of 0.02. A visualization of the outputs from our model and further details can be found in Appendix C
Rule
Rule
Rule 2
Rule 3
Rule 4
Coordinate Arithmetic Task. The model is tasked with performing arithmetic operations on
NPS
Bassline
2D coordinates. Given (Xo, Yo) and (X1, Yi),
Frsquancy Frequency Frequacy
400
400
X Addition X rultrintionn
we can apply the following operations: IX Ad-
dition: (X,, Y2) = (Xo + X1, Yo). X Sub-
traction: (X,,. Y,)
(Xa X1. Yo). Y Ad-
dition: (X,, Y,) = (Xo, Yo + Yi). Y.YY Subtrac-
tion: X,, Y, = (Xo, Yo Yo-Yo Yi) where (X,, Y.)
is the resultant coordinate.
600
In this task, the model is given 2 input coordi- nates X = [<,, 3:), (z,, v) and the expected output coordinates Y = [ R.), (1), ú))] The model is supposed to infer the correct rule to produce the correct output coordinates. Dur- ing data collection, the true output is obtained by performing a random transformation on randomly selected coordinate in X (primary co- ordinate), taking another randomly selected co- ordinate from X (contextual coordinate) into account. The detailed setup is described in Ap- pendix D We use an NPS model with 4 rules for this task. We use the the selection procedure in step 2 and step 3 of algorithm to select the primary coordinate, contextual coordinate, and the rule. For the baseline we replace the selec- tion procedure in NPS (i.e. step 2 and step 3 in algorithm D with routing MLP similar to Fedus al. (2021).
200
200
Frequency
400
400
200
100
200
200
100
200
Spoche
Epochs
Figure Coordinate Arithmetic Task. Here, we compare NPS to the baseline model in terms of segregation of rules as the training progresses. X-axis shows the epochs and Y-axis shows the frequency with which Rule i is used for the given operation. We can see that NPS disentangles the operations perfectly as training progresses with a unique rule specializing to every operation while the baseline model fails to do so.
This routing MLP has 3 heads (one each for selecting the primary coordinate, contextual coordinate, and the rule). The baseline has 4 times more parameters than NPS. The final output is produced by

---

the rule MLP which does not have access to the true output, hence the model cannot simply copy the true output to produce the actual output. Unlike the MNIST transformation task, we do not provide the operation to be performed as a one-hot vector input to the model, therefore it needs to infer the available operations from the data demonstrations.
We show the segregation of rules for NPS and the baseline in Figure We can see that NPS learns to use a unique rule for each operation while the baseline struggles to disentangle the underlying operations properly. NPS also outperforms the baseline in ferms of MSE achiev- ing an MSE of +.000 while the baseline achieves an MSE of 1.04...008 To further confirm that NPS learns all the available operations correctly from raw data demon- strations, we use an NPS model with 5 rules. We expect that in this case NPS should utilize only 4 rules since the data describes only 4 unique operations and indeed we observe that NPS ends up mostly utilizing 4 of the available 5 rules as shown in Table
Table 2:
This table shows segregation of rules when we use NPS with 5 rules but the data generation distributions de- scribes only 4 possible operations. We can see that only 4 rules get majorly uti- lized thus confirming that NPS success- fully recovers all possible operations de- scribed by the data.
Rule
Rule
Rule 3 Rule 4
Rule 5
X Addition
360
99
45
13
X Subtraction
0
482
0
Subtraction
39
Y Addition
0
.7
15
99
335
4.2 Parallel vs Sequential Rule Application
We compare the parallel and sequential rule application procedures, to understand the settings that favour one or the other, over two tasks: (1) Bouncing Balls, (2) Shapes Stack. We use the term PNPS to refer to parallel rule application and SNPS to refer sequential rule application.
Shapes Stack. We use the shapes stack dataset intro- duced by |roohh et al. (2018). This dataset consists of objects stacked on top of each other as shown in Figure 5) These objects fall under the influence of gravity. For our experiments, We follow the same setup Qi et al. (2021). In this task, given the first frame, the model is tasked with predicting the object bounding boxes for the next t timesteps. The first frame is encoded using a convo- lutional network followed by RolPooling (Girshick (2015)) (2013) to extract object-centric visual features. The object-centric features are then passed to the dynamics model to predict object bounding boxes of the next t steps. Qi et al.] (2021) propose a Region Proposal Interaction Network (RPIN) to solve this task. The dynamics model in RPIN consists of an Interaction Network proposed in Battaglia lia et al.
Model Name
Test
Transfer
RPIN Qi ct al
2021
1.254400006
6.377 +0325
PNPS
1.250
1.41
SNPS
1.68
g.80
Table 3: Prediction error of the com- pared models on the shapes stack dataset (lower is better) for the test as well as transfer setting. In the test set- ting the number of rollout steps t is set to 15 and in the transfer setting it is set to 30. We can see that PNPS outper- forms the RPIN baseline in both the test and transfer setting while SNPS fails to do so. Results across 15 seeds.
(2066).
To better utilize spatial information, Qi et al. [2021 propose an extension of the interaction operators in interaction net to operate on 3D tensors. This is achieved by replacing the MLP operations in the original interaction networks with convolutions. They call this new network Convolutional Interaction Network (CIN). For the proposed model, we replace this CIN in RPIN by NPS. To ensure a fair comparison to CIN, we use CNNs represent rules in NPS instead of MLPs. CIN captures all pairwise interactions between objects using a convolutional network. In NPS, we capture sparse interactions (contextual sparsity) compared to dense pairwise interactions captured by CIN. Also, in NPS we update only a few subset of slots per step instead of all slots (application sparsity).
We consider two evaluation settings. (1) Test setting: The number of rollout timesteps is same as that seen during training (i.e. t = 15); (2) Transfer Setting: The number of rollout timesteps is higher than that seen during training (i.e. t 30).
We present our results on the shapes stack dataset in Table We can see that both PNPS and SNPS outperform the baseline RPIN in the transfer setting, while only PNPS outperforms the baseline in the test setting and SNPS fails to do so. We can see that PNPS outperforms SNPS. We attribute this to the reduced application sparsity with PNPS, i.e., it is more likely that the state of slot gets updated in PNPS as compared to SNPS. For instance, consider an NPS model with N uniformly chosen rules and M slots. The probability that the state of slot gets updated in PNPS is PPNPS N - 1/N (since rule is the null rule), while the same probability for SNPS is PSNPS- = 1/M (since only slot gets updated per rule application step).
7

---

For this task, we run both PNPS and SNPS for N = (1, 4, rules and M = 3. For any given N, we observe that PPNPS > PSN Ps. Even when we have multiple rule application steps in SNPS, it might end up selecting the same slot to be updated in more than one of these steps. We report the best performance obtained for PNPS and SNPS across all N, which N = (2 +1 Null Rule) Shapes stack is dataset that would prefer model
for PNPS and N = 4 for SNPS, in Table [] with less application sparsity since all the objects are tightly bound to each other (objects are placed on top of each other), therefore all objects spend the majority of their time interacting with the objects directly above or below them. We attribute the higher performance of PNPS compared to RPIN to the higher contextual sparsity of PNPS. Each example in the shapes stack task consists of 3 objects. Even though the blocks are tightly bound to each other, each block is only affected by the objects it is in direct contact with. For example, the top-most object is only affected by the object directly below it. The contextual sparsity offered by PNPS is a strong inductive bias to model such sparse interactions while RPIN models all pairwise interactions between the objects. Figure 5 shows an intuitive illustration of the PNPS model for the shapes stack dataset. In the figure, Rule 2 actually refers to the Null Rule, while Rule / refers to all the other non-null rules. The bottom-most block picks the Null Rule most times, as the bottom-most block generally does not move.
Bouncing Balls. We consider a bouncing- balls environment in which multiple balls move with billiard- ball dynamics. We validate our model on a colored version of this dataset. This is a next-step prediction task in which the model is tasked with predicting the final binary mask of each ball. We compare the following methods: (a) SCOFF (Goyal al., 2020): factorization of knowledge in terms of slots (object proper- ties) and schemata, the latter capturing object dynamics; (b) SCOFF++: we extend SCOFF by using the idea of iterative competition as pro- posed in slot attention (SA) (Locatello et al. 2020a); SCOFF + PNPS/SNPS: We replace pair- wise slot-to-slot interaction in SCOFF++ with parallel or sequential rule application. For com- paring different methods, we use the Adjusted Rand Index or ARI (Rand) 197I). To investigate how the factorization in the form of rules allows for extrapolating knowledge from fewer to more objects, we increase the number of objects from 4 during training to 6-8 during testing.
0.42
0.36
Figure 5: Here we show the rule selection statis- tics from the proposed model for all entities in the shapes stack dataset across all examples. Each ex- ample contains 3 entities as shown above. Each cell in the table shows the probability with which the given rule is triggered for the corresponding entity. We can see that the bottom- most entity triggers rule 2 most of the time while the other entities trigger rule I most often. This is quite intu- itive as, for most examples, the pottom-most entity remains static and does not move at all while the upper entities fall. Therefore, rule 2 captures in- formation which is relevant to static entities, while rule 1 captures physical rules relevant to the inter- actions and motion of the upper entities.
We present the results of our experiments in Table 4 Contrary to the shapes stack task, we see that SNPS outperforms PNPS for the bouncing balls task. The balls are not tightly bound together into a single tower as in the shapes stack. Most of the time, a single ball follows its own dynamics, only occasionally interacting with another ball. Rules in NPS capture interaction dynamics between entities, hence they would only be required to change the state of an entity when it interacts with another entity. In the case of bouncing balls, this interaction takes place through collision between multiple balls. Since for a single ball, such collisions are rare, SNPS, which has higher application sparsity (less probability of modifying the state of an entity), performs better as compared to PNPS (lower application sparsity). Also note that, SNPS has the ability to compose multiple rules together by virtue of having multiple rule application stages. A visualization of the rule and entity selections by the proposed algorithm can be found in Appendix Figure 9
Given the analysis in this section, we can conclude that PNPS is expected to work better when interactions among entities are more frequent while SNPS is expected work better when interactions are rare and most of the time, each entity follows its own dynamics. Note that, for both SNPS and PNPS, the rule application considers only other entity as context. Therefore, both approaches have equal contextual sparsity while the baselines that we consider (SCOFF and RPIN) capture dense pairwise interactions. We discuss the benefits of contextual sparsity in more detail in the next section. More details regarding our setup for the above experiments can be found in Appendix.
8

---

4.3 Benefits of Sparse Interactions Offered by NPS
In NPS, one can view the computational graph as a dynamically constructed GNN resulting from applying dynamically selected rules, where the states of the slots are represented on the different nodes of the graph, and different rules dynamically instantiate an hyper-edge between a set of slots (the primary slot and the contextual slot). It is important to emphasize that the topology of the graph induced NPS dynamic and sparse (only a few nodes affected), while in most GNNs the topology is fixed and dense (all nodes affected). In this section, through thorough set of experiments, we show that learning sparse and dynamic interactions using NPS indeed works better for the problems we consider than learning dense interactions using GNNs. We consider two types of tasks; (I) Learning Action Conditioned World Models (2) Physical Reasoning. We use SNPS for all these experiments since in the environments that we consider here, interactions among entities are rare.
Model Name
Test
Transfer
Learning Ietion-Conditionel World Models. For learning cetion-conditionee world models, we follow the same experimental setup Kipf et al. (2019). There- fore, all the tasks in this section are next-K step (K = {1,5, 10}) prediction tasks, given the intermediate actions, and with the predictions being performed in the latent space. We use the Hits at Rank (H@1) @ 1) metrics described by KKipf eet al. (2019) for evaluation. H@1 is 1 for a particular ex- ample if the predicted state representation is nearest to the encoded true observation and O otherwise. We report the average of this score over the test set (higher is better).
SCOFF
0.28
0.15
SCOFT++
0.8437
0.2632
PNPS (10 Rules+1 Null Rule)
0.7813
0.1997
SNPS (10 Rules)
0.8518
0.3553
Table 4:
Here we show the ARI achieved by the models on the bouncing balls dataset (higher is better). We can see that SNPS outperforms SCOFF and SCOFF++ while PNPS has a poor per- formance in this task. Results average across 2 seeds.
Physics Environment. The physics enviromment (Ke et al. 2021) simulates a simple physical world. It consists of blocks of unique but unknown weights. The dynamics for the interaction between blocks is that the movement of heavier blocks pushes lighter blocks on their path. This rule creates an acyelic causal graph between the blocks. For an accurate world model, the learner needs infer the correct weights through demonstrations. Interactions in this environment are sparse and only involve two blocks time, therefore we expect NPS to outperform dense architectures like GNNs. This environment is demonstrated in Appendix Fig 11
We follow the same setup Kipf et al. (011)). We their C-SWM model as baseline. For the proposed model, we only replace the GNN from C-SWM by NPS. GNNs generally share parameters across edges, but in NPS each rule has separate parameters, For fair comparison to GNN, use an NPS model with rule. Note that this setting is still different from GŇNs as in GNNs at each step every slot is updated by instantiating edges between all pairs of slots, while in NPS an edge is dynamically instantiated between a single pair of slots and only the state of the selected slot (i.e., primary slot) gets updated.
PODE
(a) Physies Env
(b) Atari Games
Figure 6: Action-Condition Conditionnd World Models, with number of future steps to be predicted for the world-model on the horizontal axes. (a) Here we show a comparison between GNNs and the proposed NPS on the physics environment using the H@1 metric (higher is better). (b) Comparison of average H@1 scores across 5 Atari games for the proposed model NPS and GNN.
The results of our experiments are presented Figure 6 a). We can see that NPS outperforms GNNs for all rollouts. Multi-step settings are more difficult to model errors may get compounded over time steps. The sparsity of NPS (only a single slot affected per step) reduces compounding of errors and enhances cummetry-breaking in the assignment of transformations to rules, while in the

---

case of GNNs, since all entities are affected per step, there is a higher possibility of enrors getting compounded. We can see that even with a single rule, we significantly outperform GNNs thus proving the effectiveness of dynamically instantiating edges between entities.
Atari Games. We also test the proposed model in the more complicated setting of Atari, Atari games also have sparse interactions between entities. For instance, in Pong, any interaction involves only 2 entities: (1) paddle and ball or (2) ball and the wall. Therefore, we expect sparse interactions captured by NPS to outperform GNNs here as well.
We follow the same setup as for the physics environment described in the previous section. We present the results for the Atari experiments in Figure showing the average H@ I score across that the
5 games: Pong, Space Invaders, Freeway, Breakout, and QBert. As expected, proposed model achieves a higher score than the GNN-based C-SWM. The results for the Atari experiments reinforce the claim that NPS is especially good at learning sparse interactions.
Learning Rules for Physical Reasoning. To show the effectiveness of the proposed approach for physical reasoning tasks, we evaluate NPS on another dataset: Sprites-MOT (He et al., 2018). The SPrites-MOT dataset was introduced by He et al. (2018). The dataset contains a set of moving objects of various shapes. This dataset aims to test whether a model can handle occlusions correctly. Each frame has consistent bounding boxes which may cause the objects to appear or disappear from the scene. A model which performs well should be able to track the motion of all objects irrespective of whether they are occluded or not. We follow the same setup as Weis et al. (2020). We use the OP3 model /Veerapaneni al. 2019) as our baseline. To test the proposed model, we replace the GNN-based transition model in OP3 with the proposed NPS.
We use the same evaluation protocol as followed by Weis al. (2020) which based on the MOT (Multi-object tracking) challenge (Milan et al. 2016). The results on the MOTA and MOTP metries for this task are presented in Table The results on the other metrics are presented in appendix TO] We ask the reader to refer to appendix for more details about these metrics. We can see that for almost all metrics, NPS outperforms the OP3 baseline. Although this dataset does not contain physical interactions between the objects, sparse rule application should still be useful in dealing with occlusions. At any time step, only a single object is affected by occlusions i.c., it may get occluded due another object or due prespecified bounding box, while the other objects follow their default dynamics. Therefore, a rule should be applied to only the object (or entity) affected (i.e., not visible) due occlusion and may take into account any other object or entity that is responsible for the occlusion.
Model
MOTA f
MOTP
OP3
89.1
78.4
NPS
90.72+55.5
79.91+03
Table 5: Sprites-MOT. Comparison be-
tween the proposed NPS and the base- line OP3 using the MOTA and MOTP metrics on the Sprites-MOT dataset (f: higher is better). Average over random seeds.
5 Discussion and Conclusion
For Al agents such as robots trying to make sense of their environment, the only observables are low-level variables like pixels in images. To generalize well, an agent must induce high-level entities as well as discover and disentangle the rules that govern how these entities actually interact with each other. Here we have focused on perceptual inference problems and proposed NPS, neural instantiation of production systems by introducing important inductive bias in the architecture following the proposals of Bengio (2017); Goyal & Bengio 2020);; Ke et al. (2021).
Limitations & Looking Forward. Our experiments highlight the advantages brought by the factor- ization of knowledge into a small set of entities and sparse sequentially applied rules. Immediate future work would investigate how to take advantage of these inductive biases for more complex physical environments (Ahmed et al. 2020) and novel planning methods, which might be more sample efficient than standard ones (Schrittwieser et al. (020).
We also find that Sequential and Parallel NPS have different properties suited towards different domains. Future work should explore how to effectively combine these two approaches. We discuss this in more detail in Appendix section B.3
10

---

Acknowledgements
The authors would like to thank Matthew Botvinick for useful discussions. The authors would also like to thank Alex Lamb, Stefan Bauer, Nicolas Chapados, Danilo Rezende and Kelsey Allen for brainstorming sessions. We are also thankful to Dianbo Liu, Damjan Kalajdzievski and Osama Ahmed for proofreading. We would like to thank Samsung Electronics Co. Ltd. and CIFAR for funding this research. We would also like to thank Google for providing Google cloud credits used in this work.
References
Ahmed, O., Träuble, F., Goyal, A., Neitz, A., Wüthrich, M., Bengio, Y., Schölkopf. B., and Bauer, S. Corsalworld: A robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint ar (xXx:20000004296, 2020.
Anderson, R. Skill acquisition: Compilation of weak-method problem situations. Psychological review, 94(2):192, 1987.
Andreas, J., Rohrbach, M., Darrell, T., and Klein, Neural module networks. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 39-48, 2016.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409:044, 2014.
Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D. and Kavukcuoglu, K. Interaction networks for learning about objects. relations and physics. CoRR, abs/1612.00022,, 2016. URL http: //arxiv. org/ abs 1tt//1112.00022
Battaglia, P. W., Hamrick, J. B., Bapst, V.,
A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806.0/ 02200, 2018.
Bengio, Y. The consciousness prior. arXiv preprint arXiv: 709.08568. 2017.
Bottou, L. and Gallinari, P. A framework for the cooperation learning algorithms. In Advances in neural information processing systems, pp. 781-788, 1991.
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Bunel, R., Hausknecht, M., Devlin, Singh, R., and Kohli, P. Leveraging grammar and reinforcement learning for neural program synthesis. arXiv preprint
2018
Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick. M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. arXiv preprint 1XX1:1901.11390 2019.
Cai, J., Shin, R., and Song, D. Making neural programming architectures generalize via recursion. arXiv preprint arXiv: 2017.
Chen, R. T., Li, X.. Grosse, R., and Duvenaud, D. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04944 2018.
Ding, D., Hill, F., Santoro, A., and Botvinick, M. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08500 2020.
Du, Y., Smith, K., Ulman, T., Tenenbaum, J., and Wu, J. Unsupervised discovery of 3d physical objects from video. arXiv preprint arXiv:2007.12:: 2020.
Engelcke, M., Kosiorek, A. R., Jones, O. P., and Posner, I. Genesis: Generative scene inference and sampling with object-centric latent representations, arXiv preprint arXiv: 2019.

---

Eslami, S., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Kavukcuoglu. K., and Hinton, G. E. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint 1rXiv:1003.88.. 2016.
Evans, R., Hermández-Orallo, J., Welbi, J., Kohli, P., and Sergot, M. Making sense of sensory input. Artificial Intelligence, pp. 103438, 2019.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://axxii...tt /bb// 2101.0380 03961
Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A. A., Pritzel, A., and Wierstra, D. Pathnet: Evolution channels gradient descent super neural networks. arXiv preprint arXiv:1001.08734, 2017.
Girshick, R. B. Fast R-CNN. CoRR, abs/ 500.08884 2015. UBL./cpp///arziv.org//bbb/// 08083
Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:2011.1500/ 2020.
Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., and Schölkopf, B. Recurrent independent mechanisms, 2019.
Goyal, A., Lamb, A., Gampa, P., Beaudoin, Levine, S., Blundell, C., Bengio, Y., and Mozer, M. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint (XXx:20066.1622) 2020.
Graves, A., Wayne, G., and Danihelka, Neural turing machines. arXiv preprint arXiv:1410.5400 2014.
Greff, K., Rasmus, A., Berglund, M., Hao, T. H., Schmidhuber, J., and Valpola, H. Tagger: Deep 2016.
unsupervised perceptual grouping. arXiv preprint
Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C.,, Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv: 1903. 100.0000,0 2019.
Groth, O., Fuchs, F., Posner, I., and Vedaldi, A. Shapestacks: Learning vision-based physical intuition for generalised object stacking. CoRR, 0os//804.08015 2018. URL http://///aa www.axiiiiioogab.....aaaasb 1804. 08018
He, Z.. Li, J., Liu. D.. He, H., and Barber, D. Tracking by animation: Unsupervised learning of multi-object attentive trackers. CoRR, 031337
2018. URL http:://axxii..o// org/ abs/
Higgins, I., Matthey, L.,, Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Jacobs, R. A., Jordan, M. Nowlan, S. J., Hinton, G. E., et al. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01 144, 2016.
Ke. N. R., Didolkar, A. R., Mittal, S., Goyal, A., Lajoic, G., Bauer, S., Rezende, D. J., Mozer, M. C., Bengio, Y., and Pal, C. Systematic evaluation of causal discovery in visual model based reinforcement learning. 2021.
Kipf, T., Fetaya, E., Wang, K.CC,, Welling, M., and Zemel, R. Neural relational inference for interacting systems. arXiv preprint arXiv:1802.0466 04687, 2018.
Kipf, T., van der Pol, E., and Welling, M. Contrastive learning of structured world models. arXiv 2019.
preprint
12

---

Kirsch, L., Kunze, J., and Barber, D. Modular networks: Learning to decompose neural computation. In Advances Neural Information Processing Systems, pp. 2:408-2418, 2018.
Kosiorek, A., Kim, H., Teh, Y. W., and Posner, I. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31:8606-8611 2018.
Laird, E., Rosenbloom, P. S., and Newell, A. Chunking in soar: The anatomy of a general learning mechanism. Machine learning. 1(1):11 (1):11-46, 1986.
Lamb, A., Goyal, A., Stowik, A., Mozer, M., Beaudoin, P., and Bengio, Y. Neural function modules with sparse arguments: A dynamic approach integrating information across layers. arXiv preprint 0rX1v:2010.08012, 2020.
Le Roux, N., Heess, N., Shotton, J., and Winn, J. Learning a generative model of images by factoring appearance and shape. Neural Computation, 23(3):593-650, 2011.
Li, Y., Gimeno, F., Kohli, P., and Vinyals, O. Strong generalization and efficiency in neural programs. arXiv preprint (rXiv:2007.03629, 2020.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A.. and Kipf, T. Object-centric learning with slot attention. arXiv preprint arXiv:2006. 1005, 2020a.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention, 2020b.
Lovett, M. and Anderson, J. R. Thinking as a production system. The Cambridge handbook of 2005.
thinking and reasoning, pp. 401-429, 2005.
McMillan, C., Mozer, M. C., and Smolensky, P. The connectionist scientist game: rule extraction and refinement in a neural network. In Proceedings of the 13th Annual Conference of the Cognitive Science Society, pp. 424 430, 1991.
Milan, A., LealeTaixé, L., Reid, I. D., Roth, S., and Schindler, K. MOT16: A benchmark for multi- object tracking. CoRR, abbs/1603.000/ 00083, 2016. BtLphttp://axiiv org/abs/////eoooooooooooooe 1603. 0t////003000011111
Neelakantan, A., Le, Q V., and Sutskever, I. Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv: 1511104314,4 2015.
Qi, H., Wang, X., Pathak, D.. Ma, Y., and Malik, J. Learning long-term visual dynamics with region proposal interaction networks. In International Conference on Learning Representations, 2021. URL httttpss//oppnrrevie..eeee net.
_______Akka 4Akcd8Re
Rahaman, N., Goyal, A., Gondal, M. W., Wuthrich, M., Bauer, S.. Sharma, Y., Bengio. Y., and
Schölkopf, B. S2rms: Spatially structured recurrent modules. arXiv preprint arXiv:0007.06533, 2020.
Rand, W. M. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):8468855) 4500 1971.
Raposo, D.,. Santoro, A., Barrett, D.,, Pascanu, R., Lillicrap, T., and Battaglia, P. Discovering objects and their relations from entangled scene representations. arXiv preprint arXiv:1702.0000 2017.
Reed, S. and De Freitas, N. Neural programmeerinterpretes.. arXiv preprint arXiv:1511.0000 2015.
Ronco, E., Gollee, H., and Gawthrop, P. J. Modular neural networks and self-decompositiin.. Technical Report CSC-96012, 1997.
Rosenbaum, C., Klinger, T., and Riemer, M. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.0123 2017.
Rosenbaum, C., Cases, I., Riemer, M., and Klinger, T. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv:190.112774.0 2019.
13

---

Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M.. and Monfardini, G. The graph neural network model. IEEE Transactions on Neural Networks, 20( (0)6660-88 2008.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L.,, Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel. T., et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839)6604-009, 2020.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 2017.
Tacchetti, A., Song, F., Mediano, P. A., Zambaldi, V., Rabinowitz, N. C., Graepel, T., Botvinick, M.. and Battaglia, P. W. Relational forward models for multi-agent learning. arXiv preprint arXiv: 11044, 2018.
Touretzky, D. and Hinton, G. E. A distributed connectionist production system. Cognitive science, 12(3):423-4055 1988.
Trask, A., Hill, F., Reed, S. E.,, Rae, J., Dyer, C.,, and Blunsom, P. Neural arithmetic logic units. In Advances in Neural Information Processing Systems. pp. 8035-8044, 2018.
Van Steenkiste, S., Chang, M., Greff, K., and Schmidhuber, J. Relational neural expectation maximiza- tion: Unsupervised discovery of objects and their interactions. arXiv preprint 1802. 2018.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need, 2017.
Veerapaneni, R., Co-Reyes, D., Chang, M., Janner, M., Finn, C., Wu, J., Tenenbaum, J. B., and Levine, S. Entity abstraction visual model-based reinforcement learnnnnn CoRR, abs/1910.12827, 2019. URL http::t::///axii org/ abs 1910. 12827
Veerapaneni, R., Co-Reyes, D., Chang, M., Janner, M., Finn, C., Wu, J., Tenenbaum, J., and Levine, S. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., and Tacchetti, A. Visual interaction networks: Learning a physics simulator from video. In Advances in neural information processing systems. pp. 4539 4539-4547, 2017.
Weis, M. A., Chitta, K., Sharma. Y., Brendel, W., Bethge, M., Geiger, A., and Ecker, A. Unmasking the inductive biases of unsupervised object representations for video sequences, 2020.
Xu, D., Nair, S., Zhu, Y., Gao, J., Garg, A., Fei-Fei, L., and Savarese, S. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Zablotskaia, P., Dominici, E. A., Sigal, L., and Lehrmann, A. M. Unsupervised video decomposition using spatio-temporal iterative inference. arXiv preprint arXiv:2006. /4727, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
14

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See section 4.3
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasin new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] All datasets we used are open-sourced
(c) Did you include any assets either in the supplemental material
URL? [Yes]
We include the code for reproducing our experiments
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All datasets we use are synthetic and created using simulation hence do not include any personal or user data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We not use any dataset that includes any kind of identifying information.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "ubmurL82qW", "reviewer": "Reviewer_CUnV", "summary": "The authors present a new neural network system that can reason over entities. The solution should be applicable to any tasks that require visual reasoning.\n\nSpecifically, the NPS algorithm is:\n```\nfor every step in a sequence:\n    update the slots (entities)\n    compute attention between slot queries and rule keys to select a rule\n    compute attention between primary slot queries and all slot keys to get contextual slot\n    apply the rule MLP to the primary and contextual slot values\n    add the result to the primary slot\n```\nSlots can either be updated in parallel (i.e., there is only one slot updated per step) or in sequence (i.e., multiple slots can be updated per step).\n\nThey compare sequential vs parallel rule application on 4 tasks:\n- given a transformation and mnist digit, predict the digit with the transformation applied\n- given two x,y coordinates predict the result of an arithmetic change (e.g., subtract the ys from one another) applied\n- for a stack of falling shapes, predict the bounding box of each object in future frames\n- predict the next frame of billiard balls bouncing around\n\nThey compare NPS to GNNs and related methods on four tasks. The goal of each task is to predict latent states in future frames:\n- physics env: blocks of unknown weight interacting with one another\n- Atari games (pong, space invaders, freeway, breakout, qBert)\n- sprites (moving objects of different shapes)", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: NPS is a novel system for visual reasoning. The computational components borrow heavily from attention mechanisms but separately represent rules to apply.\n\nQuality: The authors test a large number of total tasks, targeting different aspects of NPS. E.g., using the first set of experimental results they are able to provide the recommendation that sequential application is better for sparse interactions.\n\nClarity: I would have appreciated a more straightforward introduction that was heavier on visuals. E.g., the paragraph beginning at section 64 discusses going from raw observations to slots, which is not what NPS does. It takes up a lot of real estate in the introduction and gave me the impression that the paper focused on going from raw data to slots. It's accompanied by Figure 1, which provides a visualization of rules before they're introduced, which I found confusing. I think it would help with clarity to add more visualizations of the NPS system into the main text (or to at least pull the algorithm into the main text). I also raise some minor inconsistencies that caused confusion under minor comments.\n\nSignificance: The potential impact for differentiable reasoning is enormous. The aim of this paper is to take a step towards goal by showing a reasoning system on a variety of restricted settings that are straightforward to analyze. On many tasks, NPS is compared to only one baseline and provides a small performance improvement. E.g., on shapes stack, bouncing balls (test), and sprites, NPS is within 1-2 points of the baseline (and well within the error bars). On physics env and Atari, NPS has a more sizable gain, but is still within the error bars.\n\nMinor comments:\n- Algorithm 1 should specify that j indexes M (num slots)\n- The steps in algorithm 1 should be consistent with main text (it omits the primary slot selection)\n- L208: an -> a?\n- L402 figure cuts off the text\n- \"Entity abstraction in visual model-based reinforcement learning.\" is cited twice?\n\nQuestions to authors:\n- how would you contrast NPS to a transformer trained on the same tasks, but with a rule \"memory store\"?\n\nRelated work of interest:\n- Visual Grounding of Learned Physical Models. Li et. al, 2020.\n- Learning visual predictive models of physics for playing billiards. Fragkiadaki et. al., 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed limitation and impact.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "obDjfhJukQ", "reviewer": "Reviewer_LeDU", "summary": "The paper presents Neural Production Systems (NPS), a neural model to build entiity-centric representations and model the latent rules that determine their interactions in visual environments, inspired by production systems.", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I found this paper very interesting. Although inspired by an old concept (i.e., production systems), its deep learning adaptation is far from trivial.\n- The reported experiments are convincing. In particular, the authors show that NPS can learn arithmetic rules, map rules to MNIST transformation, work in richer visual settings and perform future-state predictions (such as predicting the number of future steps in action-conditioned world models) better than GNNs.\n- Rules are differentiable, and could be learned in the context of a broader framework.\n\nQuestions:\n- How does the model scale with respect to the number of entities/rules considered?\n\nPresentation comments:\n- Figure 1 is not colour-blind friendly.\n- It might be useful to report results for individual games in Table 5, with the number of entities/rules considered.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RvBpJYH7MSj", "reviewer": "Reviewer_hBZe", "summary": "This paper proposes Neural Production Systems (NPS) which is a (neural) rule-based learning system that operate on explicit entities. The main idea is to extend traditional Production systems to be end-to-end learnable with neural networks. Unlike previous NN systems that model entity-wise interactions such as GNN, NPS allows sparse interactions between entities which is favorable in certain scenarios. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality & Significance\n- The idea of extending the traditional rule-based system to neural net rule-based system is intriguing as it can combine the benefit of excellent perception of NN and generalization of rule-based systems. Also, rules in a given environment can be naturally discovered through the learning process. This paper shows promising results on how rules can be learned. However, I have a concern that this paper seems like a marginal improvement over the previously proposed SCOFF (Goyal et al. 2020) model which I further discuss below. \n\nQuality\n- SCOFF model also has soft-attention based selection process for entities and hard selection with Gumbel-softmax for schema. Although the intuition and exact formulation is different, the rule application process in NPS can be modelled using schema+soft competition among entities in SCOFF. In that case, the main benefit of NPS comes from having the sparse interaction between entities because it can be more efficient and easier to learn which could be valuable if there is a significant improvement in performance. Therefore, the experiments section should demonstrate that better in my opinion. \n- Exp could include more analysis on how NPS can learn certain rules that SCOFF cannot. It would be great to see if the improvements in NPS is enabling the perfect separation of rules in Sec4.1 and if SCOFF cannot do it perfectly. \n- The shapes stack and Sec4.3 could also compare with SCOFF as a stronger baseline.\n- The internal behavior of NPS is analyzed for simple tasks such as MNIST transformation, but systems like NPS can really shine when applied to complex environments with many entities such as Bouncing Balls. The results would be strengthend if there are detailed analysis or visualization of how entities are selected and how rules are applied to the selected entities.\n- Line 402 and Figure 5 are overlapping.\n\nClarity\n- The paper is easy to read and well-written.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5qeJ6tVk4Oo", "reviewer": "Reviewer_CM2F", "summary": "This paper proposes a novel method that combines some of the strengths of deep learning with *production systems*, a classic approach in symbolic AI and cognitive science, and evaluates the method on physical reasoning tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is very exciting and I think it has a lot of promise. Furthermore, the algorithms proposed in the paper creatively combine a range of deep learning techniques (slot-based object-centric representations, key/value attention, gumbel softmax) in an attempt to accomplish this unification. Unfortunately, I think in its current state this work suffers from two limitations:\n\n1) The results: \n* The current set of experiments seem to demonstrate only marginal improvements over the baselines in some cases. In many instances there is no specification of how many random seeds are being averaged over, or the number of seeds is very small, and it is not clear whether there is even a statistically significant advantage over the baseline methods. \n* Furthermore there is typically only one baseline evaluated for each experiment, and it is not clearly specified why this baseline was chosen or whether it represents the current state-of-the-art for the particular metric being evaluated. This is not the only important consideration, but it is certainly important information to include.\n\n2. The algorithm(s): There are significant questions about the generality of the proposed algorithm, at least in its current formulation. \n* First, the algorithm is applied somewhat inconsistently to the different experiments, in a way that isn’t entirely transparent. For instance, in the MNIST transformation task, it sounds like the task instruction (specifying the transformation to be performed) is used to query the rules, which, according to their formulation, should make the task instruction itself the ‘primary slot’, meaning that it should be updated by the selected rule. But presumably it is actually the image embedding that is treated as the primary slot, so as to apply the selected transformation. This experiment is only intended as a preliminary test of the model, but I think it actually reveals a way in which important aspects of the implementation have to be manually tailored to a particular task, because the rule selection and application needs to be handled in an entirely different way if rules are selected on the basis of an instruction vs. on the basis of the objects themselves. Another example comes from the coordinate arithmetic task, in which the coordinates need to be assigned to slots in a particular way in order for the model to successfully use the proposed rule selection method. Specifically, the input and output coordinates for each dimension need to be concatenated into a single slot in order to allow the rule to be inferred from pairwise comparisons between each slot and each rule embedding. If, instead, the input and output coordinates were each represented as separate slots (which seems like a natural way to apply the algorithm as currently specified) there would be no way to identify the rule governing the transformation from input to output.\n* Additionally, there are two versions of the proposed method, sequential vs. parallel NPS, and which version works better is somewhat task-dependent. The authors offer reasonable conjectures as to why a particular method works better on a particular task, but this task-dependency does somewhat undermine their claim that sparse interactions are a generally useful inductive bias. The parallel version, which is somewhat less sparse than the sequential version, works better on tasks involving multiple closely interacting objects, whereas the more sparse sequential version works better on tasks involving more spatially distributed objects that mostly don’t interact with each other. But presumably the task of real-world physical reasoning will involve a mixture of these scenarios, and there is no proposal for how this decision should be made other than by the intuitive judgment of the modeler.\n* Finally, the current formulation seems to be overly tailored to physical reasoning tasks, and lacking the more general expressive power of the original production systems. For instance, the current formulation only accommodates binary interactions, and there is no way to achieve any kind of hierarchy (i.e. in which the output of one rule serves as the input to another). Many of the design decisions regarding how rules are selected and applied seem as though they have been made specifically with physical reasoning in mind, and it’s not clear how they will be modified to handle other sorts of tasks to which production systems have traditionally been applied, especially tasks involving more abstract reasoning. \n\nIn summary, I think the current approach might be more conservatively described as a method for efficiently modeling sparse interactions for the purposes of physical reasoning, rather than as a ‘neural production system’ which seems to imply much more general functionality than it currently has. Nevertheless, I think that it might still merit acceptance if the issues regarding the results and baselines are resolved.\n\nMinor points:\n* Some of the notation is inconsistent or unclear. For instance, in the description of step 2 (line 149), the key is obtained using a variable denoted as $R_i$, is this the same as the learned rule embedding vector (described in line 124) denoted as $\\vec{\\textbf{R}}_i$? Is $\\textbf{k}_i$ simply equal to $\\vec{\\textbf{R}}_i$ as indicated in algorithms 1 and 2, or is it multiplied by the weights $W^k$ as indicated on line 149? In Algorithm 2, are the weights $W^k$ really used to obtain $\\textbf{q}_p$ or are these supposed to be $W^q$ or $\\widetilde{W}^q$? Also in Algorithm 2, is $\\hat{W}^k$ an additional set of weights not specified in the input description, or is it supposed to be $W^k$?\n* In the Appendix it says that Table 3 in the paper is wrong and that Table 9 (in the Appendix) reflects the correct results. Hopefully Table 9 will go into the main body of the paper, including the results for both the test and transfer regimes?\n* It would be helpful to include pointers to the specific relevant sections in the Appendix, and to try and arrange the sections of the Appendix in the same order as the corresponding sections in the main body of the paper.\n\n**Update after discussion period**\n\nThe authors have made improvements to the algorithm, clarified the criteria for selecting baselines and how these relate to the current state-of-the-art, and performed more extensive experiments which demonstrate that the improved algorithm clearly outperforms other approaches on a range of tasks. Notably, these involved experiments on a Raven's Progressive Matrices benchmark, showing some improvements in out-of-distribution generalization, suggesting that NPS may useful in more symbolic domains in addition to the physical reasoning tasks that are the focus of most of the paper. I believe the paper merits acceptance and am updating my score to a 7.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a differentiable and learnable version of the classic production system architecture. This is a worthwhile and interesting attempt at enabling more flexible reasoning in deep learning systems. The reviewers found the initial evaluations of the system uncompelling, however the substantial additions provided during the discussion phase were deemed sufficient by all reviewers. This means the paper is an unusual case where it will need substantial revision to the results, but the reviewers believe this is feasible -- so I too will recommend acceptance. I urge the authors to also attempt to clarify the model presentation, as indicated by the reviewers.
Decision: Accept (Poster)