Emergent Symbols through Binding in External Memory
Authors: Taylor Whittington Webb
Ishan Sinha
Jonathan Cohen
Venue: ICLR 2021 Spotlight (2021)
Keywords: abstract rules
out-of-distribution generalization
external memory
indirection
variable binding

Abstract: A key aspect of human intelligence is the ability to infer abstract rules directly from high-dimensional sensory data, and to do so given only a limited amount of training experience. Deep neural network algorithms have proven to be a powerful tool for learning directly from high-dimensional data, but currently lack this capacity for data-efficient induction of abstract rules, leading some to argue that symbol-processing mechanisms will be necessary to account for this capacity. In this work, we take a step toward bridging this gap by introducing the Emergent Symbol Binding Network (ESBN), a recurrent network augmented with an external memory that enables a form of variable-binding and indirection. This binding mechanism allows symbol-like representations to emerge through the learning process without the need to explicitly incorporate symbol-processing machinery, enabling the ESBN to learn rules in a manner that is abstracted away from the particular entities to which those rules apply. Across a series of tasks, we show that this architecture displays nearly perfect generalization of learned rules to novel entities given only a limited number of training examples, and outperforms a number of other competitive neural network architectures.

Published as a conference paper at ICLR 2021
EMERGENT SYMBOLS THROUGH BINDING IN EXTERNAL MEMORY
Taylor W. Webb University of California Los Angeles Los Angeles, CA taylor. .w. webb@gma i com
Ishan Sinha, Jonathan D. Cohen Princeton University Princeton, NJ
ABSTRACT
A key aspect of human intelligence is the ability to infer abstract rules directly from high-dimensional sensory data, and to do so given only a limited amount of training experience. Deep neural network algorithms have proven to be a power- ful tool for learning directly from high-dimensional data, but currently lack this capacity for data-efficient induction of abstract rules, leading some to argue that symbol-processing mechanisms will be necessary account for this capacity. In this work, we take a step toward bridging this gap by introducing the Emergent Symbol Binding Network (ESBN), recurrent network augmented with an exter- nal memory that enables a form of variable-binding and indirection. This bind- ing mechanism allows symbol-like representations to emerge through the learning process without the need to explicitly incorporate symbol-processee processing machinery. enabling the ESBN learn rules in a manner that is abstracted away from the par- ticular entities to which those rules apply. Across a series of tasks, we show that this architecture displays nearly perfect generalization of learned rules to novel en- tities given only a limited number of training examples, and outperforms number other competitive neural network architectures.
INTRODUCTION
Human intelligence is characterized by a remarkable capacity to detect the presence of simple, ab- stract rules that govern high-dimensional sensory data, such images sounds, and then apply these to novel data. This capacity has been extensively studied by psychologists in both the visual domain, in tasks such Raven's Progressive Matrices (Raven Court 1938). and the auditory domain, in tasks that employ novel, artificial languages (Marcus et al. 1999).
In recent years, deep neural network algorithms have reemerged as a powerful tool for learning directly from high-dimensional data, though many studies have now demonstrated that these models suffer from similar limitations as those faced by the earlier generation of neural networks: requiring enormous amounts of training data and tending to generalize poorly outside the distribution of those training data (Lake & Baroni 2018 Barrett al., 2018). This stands in sharp contrast the ability of human learners to infer abstract structure from a limited number of training examples and then systematically generalize that structure to problems involving novel entities.
It has long been argued that the human ability to generalize in this manner depends crucially capacity for variable-binding, that is, the ability to represent a problem terms of abstract symbol- like variables that are bound to concrete entities (Holyoak & Hummel] 2000 Marcus 2001). This in turn can be broken down into two components: 1) mechanism for indirection, the ability to bind two representations together and then ct al. 2013), and 2) representational scheme whereby one of the bound representations codes for abstract variables, and the other codes for the values of those variables.
representation refer and retrieve the other (Kriete

---

Published as a conference paper at ICLR 2021
In this work, we present novel architecture designed around the goal of having capacity for ab- stract variable-binding. This is accomplished through two important design considerations. First, the architecture possesses explicit mechanism for indirection, in the form of a two-column ternal memory. Second, the architecture is separated into two information-processiii that maintains learned embeddings of concrete entities (in our case, images), and one in which recurrent controller learns to represent and operate over task-relevant variables. These two streams only interact in the form of bindings in the external memory, allowing the controller learn to per- form tasks in a manner that is abstracted away from the particular entities involved. We refer to this architecture the Emergent Symbol Binding Network (ESBN), due the fact that this arrangement allows abstract, symbol-like representations to emerge during the learning process, without the need to incorporate symbolic machinery.
streams, one
We evaluate this architecture on suite of tasks involving relationships among images that are gov- erned by abstract rules. Across these tasks, we show that the ESBN capable of learning abstract rules from a limited number of training examples and systematically generalizing these rules to novel entities. By contrast, the other architectures that we evaluate are capable of learning these rules in some cases, but fail generalize them successfully when trained limited number of problems involving a limited number of entities. We conclude from these results that a capacity for variable-binding
necessary component for human-like abstraction and generalization, and that the ESBN is a promising candidate for how to incorporate such a capacity into neural network algorithms.
2 TASKS
2
(a) Same/different
(b) RMTS
(c) Distribution- thethee
(d) Identity rules
Figure 1: Abstract rule learning tasks. Each task involves generalizing rules to objects not seen during training. (a) Same/different discrimination task. (b) Relational match-to-sample task (answer is 2). (c) Distribution-of-tree task (answer is 2). (d) Identity rules task (ABA pattem, answer is 1).
We consider series of tasks, each involving the application of an abstract rule to set of images. For all tasks, we employ the same set of n 100 images, in which each image is a distinct Unicode character (the specific characters used are shown in A.7). We construct training sets in which m images are withheld (where S m S n O, and o is the minimum number images necessary create problem in given task) consisting of problems that employ only the remaining (n m) images, and then test on problems that employ only the m withheld images, thus requiring generalization to novel entities. In the casiest generalization regime (m 0) the test set contains problems composed of the same entities as observed during training (though the exact order of these entities differs). In the most extreme generalization regime, we evaluate models that have only been trained on the minimum number of entities for given task, and then must generalize what they learn to the majority the n images in the complete set. This regime poses an extremely challenging test of the ability to learn to perform these tasks from limited training experience, in a manner that is abstracted away from the specific entities observed during training.
The first task that study is a same/different discrimination task (Figure lak. In this task, two images are presented, and the task is to determine whether they are the same or different. Though this task may appear quite simple, it has been shown that the ability to generalize this simple rule to novel entities is actually a significant challenge for deep neural networks (Kim et al. 2018). a pattern that we also observe in results.
The second task that we consider is relational match-to-san satch-eo--amplle (RMTS) task (Figure 1b). es- sentially higher-order version of same/different task. In this task, source pair of objects is
2

---

Published as a conference paper at ICLR 2021
compared to two targer pairs. The task is to identify the target pair with the same relation as the source pair; e.g., if the source pair contains two of the same object, to identify the target pair that contains two of the same object. It was initially believed that the ability to perform this task is not unique to humans (Premack) 1983), but it has now been shown that this ability depends on a visual entropy confound that arises from using large arrays of objects rather than simple pairs (Fagot al. 2001]. When the task is presented in a manner that does not allow this confound to be exploited (as is the case in our experiments), the ability to perform the task with novel entities appears to be unique to humans, and therefore is good test of the human ability for abstract rule learning.
Next we consider a task based on Raven's Progressive Matrices (RPM: Raven & Court (938 RPM is a commonly used visual problem-solving task, and is one of the most widely used tests of fluid intelligence (Snow et al. 1984), the ability to reason and make inferences in a novel domain (as opposed to crystallized intelligence, the ability to solve familiar tasks). In this task, a 3 x 3 array of figural elements is presented, in which the elements are governed by a simple rule, or set of rules, with the lower right element of the array left blank. The task is to infer the rule that governs the elements in the array, and then use that rule to select from among 8 candidate completions. Many of the rules that govern RPM problems are relations involving sets. One such rule is sometimes referred to as distribution--f of (Carpenter et al. 1990). according to which the same set of three elements (e.g. a triangle, square, and circle) will appear in each row, though the order doesn't 't matter. The task in this case is simply to identify the set, determining which element is missing from the final row, and locating this element among the choices.
Though multiple RPM-inspired datasets have recently been proposed (Barrett et al. 2018 Zhang al. 2019), in this work we choose to strip away unnecessary complexity, focusing on 2 3 arrays governed by a single rule (Figure [T]). in order to focus specifically on the capacity for generaliza- tion of an abstract rule novel entities. We find that, even in this simplified setting, this form of generalization is extremely challenging.
The final task that we consider is visual version of the identity rules task studied by Marcus et al. (1999). In this task, an abstract pattern (e.g. ABA or ABB) must be inferred from a sequence of elements. For instance, in the original study, the following sequence 'ga ni ga, na li, wo fe wo' is governed by an ABA rule, whereas the sequence 'ga ni ni, li na na, wo fe fe' is governed by an ABB rule. This study played an important role in debates concerning the presence of algebraic rule- like processes in human cognition, because it demonstrated that even 7-month-old human infants are capable of detecting this abstract regularity and generalizing it to novel entities, whereas neural networks tend overfit to the specific entities involved and fail to generalize the rule.
In our implementation, we use visual images rather than sounds, and present the task as a x array (Figure Td). In this task, each problem is governed by either ABA, ABB, AAA rule. The task is to determine which of these patterns is present in the first row, and then to apply that pattern by selecting an element from set of choices to complete the second row.
For all four tasks, we consider generalization regimes in which some number of images (m E (0, 50, 85, 95} out of n 100) are withheld from training. For the same/different discrimination task, on which only two images are necessary to construct a problem, we also consider the case in which m 98 (such that the training set consists of problems involving only n m 2 images, the minimum number necessary to construct the task).
In most settings, we construct training sets consisting of 10* problems. This is a tiny fraction of all possible problems (on the order of 10 when the multiple choice options are considered] Thus, even the easiest generalization regime (m = 0) this an extremely small amount of training data relative to the size of the task space. In the most extreme regimes, in which 2 95, it is only possible to construct few hundred problems, resulting in even more limited training experience.
3 APPROACH
For each task, we treat the problem sequence of images #:.1... with an associated target y. In the same/different discrimination task, there are T = 2 images, and y is a binary target indicating whether the images are the same or different. In the RMTS task, there are T = 6 images,
Except for the same/different task, detailed in A.2
3

---

Published as a conference paper at ICLR 2021
consisting of the source pair followed by two target pairs, and is a binary target indicating which target pair matches the source pair. In both the distribution-on theeee task and the identity rules task, there are T 9 images, consisting of the three entries in the first row, the two non-empty entries in the second row, and the four multiple-choice options, and y is a four-way classification target, indicating which of the multiple-choice options is correct.
All images are 32 x 32 grayscale images containing a single Unicode character. For each problem, first process each image independently by a shared encoder fe, generating image embeddings Zt=1... Et= and then pass these embeddings to a sequential model component fa that generates a response (either through a sigmoid output layer for tasks with a binary target, or a softmax layer for tasks with a four-way classification target). The sequential component is either the ESBN or one of a number of alternative architectures described below. We use the same encoder architecture fa (detailed in A.3) for all models. All components are trained end-to-end, including the encoder.
3.1 TEMPORAL CONTEXT NORMALIZATION
We use temporal context normalization (TCN), recently shown to improve out-of-distribution gen- eralization in relational reasoning tasks (Webb al. 2020). TCN is similar to batch normalization, but, instead of normalizing over the batch dimension, normalizes over a task-relevant temporal win- dow. This has the effect of preserving information about the relations between the entities present within this window (e.g. the size of those entities relative to one another), resulting in better gener- alization of learned relations to novel contexts (i.e. auttof-ddisrrbutionn
We found that TCN significantly improved generalization for all of the models on all of the tasks studied in the present work Therefore, the primary results we report all incorporate this technique (A.5.T includes comparison of the performance of all models on all tasks with and without TCN). Specifically, we applied TCN to the embeddings 2t=1... 2t=T extracted by the encoder. |Webb et al. 2020) also reported that it is sometimes useful to apply TCN separately different components of a sequence. We found that this was the case for the RMTS task that we studied, which we found it useful to apply TCN separately to the embeddings for the source pair and each target pair. For all of the other tasks that we studied, TCN was applied over the entire sequence for each problem.
3.2 EMERGENT SYMBOL BINDING NETWORK
M
M,
Figure 2: Emergent Symbol Binding Network. fa consists of LSTM controller plus output layers for y. kw and (not shown). fe is a multilayer feedforward encoder that translates an image æ into a oiw-dinesiional embedding z. These two pathways only interact indirectly via key/value memory.
The ESBN (Figure 2 Algorithm I uses an LSTM controller (fa) with differentiable external memory that is explicitly separated into keys (M₃) and values (M.). At each time step t. a key/value pair written to memory. The keys written to memory, kw are generated by an output layer from the LSTM controller, and the values are the individual input embeddings, 2t, of the input sequence, unmodified by the LSTM. Our hypothesis was that factoring the model into two separate information processing streams would allow the LSTM to learn how to represent abstract variables in the keys
Except for the Net the same/different task, detailed in A.5.1
4

---

Published as a conference paper at ICLR 2021
it generates, which could then be explicitly bound to associated values (image embeddings) learned by the separate encoder network (fe), allowing the ESBN to employ a form of indirection.
To retrieve keys from memory, similarity scores computed by comparing (via dot product) the image embedding & to all of the values in memory M These similarity scores are passed through a softmax nonlinearity generate weights wk,, and passed through sigmoid nonlinearity (with learned gain and bias parameters, Y and 3) to generate confidence values Ck2 (one weight and confidence value per entry in memory). The weights are used compute 1) a weighted sum of all keys in memory M&, 1' and 2) a weighted sum of all associated confidence values Cke- Finally, the retrieved key and associated confidence value are concatenated and multiplied by a learned sigmoidal gate gr to form k,, the input to the LSTM controller at the next time step.
Algorithm 1: Emergent Symbol Binding Network. (Il) indicates the concatenation of vector and a scalar, forming a vector with one additional dimension. { indicates the concatenation of matrix and a vector, forming matrix with one additional row. o is the logistic sigmoid function.
kne + 0:
h==0 0:
Ma <- []:
M t- (}:
for l in 1. ...
21 4- fe(x));
D. 9t. kwo hi fhi-11 kre-));
if t is then
kr. f 0:
else
Wk. + softmax M 21);
Ck. + (MM 21) + +3;;
kn - gt Wk, (1) (i)loc,,)) (1)
end
MR. f- (M b
M f (M 1 z1);
end
return D:=T
3.3 ALTERNATIVE ARCHITECTURES
The simplest alternative architecture that we consider is an LSTM (Hochreiter & Schmidhuber 1997) without external memory. We pass the low-dimensional embeddings 2:=1... Z;=T directly to the LSTM, and generate prediction ÿ by passing the final hidden state through an output layer.
alternative external memory architectures: the Neural Turing Machine (NTM: Graves et al. (2014) and Metalearned Neural Memory (MNM: Munkhdalai et al. (2019)) ). This comparison allows us to determine what extent results depend on the specific details of the ESBN's external memory, and, in particular, the separation between its two information- processing pathways vs. the mere presence of an external memory. Our NTM implementation consists of an LSTM controller (which takes image embeddings input, and generates a prediction ÿ as output) that interacts with an external memory using both content-ban based and location-based read/write mechanisms. Our MNM implementation employs the publicly available code from the original paper, modified so as to employ the same encoder architecture and TCN procedure as the other architectures that test. Just with the ESBN, allow both of these architectures extra time step to process the information retrieved from memory following the final input.
Next we consider
We also consider the Relation Net (RN; Santoro al. (2017)), architecture that has proven to be an effective approach for wide range relational reasoning tasks. In our implementation, we treat the low-dimensional image embeddings as individual 'objects" in the RN framework, using
5

---

Published as a conference paper at ICLR 2021
shared MLP to process all pair-wise combinations of these embeddings, summing the outputs from this MLP, and then passing them to another MLP that generates the prediction ÿ.
We also compare our model against the Transformer (Vaswani et al. 2017), an architecture originally developed in the domain of natural language processing, but has proven to be effective for a wide range of sequential data, and demonstrated capacity for some degree of extrapolation (Saxton et al., 2019). After applying the transformer architecture to the sequence of image embeddings (allowing self-attention between these embeddings), we compute an average of the (transformed) embeddings, and then pass this to a small MLP that then generates the task output ÿ.
Finally, consider the PrediNer (Shanahan et al. 2019). PrediNet was designed with the goal of being "explicitly relational,' and has been shown to be effective at generalizing learned relations to novel entities. We apply the PrediNet's multi-head attention over the ID temporal sequence of image embeddings (as opposed to applying attention over 2D image, as in the original work), and then pass the output of the PrediNet module to a small MLP that generates ÿ.
4 RESULTS
CSON
CSEM
Varnforrer
Thanstrrr
LSTT
LTTM Prectiuat
Pred Mee
ridject williadd (raning)
Talgeet withibt
(a) Sameldifferent
(b) RMTS
EGON
Yonsforrer
Toreform
LSTN
LCTN
inad sae
Prode Seee
23
23-
riddeed tht
redpects thilddd
(c) Distribution-of-tie
(d) Identity rules
Figure 3: Results for all four tasks with m objects withheld (out of 100) during training. Results reflect test accuracy averaged over 10 trained networks (+: the standard error of the mean).
Figure 3 shows the generalization results for all four tasks. Our primary finding is that the ESBN displayed nearly perfect generalization (2 95%) of the learned rule all four tasks, even when trained on very limited number of problems (just hundreds of problems, in the case of the most extreme generalization regimes) involving a limited number of entities (as few as just two entities, in the case of the same/different task), and tested on completely novel entities. Some of the alter- native architectures that we evaluated showed a surprising capability to generalize to novel entities in some tasks as seen, for instance, in the generalization results for the Transformer and RN on the same/different and RMTS tasks (though we note that all architectures incorporate TCN, without which generalization is significantly worse, shown in A.5.Ip. Nevertheless, native architectures were able to generalize what they learned in the most extreme generalization regimes, whereas the ESBN performed comparably well across all regimes.
of these alter-

---

Published as a conference paper at ICLR 2021
Notably, the RN performed very poorly the distribution-of- and identity rules tasks, even in the easiest regime (m = 0). We speculate that this results from the fact that the RN is biased toward pair-wise relations, whereas these tasks are both based ternary relation. Is possible to represent this ternary relation as a combination of pair-wise relations, but doing so requires a more complex strategy and therefore likely more training data. We include results in A.5.2 demoonstannt that the RN is capable of successfully generalizing in this task (though not in the most extreme regimes) when trained on an order of magnitude more data (10% instead of 10 examples). We also present results for the Temporal Relation Network (Zhou et al. 2018), an RN variant that incorporates ternary relations via subsampling, though we find that this doesn't help as much as increasing the amount of training data.
RMTS
(b) Disfrieution-of-hhree
(c) Identity rules
Figure 4: Training accuracy time courses for all models on the m = 0 regime. Each time course reflects an average over 10 trained networks. Error bars reflect the standard error of the mean.
In addition to requiring very small amount of training data and generalizing systematically to novel entities, the ESBN also requires very little training time. Figure [ shows training accuracy time courses for the RMTS, distribution-of-thre ESBN converged to nearly perfect training accuracy within 100 to 200 training updates on all four tasks, whereas the other models required thousands, even tens of thousands of training updates to reach convergence
and identity rules tasks for all models [ The
We also performed some experiments to better understand how the ESBN operates, and why it was so effective. First, we tested whether the systematic generalization exhibited by the ESBN was de- pendent on the use of convolutional layers in the encoder, which naturally confer significant degree of generalization in tasks that involve shape recognition. We found that the ESBN generalized to novel entities comparably well when using either an MLP encoder or random projection (sce/A.5.4) for details), suggesting that the ESBN is capable of generalizing learned rules to any arbitrary set of entities, regardless of how those entities are encoded. For comparison, we also performed the same experiments with the Transformer (the best performing alternative architecture on our tasks) and found that, by contrast, its performance was significantly impaired by the use of random projection instead of convolutional encoder.
Second, we performed an ablation experiment on the confidence value appended to retrieved mem- ories. We found that ablation of these confidence values impaired the ESBN's performance in both the same/different and RMTS tasks, but not the distribution-of-the for details). A likely reason for this result is that the Sestribution-of-thres and identity rules tasks only require retrieval of the best match from memory, whereas the same/different and RMTS tasks require
or identity rules tasks A.55)A.5.5)
of how good of a match that memory is, which is exactly the information that the confidence value conveys. This dissociation mirrors the distinction sometimes made in cognitive psychology between recollection and familiarity (Yonelinas) 2001).
Third, we performed an analysis the key representations learned by the controller. We hypoth- esized that the controller would leam represent abstract variables in the keys that it writes to memory, and that these representations therefore shouldn't 't vary based on the values to which they bound. This analysis revealed a high degree of overlap between the keys written during training
'e found that all models converged within few hundred training updates on the relatively easy same/different discrimination task, so those time courses are omitted here, but shown in A.5.3
*Note that all models eventually reached convergence, though this required that some of them be trained longer than is depicted in these figures, detailed in A.4

---

Published as a conference paper at ICLR 2021
and test (involving entirely different entities), suggesting that this was indeed the case (see A.6 for details). This ability to arbitrarily bind values to variables, without affecting the representations of those variables, is a key property of systematic generalization exhibited by the ESBN.
systems, and is likely the basis of the strong
RELATED WORK
There have been number of proposals for augmenting neural networks with external mem- ory. An influential early line of work, Complementary Learning Systems (McClelland et al. 1995), proposed that neural systems benefit from having components that learn different time scales, and argues that this combination allows neural networks both to learn general, abstract structure (using standard learning algorithms) and to rapidly encode arbitrary new items (using external memory). In recent years, there have been a number of proposals for how to implement the lat- ter efficiently, including Fast Weights (Ba et al. 2016ap, the NTM and closely related Differentiable Neural Computer (Graves et al. 2016), and the Differentiable Neural Dictionary (DND;Pritzel Pritzel et al.) 2017 Our external memory approach is most closely related to the DND, which also involves a two-column key/value memory. Variations on key/value memory have also been employed in other more recently proposed approaches, such as the Memory Recall Agent (Fortunato et al. 2019) and the Dual-Coding Episodic Memory (Hill et al. 2020). where it afforded various benefits in terms of generalization. One critical difference between our model and this previous work is that the ESBN's controller is forced to interact with perceptual inputs only indirectly through its memory, design decision that we argue is crucial to its ability to systematically generalize what it learns.
It is worth noting that architectures such as Fast Weights and the NTM are, in principle, capable of implementing variable- variable-binding, though it is a separate question whether such a strategy will result from learning in any particular task. Along these lines, recent study from Chen et al. (20T9) found that both of these architectures are capable of generalizing learned structure to novel entities when allowed a sufficiently dense sampling of the space of potential objects (the 'objects' in their study were randomly sampled 50-dimensional vectors). This contrasts with our findings, in which the NTM performed poorly when trained on far fewer samples from a much nigher-dimensional space (in the m = 95 regime). This suggests that indirection and variable-binding principle for architectures such as the NTM, do not emerge in practice when given only limited amount of training experience, whereas this capacity is explicitly built into the ESBN.
though possible in
At a high level, the idea of factoring a model into two distinet information processing streams, one that codes abstract task-relevant variables or roles and one that codes concrete entities, has been ex- plored before. Kricte et al. (2013) proposed the PBWM Indirection model, in which one population of neurons acted as pointer to another population of neurons by gating its activity, and showed that this model enabled a significant degree of generalization to novel role/filler bindings. [Whittington et al. (2019) proposed the Tolman- Eichenbauh machine, model that is capable of learning abstract relational structure (such as 2D spatial maps), and showed that this model captured a number of phenomena relating to grid cells and place cells. |Russin et al. (2019) proposed Syntactic Attention, an architecture involving separate pathways for processing syntax vs. semantics, and showed that this approach was capable of a significant degree of compositional generalization on the challeng- ing SCAN dataset. Relative to this previous work, our central contribution is the development of a simple model that can learn abstract rules directly from high dimensional data (images), exploiting this same high-level idea to enable nearly perfect generalization of those rules to novel entities.
There has been extensive modeling work focusing on some of the tasks that we study. The recent development of two datasets modeled after Raven's 's Progressive Matrices, Procedurally Generated Matrices (Barrett et al., 2018), and RAVEN (Zhang et al.) 2019), has spurred the development of models that are capable of solving RPM-like problems (Jahrens & Martinetz) [220 Wu et al.) 2020). However, these models typically require very large training sets (on the order of 10th training exam- ples), and largely fail to generalize outside of the specific conditions under which they are trained, whereas the ESBN exhibits the ability to learn rapidly and generalize nut-of-distribution
There have also been number of models proposed to account for the human ability to rapidly learn identity rules (Alhama & Zuidema, 2019). Though some of these models achieved significant generalization of learned identity rules to novel entities, they did so mostly through the inclusion of
8

---

Published as a conference paper at ICLR 2021
highly task-specific mechanisms. By contrast, our aim in the present work was present general approach that could be applied to a wider range of tasks.
Finally, there have been number of recent proposals for so-called 'neurosymbolic" models, incor- porating elements from both the neural network and symbolic modeling frameworks (Mao et al. 2019 Nye et al., 2020). Though we have emphasized the notion of "emergent symbols" in the present work, we stress that this is quite distinct from neurosymbolic modeling efforts since we do not explicitly incorporate any symbolic machinery into the ESBN. Instead, our approach was to show how the functional equivalent of symbols can emerge in a neural network model with an appropriate architecture and binding mechanism.
6 DISCUSSION
6.1 LIMITATIONS AND FUTURE WORK
One open question is whether the strict division between the two information processing streams in the ESBN is necessary, and whether it limits the sorts of relations and rules that can learn. future work, it may be desirable to soften this division, for instance by encouraging it in a regularization term, rather than strictly enforcing architecturally.
A second limitation is that the tasks we study are not as complex as other similar tasks that have recently been studied, such as the two recently proposed RPM-like benchmarks (Barrett et al. 2018 Zhang et al. 2019). In the present work, intentionally stripped away some of this complexity in order to make progress on the issuc of out-of-distribution generalization. Extending the ESBN to more complex tasks will likely require the incorporation of visual attention mechanisms enable selective sequential processing of individual elements within a scene. There are many recently proposed approaches for doing this (Gregor et al.) 2015: Locatello et al.) 2020). In future work, look forward to extending the ESBN in this manner and testing it on more complex tasks.
6.2 RELATION TO WORK NEUROSCIENCE
It is worth considering how the present work relates to pre-existing theories of how the brain might implement variable-binding. Classic proposals for variable-binding in neural systems emphasize dynamic binding of representations. either by computing the tensor product between those rep- resentations (Smolensky) 1990). or by establishing synchronous activation between two pools of units (Hummel & Holyoak 1997). An alternative proposal is that variable- binding is accomplished via semi-permanent synaptic changes in the hippocampus, relying on the same mechanism that plays central role in episodic memory (Cer & O'Reilly) 2006). This approach relies on contextual in- formation and retrieval processes to prevent potential interference between conflicting memories, rather than explicit unbinding mechanisms. Our model is more line with the latter account, since it does not possess an unbinding mechanism. As such, our model can be seen as part of a recent trend toward the reinterpretation of putatively working memory functions in terms episodic mem- ory (Beukers et al. 2020).
CONCLUSION
In this work, we have presented a model of abstract rule learning based on a novel architecture, the ESBN, and shown that this model is capable of rapidly learning abstract rules directly from images given only small amount of training experience, and then successfully generalizing those rules to novel entities. Key to the model's 's performance is its separation into two streams that only interact through indirection, allowing the ESBN to learn tasks in a manner that is abstracted away from the specific entities involved, and resulting in the emergence of symbol-like representations. We believe that these results suggest that such variable- variable-binding capacity is essential ingredient for achieving human-like abstraction and generalization, and hope that the ESBN will be a useful tool for doing so.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like to thank Zachary Dulberg, Steven Frankland, Randall O' O'Reilly, Alexander Petrov, and Simon Segert for their helpful feedback and discussions.
REFERENCES
Raquel Alhama and Willem Zuidema. review computational models basic rule learning: The neural-symbolic debate and beyond. Psychonomic bulletin & review,
2019.
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, pp. 4331-4339, 2016a.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 2016b.
David GT Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. arXiv preprint orXiv:/807.04220 2018.
Andrew Beukers, Kenneth A Norman, and Jonathan D Cohen. Is activity silent working memory just episodic memory?, 2020.
Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: theo- retical account of the processing in the raven progressive matrices test. Psychological review, 97 (3):404, 1990.
Daniel M Cer and Randall C O'Reilly. Neural mechanisms of binding in the hippocampus and neocortex: insights from computational models., 2006.
Catherine Chen, Qihong Lu, Andre Beukers, Christopher Baldassano, and Kenneth Nor- man. Learning to perform role-filler binding with schematic knowledge. arXiv preprint 1902.09900000 2019.
Joël Fagot, Edward A Wasserman, and Michael E Young. Discriminating the relation between relations: the role of entropy in abstract conceptualization by baboons (papio papio) and humans (homo sapiens). Journal of Experimental Psychology: Animal Behavior Processes, 27(4)3316, 2001.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomênech Badia, Gavin Buttimore, Charles Deck, Joel Z. Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. Advances in Neural Information Processing Systems, pp. 12469-12478. 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 110:5:1410.5401, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-47,, 2016.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint 1rXiv:152..04623 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1034-1034, 2015.
10

---

Published as a conference paper at ICLR 2021
Felix Hill, Olivier Tieleman, Tamara Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.01719, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8):
1735-1780, 1997.
Keith J Holyoak and John E Hummel. The proper treatment of symbols in connectionist architec- ture. Cognitive dynamics: Conceptual change in humans and machines, 229:263, 2000.
John E Hummel and Keith J Holyoak. Distributed representations of structure: A theory of analog-
ical access and mapping. Psychological review, 104(3):427, 1997.
Marius Jahrens and Thomas Martinetz. Solving raven's 's progressive matrices with multi-layer rela- tion networks. arXiv preprint 110v: 2020.
Junkyung Kim, Matthew Ricci, and Thomas Serre. Not-so-clevr: learning same came-differemt relations
strains feedforward neural networks. Interface focus, 814): 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1::1412.6980, 2014.
Trenton Kriete, David Noelle, Jonathan Cohen, and Randall C O'Reilly. Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences, 110(4)):13300-1333 2013.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequencce-o--eequeene recurrent networks. In International Conference on Machine Learning, pp. 2873-2882, PMLR, 2018.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint urXiv:2006. 15055, 2020.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 1904. 12584, 2019.
Gary Mareus. The algebraic mind, 2001.
Gary F Marcus, Sugumaran Vijayan, S Bandi Rao, and Peter Vishton. Rule learning by seven- month-old infants. Science, 283(5398) 1999.
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearndd neural memory. Advances in Neural Information Processing Systems, pp. 13331-13342, 2019.
Maxwell Nye, Armando Solar-Lezama, Joshua B Tenenbaum, and Brenden M Lake. Learning compositional rules via neural program synthesis. arXiv preprint arXiv:2003.05502 2020.
David Premack. The codes of man and beasts. Behavioral and Brain Sciences, 6(1):125-136, 1983.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. arXiv preprint 17X1v:1703.01988. 2017.
John C Raven and JH Court. Raven's 's progressive matrices. Western Psychological Services Los Angeles, CA, 1938.
Jake Russin, Jason Jo, Randall C O' O'Reilly, and Yoshua Bengio. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv: X0XXix::440000 /904.09704... 2019.

---

Published as a conference paper at ICLR 2021
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities of neural models. arXiv preprint arXiv:1904.01557, 2019.
Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell, Christos Kaplanis, David Barrett, and Marta Garnelo. An explicitly relational neural network architecture. arXiv preprint 1rXIv::905.10307, 2019.
Ishan Sinha, Taylor W Webb, and Jonathan D Cohen. memory-angmented neural network model of abstract rule learning. arXiv preprint arXiv:2012.07172, 2020.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159-216, 1990.
Richard Snow, Patrick C Kyllonen, and Brachia Marshalek. The topography of ability and learning correlations. Advances in the psychology of human intelligence, 2(S 47): 1984.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Taylor W Webb, Zachary Dulberg, Steven M Frankland, Alexander Petrov, Randall O O'Reilly, and Jonathan D Cohen. Learning representations that support extrapolation. arXiv preprini arXiv:2007.05052 2020.
James CR Whittington, Timothy H Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy Behrens. The lolman-eichenbaum machine: Unifying space and relational mem- ory through generalisation in the hippocampal formation. BioRxiv, pp. 770495, 2019.
Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arX1v:2007.04212, 2020.
Andrew P Yonelinas.
Consciousness, control, and confidence: the cs of recognition memory. Journal of Experimental Psychology: General, 130(3):361, 2001.
2001.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Kong-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
A APPENDIX
A.1 CODE AVAILABILITY
All code, including code for dataset generation, model implementation, training, and evaluation, is available GitHub.
A.2 DATASET GENERATION
In this section, provide details on the dataset generation process for all tasks. In all of our simulations, a dataset was generated from scratch (according to the procedures described below) at the beginning of each training run, such that different statistics were the same across these datasets. We did this to prevent the possibility that our results would reflect biases present in particular dataset.
involved different datasets, though the
12

---

Published as a conference paper at ICLR 2021
Table 1: Training and sizes for the same/different discrimination task.
50
85
95
98
Training
18,810
4,900 4,900
420
40
4 10,000
Test 990
10,000
10,000
A.2.1 SAME/DIFFERENT DISCRIMINATION
Given n = 100 total images, with m O withheld during training, there are n² = 10 possible same/different problems. To prevent the potential for networks to be biased by the fact that the overwhelming majority of these are 'different" problems, we created balanced datasets by including duplicates of the 'same' problems. Specifically, we randomly sampled (with replacement) n(n of the unique 'same' trials and combined them with the n - 1) unique "different" trials, resulting in 2n(n - 1) = 19, 800 total problems. We reserved 990 of these problems for test, yielding training sets including 18, 810 problems (ensuring that duplicates of the same problem did not appear in both the training and test sets).
We followed similar procedure for the other regimes, generating balanced datasets by duplicating the 'same' problems when necessary. These datasets incorporated either all of the problems that resulted from this procedure (given the n-m images available for training, the images available for test), or 10, 000 problems, whichever was smaller. The exact size of each of these datasets is shown in Table
A.2.2 RMTS
Table 2: Training and test set sizes for the RMTS task.
m
0 m
m
85 m 95
Training
10,000
10,000
10,000
480
Test
10,000
10,000
10,000
10,000
For the RMTS task, we constructed balanced training and test sets ensuring that there were an equal number of problems with 'same" 'different' source pair. These datasets contained either 10, 000 problems, or the minimum number of problems possible in given regime, whichever was smaller (Table For most regimes. 10, 000 problems constitutes tiny fraction of the full space of possible problems (ranging from 109h for the m 0 regime to 10 for the training set in the m 85 regime), and thus there was no need to duplicate problems to achieve balanced datasets. For the m 95 regime, there are only 480 possible training problems, which happen to include the same number of 'same' and 'different' trial types.
A.2.3 DISTRIBUTION-OF-THRER
Table 3: Training and test sizes for the distribution-of-thres task.
m = 0 m = 50
m = 85
95
Training
10,000
10,000
10,000
360
Test 10,000
10,000
10,000
10,000
For the distribution-of-three task, generated problems by randomly selecting three of the avail- able images in a given regime (either n m images during training, or m images during test), and then randomly sampling permutations of those images for the rows (allowing the possibility
13

---

Published as a conference paper at ICLR 2021
that the same permutation appears in both rows) of the 2 x 3 matrix. We then randomly selected fourth image to appear with the other three as possible answers, and randomly permuted these four answer choices. When taking into account the identity of this fourth image, and the permutation of the answer choices, the number of unique distribution-of-thee problems in the m = 0 regime is on the order of 10
For most regimes, we randomly created training and test sets consisting of 10, 000 randomly sampled problems. For the m = 95 regime, the training set consisted of 360 problems (the total number of unique problems possible in this regime when not considering the identity and order of the answer choices, which were randomly selected).
A.2.4 IDENTITY RULES
Table 4: Training and test set sizes for the identity rules task.
0
50
85
95
Training
10,000
10,000
10,000
8,640
Test
10,000
10,000
10,000
10,000
For the identity rules task, we constructed datasets with an approximately balanced (through uniform random sampling) number of ABA, ABB, and AAA problems. These datasets consisted of either 10, 000 problems, or the minimum number of possible problems in a given regime, whichever was smaller (Table For the training set in the m 95 regime, datasets consisting of 8, 640 problems were constructed from the 7, 200 possible unique problems in this regime, by duplicating the AAA problems to match the number of ABA/ABB problems. For all other datasets, 10, 000 problems constituted small fraction of the total number of possible problems (ranging from 10 for the m regime to 10 for the training set in the m = 85 regime), and no duplication was necessary to achieve balanced problem types.
A.3 IMPLEMENTATION DETAILS FOR ALL MODELS
A.3.1 ENCODER
We used the same feedforward encoder architecture to process each of the images in sequence Tt=1... Tr=T, generating low-dimensional embeddings 21=1... that were then passed to the core sequential component of each model (either the ESBN or one of the alternative architectures [ This encoder consisted of three convolutional layers, each with 32 channels, a 4 x 4 kernel, and a stride of followed by two fully-connected layers with 256 units and 128 units respectively. All layers used ReLU nonlinearities. All weights were initialized using a Kaiming normal distribution (He et al. 2015), and all biases were initialized to O.
A.3.2 TASK OUTPUT LAYER
All models had an output layer for generating ÿ. The number of units and the nonlinearity depended on the task. For the same/different and RMTS tasks, the output layer had unit and a sigmoid nonlinearity (producing a number between 0 and to code for 'same" vs. 'different', or pair vs. pair 2). For the distribution-of-thie nonlinearity (to select of the answer choices). The weights of the output layer were initialized using an Xavier normal distribution (Glorot & Bengio 2010), and the biases were initialized to 0.
and identity rules tasks, the output layer had 4 units and softmax
A.3.3 ESBN
The details of the ESBN's operations are given in Algorithm ! The LSTM controller had 1 layer with 512 units, and employed the standard tanh nonlinearities and sigmoidal gates. The controller
'The encoder architecture was the same for all models, but the parameters of the encoder were learned end-to-end from scratch each training run.
14

---

Published as a conference paper at ICLR 2021
also had output layers for kw (256 units with ReLU nonlinearity), (1 unit with sigmoid nonlin- earity), and ÿ. The input to the controller at each time step was k,,, the key retrieved from memory at the previous time step (along with the associated confidence value, cg). At the beginning of each sequence, k and the controller's hidden state h were initialized to 0.
After processing full sequence, the ESBN was allowed an additional time step for the controller to process the retrieved key associated with the final input. After this additional time step, the final hidden state of the LSTM was passed through the task output layer to generate the prediction ÿ. We note that it is also possible to retrieve values from memory (from M,) using a similar procedure and then decode these values make predictions in image space (Sinha et al. 2020), but in the present work we focus only on the classification component of the model.
The input weights for the LSTM controller were initialized using an Xavier normal distribution with gain value of 5/3. The weights for the LSTM's s gates, as well as the weights for the output layer for the gate g. were initialized with an Xavier normal distribution (with a gain of 1). The weights for the output layer that produced kn were initialized using a Kaiming normal distribution. All biases were initialized to O. The parameters 7 and 3 were initialized to and 0 respectively.
A.3.4 LSTM
The LSTM architecture had 1 layer with 512 units. Image embeddings were passed the LSTM as sequence, after which was generated through a task output layer. The LSTM's hidden state was initialized
the beginning of each sequence. The LSTM's weights were initialized using the same scheme as the LSTM controller in the ESBN (using an Xavier normal distribution, with gain of 5/3 for the input weights and for the gates), and biases were initialized to O.
A.3.5 NTM
The NTM had an LSTM controller (1 layer with 512 units). The LSTM's hidden state was initialized to at the beginning of each sequence, and the LSTM's parameters were initialized in the same way as the LSTM architecture and the controller for the ESBN. The NTM had one write head and one read head. The read head had the following output layers:
1. read key: 256 units, tanh nonlinearity, weights initialized using an Xavier normal distribu- tion with a gain of 5/3.
2. key strength: unit, softplus nonlinearity, weights initialized using a Kaiming normal distribution.
3. interpolation gate: unit, sigmoid nonlinearity, weights initialized using an Xavier normal distribution with gain of 1.
4. shift weights: 3 units (corresponding to the allowable shifts 1,, 0, and 1), softmax nonlin- earity, weights initialized using Xavier normal distribution with a gain of 1.
The write head had the following output layers:
1. erase vector: 256 units, sigmoid nonlinearity, weights initialized using an Xavier normal distribution with a gain of
2. add vector: 256 units, tanh nonlinearity, weights initialized using an Xavier normal distri-
bution with a gain of 5/3.
3. write key: 256 units, tanh nonlinearity, weights initialized using an Xavier normal distribu-
tion with a gain of 5/3.
4. key strength: 1 unit, softplus nonlinearity, weights initialized using a Kaiming normal distribution.
5. interpolation gate: unit, sigmoid nonlinearity, weights initialized using an Xavier normal distribution with gain of 1.
6. shift weights: 3 units (corresponding to the allowable shifts -1, O, and 1), softmax nonlin- earity, weights initialized using Xavier normal distribution with gain of 1.
All biases for these output layers were initialized to O. The NTM used these outputs to interact with its external memory, employing all of the location- and content-based mechanisms described in the original work (Graves et al. 2014). Cosine similarity was used as a similarity measure for the content-based mechanisms. The memory matrix had 10 rows of size 256. The initial state of the
15

---

Published as a conference paper at ICLR 2021
memory at the beginning of each sequence was learned. The learned initial state was initialized (at the beginning of training) using an Xavier normal distribution.
The imput to the LSTM controller at each time step consisted of the image embedding corresponding to that time step and the read vector from the previous time step. At the beginning of each sequence, the read vector, read weights, and write weights NTM was allowed an additional time step to process the read vector retrieved from memory after observing the final image embedding, after which ý was generated through an output layer from the LSTM controller.
initialized to O. Just with the ESBN, the
A.3.6 MNM
We implemented the MNM using publicly available code released with the original publica- tion (Munkhdalai et al., 2019). Specifically, we used the version MNM that employs a learned local update ('MNM-p' in the original paper). Before passing the images in our tasks to the MNM model, we applied the same encoder and TCN procedure used for the other architectures that we tested. Other than this modification, the original implementation, including all architectural hyper- parameters, was unmodified.
A.3.7 RN
The RN implementation consisted of two MLPs. The first MLP (used to process pair-wise com- binations of image embeddings) had hidden layer of size 512 and an output layer of size 256. The outputs from the first MLP were summed, and then passed to the second MLP, which had a hidden layer of size 256 and an output layer for generating ÿ. All layers (except the output layer) used ReLUU nonlinearities. All weights were initialized using a Kaiming normal distribution (except the output layer, which initialized to 0. Before passing the image embeddings to the first MIP, they were appended with a tag (an integer from 0 T 1) indicating their position in the input sequence.
initialized according to the description in A.3.2). and all biases were
A.3.8 TRN
The TRN employs two key design decisions intended prevent the combinatorial explosion that would naturally result from the inclusion of n-ary relations:
1. Only considering temporally ordered, non-redundant sets (whereas the original RN consid- ers all possible pairs of objects, including both permutations of the same pair, and the pair of each object with itself).
2. Subsampling from these sets.
We found that it was computationally feasible to implement TRN with ternary relations in our tasks by only using (1), without the need to subsample. Thus, our implementation considers all temporally ordered, non-redundant sets of two and three.
Each pair of image embeddings was processed by an MLP with hidden layer of size 512 and an output layer of size 256. The outputs of this MLP for all pairs were then summed, and processed by an additional fully-connected layer with 256 units, yielding single vector representing all pairs.
Each set of three image embeddings was processed by separate MLP with the same hyperparam- eters (hidden layer of 512 units, output layer of 256 units). The outputs of this MLP for all sets of three were then summed, and processed by a separate fully-connected layer with 256 units, yielding single vector representing all sets of three.
These two vectors, representing all pairs and sets of three, were then summed and passed to an additional fully-connected layer with 256 units, and then to the output layer to generate ÿ. All layers (except for the output layer) used ReLU nonlinearities. All weights in these layers were initialized using Kaiming normal distribution, and all biases were initialized to 0.
Just as with the RN, we append the image embeddings with a tag indicating their position in the sequence before passing them to the first MLP in the TRN.
16

---

Published as a conference paper at ICLR 2021
A.3.9 TRANSFORMER
The Transformer implementation consisted of single Transformer encoder layer. We also experi- mented with 2- and 3-layer Transformers but these did not generalize as well as the 1-layer Trans- former in the tasks that we studied. Positional encoding (as described by Vaswani et (2017) was applied to the sequence of image embeddings, which were then passed to the Transformer layer. The self-attention layer had 8 heads. The MLP had a single hidden layer with 512 units, and used ReLU nonlinearities. Residual connections and layer normalization (Ba et al., (016b) were used following both the self-attention layer and the MLP. The self-attention weights (for generating the keys, queries, and values) were initialized using Xavier normal distribution. The MLP weights were initialized using a Kaiming normal distribution.
After applying the Transformer layer, the (transformed) embeddings were averaged and passed to an output MLP. The output MLP had a single hidden layer with 256 units, and an output layer for generating ÿ. The hidden layer used ReLU nonlinearities, and the weights were initialized using Kaiming normal distribution. All biases were initialized to O.
A.3.10 PREDINET
The PrediNet implementation was as close as possible to the model described in the original work (Shanahan et al. 2019), except that the multi-head attention poral sequence of image embeddings, rather than over a 2D feature map (since there was no spatial component to the tasks that studied). Before being passed to the PrediNet module, the image embeddings were appended with a tag (an integer from 0 to T - 1) indicating their temporal position. The PrediNet module used keys of size 16, 32 heads, and 16 relations. All weights in the PrediNet module were initialized using an Xavier normal distribution.
applied over the ID tem-
The output of all PrediNet heads was concatenated and passed to output MLP. This MLP had single hidden layer with 8 units, and an output layer for generating ÿ. The hidden layer used ReLU nonlinearities, and the weights were initialized using a Kaiming normal distribution. All biases were initialized to O.
A.4 TRAINING DETAILS
Table 5: Learning rates for all models trained without TCN.
Same/different
RMTS
Distribution-off offthhee
Identity rules
ESBN
5e 5
5c
5e 5
5c 5
Transformer
5e 4
5e 4
5e 4
5e 4
NTM
5e 4
5e
5e
5e 4
MNM
5
5e 4
5e 4
5e 4
LSTM
5
5e-4 4
5e 4
5e
PrediNet
5e 4
5e
5e 5
5e 5
RN
5 4
5
5e-4 4
5e 4
All models were trained with a batch size of 32 using the ADAM optimizer (Kingma & Ba, 2014).
The learning rate for all models trained with TCN was 5e 4. Some of the models failed to converge when trained without TCN, requiring smaller learning rate of 5e 5. The learning rates used for all models when trained without TCN are shown in T
Because different generalization regimes (different values for m) involved different training set sizes, and therefore involved fewer training updates per epoch, the number of training epochs re- quired to reliably achieve convergence varied based on the regime. The default number of training epochs for all tasks and regimes shown in Table
Some models required additional training the RN required longer training on the distribution-of- task (Table 7). and the PrediNet, RN, and Transformer required longer training the identity rules task (Table
tasks reach convergence. The PrediNet and
17

---

Published as a conference paper at ICLR 2021
Table 6: Default number of training epochs for all tasks and regimes.
0
50
85
95
98
Same/different
50
50
50
100
100
RMTS
50
50
50
200
Distribution-of-tle
50
50
50
150
Identity rules
50
50
50
50
Table 7: Number of training epochs for the PrediNet and RN on the distribution-of-three task.
m
50
85
m
95
PrediNet
100
100
100
150
RN
150
150
150
800
Table 8: Number of training epochs for the PrediNet, RN, and Transformer on the identity rules task.
m
0
172
50
17
85
m
95
100
100
100
150
When training the RN on larger datasets for the citribution-of-theee and identity rules tasks, the same learning rate and number of training epochs used when training on smaller datasets was sufficient to reach convergence.
A.5
SUPPLEMENTARY RESULTS
A.5.1 RESULTS WITH AND WITHOUT TCN
Tables 9 12 show the results for all models trained both with and without TCN. With the exception of the PrediNet on the same/different task, every model benefited on every task from the incorpo- ration of TCN, in many cases substantially. Results for models trained with TCN (indicated by *+ TCN') correspond to the results presented in Figure 3 (except for the results of the PrediNet on the same/different task, for which the version of the model trained without TCN is plotted in Figure 3)
We note that, even with lower learning rate 5e 5, some models failed to converge without TCN, such the ESBN the same/different task, or the RN on the RMTS task. It is possible that some of these models might have performed better we had optimized them further by training for longer trying different learning rates, but opted not to do that since TCN was so effective across all of the models and tasks that we studied.
18

---

Published as a conference paper at ICLR 2021
Table 9: Results for same/different task. Results reflect test accuracy averaged over 10 trained networks (#t the standard error of the mean).
m 0
772 = 50
772 = 85
771
95
m = 98
ESBN TCN
100.0 0.0
100.0 0.0
100.0 + 0.0
1.0
100.0 10.0::000
ESBN
50.0 0.02
+.0
50.1
4.2
5.1
Transformer + TCN
0.0
100.0
1.0
1.0
72.3
Transformer
0.0
9.02
9.64
1.8
1.3
NTM + TCN
0.0
9,099
9.69
6.57
53.3 1.4
NTM
9.9
0.3
2.
2..0
5.9
MNM + TCN
0.0
99.95
9.4
7..0
52.3 0.5
MNM
9.1
1.8
8.16
1.61
51.7
LSTM + TCN
100.0
99.97 0.01
9
1.5
54.8 + 1.1
LSTM
88.2 3.2
0.5
8 2.4
+.7
56.5 + 1.6
PrediNet TCN
100.0 0.0
0.1
9.3
6.92
2.3
PrediNet
0.00
0.09
9..
9.60
2.
RN TCN
100.0 0.0
0.0
0.0
9..9
66.8 - 6.6
RN
99.98 0.02
0.4
5..2
50.5
52.3 0.7
Table 10: Results for relational mateh-to-sample task. Results reflect test accuracy averaged over 10 trained networks (=: the standard error of the mean).
50
85
95
ESBN + TCN
100.0 + 0.0
100.0 0.0
1.0.0
+ 0.7
ESBN
66.46..1
6.5
5..0
51.0
Transformer
1.0
99.98 0.01
9..1
2.5
Transformer
9..4
0.7
86.4
4.2
NTM + TCN
100.0
99.97
96.8 4 0.5
80.1 2.3
NTM
99.5 +
9.7
81.2 1.5
50.1
MNM + TCN
99.99 =
0.03
98.7
5.2
MNN
74.6
6.76
78.3 = 3.7
5..0
LSTM TCN
99.99 =
9.03
94.9 = 1.3
6.7
LSTM
99.1
2.0
80.9 1.1
5.1
PrediNet TCN
99.7
9.1
94.6
2.74
PrediNet
54.9
50.1 0.2
65.9
49.7 0.2
0.2
RN TCN
+.00
99.99 +.0
99.5 0.3
79.6 + 2.1
RN
50.1 + 0.2
4.2
50.2
50.0 + 0.1
19

---

Published as a conference paper at ICLR 2021
Table 11: Results for distribution-of-trre task. Results reflect test accuracy averaged over 10 trained networks (#t the standard error of the mean).
772
772
50
772 = 85
172 = 95
ESBN TCN
98.7 0.4
99.0 0.3
99.5
99.7 0.1
ESBN
99.98 + 0.0
9.2
62.0 9.1
Transformer + TCN
2.6
9.5 9.2
32.1
Transformer
3.3
6..6
7..6
2.,0
NTM + TCN
0.4
9,4
9..3
34.0
NTM
0.5
8..1
7.
2.3
MNM + TCN
9.3
9.4
9..6
3.6
MNM
8.9
6.2
4..4
2..6
LSTM + TCN
9.6
0.5
9.
3.8
LSTM
9.6
1.5
4.
27.5
0.3
PrediNet TCN
0.3
9..6
93.
27.8
PrediNet
3.0
6..7
7.00
2.1
RN TCN
3.0
7.,6
2.3 7.2
RN
25.1 0.1
2..9
2.37
2..2
Table 12: Results for identity rules task. Results reflect accuracy averaged over 10 trained networks (#t the standard error of the mean).
m =
7 = 50
772 = 85
m7 = 95
ESBN TCN
99.6
99.6 0.1
99.9 0.04
99.2 0.4
ESBN
99.4 0.1 0.0
9..2 9.2
Transformer TCN
9.73
9.0
9.70
67.1
Transformer
4.1
7.1
4.64
38.6
NTM TCN
0.6
0.5
9.6
6.2
NTM
9.3
90.1
8.2
2.1
MNM + TCN
9.42
93.
9.5
1.5
MNM
9.7
4..
0.2
LSTM + TCN
0.1
97.7
9.71
62.5 1.1
LSTM
0.5
0.6
7.77
24.8
PrediNet + TCN
9.8
9,7
8..8
29.9
PrediNet
4.4
1.9
40.3
0.66
RN + TCN
6.7
4.0
48.7
41.4
RN
41.17.2
3..3
31.6
0.4
20

---

Published as a conference paper at ICLR 2021
A.5.2 PERFORMANCE OF RN ON TERNARY RELATIONS
Table 13 shows the results for the RN (w/ TCN) on the distribution-of-hhree and identity rules tasks when trained on larger training sets (10' instead of 10 training examples). These results show that with more training data, the RN, which is biased toward processing pair-wise relations, is able to learn these tasks (which are based on ternary relations) in a manner that enables some degree of generalization. Note that these results not include the m 95 regime, because there are not enough images in that regime to create larger training sets than were originally used.
Table 13: Results for the RN on the distribution-on theeh and identity rules tasks when trained on a larger training set. Results reflect test accuracy averaged over 10 trained networks (+- the standard error of the mean).
m
m 50
m 85
Distribution-o-tthree
84.5 + 10.0
84.6 + 9.9
72.4 + 9.7
Identity rules
89.2 + 5.0
99.7 + 0.1 86.6 + 4.1
We also tested the TRN, which incorporates ternary relations through subsampling, on these tasks (with the standard training set size of 10 training examples). Table 14 shows the results. This yielded slight improvement over the RN (when trained on 10* training examples). though not as much of an improvement as resulted from training the RN with larger training set. This result may seem surprising given that the TRN explicitly incorporates ternary relations. We note two possible explanations for this result:
1. The systematic comparison of every pair of objects, including permutations and compar- isons of each object with itself, allows the RN to take advantage of very powerful form of data augmentation, enforcing a certain degree of systematicity in the relations that it learns. By only considering temporally ordered and non-redundant sets, the TRN not able to take advantage of this to the same extent, and therefore might not learn relations that generalize as well.
2. The distribution-of-thiee and identity rules tasks both involve not only ternary sets, but the higher-order comparison of multiple pairs of ternary sets (the first row vs. the combination of the second row with each candidate answer). One could presumably engineer solution to this problem within the RN framework, but we take it as strength of the ESBN that no such special engincering necessary in this case.
Table 14: Results for the TRN on the distribution-of-thre and identity rules tasks. Results reflect test accuracy averaged over trained networks (#: the standard error of the mean).
m 50 1
85
m 95
Distribution-of-thre
60.2 5.4
77.5 1.5530 5.7
88.7 + 0.8
27.8 - 0.5
Identity rules
40.3 + 2.0
43.6 2.0
52.8 2.7
44.9 + 1.0
A.5.3 TRAINING TIME COURSES FOR SAME/DIFFERENT TASK
Figure 5 shows the training time courses for all models on the same/different task. Unlike the other three tasks we studied (for which training time courses are shown Figure D. all models were able to learn this task within few hundred training updates (though all models except the ESBN failed generalize in the most extreme regime).
A.5.4 ALTERNATIVE ENCODER ARCHITECTURES
In order determine whether the systematic generalization exhibited by the ESBN depended to some extent on the convolutional layers in its encoder, we performed experiments with two alterna- tive encoder architectures: a multilayer perceptron (MIP) encoder, and random projection.
21

---

Published as a conference paper at ICLR 2021
100
ESBN
Transformer
MNM
LSTM
PrediNet
training accuracy
60
RN
40
0-
0
50
100
150
200
250
300
batches
Figure 5: Training accuracy time courses on m = 0 regime of the same/different task. Each time course reflects an average over 10 trained networks. Error bars reflect the standard error of the mean.
CSER temm MNL
223
mnnndiiwe jound. Joua
wereerahnng treneg!
(a) Sameldifferent
(b) RMTS
CCON Icemo
CSON toend
Vonaboma Scand
Vonatoma
Vorsffrmmr ponct
(c) Distribution-or cinibbution-of-hrree th-rhr
(d) Identity rules
Figure 6: Results for all four tasks with convolutional (conv), multilayer perceptron (MLP), or random (rand) encoders. Results reflect test accuracy averaged over 10 trained networks (#: the standard error of the mean).
The MI. encoder consisted of 3 fully-connected layers, with 512, 256, and 128 units, each of which used ReL.U nonlinearities. All weights were initialized using Kaiming normal distribution, and all biases were set to 0.
The random projection encoder involved only single, untrained, fully-connected layer that pro- jected from the flattened image to 128 units, followed by a ReLU nonlinearity. Weights were sam- pled from Kaiming normal distribution, and biases were set
22

---

Published as a conference paper at ICLR 2021
Figure 6 and Tables 15 18 show the results for these experiments, along with the original version of the model (with convolutional encoder) for comparison. To enable a fair comparison with the original model, all experiments employed TCN. The results show that the ESBN performed comparably well with all three of the encoder architectures. This was confirmed by performing paired t-tests the average test accuracy in each task/generalization condition (each combination of task and value of m) for the MLP vs. convolutional encoder (t -177, 0.1) and for the random vs. convolutional encoder (f 1.6, p 0.13).
For comparison, we also performed experiments with these alternative encoders in the Transformer architecture. These experiments revealed that, contrast with the ESBN, the Transformer's per- formance was significantly impaired by the use of a random vs. convolutional encoder (t = -4.0,
p = 0..01), though it appeared to perform comparably well with MLP vs. convolutional encoder (t = 1.6,, p = 0.14).
Table 15: Results for same/different task with convolutional (conv), multilayer perceptron (MLP), or random (rand) encoders. Results reflect test accuracy averaged over 10 trained networks (=: the standard error of the mean).
m 0
m = 50
m7 = 85
171 = 95
771 98
ESBN (conv)
100.0 0.0
100.0
100.0 0.0
100.0 + 0.0
100.0 0.0
ESBN (MLP)
100,0 = 0.0
100.0 0.0
1.0
=
100,0
ESBN (rand)
100.0 = 0.0
0.0
100
100.0
1.0
Transformer (conv)
100.0 = 0.0
100.0
1.0
100.0
7.2
Transformer (MLP)
100.0 0.0
100.0
100.0
100.0
77.6 4.6
Transformer (rand)
100.0 + 0.0
100.0 0.0
100.0 10.0
100.0 0.0
50.6 0.3
Table 16: Results for relational match-to-sample task with convolutional (conv), multilayer per- ceptron (MLP), or random (rand) encoders. Results reflect test accuracy averaged over 10 trained networks (# the standard error of the mean).
0
50
85
95
ESBN (conv)
100.0 + 0.0
100.0 + 0.0
100.0 = 0.0
95.0 + 0.7
ESBN (MLP)
100.0 0.0
100
100.0
9.2
ESBN (rand)
100.0
1.0
100.0=
93.8
Transformer (conv)
100.0
99.98
99.1
79.8 2.5
Transformer (MLP)
100.0
100.0
9.19
73.6 5.3
Transformer (rand)
99.99 + 0.01
99.9 0.04
95.4
46.8 1.7
Results for distribution-of-thne task with convolutional (conv), multilayer perceptron (MLP),, or random (rand) encoders. Results reflect test accuracy averaged over 10 trained networks (=: the standard error of the mean).
Table 17:
0
50
85
95
ESBN (conv)
98.7 + 0.4
99.0 + 0.3
99.5 + 0.2
99.7 + 0.1
ESBN (MLP)
99.0 0.1
0.3
98.0
95.9 + 0.5
ESBN (rand)
100.0
1.0
100.0 1.0
Transformer (conv)
8..7
9.2
1.
Transformer (MLP)
92.7 2.1
9.53
92.1 221+0.. 0.8
35.3
Transformer (rand)
66.0 + 6.2
80.8
60.9 2.4
26.9
23

---

Published as a conference paper at ICLR 2021
Table 18: Results for identity rules task with convolutional (conv), multilayer perceptron (MLP), or random (rand) encoders. Results reflect test accuracy averaged over 10 trained networks (#: the standard error of the mean).
50
85
95
ESBN (conv.)
99.6 0.2
99.6 + 0.1
99.9 + 0.04
99.2 + 0.4
ESBN (MIP)
99.3 3.3+-.22
98.6 0.3
97.7
95.5
ESBN (random)
100.0 0.0
1.0
1.0
0.0
Transformer (conv.)
98.3
9.01
9.70
6..1
Transformer (MLP)
8..5
8488
2..4
5.88
Transformer (random)
47.8
1.94
4.1
0.6
A.5.5 CONFIDENCE ABLATION EXPERIMENT
In order to determine the importance of the confidence values appended to retrieved memories, tested a version of the ESBN without these confidence values. These results are shown in Table [9] and Figure?? 7 The ablation of confidence values prevented the ESBN from being able perform the same/different task at all, and resulted much slower training on the RMTS task. By contrast, ablation of confidence values did not affect performance, either in terms of generalization training time, for the cotribttionnofftheee or identity rules tasks. This can be explained by the fact that these tasks only require the retrieval of the best match from memory. whereas the same/different and RMTS tasks require the model to know how good of a match the best match is, which is precisely information conveyed by confidence values.
Table 19: Results for the confidence ablation experiment. Results reflect test accuracy averaged over 10 trained networks (# the standard error of the mean).
m
m = 50
= 85
m7
95
771 = 98
Same/different
50.0 + 0.02
50.0 0.0
50.0 0000000
49.8
50.0 + 0.1
RMTS
99.95 0.01
99.9
9.02
0.6
99.2 0.2 Distribution-of-thee
99.0
9.3
9.1
Identity rules
99.6 - 0.1
99.6 0.2
9.1
99.2
200
2400
380
(a) RMTS
(b) Distribution- -f-enrro--hcccct--
(c) Identity rules
Figure 7: Training accuracy time courses for the ESBN model without confidence values the m 0 regime, shown with the time courses for all other models for comparison. Each time course reflects average over 10 trained networks. Error bars reflect the standard emror of the mean.
It is also worth noting one potential alternative to explicit, inbuilt confidence value. In our im- plementation, the ESBN's memory is empty at the beginning of each sequence that it processes. However, when multiple entries present in memory, will generally be the case in realistic, temporally extended settings, the presentation a previously unseen item will result in the retrieval of mixture of (weakly matched) memories. This mixed representation therefore serve
24

---

Published as a conference paper at ICLR 2021
reliable cue for the degree which the current percept matches stored memory, obviating the need for an explicit confidence value. To demonstrate this, we implemented a version of the ESBN that begins each sequence with single, learned key/value entry stored in memory (initialized to 0 the beginning of training). Table shows that this approach allows the ESBN to learn and perfectly generalize the same/different task. Figure & shows that this approach allows the ESBN to retain the short training time of the original model on the RMTS task.
Table 20: Results on the same/different task for the ESBN model with learned default memory instead of confidence values. Results reflect test accuracy averaged over 10 trained networks (#: the standard error of the mean).
m
772
50
772 85
772 95
772 = 98
100.0 0.0 100.0 + 0.0
100.0 + 0.0
100.0 0.0 100.0 0.0
100
training accuracy
40
ESBN
LSTM
Transformer
PrediNet
NTM
RN
MNM
0-
0
250
500
750
1000 batches
1250
1500
1750
2000
Figure 8: Training accuracy time courses on m 0 regime of the RMTS task for the ESBN model with learned default memory instead of confidence values. Each time course reflects average over 10 trained networks. Error bars reflect the standard error of the mean.
25

---

Published as a conference paper at ICLR 2021
A.6
ANALYSIS OF LEARNED REPRESENTATIONS
t=
12.5
10.0
7,5
2nd PC
5.0
2.5
0.0
-25
-5.0
-20
-10
10
30
1st
(a) kw (training)
-311
-335
+2.0
-2.00
-225
-225
21d
-3.00
-3.30
-2.33
Baring
-2.23
Baring
1-
1st
(b) kw (training test)
(c) l (training test)
Figure 9: Representations learned by ESBN (projected along first two principal components). (a) Keys written memory during time steps 1-9 (training set). (b) Keys written memory during time steps 1-3 (training set vs. test set). (c) Keys retrieved from memory following second appearance of objects that first appeared during time steps 1-3 (training set test set).
To better understand how the ESBN works, we performed an analysis the representations that learned the distribution-of-thie task. Specifically, performed analysis of network trained on the most difficult generalization regime (m 95), by performing principal component analysis (PCA) all key vectors written and retrieved from memory for both the training and sets, and visualizing these vectors along the first two principal components.
First, we looked the keys that were written to memory (kw)) We found that the keys for the first three time steps were tightly clustered, whereas the keys for the subsequent time steps (4-9) were more diffuse (Figures 9a and 9b). This makes sense because, in the distribution-of-thee task, the ESBN only needs to be able to reliably retrieve what it wrote during the first three time steps (when the objects in the first row were presented). For time steps 4-9, the only important consideration is that the keys written memory not overlap with those written during the first three time steps, which also appears to the case.
Second, we compared the keys written memory for the first three time steps in the training test sets (Figure 9b). This revealed that, for a given time step, the keys written to memory in the
26

---

Published as a conference paper at ICLR 2021
training test sets remarkably similar (so much so that they completely overlapping for time steps and 2).
Third, we looked at the keys that were retrieved from memory following the second appearance of the objects that appeared on time steps 1-3. We found that 1) these closely matched the distribution of keys written to memory during time steps 1-3, and 2) these were highly overlapping for the training vs. test sets (Figure
Taken together, these results help explain why the ESBN regime, despite the very small degree of overlap between the distribution of training and test images. Because the ESBN's controller was relatively isolated from the part of the model that deals with image embeddings, it was able to learn to encode abstract symbol-like representations (such as 'first image', 'second image', and 'third image') ) that did not depend on the identity of the images. Then, when queried with an image, was able to successfully retrieve the image's corresponding abstract encoding. even when that image was quite different than those observed during training. That is, the model learned representations to use keys that could be used for binding and indirection in the same way that symbols are used in traditional computational architectures.
successful in this generalization
27

---

Published as a conference paper at ICLR 2021
A.7 UNICODE CHARACTERS
Figure shows all 100 images that were used to construct the abstract rule learning tasks.
Σ
&
%
?
Figure 10
28


Reviews:
[{"review_id": "hwjIemc2npn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors present an architecture capable learning symbolic representations and rules over those symbols.  Also presented is a set of  tasks involving manipulation of symbolic rules over a set of symbols where held symbol sets may be used to measure generalization performance on the task set.  The goal of this approach is learn from high dimensional inputs useful symbolic representations which are difficult to generalize across from pixels alone.\n\nThe  architecture described in this paper is named the emergent symbol binding network (ESBN) which is composed of a two column external memory for indirection and two information processing streams, one for representing embeddings of concrete variables and another that is recurrent and trained to operate over task relevant variables.  The architecture uses Temporal Context Normalization across sequences to strengthen the models sensitivity to out-of-distribution samples.  This architecture is then trained and evaluated across a set of tasks involving symbolic entities and rules over which the ESBN must learn to generalize and is compared to other memory based architectures with a capacity for relational reasoning.  \n\n \nStrength & Weaknesses:\n\nThe architecture divides the  memory into symbolic keys and pixel encoded values and allows gradients to flow between them (symbols -> encodings) via a similarity weighting function.  As keys are generated from an LSTM they should also encode the sequence history and learn to store the correct information over the sequence to perform the symbolic manipulation required to determine the correct answers to the tasks set out. \n\nThe architecture to me seems not entirely novel to me.  There have been other approaches using key-value based episodic memories, for example the architecture in Fortunato et al. (2019) \"Generalization of reinforcement learners with working and episodic memory\".  The particular way that symbols are learned is interesting but I don't think necessarily exclusive to this model. I also have some concerns the model overfit the task.  It may have been useful to consider other the more complex tasks involving symbolic reasoning such as CLEVR or bAbI.  The tasks presented seem mainly to be toy tasks.  \n\nThe addition of the Temporal Context Normalization for improving generalization over out-of-distribution samples is a nice.  It'd be interesting to see an ablation to quantify the effect of this, for instance, how does it compare with Batch normal or when simply removed for example.\n\nI'm surprised that the Relation Net failed at this task - were positional encodings included in the representation?  same with the transformer - this is crucial when order matters in the input sequence.\n\nThe RN performing poorly on the distribution-of-three and identity-rules tasks is surprising and might be due to 1) the relationships are limited to two entities or 2) the exclusion of sequence order info.  As such, I believe the results may not fully be reflecting the Relation Nets capacity on such a problem as I'd expect them to be able to solve these problems  e.g. The RN could easily be extended to consider all ternary relations and is mainly limited by the task definition.\n\nMore equations would have been helpful.  Possibly consider including Algorithm 1 in the main paper as it is the only description of how the model works.\n\n\nRecommendation:\n\nWhile I believe that this is a very interesting problem domain I think that the solution provided would have been more compelling if the scope of the problem was more ambitious and also I believe that the baselines could have been stronger.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ev_7XvsGdl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper addresses abstract rule learning in high-dimensional data through constructing a recurrent neural network that exhibits a variable-binding ability. The proposed method, ESBN, is a RNN augmented with two memories, one for keys and one for values. The key memory captures the relations between items, which are important to produce symbolic answer (e.g.,  item/group index). The value memory stores the item embedding. Tested on several abstract rule learning tasks on visual inputs, ESBN exhibits excellent generalization with limited training data.\n\nPros:\n- The problem of learning abstract rules that are decoupled from actual content (visual symbols in this case) is important, and is currently under-studied in machine learning.\n- The solution, ESBN, is simple and effective. Empirical evaluations clearly show that ESBN woks well on tested tasks, while several well-studied neural  networks (LSTM, NTM, Relational network, Transformers) fail.\n\nCons:\n- While the results are very encouraging on the simplified tasks, we still do not really know WHY the model works the way it does. After all, the relation between keys are transferred from the similarity between corresponding values. The tasks tested seem to emphasize on the object matching, and thus may not work on other types of rules, or with more variations of inputs.\n- The paper is motivated well and the high-level idea is described, but certain important algorithmic/design details are missing in the main text (e.g., the role of confidence, gate). Algorithm 1 is at core of ESBN, but is left to Appendix, making it hard to follow the unrolling of the controller in step t. The concatenation of the key memory with the confidence score seems to make the dimension of k_r grows with t. Is this intended? If not, then how a fixed size input to the RNN controller is implemented?\n\nOthers:\n- The paper conjectures that that the poor result of the Relation Network (RN) is due to its focus on pair-wise relations. More recent follow-ups have introduced higher-order relations (e.g., Temporal Relation Network [1] and Conditional Relation Network [2]) may be able to mitigate the issue. It would be nice to have these tested, or discussed in the context of abstract rules learning.\n-  It would be important to explain/discuss the effect of temporal context normalization on ESBN.\n\n\nReference:\n[1] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. \"Temporal relational reasoning in videos\". In Proceedings ofthe European Conference on Computer Vision (ECCV), pages 803–818, 2018.\n[2] Le, Thao Minh, et al. \"Hierarchical Conditional Relation Networks for Video Question Answering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ef-oI3XYgr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a neural network architecture comprising a two-stream memory structure: one block is populated with visual representations, and the other is populated by hidden state vectors from a controller. The interaction between these two blocks is suggested to be akin to a symbolic-like mechanism for indirection. Reasoning that indirection is a central component for solving problems in the abstract, the authors go on to show that, indeed, such a network can solve abstract reasoning problems out-of-distribution.\n\nA crucial aspect of the proposed network is that the memory keys are not at all conditioned on the current visual input. This means that the model needs to learn how to produce representations that will allow it to solve tasks in an abstract sense. This is a clever maneuver, because it solves the \"out of distribution\" problem by not allowing the information used to solve a task to ever become out of distribution: the keys, which condition the model's output, are always produced by the LSTM, and the LSTM is never conditioned on anything that is different at test time from what it sees during training time. The difficult task of handling out of distribution data, then, reduces to the vision encoder being able to simply acknowledge that one image is different from another, and it must do so using images it has never seen before. Such a mechanism is in contrast to fully connected networks that need to produce representations from out-of-distribution images *and* reason with those representations. \n\nThat being said, the out-of-distribution demands imposed on the visual encoder aren't as extreme as it would seem. This is because, for the current tasks, what is out-of-distribution is an object's *shape*. Convolutional encoders are predisposed through their inductive biases to detect shape differences. I would be quite interested to see how different encoders fair in this task, in this exact same two-memory network. For example, a simple random projection of the image, or an MLP would be interesting. I would hypothesize that the random projection method would work if images of the same type are identical, but would fail if images of the same type have subtle variation, as do, for example, a dataset of hand-written characters. Thinking about this further, it might be critical to also test the convolutional encoder on a dataset wherein shape types have variation between instances, because even the convolutional encoder might break down when \"matching up\" various instances of a type that are out-of-distribrution (whereas, in the work here, each representation it produces for a character type will necessarily be identical, even if the characters are out of distribution because each image instance is exactly the same). \n\nThe reason I emphasize these latter points about the encoder is because the work is pitched as providing a mechanism for allowing a certain type of symbolic-like reasoning, which in theory, should be insensitive to the types of encoders. If indirection is truly the causal mechanism at play, then the representations that do the \"referring\" should be completely insensitive to the representations that are \"referred\". But, we need evidence to demonstrate that this is the case.  Otherwise, we cannot claim that abstract variable binding occurs. \n\nAltogether, the work is wonderfully put together. The experiments are simple, yet crisp, and they do the job. The work is well motivated, and well written, and there is ample background details to understand the models and experiments. Background work is adequately represented, and results are explained. My only reservations are in regards to the sensitivity of the encoder, and the sensitivity to the task design (particularly, the choice of using identical images per character type) to the overall performance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "71iwlh8Dxsz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a set of tasks that require a simple symbolic reasoning and a new model - Emergent Symbol Binding Network (ESBN) to solve them. The tasks are designed such that it tests the model’s extrapolation ability. Specifically, they introduce novel input symbols that are unseen during training to evaluate the model. They show that previous memory-augmented neural networks fail on the tasks whereas their approach with an external memory that supports a specific variable-binding achieves an excellent performance. \n\nThe main components of ESBN are an input encoder, a key-value memory and a LSTM controller.  The encoder is a CNN that takes an input symbol ((i.e. grayscale image) and outputs a value vector for the memory. The LSTM controller then produces a key vector by taking in a memory entry retrieved by using an attention mechanism. This retrieved memory entry consists of two parts: a key vector corresponding to a value that is selected by the attention and a confidence score. They calculate the attention and confidence scores based on the similarity between the current value and the other value vectors in the memory. The confidence score seems to be an important feature because it tells the LSTM that whether the new input symbol already existed in the memory or not.  Therefore, by design, the LSTM controller never sees the actual input symbols and it only sees their corresponding key vectors, which are in turn generated by the LSTM itself. This allows the LSMT to operate on a representation that is abstracted away from the (sensory) input and solve the reasoning tasks that require extrapolation over input symbols.\n\nPros:\n- This is interesting work that addresses an important question in neuroscience and deep learning. \n- The tasks proposed here can be used to evaluate the other models\n\nCons:\n- They test the model only on synthetic tasks\n- The model feels ad-hoc for the task. It is unclear if the model can be useful for other more complex problems.\n- I believe the confidence score is crucial as it tells LSTM whether the same object was already in the memory. Some ablation on this is useful. Specially, the confidence score seems to provide a quite reliable high-level feature for the LSTM controller, which is not necessarily a weakness but providing ablation could be nice.\n\nAdditional comments:\n- Curious to see how more recent associative MANNs perform on these tasks, such as TPR-RNN (https://github.com/ischlag/TPR-RNN) and MNM (https://bitbucket.org/tsendeemts/mnm/src/master/). Since the code is already out there, it shouldn’t be too difficult to include them as baseline.\n- Smolensky’s TPR, which is one of the earliest works on the topic is currently missing from the discussion\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a recurrent neural network architecture for abstract rule learning. An LSTM is augmented with a two-stream memory structure: one block is populated with visual representations, and the other is populated by hidden state vectors from the RNN controller.

The authors also introduce a set of tasks that require a simple symbolic reasoning on visual inputs and strong extrapolation ability. They show that previous memory-augmented neural networks fail on these tasks, whereas their model exhibits excellent generalization with limited training data.

Pro: The work addresses an important and open question in neuroscience and deep learning. The proposed solution is simple and effective. The manuscript is well-written. It was also improved in a revised version after the first review round.

Con: The main criticism raised by the reviewers was that the considered tasks may be a too simplified synthetic task.  It would have been good to consider other the more complex tasks involving symbolic reasoning such as CLEVR or bAbI.

While this is a valid criticism, all reviewers agreed that this is an interesting and important work worth publishing. In particular, the considered question is of pivotal importance for the community and the work presents a significant progress.
Decision: Accept (Spotlight)