An Explicitly Relational Neural Network Architecture
Authors: Murray Shanahan
Kyriacos Nikiforou
Antonia Creswell
Christos Kaplanis
David Barrett
Marta Garnelo
Venue:  (2020)
Keywords: relational representation

Abstract: With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data. In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity. We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures. The workings of a successfully trained model are visualised to shed some light on how the architecture functions.

Under review as a conference paper at ICLR 2020
AN EXPLICITLY RELATIONAl NEURAL NETWORK AR- CHITECTURE
Anonymous authors Paper under double-blind review
ABSTRACT
With view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data. In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity. We show that the proposed architecture, when pre- pre-trained on curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures. The workings of a successfully trained model are visualised shed some light on how the architecture functions.
INTRODUCTION
When humans face novel problems, they are able to draw effectively on past experience with other problems that are superficially very different, but that have similarities more abstract, structural level. This ability is essential for lifelong, continual learning, and confers on humans degree of data efficiency. powers of transfer learning, and a capacity for out-of-distribution generalisation that contemporary machine learning has yet to match (Grneloo et al. 2016 Lake et al. 2017 Marcus 2018 Smith 2019) case may be made that all these issues are different facets of the same underlying challenge, namely the challenge of devising systems that learn to construct general-purpose purpose, reusable
representations (McCarthy 1987| Bengio et al., 2013). representation general-purpose and reusable to the extent that it contains information whose domain of application exceeds the context within which it was acquired.
Representations that are general-purpose and reusable improve data efficiency because system that already knows how to build representations relevant to a novel task (despite its novelty) doesn') 't have to learn that task from scratch. Ideally, system that efficiently exploits general-purpor -puppsse, reusable representations in this way should be the very same system that learned how to construct them the first place. Moreover, in learning to solve novel task using such representations, we should expect the system to learn further representations that are themselves ceneral-purpose and reusable. So, with the exception of the very first representations the system learns, all learning in such system would in effect be transfer learning, and the process of learning would be inherently cumulative, continual, and lifelong.
One approach to building such a system is to take inspiration from the paradigm of classical, symbolic AI (Garnelo & Shanahan 2019). Building on the mathematical foundations of first-order predicate calculus, a typical symbolic Al system works by applying logic-like rules of inference to language- propositional representations whose elements are objects and relations. Thanks their declarative character and compositional structure, these representations lend themselves naturally to generality and reusability. However, contrast to contemporary deep learning systems, the representations deployed in classical AI are not usually learned from data but hand-crafted (Harnad [1990). The aim of the present work is to get the best of both worlds with an end-to-end differentiable neural network architecture that builds in propositional, relational priors in much the same way that convolutional network builds in spatial and locality priors.
The architecture introduced here builds recent work with non-local network architectures that learn to discover and exploit relational information (Wang et al. 2018). notably relation nets (Santoro al. 2017 Palm et al. 2018) and architectures based on multi head attention (Vaswani et a 2017

---

Under review as a conference paper at ICLR 2020
kheads
SottenLLi
softmox/a,KK
Wen
K
W
comporniar
W
Tuttonk)
Waa
mn
A
B
C
Moans C=AB
The PrediNet architecture. WK and Ws are shared across heads, whereas WQ1 and WQ² are local to each head. See main text for more details.
Figure 1:
Santoro al. 2018; Zambaldi et al., 2019). However, these architectures generate representations that lack explicit structure. There is, general, no straightforward mapping from the parts of representation the usual elements of symbolic medium such as predicate calculus: propositions, relations, and objects. To the extent that these elements are present, they are smeared across the embedding vector, which makes representations hard to interpret and makes it more difficult for downstream processing to take advantage of compositionality.
Here present an architecture, which call PrediNet, that learns representations whose parts map directly onto propositions, relations, and objects. To build a sound, scientific understanding of the proposed architecture, and facilitate a detailed comparison with other architectures, the present study focuses on simple tasks requiring relatively little data and computation. We develop family of small, simple visual datasets that can be combined into a variety of multi-task curricula and used to assess the extent to which an architecture learns representations that are general general-purpose purpose and number of experiments using these datasets that demonstrate
reusable. We report the results potential of an explicitly relational network architecture to improve data efficiency and generalisation, to facilitate transfer, and to learn reusable representations.
The main contribution of the present paper is novel architecture that learns to discover objects and relations high-dimensional data, and to represent them in a form that is beneficial for downstream processing. The PrediNet architecture does not itself carry out logical inference, but rather extracts relational structure from raw data that has the potential to be exploited by subsequent processing. Here, for the purpose of evaluation, PrediNet and train it on a simple set of spatial reasoning problems. The aim is to acquire a sufficient scientific understanding of the architecture and its properties in this minimalist setting before applying it to more complex problems using more sophisticated forms of downstream inference.
graft simple multi-layer perceptron output module the
2 THE PREDINET ARCHITECTURE
The idea that propositions are the building blocks of knowledge dates back to the ancient Greeks, and provides the foundation for symbolic AI, via the 19th century mathematical work of Boole and Frege (Russell & Norvig, 2009). An elementary proposition asserts that a relationship holds between set of objects. Propositions can be combined using logical connectives (and, or, not, etc), and can participate in inference processes such as deduction. The task of the PrediNet is to (learn to) transform high-dimensional data such as images into propositional representations that are useful for downstream processing. A PrediNet module (Fig. [] can be thought of as pipeline comprising three stages: attention, binding, and evaluation. The attention stage selects pairs of objects interest, the binding stage instantiates the first arguments of set of three- three-place predicates (relations) with selected object pairs, and the evaluation stage computes values for each predicate's 's remaining (scalar) argument such that the resulting proposition is true.
2

---

Under review as a conference paper at ICLR 2020
More precisely, a PrediNet module comprises k heads, each of which computes j relations between pairs of objects (Fig. D The imput to the PrediNet is a matrix, L, comprising n rows feature vectors, where each feature vector has length m. In the present work. L is computed by convolutional neural network (CNN). The CNN outputs a feature map consisting of n feature vectors that tile the input image. The last two elements of the feature vector the xy co-ordinates of the associated patch in the image. So the length m of each feature vector corresponds to the number of filters in the final CNN layer plus 2 (for the co-ordinates), and the ith element of feature vector (for m - 2) is the output of the 1th filter. For a given input L, each head h computes the same set of relations (using shared weights Ws) but selects different pair of objects, using dot-product attention based on key-query matching (Vaswani et al. 2017). Each head computes separate pair of queries Q' and Q (via W Q1 and Wo2). But the key space K (defined by Wx) is shared, so that the set of entities that are candidates for attention is consistent across heads. The whole (flattened) image is used to generate queries, allowing attention masks to depend on the image's 's full (non-local) content.
Ω = flattenLD)W')
O = flatten (L)
K = LWK
Applying the resulting pair of attention masks directly L yields pair of objects E and E2. each represented by a weighted sum of feature vectors.
E = coftmax(Q') K )
E = softmax
All j relations between E and E are then evaluated. There many ways to compute relationship between a pair of objects represented as feature vectors. We chose to compute the values of relations by taking vector differences, which has been shown to be effective in the context of relationally structured knowledge bases (Bordes et al. 2011 Socher et al. 2013)). In the current architecture, E and E are subject linear mapping (via W into ID spaces, one per relation, and the resulting vector is passed through an dement-wise comparator, yielding a vector of differences D
D = W =
Wss
The last two elements of E and E (the positions P and Ph respectively) are concatenated to the vector D of differences to give the head's output Ra = (D*, P P.). Finally, the outputs of all & heads are concatenated, yielding the output of the PrediNet module, vector R of length k(j + 4). In predicate calculus terms, the final output of a PrediNet module with k heads and relations represents the conjunction of elementary propositions
A A=1i=1
where Vi(dd,, of, "i exa) asserts that di is the distance between objects ot and e² in the ID space defined by column of the weight matrix Ws. and the denotations of et and el are captured by the vectors Qt and Q respectively, given the key-space defined by K.
Equation supplies semantics for the PrediNet's 's final output vector H' that maps each of its elements onto a well-defined logical formula, something that cannot be claimed for other architectures, such as the relation net or multi-head attention net. In the experiments reported here, only R* is used for downstream processing, and this vector by itself doesn't have the logical structure described by Equation| [ However, the PrediNet module can easily be extended to deliver additional output in explicitly propositional form, with predicate-argument structure corresponding to the RHS of [q In the present paper, the pared-down vector form facilitates our experimental investigation, but in its explicitly propositional form, the PrediNet's output could be piped directly to (say) Prolog interpreter (Fig 7 to an inductive logic programming system, to statistical relational learning system, or indeed to another differentiable neural module.
3 DATASETS AND TASKS
would be premature to apply the PrediNet architecture to rich, complex data before we have a basic understanding of its properties and its behaviour. To facilitate in-depth scientific study, need small, simple datasets that allow the operation of the architecture to examined in detail and the fundamental premises of its design to be assessed. Our experimental goals in the present paper are
3

---

Under review as a conference paper at ICLR 2020
to test the hypothesis that the PrediNet architecture learns representations that are purpose and reusable, and 2) insofar as this is true, to investigate why. Existing datasets for relational reasoning tasks, such CLEVR (Johnson et al. (2017)) and sort-of-CLEVR (Santoro et al. (2017), were ruled out because they include confounding complexities, such as occlusion and shadows or language input, and/or because they don't lend themselves to the fine-grained task-level splits required. Consequently, we devised new configurable family of simple classification tasks that we collectively call the Relations Game.
A Relations Game task involves the presentation of an image containing a number of objects laid out on 3 3 grid, and the aim (in most tasks) is to label the image as True or False according to whether a given relationship holds among the objects in the image. While the elementary propositions learned by the PrediNet only assert simple relationships between pairs of entities, Relations Game tasks generally involve learning compound relations involving multiple relationships among many objects. The objects in question are drawn from either a training set or one of two held-out sets (Fig. 2t). None of the shapes or colours in the training set occurs in either of the held-out sets. The training object set contains uniformly coloured pentominoes and their rotations and reflections (37 shapes in all) with 25 possible colours. The first held-out object set contains uniformly coloured hexominoes and their rotations and reflections (46 shapes in all) with 25 possible colours, and the second held-out object set contains only squares, but with a striped pattern of held-out colours.
Each Relations Game task is tied to a given relation. Even with such a simple setup, the number of definable relations among all possible combinations of objects is astronomical (2(*+11)) for n distinct objects), although only a few of them will make intuitive sense. For the present study, we defined a handful of intuitively meaningful relations and generated corresponding labelled datasets comprising 50% positive and 50% negative examples. selection is shown Fig. Zp. The 'between' relation holds iff the image contains three objects in line in which the outer two objects have the same shape, orientation, and colour. The 'occurs' relation holds iff there is an object in the bottom row of three objects that has the same shape, orientation, and colour as the (single) object in the top row. The 'same' relation holds iff the image contains two objects of the same shape, orientation, and colour. In each case, we balanced the set of negative examples ensure that "tricky" images involving pairs of objects with the same colour but different shape or the same shape but different colour occur just as frequently as those with objects that differ in both colour and shape.
4 EXPERIMENTAL SETUP
At the top level, each architecture we consider in this paper comprises 1) a single convolutional input layer (CNN), 2) a central module (which might be a PrediNet or baseline), and 3) a small output multi-layer perceptron (MLP) (Fig. 3) A pair of xy co-ordinates is appended to each CNN feature vector, denoting its position in convolved image space and, where applicable, a one-hot task identifier is appended to the output of the central module. For most tasks, the final output of the MLP is a one-hot label denoting True or False The PrediNet was evaluated by comparing it to several baselines: two MLP baselines (MLP1 and MLP2), a relation net baseline (Santoro al. 2017) (RN), and a melti-head attention baseline (Vaswani et al., 2017 Zambaldi et al. 2019) (MHA).
To facilitate fair comparison, the top-level schematic is identical for the PrediNet and for all baselines (Fig. 3.. All use the same input CNN architecture and the same output MLP architecture, and differ only the central module. In MLP1, the central module is a single fully-connected layer with ReLu activations, while in MLP2 it has two layers. In RN, the central module computes the set of all possible pairs of feature vectors, each of which is passed through a 2-layer MLP; the resulting vectors are then aggregated by taking their element-wise means to yield the output vector. Finally. MHA comprises multiple heads, each of which generates mappings from the input feature vectors to sets of keys K, queries Q, and values V, and then computes softmax Q K VV. Each head's output is a weighted sum of the resulting vectors, and the output of the MHA central module the concatenation of all its heads' outputs. The PrediNet used here comprises & = 32 heads and j = 16 relations (Fig. D. All reported experiments were carried out using stochastic gradient descent, and all results shown are averages over 10 runs. Further experimental details are given in the Supplementary Material, which also shows results for experiments with different numbers of heads and relations, and with the Adam optimiser, all of which present qualitatively similar results.
4

---

Under review as a conference paper at ICLR 2020
(8)
(b)
object pentominoes
objest
stripe
passible
AAA
ABA
2
AAB
ABC
(d)
match rows
colurn patterns (multi-task)
between
1
L
AAA
False
2: False
False
True
False
ABB
4
ABA
H
True
False
True
True
+
4: True
4.
ABA
C
JEE
AAB
True
False
True
3: False
AAB
Figure 2: Relations Game object and tasks. Example objects from the training and held-out test sets. (b) There are five possible row / column patterns. In a multi-task setting, recognising each row pattern is separate task. (c) Three examples tasks for the single-task setting. (d) An example target task (left) and curriculum (right) for the multi-task setting. The curriculum task ids (right) for each of the three examples (2, 4, and 3) correspond to the respective patterns in (b), and the task in each case is to confirm whether or not the column of objects in the image conform to the designated pattern. The aim of the target task (left) to test whether the two rows of objects have the same pattern according to (b).
pre-training
Task
2: Pre-training task(s)
Task 1, task
fask
Task inj/s)
CNN
Central mocule
(eg: PredINet]
MLP
CNN
Input central nets pre-trained
Input pre-trained
Frazen
Frozen
Task
Task
CNN
Central module
MLP
CSN
Central module
MLP
Figure 3: The four-stage experimental protocol for multi-takk curriculum training. The same input module (CNN) and output module (MLP) are used for PrediNet and all baseline architectures: only the central module varies. Task identifiers are appended to the central module's output vector.

---

Under review as a conference paper at ICLR 2020
Table 1: Data efficiency in single-task Relations Game setting.
Relation
Object set
MLP1
MLP2
RN
MHA
PrediNet
Hexominoes
96.1=0.007
96.420.006
73.2+0.05
94.7=0.1
100:0.0
same
Stripes
93.310.01
14.000.01
72.9x0.05
93.720.1
100:0.0
Hexominoes
98.7+0.005
98.8+0.004
70.8#0.01
89.220.1
99.2:0.004
between
Stripes
96.940.008
97.340.004
65.240.05
85.5a0.1
98.7s0.007
Hexominoes
88.0+0.01
94.8+0.03
61.6#0.01
88.4+0.2
98.5+0.009
occurs
Stripes
73.2#0.03
87.3+0.07
62.6+0.02
80.8+0.1
96.9+0.01
Hexominoes
31.5+0.02
64.4+0.04
55.0+0.009
54.7=0.008
95.4+0.01
xoccurs
Stripes
78.2+0.03
80.8+0.05
54.0#0.01
00.6000007
95.5+0.01
colour/shape
Hexominoes
53.420.07
55.8+0.05
45.100.05
88.620.03
94.3#0.01
To assess the generality and reusability of the representations produced by the PrediNet, we adopted four-stage experimental protocol wherein 1) the network is pre-trained on a curriculum of one or more tasks, 2) the weights in the input CNN and PrediNet are frozen while the weights the output MLP are re-initialised with random values, and 3) the network is retrained set of tasks (Fig. 3 step 3, only the weights in the output MLP change, so the target task can only be learned to the extent that the PrediNet delivers re-usable representations to it, representations the PrediNet has learned to produce without exposure to the target task. To assess this, we can compare the learning curves for the target task with and without pre-training. We expect pre-training to improve data efficiency, so we should see accuracy increasing more quickly with pre-training than without it. For evidence of transfer, and to confirm the hypothesis of reusability, we are also interested in the final performance on the target task after pre-training, given that the weights of the pre-traue rreitra input CNN and PrediNet are frozen. This measure indicates how well a network has learned to form useful representations. The more different the target task is from the pre-training curriculum, the more impressed we should be that the network is able learn the target task.
target task or
RESULTS
As prelude investigating the issues of generality and reusabilty, we studied the data efficiency of the PrediNet architecture in single-task Relations Game setting. Results obtained on a selection of five tasks 'same', 'between', 'occurs', 'xoccurs', and 'colour shape' are summarised in Table The first three tasks are as described in Fig. 2 The 'xoccurs' relation is similar to occurs. It holds iff the object the top row occurs in the bottom row and the other two objects in the bottom row different. The 'colour/ shape' task involves four labels, rather than the usual two: same-shape same-colour; different-colour same-shape; same-colour different shape; diffcrent-colour/ different shape. In the dataset for this task, each image contains two objects randomly placed, and one of the four labels must be assigned appropriately. Table I shows the accuracy obtained by each of the five architectures after 100,000 batches when tested on the two held-out object sets. The PrediNet the only architecture that achieves over 90% accuracy on all tasks with both held-out object sets after 100,000 batches. On the 'xoccurs' task, the PrediNet out-performs the baselines by more than 10%, and on the 'colour/ / shape' task (where chance is 25%), it out- out-performs all the baselines except MHA by 25% or more.
Next, using the protocol outlined in Fig. we compared the PrediNet's ability to learn re-usable representations with each of the baselines. We looked at number of combinations of target tasks and pre-training curriculum tasks. Fig. depicts our findings for one these combinations in detail, specifically three target tasks corresponding to three of the five possible column patterns (ABA, AAB, and ABB (Fig. 21)). and pre-training curriculum comprising the single 'between' task. The plots present learning curves for each of the five architectures at each of the four stages of the experimental protocol. In all cases, accuracy is shown for the 'stripes' held-out object set (not the training set). Of particular interest are the (green) curves corresponding to Stage of the experimental protocol. These show how well each architecture learns the target task(s) after the central module has been

---

Under review as a conference paper at ICLR 2020
MLP2
RN
Text 000uO0
1850
20000
20000
2000
20000
Colles
00000
2000
side
2008
PradiNat
Pro training task(s)
Text SCULNCY
pre-triining
Input central nets treitrained
Input net pre-trained
Batch
Bal.3
Figure Multi-task curriculum training. The target tasks are three column patterns (AAB, ABA, and ABB) and the sole curriculum task is the 'between" relation.
50,00
10000
T000
20000
20000
X000
10000
25300
84
16003
15000
2000
Batch
Batce
Batch
1.
20
e
32
32
2000
13000
X000
200000
20005
.006
10000
100000
20000
20000
Batch
Figure 5: Reusability of representations learned with variety of target and pre-training tasks.
pre-trained the curriculum task(s) and its weights are frozen. The PrediNet learns faster than any of the baselines, and is the only one to achieve an accuracy of 90%. The rapid reusability of the representations learned by both the MHA baseline and the PrediNet is noteworthy because the 'between' relation by itself seems an unpromising curriculum for subsequently learning the AAB and ABB column patterns. As the (red) curve for Stage of the protocol shows, the reusability of the PrediNet's 's representations cannot be accounted for by the pre-training of the input CNN alone.
Fig. 5 shows larger range of target task curriculum task combinations, concentrating exclusively on the Stage 3 learning curves. Here a more complete picture emerges. In both Fig, 5h and Fig. 5 the target task is 'match rows' (d),. but they differ their pre-training curricula. The curriculum for Fig. [3 is three of the five row patterns (ABA, AAB, and ABB). This is the only case where the PrediNet does not learn representations that are more useful for the target task than those of all the baselines, outperforming only two of the four. However, when the curriculum is the three analogous column patterns rather than row patterns, the performance of all four baselines collapses to chance, while the PrediNet does well, attaining similar performance for the row-based curriculum (Fig. 5p). This suggests the PrediNet is able to learn representations that are orientation invariant, which aids transfer. This hypothesis is supported by Fig. 55. where the target tasks are all five row patterns, while the curriculum is all five column patterns. None of the baselines able to learn reusable representations in this context; all remain chance, whereas the PrediNet achieves 85% accuracy.
7

---

Under review as a conference paper at ICLR 2020
(a)
Altention
Atlention
T T
444
Raw image
Rew image
(c)
PradiNal
PrediNet Teat sell (Twxominows)
MHA
Training aud (pantominoos)
Trwining aud (pantominows)
Sama colour
Seme stape
Some colorr
Dfficerd staps
Figure 6: (a) Attention heat maps for the first four heads of trained PrediNet. Left: trained on the "same' task. Right: trained on the 'occurs' task. (b) Principal component analysis. Left: PCA on the output of selected head for PrediNet trained the 'colour / shape' task for pentominoes images (training set). Centre: The same PrediNet applied to hexominoes (held-out test set). Right: PCA applied to a representative head of the MHA baseline with pentominoes (training set). (c) Ablation study. Accuracy for PrediNet and MHA on the "colour shape' task when random subsets of the heads are used at test time. PrediNet* only samples from heads that attend the two objects.
To better understand the operation of the PrediNet, we carried out a number of visualisations. One way find out what the PrediNet's heads learn to attend is to submit images to trained network and, for each head h, apply the two attention masks softmax(Q" KT) and softmax (Q2 K to each of heat map
the n feature vectors in the convolved image L. The resulting matrix to show how attention is distrubuted over the image. We did this for number of networks trained in the single-task setting. Fig. o6 shows two examples, and the Supplementary Material contains more extensive selection. As we might expect, most of the attention focuses on the centres of single objects, and many of the heads pick out pairs of distinct objects various combinations. But some heads attend to halves corners of objects. Although most attention is focal, whether directed at object centres or object parts, some heads exhibit diffuse attention, which possible thanks to the soft key-query matching mechanism. So the PrediNet can (but isn't 't forced to) treat the background as a single entity, or to treat an identical pair of objects as a single entity.
then be plotted
To gain some insight into how the PrediNet encodes relations, we carried out principal component analysis (PCA) on each head of the central module's 's output vectors for number of trained networks, again in the single-task setting (Fig. 6p). We chose the four-label 'colour / shape' task train on, and mapped 10,000 example images onto the first two principal components, colouring each with their ground-truth label. We found that, for some heads, differences in colour and shape appear align along separate axes (Fig. 66). This contrasts with the MHA baseline, whose heads don't seem to individually cluster the labels meaningful way. For the other baselines, which lack the multi-head organisation of the PrediNet and the MHA network, the only option is to carry out PCA on the whole output vector of the central module. Doing this, however, does not produce interpretable results for any of the architectures (Fig (8ig We also identified the heads in the PrediNet that attended to both objects in the image and found that they overlapped almost entirely with those that meaningfully clustered the labels S10). Finally, still using the 'shape colour task", we carried out an ablation study, which showed that the PrediNet is significantly more robust than the MHA network pruning random subset of heads at test time. Moreover, if pruned to leave only those heads that attended to the two objects, the performance of the full network could be captured with just a handful of heads (Fig. Ge). Taken together, these results are suggestive of something we might term relational disentangling in the PrediNet.
Finally, to flesh out the claim that the PrediNet generates explicitly relational representations according we extended the PrediNet module to generate an additional output in the form of Prolog program (Fig. 7). This involves assigning symbolic identifiers 1) to each of the
to the semantics of Equation]] [
8

---

Under review as a conference paper at ICLR 2020
ob
CD
(a)
holde (B, V, sbb__, X) small (V1.
60
ob_5
Prolog query
Input image
(c)
(I)
PrediNet attention masks and object ids
=0 (-D.11,, ab 11 (-0.34, ob ob_0, ob_11
(d)
rO,
-0.15,
(-0,50, ob ob_0, ub 23 (-0.54, ob 20,00.0, ob 0, ob 11.
(xC, :- r0 X, X1 201da(t1,90)
Ed,
(:2,V,X,
(2, (2,38, 2)
-0.077
holds (23,,1,,,11
X ob. 0
holcs (24, 1V,
P3,
13,00.6, ob ob_01. =4(-0,53, ob_2, ob_01. +5 (5440.45, ob ob_ ob_00 01 :6,1.00, ob
(x5, :- r5,,,, X1.
0.06,
(26,9,
r6 9).
cb_0
holds (27, :- (7,
-0.1,
=71-0,59, ob ob_2, 0b.
small (9) : -0.2, V 0.2.
PrediNet output Prolog
Hand-written Prolog
Prolog
Figure 7: PrediNet output in propositional form. (a) A small PrediNet (8 heads, 8 relations) trained on the 'between' task given image. (b) Mean shift clustering is applied to the of all attention masks computed by the heads. Each of the resulting clusters is assigned symbolic identifier. (c) Each relation is also given a symbolic identifier, and all 64 propositions computed by the PrediNet are enumerated in Prolog syntax, in accordance with Equation I (A subset is shown.) (d) The results can be combined with further hand-written Prolog clauses. (Upper-case letters denote variables, while constants start with lower-case letters.) which relations r hold with a small value between ob_ 2 and any other object x. (f) The query yields four answers.
(e) Prolog queries can then be submitted. Here we are asking
PrediNet's j relations, and 2) to every object picked out by its heads via the attention masks they compute. Then the corresponding j k propositions symbolic identifiers to the relations is trivial. But because attention masks can differ slightly even when they ostensibly pick out the same region of the input image, it's necessary cluster them before assigning symbolic identifiers to the corresponding objects. We used mean shift clustering for this. Fig. / presents a sample of the PrediNet's output in Prolog form, along with an example of deductive inference carried out with this program. The example shown is not intended to be especially meaningful; without further analysis, we lack any intuitive understanding of the relations the PrediNet has discovered. But it demonstrates that the representations the PrediNet produces can be understood predicate calculus terms, and that symbolic deductive inference is one way (though not the only way) in which they might be deployed downstream.
be enumerated in Prolog syntax. Assigning
RELATED WORK
The need for good representations has long been recognised in AI (McCarthy 1987 Russell & Norvig 2009), and is fundamental to deep learning (Bengio et al.) (003)). The importance of reusability and abstraction, especially in the context of transfer, is emphasised by Bengio, et al. (Bengio er al. 2013), who argue for feature sets that are "invariant the irrelevant features and disentangle the relevant features"'. Our work here shares this motivation. Other work has looked at learning representations that are disentangled at the feature level (Higgins et al.) 2017a 2018). The novelty the PrediNet to incorporate architectural priors that favour representations that are disentangled at the relational and propositional levels. Previous work with relation nets and multi-head attention nets has shown how non-local information can be extracted from raw pixel data and used to solve tasks that require relational reasoning. (Santoro et al. 2017 Palm et al. 2018 Santoro et al., 2018 Zambaldi et al. 2019) But unlike the PrediNet, these networks don't produce representations with explicitly relational, propositional structure. By addressing the problem of acquiring structured representations, the PrediNet complements another thread of related work, which is concerned with learning how

---

Under review as a conference paper at ICLR 2020
earry out inference with structured representations. but which assumes the job of acquiring those representations is done elsewhere (Getoor & Taskar 2007: Battaglia et al. [0166] Rocktäschel &
Riedel 2017
Evans & Grefenstette 2018
Dong et al. 2019
In part, the present work is motivated by the conviction that curricula will be essential to lifelong, continual learning in future generation of RL agents if they are to exhibit more general intelligence, just as they are for human children. Curricular pre-training has a decade-long pedigree in deep learning (Bengio et al. 2009). Closely related to curriculum learning is the topic of transfer (Bengio 2012). a hallmark of general intelligence and the subject of much recent attention (Higgins et al. 2017b Kansky et al., | Schwarz et al., 2018). The PrediNet exemplifies a different (though not incompatible) viewpoint on curriculum learning and transfer from that usually found in the neural network literature. Rather than (or as well as) a means to guide the network, step by step, into a favourable portion of weight space, curriculum learning is here viewed in terms of the incremental accumulation of propositional knowledge. This necessitates the development of different style of architecture, one that supports the acquisition of propositional, relational representations, which also naturally subserve transfer.
Asai, whose paper was published while the present work was in progress, describes an architecture with some similarities to the PrediNet, but also some notable differences (Asai, 2019). For example, Asai's architecture assumes an input representation in symbolic form where the objects have already been segmented. By contrast, in the present architecture, the input CNN and the PrediNet's 's dot- product attention mechanism together learn what constitutes object.
7 CONCLUSION AND FURTHER WORK
We have presented neural network architecture capable, in principle, of supporting predicate logic's powers of abstraction without compromising the ideal of end-to-end learning, where the network itself discovers objects and relations in the raw data and thus avoids the symbol grounding problem entailed by symbolic Al's practice of hand-crafting representations (Harnad 1990). Our empirical results support the view that network architecturally constrained to learn explicitly propositional, relational representations will have beneficial data efficiency, generalisation, and transfer properties. Although, the present experiments don't use the fully propositional version of the PrediNet output, the concatenated vector form inherits many of its beneficial properties, notably a degree of compositionality. In particular, one important respect in which the PrediNet differs from other network architectures is the extent to which it canalises information flow; at the core of the network, information is organised into small chunks which are processed in parallel channels that limit the ways the chunks can interact. We believe this pressures the network to learn representations where each separate chunk of information (such as a single value in the vector R*) has independent meaning and utility. (We see evidence of this in the relational disentanglement of Fig. 6 The result a representation whose component parts are amenable to recombination, and therefore re-use in a novel task. But the findings reported here are just the first foray into unexplored architectural territory, and much work needs to be done to gauge the architecture's 's full potential.
The focus of the present paper is the acquisition of propositional representations rather than their use. But thanks to the structural priors of its architecture, representations generated by a PrediNet module have natural semantics compatible with predicate calculus (Equation [T]. which makes them an ideal medium for logic-like downstream processes such as rule-based deduction, causal or counterfactual reasoning, and inference the best explanation (abduction). One approach here would be to stack PrediNet modules and / or make them recurrent, enabling them to carry out the sort of iterated, sequential computations required for such processes (Palm al. 2018:: Dehghani et al. 2019). Another worthwhile direction for further research would be to develop reinforcement learning (RL.) agents using the PrediNet architecture. One form of inference of particular interest in this context is model-based prediction. which can be used to endow an RL agent with look-ahead and planning abilities (Racanière et al., 2017 Zambaldi et al. 2019). Our expectation is that RL agents in which explicitly propositional, relational representations underpin these capacities will manifest more of the beneficial data efficiency, generalisation, and transfer properties suggested by the present results. As stepping stone to such RL agents, the Relations Game family of datasets could be extended into the temporal domain, and multi-task curricula developed to encourage the acquisition of temporal, as well as spatial, abstractions.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. In International Conference on Automated Planning and Scheduling, 2019.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems, pp. 4502 4510, 2016.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proc. ICML Workshop on Unsupervised and Transfer Learning, volume 27, pp. 17-37, 2012.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proc. 26th International Conference on Machine Learning, pp. 41-48, 2009.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828. 2013.
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. Learning structured embed- dings of knowledge bases. In Proc. AAAI, pp. 301-306, 2011.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In Proc. International Conference on Learning Representations, 2019.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In International Conference on Learning Representations, 2019.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Arlificial Intelligence Research, 61:1-64, 2018.
Marta Garnelo and Murray Shanahan. Reconciling deep learning with symbolic artificial intelligence: representing objects and relations. Current Opinion in Behavioral Sciences. 29:17-23, 2019.
Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv preprint: 1609.05518, 2016.
Lisa Getoor and Ben Taskar (eds.). Introduction to Statistical Relational Learning. MIT Press, 2007.
Stevan Hamad.. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-346, 1990.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: improving zero-shot transfer in reinforcement learning. In Proc. 34th International Conference on Machine Learning, pp. 1480-1490, 2017b.
Irina Higgins, Nicolas Sonnerat, Loie Matthey, Arka Pal, Christopher P Burgess, Matko Bosn- jak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. In International Conference on Learning Representations, 2018.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997, 2017.
11

---

Under review as a conference paper at ICLR 2020
Ken Kansky, Tom Silver, David A. Mély, Mohamed Eldawy, Miguel Lázaro-Gredilla Gredilla, Xinghua Lou, Nimrod Dorfiman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. In Proc. 34th International Conference on Machine Learning, volume 70, pp. 1809-1818, 2017.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017.
Gary Mareus. Deep learning: a critical appraisal. arXiv preprint arXiv: 2018.
John McCarthy, Generality in artificial intelligence. Communications of the ACM, 30()) 1987.
Rasmus Berg Palm, Ulrich Paquet, and Ole Winter. Recurrent relational networks. In Advances in Neural Information Processing Systems, pp. 3368-3378, 2018.
Sébastien Racanière, Théophane Weber, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adria Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. imagination-augment agents for deep reinforcement learning. In Advances Neural Information Processing Systems, pp- 5694-5705, 2017.
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In Advances in Neural Information Processing Systems, pp. 3788-3800, 2017.
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall Press, 3rd edition, 2009.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pp. 4974-4983, 2017.
Adam Santoro, Ryan Faulkner, David Raposo, Rae Jack, Mike Chrzanowski, Théophane Weber, Oriol Vinyals, Razvan Pascanu, and Tim Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 7299-7310, 2018.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Baabska-Barwinsk, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Proc. 35th International Conference on Machine Learning, volume 80, pp. 1528-4533 2018.
Brian Cantwell Smith. The Promise of Artificial Intelligence: Reckoning and Judgment. MIT Press, 2019.
Richard Socher, Danqi Chen, Christopher Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems, pp. 926-934, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representutions, 2019.
12

---

Under review as a conference paper at ICLR 2020
Table S2: Default hyperparameters
Parameter
Value
Input images size
36 x 36 x
I. size
25 x 34
Runs per experiment
10
Optimiser
Gradient descent
Learning rate
0.01
Batch size
10
Input CNN output channels
32
Input CNN filter size
12
Input CNN stride
6
Input CNN activation
ReLu
Bías
Yes
Output MLP hidden layer size
8
Output MIP output size
2 (4)
Output MLP activation Bias
ReLu
Yes (both)
MLP1 output size
k(j + 4)
MLP1 activation
ReLu
Bias
Yes
MLP2 hidden layer size
1024
MI.2 activations
ReLu
MLP2 output size
k(j + 4)
Bias
Yes (both)
RN MIP hidden layer size (pre-aggregation)
256 k(j + 4)
RN output size
RN activation
ReLu
RN aggregation
Element-wise mean
Bias
No
MHA no. of heads
MHA key / query size
16
MHA value size
j + k(j + 4)
MHA output size
MHA attention mechanism
softmax (oannax((Uu (QKT)Q
Bias
No
PrediNet no. of heads
32
PrediNet key / query size
16
PrediNet relations
j = 16
PrediNet output size
k(j + 4)
Bias
S1
HYPERPARAMETERS
Table $2
shows the default hyperparameters used for the experiments reported in the main text.
S2
SUPPLEMENTARY ANALYSIS
$2.1
DIMENSIONALITY REDUCTION ON INTERMEDIATE REPRESENTATIONS
To qualitatively assess the nature of the representations produced by each architecture, performed dimensionality reduction analysis on the outputs of the central module of each architecture trained on the 'colour/ shape" task. After training, batch of 10,000 images (pentominoes) was passed through
13

---

Under review as a conference paper at ICLR 2020
MLP1
MLP2
MHA
PrediNet
Figure S8: Representative central module outputs for networks trained on the 'colour shape' task when projected onto the two largest principal components.
the network and principal component analysis (PCA) was performed on the resulting representations, which were then projected onto the two largest principal components for visualisation. The projected representations were then colour-coded by the labels for the corresponding images (i.e. different/same shape, different/same colour).
PCA on the full representations (concatenating the head outputs in the case of the PrediNet and MHA models) did not yield any clear clustering of representations according to the labels for any of the models (Figure (8)..
For the PrediNet and MHA models, we also ran separate PCAs on the output relations of each head in order to see how distributed / disentangled the representations were. While in the MHA model there was no evidence of clustering by label on any of the heads, reflecting a heavily distributed representation, there were several heads in the PrediNet architecture that individually clustered the different labels (Figure S9). In some heads, colour and shape seemed to be projected along separate axes (e.g. heads 5, 26, and 27), while in others objects with different colours seemed be organised in hexagonal grid (e.g. heads 9 and 14).
We noted that the clustering was preserved (though slightly compressed in PC space) when the held-out set of images (hexominoes) was passed through the PrediNet and projected onto the same principal components derived using the training set. (In Section [22.2] we show that the PrediNet heads that seem to cluster the labels also attend to the two objects in the image rather than the background.)
$2.2 ATTENTION ANALYSIS
To assess the extent to which the various PrediNet heads attend to actual objects as opposed to the background, we produced lower resolution content mask for each image (with the same resolution as the attention mask) containing 0.0s at locations where there are no objects the corresponding pixels of the full image, 1.0s where more than 90% of the pixels contain an object, and 0.5s otherwise. By applying the attention mask to the content mask, and summing the resulting elements, we produced scalar indicating whether the attention mask was selecting region of the image with an object (value close to 1.0), or the background (value close to 0.0). This was tested over 1000 images from the training set (Fig. (STO). but similar results are obtained if the held-out images are used instead. The top plot in Fig- S100shows that both attention masks of some heads consistently attend to objects, while others to a combination of object and background. Importantly, the heads for which the PCA meaningfully clusters the labels are also the ones in which both attention masks attend to objects (Fig. (S))
We additionally provide a similar analysis with position mask, where each pixel in the mask contains a unique location index. The middle plot in Fig. S10 shows that the attention masks in the majority of the heads do not consistently attend specific locations. Finally, the mean absolute values of the hea-head input weights to the output MLP are shown in the bottom plot of the same figure. Interestingly, the heads that consistently attend only objects have higher weighting than the rest.
14

---

Under review as a conference paper at ICLR 2020
(a) PrediNet training set
(b) PrediNet test set
B
(c) MHA training set
Same colour, shape
Different colour, same shape
Same colour, different shape
Different colour, different shape
Figure S9: Per-head PCA on the heads of a PrediNet and an MHA trained on the 'colour / shape' task. For all networks, PCA was performed using the training data (pentominoes). In (a) and (c), the training data projected onto the two largest PCs and in (b) the test data (hexominoes) was used.
15

---

Under review as a conference paper at ICLR 2020
0xckg
25
The extent which the two attention masks of the different heads attend objects rather than the background. Middle: The extent to which the two attention masks of the different heads attend to specific locations in the image. Bottom: Mean absolute value of the weights from the different PrediNet heads to the output MLP.
Figure S10: Top:
S3 EXPERIMENTAL VARIATIONS
Further experimental results are provided this section, including variations in hyper-parameters. Fig. STT presents test accuracy curves for the 'stripes' object set, for which a summary is presented in Table of the main text. Fig. S12 shows the results on the same experiment but using the Adam optimiser instead of SGD, with a learning rate of 10 The TensorFlow default values for all other Adam parameters were used. While other learning rate values were also tested, value of 10-4 gave the best overall performance for all architectures. Multi-task experiments were also performed using Adam with the same learning rate (Fig. $14 & S15), yielding an overall similar performance to SGD with learning rate of 10
To assess the extent to which the number of heads and relations plays a role in the performance, we ran experiments with A: 64 heads and j 16 relations (Fig. S16 & (11)). as well as k = 16 heads and j = 32 relations (Fig. S18 & S19). The results indicate that having a greater number of heads leads to better performance than having greater number of relations, because they provide more stability during training and, perhaps, a richer propositional representation.
16

---

Under review as a conference paper at ICLR 2020
Restior: Sume
Relatir:: Decurs
Relatiorc Beiween
aeation XDscars
0.5
3.4
03
Iatch
Dathh
Earth
Relatio:: Sams
Relation: Occurs
Retation: Between
1.0
Reation: KOtcars
3.55
Bish
63
Batc3
Figure $11: Relations Game learning curves for the different models. SGD with learning rate of 0.01 was used with PrediNet of k = 32 heads and j 16 relations. The top and bottom rows show results for the 'hexominoes' held-out object set, while the middle row is for the 'stripes' held-out object set. The results for the top 10 batches are summarised in Table 1 of the main manuscript.
Relatir:: Sams
Relaton: Occurs
helt on Bebesen
Relabioss XDcours
33
03
Buch
Relation Sam
Relation Dccurs
on Between
Reladon X0cours
3.5
B0tch
Basch
Basch
Butch
Figure S12: Relations Game learning curves for the different models trained with the Adam optimiser (learning rate: 10-4), All other experimental parameters are the same as Fig. S11 The top row shows results for the 'hexominoes' held-out object set, while the bottom row for the 'stripes" held-out object set
17

---

Under review as a conference paper at ICLR 2020
2007
2019
1000
2000
Tall Fronads
bnd
NLD. Tabl
H0XX Took 1nd
STAALE
Figure S13: Multi-task curriculum training. The columns correspond to different target / pre-training task combinations, while the rows correspond the different architectures. SGD with a learning rate of 0.01 was used, with l = 32 and j 16. Training was performed using the pentominoes object set and testing using the 'stripes' object set. From left to right, the combinations of target / pre-training tasks are: ('match rows", '3 row patterns') ), ("5 column patterns', 'between') )) ("3 column patterns' "between"), ("match rows", #3 column patterns') )) and ('5 row patterns', '5 column patterns'). ) From top bottom, the different architectures are: MLP1, MIP2, relation net (RN), multi-head attention (MHA) and PrediNet.
18

---

Under review as a conference paper at ICLR 2020
1466 Twe
Joe
10000 2006
100 2000 2000
1000
1200 2006
100 1000 2006 2000
Figure S14: Multi-task curriculum training. The columns correspond to different target / pre-training task combinations, while the rows correspond to the different architectures, as in Fig. S13 The Adam optimiser with learning rate of 10 was used. Training performed using the pentominoes object set and testing using the 'stripes' object set.
19

---

Under review as a conference paper at ICLR 2020
HUP2
M4A
Precttes
Figure S15: Reusability of representations learned with variety of target and pre-training tasks, using the "stripes' object set. All architectures were trained using Adam, with a learning rate 10 The experimental setup is the same in Fig. SS1
20

---

Under review as a conference paper at ICLR 2020
Tall rudi
Tan bad
Tab
NLX Taak 14777aabe
Oeener Teat (ectaiing Toraet
peetec Talt tectering
Pecket fue Prosiieg
Figure S16: Multi-task curriculum training. The columns correspond to different target/pre-training task combinations, while the rows correspond to the different architectures. SGD with a learning rate 0.01 was used. Training was performed using the pentominoes object set and testing using the 'stripes' object set. The experimental setup is the same as for Fig. SSI] except that A 64 and j = 16. Increasing the number of heads for the PrediNet increases the stability during training and overall performance.
21

---

Under review as a conference paper at ICLR 2020
Toe with Pe
Teat In
MIP1
MUP2
MHtk
Figure S17: Reusability of representations learned with variety of target and pre-training tasks, using the 'stripes' object set. All experimental parameters are as in Fig. S16
22

---

Under review as a conference paper at ICLR 2020
Tall Dresslyy
Tan Sad
N421 Taak
Brch
Tra Todl
Sacth
Openee Teatt Pectaiing mer
/ecctuu Stac 2-01 Sehece
Pecket Sue Prosiing
Figure S18: Multi-task curriculum training. The columns correspond to different target pre-training task combinations, while the rows correspond to the different architectures. SGD with a learning rate of 0.01 was used. Training was performed using the pentominoes object set and testing using the 'stripes' object set. The experimental setup is the same as for Fig. SSB] except that & 16 and j = 32. Having fewer heads leads to a decrease in performance, even if the number of relations is increased maintain network size.
23

---

Under review as a conference paper at ICLR 2020
Toe with Noy
Telt mecarr No
MIP1
MUP2
MHtk
Figure S19: Reusability of representations learned with variety of target and pre-training tasks, using the 'stripes' object set. All experimental parameters are as in Fig. S18
24


Reviews:
[{"review_id": "rkgPum56YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents PrediNet: an architecture explicitly designed to extract representations in the form of three-place predicates (relations). They evaluate the architecture on a visual relational task called the \"Relations Game\" which involves comparing Tetris-like shapes according to their appearance, relative positions, etc.. They show that their architecture leads to useful generalizable representations in the sense that they can be used for new tasks without retraining.\n\nI think this paper contains a number of unusual and interesting ideas but is let down by its presentation. The writing is good, but provides very little intuition for why we should expect this approach to work (aside from its connection to equation 1) - I discuss this in more depth below. The experimental task is interesting (I'm okay with synthetic tasks of this form for unusual new architectures like this), but I'm not sure what it tests that isn't tested in the CLEVR and sort-of -CLEVR datasets which rely on similar relational reasoning to solve. The advantage of those datasets is they are well-established with strong baselines so we can be more certain that a fair comparison is been made. I've voted to reject this paper because I feel its premature in its current form.\n\nExpanding on this - The description of PrediNet covers the basics, but missing detail and intuition for why certain choices are made. For example, \n - why is L flattened for the queries (I think it’s because the query is independent of pixel location, but flattening seems of when L also includes co-ordinates) but not the key, K? \n - Why is the key space shared between heads (this seems more intuitive - keys should have consistent meaning… but if that’s the case, make that intention explicit)? \n - Also, writing the dimensionality of the matrices, would help (e.g. is W_S in R^{m x j} or R^{m x 1} or something else?). \n - What is the meaning of the position features in E_1 and E_2? From the softmax product it seems they should be a weighted sum of the pixels that are addressed - implying that it is the weighted average location?\n - The final representation mostly consists of an h x (j) vector (ignoring positions) containing the output of the comparators. Could you provide some intuition for why we would expect such a representation to be useful for the downstream task? This representation seems to differ substantially from what is used in the baseline methods: i.e. attention-weighted sums of the input features.\n\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJefjgYRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data. The authors testified the proposed algorithm using the Relations Game, whose aim is to label an image containing a number of objects as True or False according to whether a given relationship holds among the objects in the image. This paper is well organized. The applied methods are introduced in detail. The authors showed the improvement using the Relations Game.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "B1gh7vPTYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a network architecture based on the multi-head self-attention module to learn a new form of relational representations. The proposed method is shown to improve data efficiency and generalization ability on a sequence of curriculum learning tasks. \n\n+ The major novelty of this paper is a new attention-based network architecture that aims to discover objects and the relations between them. The idea of explicitly appending the patch positions to the representations is interesting, though I am not sure whether it can be generalized to real data.\n\n- Since the proposed network takes patches of full images as inputs, my major concern is about its effectiveness on high-dimensional images with more realistic objects as in the CLEVR dataset other than 2d-grid objects. It would be better if the authors could extend their method to natural images. \n\n- A closely related work is the NLM model [1], which can perfectly generalized to new tasks. Please compare the proposed model with it.\n\n- Minor: Figures are understandable, but some of them are too small, especially for the graphical legend. As space is an issue, I would suggest removing some plots and increasing the size of the ones provided. \n\n- In Figure 1, is g equal to n? \n\n[1] Neural Logic Machines. Dong et al., ICLR 2019.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model that can learn predicates (symbolic relations) from pixels and can be trained end to end.  They show that the relations learned generate a representation that generalizes well, and provide some interpretation of the model.

Though it is reasonable to develop a model with synthetic data, the reviewers did wonder if the findings would generalize to new data from real situations.  The authors argue that a new model should be understood (using synthetic data) before it can reasonably be applied to natural data.  I hope the reviews have shown the authors which areas of the paper need further explanation, and that the use of a synthetic dataset needs to strong justification, or perhaps show some evidence that the method will probably work on real data (e.g. how it could be extended to natural images).
Decision: Reject