An Explicitly Relational Neural Network Architecture

1 INTRODUCTION . When humans face novel problems , they are able to draw effectively on past experience with other problems that are superficially very different , but that have similarities on a more abstract , structural level . This ability is essential for lifelong , continual learning , and confers on humans a degree of data efficiency , powers of transfer learning , and a capacity for out-of-distribution generalisation that contemporary machine learning has yet to match ( Garnelo et al. , 2016 ; Lake et al. , 2017 ; Marcus , 2018 ; Smith , 2019 ) . A case may be made that all these issues are different facets of the same underlying challenge , namely the challenge of devising systems that learn to construct general-purpose , reusable representations ( McCarthy , 1987 ; Bengio et al. , 2013 ) . A representation is general-purpose and reusable to the extent that it contains information whose domain of application exceeds the context within which it was acquired . Representations that are general-purpose and reusable improve data efficiency because a system that already knows how to build representations relevant to a novel task ( despite its novelty ) doesn ’ t have to learn that task from scratch . Ideally , a system that efficiently exploits general-purpose , reusable representations in this way should be the very same system that learned how to construct them in the first place . Moreover , in learning to solve a novel task using such representations , we should expect the system to learn further representations that are themselves general-purpose and reusable . So , with the exception of the very first representations the system learns , all learning in such a system would in effect be transfer learning , and the process of learning would be inherently cumulative , continual , and lifelong . One approach to building such a system is to take inspiration from the paradigm of classical , symbolic AI ( Garnelo & Shanahan , 2019 ) . Building on the mathematical foundations of first-order predicate calculus , a typical symbolic AI system works by applying logic-like rules of inference to language-like propositional representations whose elements are objects and relations . Thanks to their declarative character and compositional structure , these representations lend themselves naturally to generality and reusability . However , in contrast to contemporary deep learning systems , the representations deployed in classical AI are not usually learned from data but hand-crafted ( Harnad , 1990 ) . The aim of the present work is to get the best of both worlds with an end-to-end differentiable neural network architecture that builds in propositional , relational priors in much the same way that a convolutional network builds in spatial and locality priors . The architecture introduced here builds on recent work with non-local network architectures that learn to discover and exploit relational information ( Wang et al. , 2018 ) , notably relation nets ( Santoro et al. , 2017 ; Palm et al. , 2018 ) and architectures based on multi-head attention ( Vaswani et al. , 2017 ; Santoro et al. , 2018 ; Zambaldi et al. , 2019 ) . However , these architectures generate representations that lack explicit structure . There is , in general , no straightforward mapping from the parts of a representation to the usual elements of a symbolic medium such as predicate calculus : propositions , relations , and objects . To the extent that these elements are present , they are smeared across the embedding vector , which makes representations hard to interpret and makes it more difficult for downstream processing to take advantage of compositionality . Here we present an architecture , which we call a PrediNet , that learns representations whose parts map directly onto propositions , relations , and objects . To build a sound , scientific understanding of the proposed architecture , and to facilitate a detailed comparison with other architectures , the present study focuses on simple tasks requiring relatively little data and computation . We develop a family of small , simple visual datasets that can be combined into a variety of multi-task curricula and used to assess the extent to which an architecture learns representations that are general-purpose and reusable . We report the results of a number of experiments using these datasets that demonstrate the potential of an explicitly relational network architecture to improve data efficiency and generalisation , to facilitate transfer , and to learn reusable representations . The main contribution of the present paper is a novel architecture that learns to discover objects and relations in high-dimensional data , and to represent them in a form that is beneficial for downstream processing . The PrediNet architecture does not itself carry out logical inference , but rather extracts relational structure from raw data that has the potential to be exploited by subsequent processing . Here , for the purpose of evaluation , we graft a simple multi-layer perceptron output module to the PrediNet and train it on a simple set of spatial reasoning problems . The aim is to acquire a sufficient scientific understanding of the architecture and its properties in this minimalist setting before applying it to more complex problems using more sophisticated forms of downstream inference . 2 THE PREDINET ARCHITECTURE . The idea that propositions are the building blocks of knowledge dates back to the ancient Greeks , and provides the foundation for symbolic AI , via the 19th century mathematical work of Boole and Frege ( Russell & Norvig , 2009 ) . An elementary proposition asserts that a relationship holds between a set of objects . Propositions can be combined using logical connectives ( and , or , not , etc ) , and can participate in inference processes such as deduction . The task of the PrediNet is to ( learn to ) transform high-dimensional data such as images into propositional representations that are useful for downstream processing . A PrediNet module ( Fig . 1 ) can be thought of as a pipeline comprising three stages : attention , binding , and evaluation . The attention stage selects pairs of objects of interest , the binding stage instantiates the first two arguments of a set of three-place predicates ( relations ) with selected object pairs , and the evaluation stage computes values for each predicate ’ s remaining ( scalar ) argument such that the resulting proposition is true . More precisely , a PrediNet module comprises k heads , each of which computes j relations between pairs of objects ( Fig . 1 ) . The input to the PrediNet is a matrix , L , comprising n rows of feature vectors , where each feature vector has length m. In the present work , L is computed by a convolutional neural network ( CNN ) . The CNN outputs a feature map consisting of n feature vectors that tile the input image . The last two elements of the feature vector are the xy co-ordinates of the associated patch in the image . So the length m of each feature vector corresponds to the number of filters in the final CNN layer plus 2 ( for the co-ordinates ) , and the ith element of a feature vector ( for i < m − 2 ) is the output of the ith filter . For a given input L , each head h computes the same set of relations ( using shared weights WS ) but selects a different pair of objects , using dot-product attention based on key-query matching ( Vaswani et al. , 2017 ) . Each head computes a separate pair of queries Qh1 and Qh2 ( via W h Q1 and W h Q2 ) . But the key space K ( defined by WK ) is shared , so that the set of entities that are candidates for attention is consistent across heads . The whole ( flattened ) image is used to generate queries , allowing attention masks to depend on the image ’ s full ( non-local ) content . Qh1 = flatten ( L ) W h Q1 Q h 2 = flatten ( L ) W h Q2 K = LWK Applying the resulting pair of attention masks directly to L yields a pair of objects Eh1 and E h 2 , each represented by a weighted sum of feature vectors . Eh1 = softmax ( Q h 1K > ) L Eh2 = softmax ( Q h 2K > ) L All j relations betweenEh1 andE h 2 are then evaluated . There are many ways to compute a relationship between a pair of objects represented as feature vectors . We chose to compute the values of relations by taking vector differences , which has been shown to be effective in the context of relationally structured knowledge bases ( Bordes et al. , 2011 ; Socher et al. , 2013 ) . In the current architecture , Eh1 and Eh2 are subject to a linear mapping ( via WS ) into j 1D spaces , one per relation , and the resulting vector is passed through an element-wise comparator , yielding a vector of differences Dh . Dh = Eh1WS − Eh2WS The last two elements of Eh1 and E h 2 ( the positions P h 1 and P h 2 , respectively ) are concatenated to the vector Dh of differences to give the head ’ s output Rh = ( Dh , Ph1 , P h 2 ) . Finally , the outputs of all k heads are concatenated , yielding the output of the PrediNet module , a vector R∗ of length k ( j + 4 ) . In predicate calculus terms , the final output of a PrediNet module with k heads and j relations represents the conjunction of elementary propositions Ψ ≡ k∧ h=1 j∧ i=1 ψi ( d h i , e h 1 , e h 2 ) ( 1 ) where ψi ( dhi , e h 1 , e h 2 ) asserts that d h i is the distance between objects e h 1 and e h 2 in the 1D space defined by column i of the weight matrix WS , and the denotations of eh1 and e h 2 are captured by the vectors Qh1 and Q h 2 respectively , given the key-space defined by K. Equation 1 supplies a semantics for the PrediNet ’ s final output vectorR∗ that maps each of its elements onto a well-defined logical formula , something that can not be claimed for other architectures , such as the relation net or multi-head attention net . In the experiments reported here , only R∗ is used for downstream processing , and this vector by itself doesn ’ t have the logical structure described by Equation 1 . However , the PrediNet module can easily be extended to deliver an additional output in explicitly propositional form , with a predicate-argument structure corresponding to the RHS of Equation 1 . In the present paper , the pared-down vector form facilitates our experimental investigation , but in its explicitly propositional form , the PrediNet ’ s output could be piped directly to ( say ) a Prolog interpreter ( Fig.7 ) , to an inductive logic programming system , to a statistical relational learning system , or indeed to another differentiable neural module . 3 DATASETS AND TASKS . It would be premature to apply the PrediNet architecture to rich , complex data before we have a basic understanding of its properties and its behaviour . To facilitate in-depth scientific study , we need small , simple datasets that allow the operation of the architecture to be examined in detail and the fundamental premises of its design to be assessed . Our experimental goals in the present paper are 1 ) to test the hypothesis that the PrediNet architecture learns representations that are general-purpose and reusable , and 2 ) insofar as this is true , to investigate why . Existing datasets for relational reasoning tasks , such as CLEVR ( Johnson et al . ( 2017 ) ) and sort-of-CLEVR ( Santoro et al . ( 2017 ) ) , were ruled out because they include confounding complexities , such as occlusion and shadows or language input , and/or because they don ’ t lend themselves to the fine-grained task-level splits we required . Consequently , we devised a new configurable family of simple classification tasks that we collectively call the Relations Game . A Relations Game task involves the presentation of an image containing a number of objects laid out on a 3 × 3 grid , and the aim ( in most tasks ) is to label the image as True or False according to whether a given relationship holds among the objects in the image . While the elementary propositions learned by the PrediNet only assert simple relationships between pairs of entities , Relations Game tasks generally involve learning compound relations involving multiple relationships among many objects . The objects in question are drawn from either a training set or one of two held-out sets ( Fig . 2a ) . None of the shapes or colours in the training set occurs in either of the held-out sets . The training object set contains 8 uniformly coloured pentominoes and their rotations and reflections ( 37 shapes in all ) with 25 possible colours . The first held-out object set contains 8 uniformly coloured hexominoes and their rotations and reflections ( 46 shapes in all ) with 25 possible colours , and the second held-out object set contains only squares , but with a striped pattern of held-out colours . Each Relations Game task is tied to a given relation . Even with such a simple setup , the number of definable relations among all possible combinations of objects is astronomical ( 2 ( n+1 ) 9 for n distinct objects ) , although only a few of them will make intuitive sense . For the present study , we defined a handful of intuitively meaningful relations and generated corresponding labelled datasets comprising 50 % positive and 50 % negative examples . A selection is shown in Fig . 2c . The ‘ between ’ relation holds iff the image contains three objects in a line in which the outer two objects have the same shape , orientation , and colour . The ‘ occurs ’ relation holds iff there is an object in the bottom row of three objects that has the same shape , orientation , and colour as the ( single ) object in the top row . The ‘ same ’ relation holds iff the image contains two objects of the same shape , orientation , and colour . In each case , we balanced the set of negative examples to ensure that “ tricky ” images involving pairs of objects with the same colour but different shape or the same shape but different colour occur just as frequently as those with objects that differ in both colour and shape .

Summary: This paper presents PrediNet: an architecture explicitly designed to extract representations in the form of three-place predicates (relations). They evaluate the architecture on a visual relational task called the "Relations Game" which involves comparing Tetris-like shapes according to their appearance, relative positions, etc.. They show that their architecture leads to useful generalizable representations in the sense that they can be used for new tasks without retraining.