Abstract Diagrammatic Reasoning with Multiplex Graph Networks

ABSTRACT DIAGRAMMATIC REASONING WITH MULTIPLEX GRAPH NETWORKS Duo Wang ∗ & Mateja Jamnik & Pietro Lio Department of Computer Science and Technology University of Cambridge Cambridge , United Kingdom { Duo.Wang , Mateja.Jamnik , Pietro.Lio } @ cl.cam.ac.uk ABSTRACT Abstract reasoning , particularly in the visual domain , is a complex human ability , but it remains a challenging problem for artificial neural learning systems . In this work we propose MXGNet , a multilayer graph neural network for multi-panel diagrammatic reasoning tasks . MXGNet combines three powerful concepts , namely , object-level representation , graph neural networks and multiplex graphs , for solving visual reasoning tasks . MXGNet first extracts object-level representations for each element in all panels of the diagrams , and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels . MXGNet summarises the multiple graphs extracted from the diagrams of the task , and uses this summarisation to pick the most probable answer from the given candidates . We have tested MXGNet on two types of diagrammatic reasoning tasks , namely Diagram Syllogisms and Raven Progressive Matrices ( RPM ) . For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8 % . For PGM and RAVEN , two comprehensive datasets for RPM reasoning , MXGNet outperforms the state-of-the-art models by a considerable margin . 1 INTRODUCTION . Abstract reasoning has long been thought of as a key part of human intelligence , and a necessary component towards Artificial General Intelligence . When presented in complex scenes , humans can quickly identify elements across different scenes and infer relations between them . For example , when you are using a pile of different types of LEGO bricks to assemble a spaceship , you are actively inferring relations between each LEGO brick , such as in what ways they can fit together . This type of abstract reasoning , particularly in the visual domain , is a crucial key to human ability to build complex things . Many tests have been proposed to measure human ability for abstract reasoning . The most popular test in the visual domain is the Raven Progressive Matrices ( RPM ) test ( Raven ( 2000 ) ) . In the RPM test , the participants are asked to view a sequence of contextual diagrams , usually given as a 3× 3 matrices of diagrams with the bottom-right diagram left blank . Participants should infer abstract relationships in rows or columns of the diagram , and pick from a set of candidate answers the correct one to fill in the blank . Figures 1 ( a ) shows an example of RPM tasks containing XOR relations across diagrams in rows . More examples can be found in Appendix C. Another widely used test for measuring reasoning in psychology is Diagram Syllogism task ( Sato et al . ( 2015 ) ) , where participants need to infer conclusions based on 2 given premises . Figure 1c shows an example of Euler Diagram Syllogism task . Barrett et al . ( 2018 ) recently published a large and comprehensive RPM-style dataset named Procedurally Generated Matrices ‘ PGM ’ , and proposed Wild Relation Network ( WReN ) , a state-of-the-art neural net for ∗Corresponding Author RPM-style tasks . While WReN outperforms other state-of-the-art vision models such as Residual Network He et al . ( 2016 ) , the performance is still far from deep neural nets ’ performance on other vision or natural language processing tasks . Recently , there has been a focus on object-level representations ( Yi et al . ( 2018 ) ; Hu et al . ( 2017 ) ; Hudson & Manning ( 2018 ) ; Mao et al . ( 2019 ) ; Teney et al . ( 2017 ) ; Zellers et al . ( 2018 ) ) for visual reasoning tasks , which enable the use of inductive-biased architectures such as symbolic programs and scene graphs to directly capture relations between objects . For RPM-style tasks , symbolic programs are less suitable as these programs are generated from given questions in the Visual-Question Answering setting . In RPM-style tasks there are no explicit questions . Encoding RPM tasks into graphs is a more natural choice . However , previous works on scene graphs ( Teney et al . ( 2017 ) ; Zellers et al . ( 2018 ) ) model a single image as graphs , which is not suitable for RPM tasks as there are many different layers of relations across different subsets of diagrams in a single task . In this paper we introduce MXGNet , a multi-layer multiplex graph neural net architecture for abstract diagram reasoning . Here ’ Multi-layer ’ means the graphs are built across different diagram panels , where each diagram is a layer . ‘ Multiplex ’ means that edges of the graphs encode multiple relations between different element attributes , such as colour , shape and position . Multiplex networks are discussed in detail by Kao & Porter ( 2018 ) . We first tested the application of multiplex graph on a Diagram Syllogism dataset ( Wang et al . ( 2018a ) ) , and confirmed that multiplex graph improves performance on the original model . For RPM task , MXGNet encodes subsets of diagram panels into multi-layer multiplex graphs , and combines summarisation of several graphs to predict the correct candidate answer . With a hierarchical summarisation scheme , each graph is summarised into feature embeddings representing relationships in the subset . These relation embeddings are then combined to predict the correct answer . For PGM dataset ( Barrett et al . ( 2018 ) ) , MXGNet outperforms WReN , the previous state-of-the-art model , by a considerable margin . For ‘ neutral ’ split of the dataset , MXGNet achieves 89.6 % test accuracy , 12.7 % higher than WReN ’ s 76.9 % . For other splits MXGNet consistently performs better with smaller margins . For the RAVEN dataset ( Zhang et al . ( 2019 ) ) , MXGNet , without any auxiliary training with additional labels , achieves 83.91 % test accuracy , outperforming 59.56 % accuracy by the best model with auxiliary training for the RAVEN dataset . We also show that MXGNet is robust to variations in forms of object-level representations . Both variants of MXGNet achieve higher test accuracies than existing best models for the two datasets . 2 RELATED WORK . Raven Progressive Matrices : Hoshen & Werman ( 2017 ) proposed a neural network model on Raven-style reasoning tasks that are a subset of complete RPM problems . Their model is based on Convolutional Network , and is demonstrated to be ineffective in complete RPM tasks ( Barrett et al . ( 2018 ) ) . Mandziuk & Zychowski also experimented with an auto-encoder based neural net on simple single-shape RPM tasks . Barrett et al . ( 2018 ) built PGM , a complete RPM dataset , and proposed WReN , a neural network architecture based on Relation Network ( Santoro et al . ( 2017 ) ) .Steenbrugge et al . ( 2018 ) replace CNN part of WReN with a pre-trained Variational Auto Encoder and slightly improved performance . Zhang et al . ( 2019 ) built RAVEN , a RPM-style dataset with structured labels of elements in the diagrams in the form of parsing trees , and proposed Dynamic Residual Trees , a simple tree neural network for learning with these additional structures . Anonymous ( 2020 ) applies Multi-head attention ( Vaswani et al . ( 2017 ) ) , originally developed for Language model , on RPM tasks . Visual Reasoning : RPM test falls in the broader category of visual reasoning . One widely explored task of visual reasoning is Visual Question Answering ( VQA ) . Johnson et al . ( 2017 ) built CLEVR dataset , a VQA dataset that focuses on visual reasoning instead of information retrieval in traditional VQA datasets . Current leading approaches ( Yi et al . ( 2018 ) ; Mao et al . ( 2019 ) ) on CLEVR dataset generate synthetic programs using questions in the VQA setting , and use these programs to process object-level representations extracted with objection detection models ( Ren et al . ( 2015 ) ) . This approach is not applicable to RPM-style problems as there is no explicit question present for program synthesis . Graph Neural Networks : Recently there has been a surge of interest in applying Graph Neural Networks ( GNN ) for datasets that are inherently structured as graphs , such as social networks . Many variants of GNNs ( Li et al . ( 2015 ) ; Hamilton et al . ( 2017 ) ; Kipf & Welling ( 2016 ) ; Veličković et al . ( 2017 ) ) have been proposed , which are all based on the same principle of learning feature representations of nodes by recursively aggregating information from neighbour nodes and edges . Recent methods ( Teney et al . ( 2017 ) ; Zellers et al . ( 2018 ) ) extract graph structures from visual scenes for visual question answering . These methods build scene graphs in which nodes represent parts of the scene , and edges capture relations between these parts . Such methods are only applied to scenes of a single image . For multi-image tasks such as video classification , Wang et al . ( 2018b ) proposed non-local neural networks , which extract dense graphs where pixels in feature maps are connected to all other feature map pixels in the space-time dimensions . 3 REASONING TASKS . 3.1 DIAGRAM SYLLOGISM . Syllogism is a reasoning task where conclusion is drawn from two given assumed propositions ( premises ) . One well-known example is ’ Socrates is a man , all man will die , therefore Socrates will die ’ . Syllogism can be conveniently represented using many types of diagrams ( Al-Fedaghi ( 2017 ) ) such as Euler diagrams and Venn diagrams . Figure 1 ( c ) shows an example of Euler diagram syllogism . Wang et al . ( 2018a ) developed Euler-Net , a neural net architecture that tackles Euler diagram syllogism tasks . However Euler-Net is just a simple Siamese Conv-Net , which does not guarantee scalability to more entities in diagrams . We show that the addition of multiplex graph both improves performance and scalability to more entities . 3.2 RAVEN PROGRESSIVE MATRICES . In this section we briefly describe Raven Progressive Matrices ( RPM ) in the context of the PGM dataset ( Barrett et al . ( 2018 ) ) and the RAVEN dataset ( Zhang et al . ( 2019 ) ) . RPM tasks usually have 8 context diagrams and 8 answer candidates . The context diagrams are laid out in a 3×3 matrix C where c1,1 , .. c3,2 are context diagrams and c3,3 is a blank diagram to be filled with 1 of the 8 answer candidates A = { a1 , . . . , a8 } . One or more relations are present in rows or/and columns of the matrix . For example , in Figure 1 ( a ) , there is XOR relation of positions of objects in rows of diagrams . With the correct answer filled in , the third row and column must satisfy all relations present in the first 2 rows and columns ( in the RAVEN dataset , relations are only present in rows ) . In addition to labels of correct candidate choice , both datasets also provide labels of meta-targets for auxiliary training . The meta-target of a task is a multi-hot vector encoding tuples of ( r , o , a ) where r is the type of a relation present , o is the object type and a is the attribute . For example , the meta-target for Figure 1 ( a ) encodes ( XOR , Shape , Position ) . The RAVEN dataset also provides additional structured labels of relations in the diagram . However , we found that structured labels do not improve results , and therefore did not use them in our implementation .

Summary: The paper proposes a novel, feedforward, end-to-end trainable, deep, neural network for abstract diagrammatic reasoning with significant improvements over the state of the art. The proposed model architecture is reasonable and is designed to exploit the information present at multiple granularities – at the level of objects in the diagram, their relations across diagrams, and diagram subsets. As a multimodule neural pipeline, it seems a reasonable design. Further, it shows significant performance gains over the state of the art.