Abstract Diagrammatic Reasoning with Multiplex Graph Networks
Authors: Duo Wang
Mateja Jamnik
Pietro Lio
Venue:  (2020)
Keywords: reasoning
Raven Progressive Matrices
graph neural networks
multiplex graphs

Abstract: Abstract reasoning, particularly in the visual domain, is a complex human ability, but it remains a challenging problem for artificial neural learning systems. In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks. MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks. MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels. MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates. We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM). For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8%.  For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of-the-art models by a considerable margin.

Published as a conference paper ICLR 2020
ABSTRACT DIAGRAMMATIC REASONING WITH MULTIPLEX GRAPH NETWORKS
Duo Wang & Mateja Jamnik & Pietro Lio
Department of Computer Science and Technology
University of Cambridge Cambridge, United Kingdom (Duc. Wang, Mateja Jamnik, Pietro. Liooool.ooocc..o..ooooooooo.. cam.)aacccac.c..... pieerr...ooBeal.ca....o.
ABSTRACT
Abstract reasoning, particularly in the visual domain, is complex human ability, but it re- mains a challenging problem for artificial neural learning systems. In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks. MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks. MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels. MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates. We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM). For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8% 99.8%. For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of sfute-of-the-ar models by considerable margin.
1 INTRODUCTION
Abstract reasoning has long been thought of as key part of human intelligence, and a necessary component towards Artificial General Intelligence. When presented in complex scenes, humans can quickly identify elements across different scenes and infer relations between them. For example, when you are using a pile of different types of LEGO bricks to assemble a spaceship, you are actively inferring relations between each LEGO brick, such as what ways they can fit together. This type of abstract reasoning, particularly the visual domain, is crucial key to human ability to build complex things.
Many tests have been proposed to measure human ability for abstract reasoning. The most popular test in the visual domain is the Raven Progressive Matrices (RPM) test (Raven (200)). In the RPM test, the participants are asked to view a sequence of contextual diagrams, usually given as a 3 x 3 matrices of diagrams with the bottom-right diagram left blank. Participants should infer abstract relationships in rows or columns of the diagram, and pick from a set of candidate answers the correct one to fill in the blank. Figures (a) shows an example of RPM tasks containing XOR relations across diagrams rows. More examples can be found in Appendix C Another widely used test for measuring reasoning in psychology is Diagram Syllogism task (Sato et al. (2015)), where participants need to infer conclusions based on given premises. Figure shows an example of Euler Diagram Syllogism task.
Barrett et al.
2018 recently published large and comprehensive RPM-style dataset named Procedurally Generated Matrices 'PGM', and proposed Wild Relation Network (WReN), a ofte-oof-hheart neural net for
*Corresponding Author

---

Published as a conference paper ICLR 2020
Context Panels
MultiLayer Graphs
Pp By
Gp
G
Fj + By
RØ
Meta- target
G
Fp + Ey
Prediction
?
1...8
Answer Panels
(a)
(b)
(c)
Figure 1: (a) shows an example of RPM tasks containing XOR relations across diagrams in rows and the overview of MXGNet architecture. Here F, is object representation module, E,, is edge embeddings module, Go is graph summarization module and Rg is reasoning network. (b) shows an example multilayer graph formed from objects in the first row of diagrams in the example. (c) An example of syllogism represented in Euler diagrams.
RPM-style tasks. While WReN outperforms other state-of-thh-arr vision models such as Residual Network He et al. 2016), the performance still far from deep neural nets' performance on other vision or natural language processing tasks. Recently, there has been a focus on object-level representations (Yi et al. 2018):
Hu et al. (2017) Hudson & Manning (2018); Mao al. (2019): Teney et al. (2017): Zellers et al. (2018) for visual reasoning tasks, which enable the use of Inductive-biased architectures such as symbolic programs and scene graphs to directly capture relations between objects. For RPM-style tasks, symbolic programs are less suitable these programs are generated from given questions in the Visual-Questen
Answering setting. In
RPM-style tasks there are no explicit questions. Encoding RPM tasks into graphs is a more natural choice. However, previous works scene graphs Teney et al. 2017); Zellers et al. (2018)) model single image graphs, which not suitable for RPM tasks as there are many different layers of relations across different subsets of diagrams in single task.
2

---

Published as a conference paper ICLR 2020
In this paper we introduce MXGNet, a multi-layer multiplex graph neural net architecture for abstract diagram reasoning. Here 'Multi-layer'' means the graphs are built across different diagram panels, where each diagram is a layer. 'Multiplex' means that edges of the graphs encode multiple relations between different element attributes, such as colour, shape and position. Multiplex networks are discussed in detail by Kao & Porter (2018). We first tested the application of multiplex graph on a Diagram Syllogism dataset (Wang et al.] (201 8ab), and confirmed that multiplex graph improves performance on the original model. For RPM task, MXGNet encodes subsets of diagram panels into multi-layer multiplex graphs, and combines summarisation of several graphs to predict the correct candidate answer. With a hierarchical summarisation scheme, each graph is summarised into feature embeddings representing relationships in the subset. These relation embeddings are then combined to predict the correct answer.
For PGM dataset (Barrett et al. (2018), MXGNet outperforms WReN, the previous state-of-the art model, by a considerable margin. For "neutral' split of the dataset, MXGNet achieves 89.6% test accuracy, 12.7% higher than WReN's 76.9%. For other splits MXGNet consistently performs better with smaller margins. For the RAVEN dataset (Zhang et al. (2019)), MXGNet, without any auxiliary training with additional labels, achieves 83.91 1% test accuracy, outperforming 59.56% accuracy by the best model with auxiliary training for the RAVEN dataset. We also show that MXGNet is robust to variations in forms of object-level representations. Both variants of MXGNet achieve higher test accuracies than existing best models for the two datasets.
2 RELATED WORK
proposed a neural network model on Raven-style reasoning tasks that are a subset complete RPM problems. Their model is based on Convolutional Network, and is demonstrated to be ineffective in complete RPM tasks (Barrett et al. (2018)). Mandziuk & Zychowski also experimented with an auto-encoder based neural net on simple single- RPM tasks. Barrett et al. (2018) built PGM, a complete RPM dataset, and proposed WReN, neural network architecture based on Relation Network (Santoro et al. (2017)) Steenbrugge et al. (2018) replace CNN part of WReN with a pre-trained Variational Auto Encoder and slightly improved performance. Zhang et al. (2019) built RAVEN, a RPM-style dataset with structured labels of elements in the diagrams in the form of parsing trees, and proposed Dynamic Residual Trees, simple tree neural network for learning with these additional structures. Anonymous (2020) applics Multi-head attention (Vaswani et al. (2017). originally developed for Language model, RPM tasks.
Raven Progressive Matrices: Hoshen & Werman 2017)
Visual Reasoning: RPM test falls in the broader category of visual reasoning. One widely explored task of visual reasoning is Visual Question Answering( V VQA). Dohnson et al. (2017) built CLEVR dataset, a VQA dataset that focuses on visual reasoning instead of information retrieval in traditional VQA datasets. Current leading approaches (Yi et al. 20188; Mao et al. (2019) on CLEVR dataset generate synthetic programs using questions in the VQA setting, and use these programs to process object-level representations extracted with objection detection models (Ren et al. (2015). ). This approach is not applicable to RPM-style problems as there is no explicit question present for program synthesis.
Graph Neural Networks: Recently there has been a surge of interest in applying Graph Neural Networks (GNN) for datasets that are inherently structured as graphs, such as social networks. Many variants of GNNs (Li et al.) (2015); Haamllon et al. (2017): Kipf & Welling (2016): Veličković et al. (2017) have been proposed, which are all based on the same principle of learning feature representations of nodes by recursively aggregating information from neighbour nodes and edges. Recent methods (Teney et al. (2017); Zellers et al. (2018) extract graph structures from visual scenes for visual question answering. These methods build scene graphs in which nodes represent parts of the scene, and edges capture relations between these parts. Such methods are only applied to scenes of single image. For multi-image tasks such video classification, Wang et af.] (2018b) proposed non-local neural networks, which extract dense graphs where pixels in feature maps are connected to all other feature map pixels in the space-time time dimensions.
3

---

Published as a conference paper ICLR 2020
3 REASONING TASKS
3.1 DIAGRAM SYLLOGISM
Syllogism is a reasoning task where conclusion is drawn from two given assumed propositions (premises). One well-known example is 'Socrates is man, all man will die, therefore Socrates will die'. Syllogism can be conveniently represented using many types of diagrams (Al-Fedaghi (2077) such Euler diagrams and Venn diagrams. Figure | (c) shows an example of Euler diagram syllogism. Wang et al. (2018a) developed Euler-Net, neural net architecture that tackles Euler diagram syllogism tasks. However Euler-Net is just a simple Siamese Conv-Net, which does not guarantee scalability to more entities in diagrams. We show that the addition of multiplex graph both improves performance and scalability to more entities.
3.2 RAVEN PROGRESSIVE MATRICES
In this section we briefly describe Raven Progressive Matrices (RPM) in the context of the PGM dataset (Barrett et al. (2018) and the RAVEN dataset (Zhang et al. (2019). RPM tasks usually have 8 context diagrams and answer candidates. The context diagrams are laid out in 3 x matrix C where C1.1...C3. are context diagrams and C3.3 is a blank diagram to be filled with I of the 8 answer candidates A {011.... as). One or more relations are present in rows or/and columns of the matrix. For example, in Figure (a), there XOR relation of positions of objects in rows of diagrams. With the correct answer filled in, the third row and column must satisfy all relations present in the first 2 rows and columns (in the RAVEN dataset, relations are only present in rows). In addition to labels of correct candidate choice, both datasets also provide labels of meta- targets for auxiliary training. The meta-target of task is a multi-ho vector encoding tuples of (r, o, a) where is the type of relation present, is the object type and is the attribute. For example, the meta- veta-target for Figure (a) encodes (XOR, Shape, Position). The RAVEN dataset also provides additional structured labels of relations in the diagram. However, we found that structured labels do not improve results, and therefore did not use them in our implementation.
4 METHOD
MXGNet is comprised of three main components: object-level representation module, graph processing module and a reasoning module. Figure| Ta shows an overview of the MXGNet architecture. The object- level representation module Fp. as the name suggests, extracts representations of objects in the diagrams as nodes in a graph. For each diagram dj C CUAA A, a set of nodes Vi,;; = 1... L.j = 1...N N is extracted where L is the number of layers and N is the number of nodes per layer. We experimented with both fixed and dynamically learnt N values. We also experimented with an additional 'background' encoder that encodes background lines (See Appendix C for an example containing background lines) into a single vector, which can be considered as a single node. The multiplex graph module Gg, for a subset of diagrams, learns the multiplex edges capturing multiple parallel relations between nodes in a multi-layer graph where each layer corresponds to one diagram in the subset, as illustrated in Figure (c). In MXGNet, we consider a subset of cardinality 3 for 3 x 3 diagram matrices. While prior knowledge of RPM rules allows us to naturally treat rows and columns in RPM as subsets, this prior does not generalise other types of visual reasoning problems. Considering all possible diagram combinations as subsets is computationally expensive. To tackle this, we developed relatively quick pre-training method to greatly reduce the search space of subsets, as described below.
Search Space Reduction: We can consider each diagram as node ef in a graph, where relations between adjacent diagrams are embedded edges ety, Note here we considering the graph of 'diagrams', which is different from the graph of "objects' in the graph processing modules. Each subset of 3 diagrams in this case can be considered as subset of 2 edges. We here make weak assumptions that edges exist between adjacent diagrams (including vertical, horizontal and diagonal direction) and edges in the same subset must be adjacent (defined as two edges linking the same node), which often used in other visual reasoning
4

---

Published as a conference paper ICLR 2020
problems. We denote the subset of edges as {ee G.. We use neural nets to embed nodes, edges and subsets. We use CNNs to embed diagram nodes into feature vectors, and MLPs to embed edges based on node embeddings and subsets based on edge embeddings. While it is possible to include graph architectures for better accuracy, we found that simple combinations of CNNs and MLPs train faster while still achieving the search space reduction results. This architecture first embeds nodes, then embeds edges based on node embedding, and finally embed subsets based on edge embedding, The subset embeddings are summed and passed through reasoning network to predict answer probability, similar to WReN (Barrett et al. (2018) For the exact configuration of the architecture used please refer to Appendix A For each subset Ci Gk we define gating variable Gijk. controlling how much does each subset contributes to the final result. In practice we use tanh function, which allows subset to contribute both positively and negatively to the final summed embeddings. In training we put L1 regularization constraint on the gating variables to suppress Gijk of non-contributing subsets close to zero. This architecture can quickly discover rows and columns as contributing subsets while leaving gating variables of other subsets not activated. We describe the experiment results in section 5.1 While this method is developed for discovering reasoning rules for RPM task, it can be readily applied to any other multi-frame reasoning task for search space reduction. In the rest of the paper, we hard-gate gate subsets by rounding the gating variables, thereby reducing subset space to only treat rows and columns as valid subsets.
We treat the first rows and columns as contextual subsets Ct, where i and are row and column indices. For the last row and column, where the answers should be filled in, we fill in each of the 8 answer candidates, and make 8 row subsets a,, C [1, 8) and 8 column subsets a,, i C (1,8).
The graph module then summarises the graph of objects in subset into embeddings representing relations present in the subset. The reasoning module Ro takes embeddings from context rows/columns and last rows/columns with different candidate answers filled in, and produce normalised probability of each answer being true. It also predicts meta-target for auxiliary training using context rows/columns, Next, we describe each module in detail.
4.1 OBJECT-LEVEL REPRESENTATION
In the PGM dataset there are two types of objects, namely 'shapes' and background 'lines'. While a natural choice to use object-level representation on shapes they varying in many attributes such as position and size, it is less efficient on background lines as they only vary in colour intensity. In this section we first describe object-level representation applied to 'shapes' objects, and then discuss object-level representation on 'lines' and an alternative background encoder which performs better.
MXGNet we experiment with two types of object-level representations for 'shapes', namely CNN grid features and representation obtained with spatial attention. For CNN grid features, we use each spatial location the final CNN feature map as the object feature vector. Thus for each feature maps of width W and height H, .N = W x H object representations are extracted. This type of representation is used widely, such as in Relation Network (Santoro et al. (2017)) and VQ- VAE (van den Oord et al. (2017)). For representation obtained with attention, for each object attended. This is similar to objection detection models such as faster R-CNN (Ren et al.) (2015)), which use a Region Proposal Network to propose bounding boxes of objects the input image. For each attended location a presence variable Zyres is predicted by attention module indicating whether an object exists in the location. Thus the total number of objects N can vary depending on the sum of zpres variables. As object-level representation is not the main innovation of this paper, we leave exact details for Appendix A.I
use spatial attention attend locations of objects, and extract representations
For background 'lines" objects, which are not varying in position and size, spatial attention is not needed. We experimented with a recurrent encoder with Long-Short Term Memory (Hochreiter & Schmidhuber/ (1997) on the output feature map of CNN, outputting M number of feature vectors. However, in the experiment
5

---

conv-net Published as a conference paper ICLR 2020
aggregated 1 to 3 edges
Node Aggregation
Diagram 1
Diagram 2
Diagram 3
aggregated edges
Figure 2: Illustration of multiplex edge embeddings and cross-gaie goos--gating function. Each edge contains a set of different sub-connections (colored differently). Multiplex edges connecting to each node the last layer are aggregated according its originating layer. Aggregated embeddings then passed gating function G, which outputs gating variables from each aggregated embeddings.
we found that this performs less well than just feature map embeddings produced by encoder.
4.2 MULTIPLEX GRAPH NETWORK
Multiplex Edge Embedding object-level representation module outputs set of representations Ui i C [1, L).j C [1, NJ for 'shapes' objects, where L is the number of layers (cardinality of subset of diagrams) and N the number nodes per layer. MXGNet uses an multiplex edge-embedding network E,, to generate edge embeddings encoding multiple parallel relation embeddings:
(.s)a..)) = (P* (vis, i
Here p is projection layer projecting concatenated node embeddings T different embeddings. E* is a small neural net processing tth projections to produce the fth sub-layer of edge embeddings. Here, we restricted the edges to be inter-layer only, as found using intra-layer edges does improve performance but increases computational costs. Figure illlustrates these multiplex edge embeddings between nodes of different layers. We hypothesise that different layers of the edge embeddings encode similarities//iiferenees in different feature spaces. Such embeddings of similaritissddifeernnes are useful in comparing nodes for subsequent reasoning tasks. For example.for Progessive relation of object sizes, part of embeddings encoding size differences can be utilized to check if nodes in later layers are larger in size. This is similar Mixture of Experts layers (Higen al. (20T3): [Shazeer al. (2017) introduced Neural Machine Translation tasks. However, in this work we developed a new cross-muliplexing gating function at the node message aggregation stage, which is described below.
After edge embeddings are generated, the graph module then summarises the graph into a feature embedding representing relations present the subset of diagrams. aggregate information in the graph to nodes of the last layer corresponding to the third diagram in row column, because in RPM tasks the relations are in the form Diagram3 = Function (DionnDaa...... Diagram2). All edges connecting nodes in a particular layer i A L,, node vz, & in the last layer I. are aggregated by function Fag composed four different types set operations, namely max, min, sum and mean:
Graph Summarisation:
fULA
Fag ( -.. (L.)).(L.))); Fag concat(mar() min(), sum(), mcan()
(2)
6

---

Published as a conference paper ICLR 2020
We use multiple aggregation functions together because different sub-tasks in reasoning may require different types of summarization. For example, counting number of objects is better suited for sum while checking if there is object with the same size is better suited for maz. The aggregated node information from each layer is then combined with a mrooipluuiiil gating function. It is named pross-multiplexing because each embeddings in the set are "multiplexing" other embeddings in the set with gating variables that regulate which stream of information pass through. This gating function accepts set of summarised node embeddings { [f ... f as input, and output gating variables for each layer of node embeddings in the set:
g1, gNNk = G(foi,k fuN,A): gi, = {gg, ... ....)
In practice G is implemented as an MLP with multi-head outputs for different embeddings, and Sigmoid activation which constrains gating variable g within the range of 0 to 1. The node embeddings of different layers are then multiplied with the gating variables, concatenated and passed through a small MLP to produce the final node embeddings: fez MM [ (f x g k) ........... Node embeddings and background embeddings are then concatenated and processed by residual neural block to produce final relation feature embeddings r of the diagram subset.
4.3 REASONING NETWORK
The reasoning network takes relation feature embeddings r from all graphs, and infers the correct answer based on these relation embeddings. We denote the relation embeddings for context rows as rf": i 1, and context columns as r3;; i 1,2. The last row and column filled with each answer candidate are denoted rf"; = I,..., . and r::: = 1, ..... 8. For the RAVEN dataset, only row relation embeddings rer and rar used, as discussed in Section 3.2 The reasoning network Rg is multi-layer residual neural net with softmax output activation that processes concatenated relation embeddings and outputs class probabilities for each answer candidate. The exact configuration of the reasoning network be found in Appendix A.3
For meta-target prediction, all relation information is contained in the context rows and columns of the RPM task. Therefore. we apply meta- meta- predicting network Rmeta with Sigmoid output activation to all context rows and columns to obtain probabilities of each meta-tare target categories:
Rmatt (ri + 2 + r + +
Pmeta
4.4 TRAINING
The full pipeline of MXGNet end-to-end trainable with any gradient descent optimiser. In practice, we used RAdam optimiser (Liu et al.] (2019)) for its fast convergence and robustness to learning rate differences. The loss function for the PGM dataset the same as used in WReN (Barrett et al. (2018)) L = Lans + 3&meta-targe where 3 balances the training between answer prediction and target prediction. For the RAVEN dataset, while the loss function can include auxiliary meta-target and structured labels L Lans atstruct + 3Lmmetattrgee we found that both auxiliary targets do not improve performance, and thus set a and (3 to O.
5
EXPERIMENTS
5.1 SEARCH SPACE REDUCTION
The Search Space Reduction model is applied on both PGM and RAVEN dataset to reduce the subset space. After 10 epochs, only gating variables of rows and columns subset for PGM and of rows for RAVEN have value larger than 0.5. The Gating variables for three rows are 0.884, 0.812 and 0.832. The gating variables for three columns are 0.901, 0.845 and 0.854. All other gating variables are below the threshold value of 0.5. Interestingly all activated (absolute value 0.5) gating variables are positive. This is possibly because it is easier for the neural net to learn an aggregation function than a comparator function. Exact experiment statistics can be found in Appendix D
7

---

Published as a conference paper ICLR 2020
5.2 DIAGRAM SYLLOGISM PERFORMANCE
We first test how well can the multiplex graph network capture relations for the simple Diagram Syllogism task. We simply add the multiplex graph to the original Conv-Net used in (Wang et al. (201 8a)). MXGNet achieved 99.8% accuracy on both 2-contour and 3-contour tasks, higher than the original paper's 99.5% and 99.4% accuracies. The same performance on 2-contour and 3-contour tasks also show that MXGNet scales better for more entities in the diagram. For more details please refer Appendix E
5.3 RPM TASK PERFORMANCES
In this section we compare all variants of MXGNet against the oftte-of-thearr models for the PGM and the RAVEN datasets. For the PGM dataset, we tested against results of WReN (Barrett et al. (2018) ) in the auxiliary training setting with 33 value of 10. In addition, we also compared MXGNet with
(Steenbrugge et al. (2018) result without auxiliary training. For the RAVEN dataset, we compared with WReN and ResNet model's performance as reported in the original paper (Zhang et al. (2019)). We evaluated MXGNet with different object- representations (Section on the test data in the 'neutral' split of the
PGM dataset.
Table D (a) shows test accuracies of model variants compared with WReN and VAE-WReN for the case without auxiliary training (3 = 0) and with auxiliary training (B = 10) for the PGM dataset. Both model variants of MXGNet outperform other models by considerable margin, showing that the multi-layer graph is indeed a more suitable way to capture relations in the reasoning task. Model variants using grid features from the CNN feature maps slightly outperform model using nattial-atenntion-aaas with and without auxiliary training settings. This is possibly because the increased number of parameters for the spatial attention variant leads over-fitting as the training losses of both model variants are very close. In our following experiments for PGM we will use model variants using CNN features to report performances.
object representations for both
Table (b) shows test accuracies of model variants compared with WReN the best performing ResNet models for RAVEN dataset. WReN surprisingly only achieves 14.69% as tested by Zhang et al. (2019). We include results of the ResNet model with or without Dynamic Residual Trees (DRT) which utilise additional structure labels of relations. We found that for the RAVEN dataset, auxiliary training of MXGNet with meta-target structure labels does not improve performance. Therefore, we report test accuracies of models trained only with the target-prediction objective. Both variants of MXGNet significantly outperform the ResNet models. Models with spatial attention object-level representations under-perform simpler CNN features slightly, most probably due to overfitting, as the observed training losses of spatial attention models are in fact lower than CNN feature models.
5.4 GENERALISATION EVALUATION FOR PGM
In the PGM dataset, other than the neutral data regime in which test dataset's sampling space is the same as the training dataset, there are also other data regimes which restrict the sampling space of training or test data to evaluate the generalisation capability of a neural network. In the main paper, due to space limitations, we selected 2 representative regimes, the 'interpolation' regime and the 'extrapolation" regime to report results. For results of other data splits of PGM, please refer to Appendix G For 'interpolation' regime, in the training dataset, when attribute a color and spectrum of values. This tests how well can a model 'interpolate' for missing values. For 'Extrapolation" regime, in the training dataset, the value of is restricted to be the lower half of the value spectrum. This tests how well can model 'extrapolate' outside of the value range in the training dataset. Table shows validation and test accuracies for all three data regimes with and without auxiliary training. In addition, differences between validation and test accuracies are also presented to show how well can models generalise. MXGNet models consistently perform better than WReN for all regimes tested. Interesting for 'Interpolation' regime, while validation accuracy of MXGNet is lower than WReN, the test accuracy is higher. In addition, for regime
size, the values of are restricted to even-indexed values the
8

---

Published as a conference paper ICLR 2020
WReN Barrett
VAE-WReN
ARNe
MXGNet
Model
al.
Steenbrugge et al. (2018)
Anonymous
2020)
CNN
Sp-Attn
(2018)
76.9
N/A
88.2
89.6
88.8
(5) = 10
acc.
62.6
64.2
N/A
66.7
66.1
(%) = 0
(a) PGM
WReN
ResNet
ResNet+DRT
ARNe
MXGNet
Model
Zhang et al. (2019)
Zhang al.
2019
Zhang et al. (0019
Anonymous
2020
CNN
Sp-Attn
acc. (%)
14.69
53.43
59.56
19.67
83.91
82.61
(b) RAVEN
Table I: (a) shows results comparing MXGNet model variants against WReN for the PGM dataset. (b) shows results comparing MXGNet model variants against ResNet models for the RAVEN dataset. The object-level representaionn has two variations which are (ol) CNN features and (o2) Spatial Attention features (Section 4.1)
'Interpolation' and 'Extrapolation' MXGNet also shows smaller difference between validation and test accuracy. These results show that MXGNet has better capability of generalising outside of the training space.
3
10
Model
Regime
Val.
test%
Diff.
Val. %
test %
Diff.
Neutral
63.0
62.6
-0.4
77.2
76.9
-0.3
WReN
Interpolation
79.0
64.4
-14.6
92.3
67.4
-24.9
Extrapolation
69.3
17.2
-52.1
93.6
15.5
-79.1
Neutral
67.1
66.7
-0.4
89.9
89.6
-0.3
MXGNet
Interpolation
74.2
65.4
-8.8
91.5
84.6
-6.9
Extrapolation 69.1
18.9
-50.2
94.3
18.4
-75.9
Table Generalisation performance comparing MXGNet model variants against WReN. DDiff'' the difference between the test and the validation performances.
DISCUSSION AND CONCLUSION
We presented MXGNet, new graph- based approach to diagrammatic reasoning problems the style of Raven Progressive Matrices (RPM). MXGNet combines three powerful ideas, namely, object-level representation, graph neural networks and multiplex graphs, to capture relations present in the reasoning task. Through experiments showed that MXGNet performs better than previous models on two RPM datasets. We also showed that MXGNet has better generalisation performance.
One important direction for future work is to make MXGNet interpretable, and thereby extract logic rules from MXGNet. Currently, the learnt representations in MXGNet are still entangled, providing little in the way of understanding its mechanism of reasoning. Rule extraction provide people with better understanding of the reasoning problem, and may allow neural networks to work seamlessly with more programmable traditional logic engines.
While the multi-layer multiplex graph neural network is designed for RPM style reasoning task, it can be readily extended to other diagrammatic reasoning tasks where relations present between multiple elements
9

---

Published as a conference paper ICLR 2020
across different diagrams. One example of real-world application scenario is robots assembling parts of an object into a whole, such building LEGO model from room of LEGO blocks. MXGNet provides a suitable way of capturing relations between parts, such as ways of piecing and locking two parts together.
REFERENCES
Sabah Al-Fedaghi. Logic representation: Aristotelian syllogism by diagram. In Applied Computing and Information Technology/ Intl Conf on Computational Science/intellignnce and Applied Informatics/ Intl Conf on Big Data, Cloud Computing, Data Science and Engineering, 2017.
Anonymous. Attention on abstract visual reasoning. In Submitted to International Conference on Learn- ing Representations, 2020. URL URU/itttps///pppeeeeeee.eccc Botte//ooooooi--elelekeeo. under
review.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference Machine Learming, pp. 511-520, 2018.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations deep mixture of experts. arXiv preprint arXiv: 1312.4314, 2013.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pp. 1034, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 175):1735-,780, 1997.
Dokhyam Hoshen and Michael Werman. Iq of neural networks. arXiv preprint arXiv:1710.01692 2017.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 804-813, 2017.
Drew A Hudson and Christopher Manning. Compositional attention networks for machine reasoning.
arXiv preprint arXiv: 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing 2015.
internal covariate shift. arXiv preprint
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-sofftmxx arXiv preprint arXiv:1611.01 144, 2016.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Ta-Chu Kao and Mason A Porter. Layer communities multiplex networks. Journal of Statistical Physics, 173(3-4): 1286 128--132,, 2018.
10

---

Published as a conference paper ICLR 2020
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: wrXiv:/609002907, 2016.
Yujia Li, Daniel Tarlow, Mare Broekschmidt, and Richard Zemel. Gated graph sequence neural networks.
arXiv preprint arXivv1511.0554 2015.
Liyun Liu, Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning and beyond. arXiv preprini arXiv: 999900003555 2019.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 161.071, 2016.
Jacek Mandziuk and Adam Zychowski. Deepiq: A human-inspired ai system for solving iq test problems.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint 1904 12584, 2019.
John Raven. The raven's progressive matrices: change and stability over culture and time. Cognitive psychology, 41(1):1-48, 2000.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Yuri Sato, Sayako Masuda, Yoshiaki Someya, Takeo Tsuji,, and Shigeru Watanabe. An fmri analysis of the efficacy of euler diagrams in logical reasoning. Visual Languages and Human-Centre Computing (VL/HCC), 2015 IEEE Symposium on, pp. 143-151. IEEE, 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06533. 2017.
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generalization for abstract reasoning tasks using disentangled feature representations. arXiv preprint 1.1i:1811104784. 2018.
Damien Teney, Lingqiao Liu, and Anton van den Hengel. Graph-structured representations for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2017.
Aaron den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6000 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Duo Wang, Mateja Jamnik, and Pietro Li. Investigating diagrammatic reasoning with deep neural networks. In International Conference Theory and Application of Diagrams, 2018 8a.
11

---

Published as a conference paper ICLR 2020
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794-7803 2018b.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in Neural Information Processing Systems, pp. 1031-1042, 2018.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 5831-5840, 2018.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. arXiv preprint arXiv: 1903.02741, 2019.
12

---

Published as a conference paper ICLR 2020
A ARCHITECTURE
In this section we present exact configurations of all model variants of MXGNet. Due to the complexity of architectures, we will describe each modules sequence. The object-level representation has two variations which are (ol) CNN features and (o2) Spatial Attention features. Also the models for PGM and RAVEN dataset differ in details. Unless otherwise stated, in all layers we apply Batch Normalization loffe & Szegedy (2015) and use Rectified Linear Unit activation function.
A.1 OBJECT-LEVEL REPRESENTATION ARCHITECTURE
CNN features: The first approach applies a CNN on the input image and use each spatial location in the final CNN feature map as the object feature vector. This type of representation is used widely, such in Relation Network Santoro et al. (2017) and VQEV van den Oord et al. (017). Formally, the output of a CNN is a feature map tensor of dimension H W D where H, W and D are respectively height, width and depth of the feature map. At each H and W location, an object vector is extracted. This type of object representation is simple and fast, but does guarantee that the receptive field at each feature map location fully bounds objects in the image.
We residual module He et al. (2016) with two residual blocks to extract CNN features, shown in figure This is because Residual connections show better performance in experiments. The structure of a single Residual Convolution Block is shown in figure Unless otherwise stated, convolutional layer in residual blocks has kernel size of 3 x 3. The output feature map processed by another residual block is treated background encoding because we found that convolutional background encoding gives better results than feature vectors.
Conv. layer
ReLU
identity
Conv. layer
ReLU
Figure 3: Architecture of single Residual Convolution Block.
Spatial Attention Object-level representation: The second approach is to use spatial attention attend to locations of objects, and extract representations for each object attended. This is similar to object detection models such as faster R-CNN Ren et al.) (2015), which use Region Proposal Network propose bounding boxes of objects in the input image. In practice, spatial attention module. [ shows the architecture used for extracting object-level representation using spatial attention. A CNN composed of conv layr and 2 residual blocks is first applied to the input image, and the last layer feature map is extracted. This part is the same as CNN grid feature module. spatial attention network composed of 2 conv layer then processes information each spatial location on the feature map, and outputs numbers of = (z zwhere), corresponding to possible objects at each location. Here, pres is a binary value indicating if an object exists in this location, and 20kere is an affine
use Spatial Transformer Jaderberg et al. (2015) as our
13

---

Published as a conference paper ICLR 2020
Conv-64 4-by-4
CNN Grid Features
Conv-32 5-by-5
Max- Pooling
ResBlock 32
ResBlock 32
Background Features
Figure 4: CNN feature object-level representation module. 'Conv' is convolution layers, 'Max-Pooling' is max-pooling layer and 'ResConv Block' is Residual Convolutional Block.
transformation matrix specifying sampling region the feature maps. grres the binary variable, is sampled from Gumbel-Sigmoid distribution Maddison et al. (2016); Jang et al. (2016), which approximates the Bernoulli distribution. We Gumbel temperature 0.7 throughout the experiments. For the PGM dataset we restricted & to be and 2 where to be translation and scaling matrix as 'shapes' objects do not overlap and do not have affine transformation attributes other than scaling and translation. For all 2,; i C [1, 2fre* is I, an object encoder network samples a patch from location specified by 2 using grid sampler with fixed window size of 4 x pixels. More details of the grid sampler can be found in Jaderberg et al. (2015). The sampled patches are then processed by conv-layer to generate object embeddings.
W). if
Conv-64
4-by-4
Z
Conv-4
1-by-1
Conv-32
Max-
ResBlock 32
ResBlock 32
Conv-64 4-by-4
Object Features
Sampler
5-by-5
Pooling
Background Features
Figure 5: Spatial attention based feature object-level representation module. 'Conv' is convolution layers, "Max-Pooling" is max-pooling layer and 'ResConv Block' is Residual Convolutional Block. is the spatial attention variable (rrres, where)) Sampler is grid sampler which samples grid points from given feature maps.
A.2 GRAPH NETWORKS
Multiplex Edge Embeddings:Figuu 2 in the main paper shows overview of the multiplex graph archi- tecture. While motivation and overview of architecture is explained section of the main paper, in this section provide exact configurations for each part of the model. Each sub-layer of the multiplex edge is
14

---

Published as a conference paper ICLR 2020
embedded by a small MLP. For PGM dataset, we use parallel layers for each multiplex edge embeddings with each layer having 32 hidden units and 8 output units. For RAVEN dataset we use 4 layers with 16 hidden units and output units because RAVEN dataset contains fewer relations types than PGM dataset. Gating function is implemented as one Sigmoid fully connected layer with hidden size equal to the length of concatenated aggregated embeddings. Gating variables are element-wise multiplied with concatenated embeddings for gating effects. Gated embeddings are then processed with final fully connected layer with hidden size 64.
This module summarizes all node summary embeddings and background embed- dings produce diagram subset embedding representing relations present in the set of diagrams. We experimented with various approaches and found that keeping embeddings as feature maps and processing them with residual blocks yields the best results. Background feature map embeddings are generated with one additional residual block of 48 on top of lower layer feature-extracting resnet. For object representations obtained from CNN-grid features, we can simply reshape node embeddings into a feature map, and process with additional conv-nets to generate feature map embeddings of the same dimension to background feature map embeddings. For object representations with spatial attention, we can use another Spatial Transformer write node summary embeddings to its corresponding locations canvas feature map. Finally we concatenate node summary embeddings and background embeddings and process with residual blocks of size 64 to produce the relation embeddings.
Graph Summarization:
A.3 REASONING NETWORK
Figure shows the reasoning network configuration for RPM tasks. We experimented with the approach introduced in Barrett al. (2018), which compute scores for each answer candidates and finally normalize the scores. We found this approach leads to severe overfitting on the RAVEN dataset, and therefore used a simpler approach to just concatenate all relation embeddings and process them with a neural net. In practice we used two residual blocks of size 128 and 256, and a final fully connected layer with 8 units corresponding answer candidates. The output is normalized with softmax layer. For Meta-target prediction, all context relation embeddings (context rows and columns for PGM while only rows for RAVEN dataset) are summed and fed into a fully connected prediction layer with Sigmoid activation. For PGM there are 12 different meta-targets while for RAVEN there are 9.
RelEmbed
RelEmbed 2
ResBlock 128
ResBlock 256
Softmax Output
Concat
FC-8
RelEmbed S
context embeddings Only
Meta-Target Prediction
Sum
FC-12
Figure 6: Architecture overview of reasoning module. 'RelEmbed' is relation embeddings, 'Concat' concatenation layer. "ResBlock" Residual Convolutional Block. 'FC' is fully connected layer.
15

---

Published as a conference paper ICLR 2020
B TRAINING DETAILS
The architecture is implemented in Pytorch framework. During training, used RAdam optimizer Liu et al. (2019) with learning rate 0.0001, 8n = 0.9.32 = 0.999. We used batch size of 64, and distributed the training across 2 Nvidia Geforce Titan X GPUs. We early-stop training when validation accuracy stops increasing.
C MORE DETAILS OF RPM DATASETS
In PGM dataset there are two types of elements present in the diagram, namely shapes and lines. These elements have different attributes such as colour and size. the PGM dataset, five types of relations can be present in the task: (Progression, AN D. OR, XOR, ConsistentU n.onn. The RAVEN dataset, compared PGM, does not have logic relations AND, OR, XOR, but has additional relations Arilhmelic,, Consland. In addition RAVEN dataset only allow relations to be present in rows.
Figure 7a and 7b show two examples from the PGM damaset(Image courtesy Barrett et al. (2018)) The first example contains a 'Progression' relation of the number objects across diagrams in columns. The second examples contains 'XOR' relation of position of objects across diagrams in rows.
In addition to shape objects, diagrams in the PGM dataset can also contain background line objects that appear at fixed locations. Figure Sa and 8b show two examples of PGM tasks containing line objects.
D MORE DETAILS ON SEARCH SPACE REDUCTION
In this section we provide detailed architecture used for Search Space reduction, and present additional experimental results.
The node embeddings are generated by applying a Conv-Net of 4 convolutional layer (32 filters in each layer) of kernel size 3, and fully connected layer mapping flattened final-layer feature maps to a feature vector of size 256. Edge embeddings are generated by a 3-layer MLP of 512 512 - 256 hidden units. Subset embeddings are generated by a fully connected layer of 512 units. The subset embeddings are gated with the gating variables and summed into a feature vector, which is then feed into the reasoning net, a 3-layer MLP with 256 256 - 13. The output layer contains units. The first unit gives probability of currently combined answer choice being true. The rest 12 units give meta-target prediction probabilities. This is the same as Barrett et al. (2018). The training loss function is:
L Lans -..nmmtttaarrre target
Σ
GijAk
(5)
(f.jA)CS
L1
our experiment we have tested various values X, and found 0.01 be the best. This model is trained with RAdam optimizer with learning rate of 0.0001 and batch size of 64. After 10 epochs of training, only gating variables of subsets that are rows and columns are above the 0.5 threshold. The Gating variables for three rows are 0.884, 0.812 and 0.832. The gating variables for three columns are 0.901, 0.845 and 0.854. All other gating variables are below 0.5. Among these, the one with highest absolute value is 0.41 Table 3 shows the top-16 16 ranked subsets, with each subset indexed by 2 connecting edges in the subset. Figure illustrates this way of indexing the subset. For example, the first column with red inter-connecting arrows is indexed as connecting diagram 0 and 3, and the other connecting diagram 3-6. Similarly the subset connected by blue arrows is indexed as 1-2-5. Note that 1-2-5 and 2-1 2-1-5 is different because the 1-2-5 contains edge 1-2 and 2-5 while 2-1-5 contains edges 1-2 and 1-5.
0-3-6. This indicates that there edges,
16

---

Published as a conference paper ICLR 2020
XOR(panel 1, panel 2)
Context Panels
+1
?
?
Answer Panels
*
C
D
A
B
D
*
*
E
F
G
H
E
F
G
H
Figure 7: Two examples in PGM dataset. (a) task contains 'Progression" relation of the number of objects across diagrams in columns while (b) contains a 'XO'' relation of position of objects across diagrams in rows.
MORE DETAILS ON EULER DIAGRAM SYLLOGISM
The original model in Wang et al. (2018a) uses Siamese Conv-Net model to process two input premise diagrams and output all consistent conclusions. Convolutional layers with shared weights are first applied two input diagrams. The top layer feature maps are then flattened and fed into reasoning network to make predictions. We simply use CNN grid features the top layer feature maps as object-level representations, and use the multi-layer multiplex graph to capture object relations between the two input premise diagrams. We use multiplex edge embeddings 4 layers, with each layer of dimension 32. The cross multiplexing here becomes self-multiplexing first diagram to second diagram). Final node embeddings are processed by convolutional layer to produce the final embedding, which is also fed into the reasoning network along with the conv-net embeddings.
there only 2 diagrams (Only embedding of node summary for edges from
17

---

Published as a conference paper ICLR 2020
?
?
A
B
D
A
B
C
D
E
F
G
H
E
F
G
H
(b)
Figure 8: Two examples in PGM dataset containing background line objects.
F ABLATION STUDY
We performed ablation study experiments to test how much does the multiplex edges affects performance. We have tested model variants, one without any graph modules, and the other model graphs using vanilla edge embeddings produced by MLPs, on PGM dataset. We found that without graph modules, the model only achieved 83.2% test accuracy. While this lower than MXGNet's 89.6%, is still higher than WReN's 76.9%. This is possibly because the search space reduction, by trimming away non-contributing subsets, allow the model to learn more efficiently. The graph model with vanilla edge embeddings achieves 88.3% accuracy, only slightly lower than MXGNet with multiplex edge embeddings. This shows that while general graph neural network is suitable model for capturing relations between objects, the multiplex edge embedding does more efficiently by allowing parallel relation multiplexing.
ADDITIONAL GENERALIZATION PERFORMANCE ON PGM DATASET
Table shows performance of MXGNet on other splits of PGM dataset. MXGNet consistently outperforms
WReN for test accuracy, except for H.O. Triple Pairs and H.O. shape-color in the case Additionally
18

---

Published as a conference paper ICLR 2020
0
1
2
3
3
4
5
6
7
8
Figure 9: Illustration of diagram ordering in the matrix and numbered representation of subsets.
Rank
Diagram subsets
GatingV ariable
0-3-6
0.901
0-1-2
0.884
2-5-8
0.854
1-4-7
0.845
6-7-8
0.832
3-4-5
0.812
1-2-5
0.411
2-1-5
0,384
3-6-7
0.381
10
3-7-4
0.364
11
6-3-7
0.360
12
1-5-4
0.357
13
0-4-6
0.285
14
3-4-
0.282
15
1-3-4
0.273
16
1-4-5
0.271
Table 3: All subsets ranked by the absolute value of their corresponding gating variables.
here provide the analysis according Sec 4.2 and Sec 4.6 in Barrett et al. 2018). unfortunately see 4.3 of this paper, namely the analysis of distractors, cannot be performed as the publicly available dataset does not include any ground truth labels about distractors, nor any labels of present objects that can be used to synthesize distractor labels. For Meta-target prediction, MXG-Net achieves 84.1% % accuracy. When Meta- target is correctly predicted, the model's target prediction accuracy increases 92.4%. When Meta-target is incorrectly predicted, the model only has 75.6% accuracy. For three logical relations the model performs best for OR relation (95.3%), and worst for XOR relation(92.66%, Accuracy for line-type tasks (86.5%) is only
19

---

Published as a conference paper ICLR 2020
slightly better than for shape tasks (80.1%), showing that object representation with graph modeling does improve on relations between shapes. The type of relation with worst performance is ConsistentU Union, with only 75.1% accuracy. This is expected ConsistentUUiion in fact memory task instead of relational reasoning task.
B
0
10 test %
Model
Regime
Val. %
test %
Diff.
Val. %
Diff.
H.O. Attribute Pairs
46.7
27.2
-9.55
73.4
51.7
-21.7
WReN
H.O. Triple Pairs
63.9
41.9
-22.0
74.5
56.3
-18.2
H.O. Triples
63.4
19.0
-44.4
80.0
20.1
-59.9
H.O. tye
59.5
14.4
-45.1
78.1
16.4
-61.7
H.O. shape-color
69.3
17.2
-52.1
93.6
15.5
-78.1
H.O. Attribute Pairs
68.3
33.6
-34.7
81.9
69.3
-12.6
MXGNet
H.O. Triple Pairs
67.1
43.3
-23.8
78.1
64.2
-13.9
H.O. Triples
63.7
19.9
-43.8
80.5
20.2
-60.3
H.O. line-type
60.1
16.7
-43.4
85.2
16.8
-61.5
H.O. shape-color
68.5
16.6
-51.9
89.2
15.6
-73.6
Table 4: Generalisation performance comparing MXGNet model variants against WReN. 'Diff.' is the difference between the test and the validation performances.
20


Reviews:
[{"review_id": "H1lWI44d9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors solve for the task of Raven Progressive Matrices (RPM) reasoning. They do so by considering multiplexed graph networks. They present an architecture for the same. The basic premise is a combination of object level representation that is obtained by a method similar to region proposal and combining them with graph network. The approach uses gated graph networks that also uses an aggregation function. These are combined and result in node embeddings. Detailed analysis of the network is provided. This provides improved results over earlier WREN method. However, the performance is slightly lesser than another paper simultaneously submitted that achieves similar results. That approach uses transformer network for spatial attention while here the spatial attention is just based on object level representation.\n\nOver all while the contribution is useful, not much analysis is provided on the interpretability of the results. For instance, the statistics in terms of the search space reduction as to how many subsets get pruned. Further, there may be subsets of graphs that could span across rows and columns. The decision in terms of restricting the reduction to span specific rows or columns may result in pertinent nodes also being pruned. Certain aspects that relate to object level representation are not very clear. I am not fully aware about results in this specific area and that may also be a reason for the same.\n\nTo conclude, I believe this paper provides a useful contribution by modeling the diagrammatic abstract reasoning as a graph based reasoning approach. The multiplex graph network could be a useful component that is also relevant for other problems. The paper provides sufficient analysis to convince us regarding the claims.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1lN8OSAYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes using a new version of graph networks – multiplex graph networks – which do object representation followed by some form of graph processing and reasoning to answer \"IQ test\" style diagrammatic reasoning, in particular including Raven Progressive Matrices that have been previously studied (a little).\n\nThe paper shows very strong results on multiple datasets, much stronger than previous results (from strong groups) on these datasets. On these grounds, I believe the paper should be accepted. \n\nHowever, the structure and writing of the paper was very frustrating to me. The paper just didn't make much of an attempt to explain and then motivate/analyze the model used. I mean, if I were writing the paper, I would have considered and done many things, such as:\n\n - shortening the introduction\n - shortening the related work\n - making the presentation of the datasets more succinct\n - having only one figure that covers most of what is currently in figures 1 and 2\n - putting details of what seem more ancillary details like the treatment of background lines objects in an appendix\n - remove Figure 3, which didn't convey much to me in the absence of more careful explanation of the model.\n\nso that I could motivate, carefully explain, and evaluate the main model in the paper. But here, all these things fill the main text, and we're told that we have to read the appendices to understand the model....  And the presentation in the appendix is more a dump-all-the-facts presentation than a careful development of the design.\n\nNevertheless, the general direction of the architecture seems sound, and the results look very strong, and there are even some useful ablations in the appendix.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgYu7iQqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel, feedforward, end-to-end trainable, deep, neural network for abstract diagrammatic reasoning with significant improvements over the state of the art. The proposed model architecture is reasonable and is designed to exploit the information present at multiple granularities – at the level of objects in the diagram, their relations across diagrams, and diagram subsets. As a multimodule neural pipeline, it seems a reasonable design. Further, it shows significant performance gains over the state of the art.  \n\nHowever, the writing quality is poor and is the primary reason for my giving it a low score. The paper is difficult to read and it’s hard to figure out the terminology and it’s grounding in the problem; the high-level abstract design and design choices that address the nature of the problem from the low level details, etc. \n\nThe paper uses terminology without explaining the reason for it - for example, why is the approach called ‘Multiplex Graph Networks’? What information is being multiplexed and how? Graphs are conceptual in the proposed approach – there doesn’t seem to be any graph algorithms or graph based processing. Once the module is run for search space reduction, the set of edges or relations (node pairs) become well-defined (in adjacent rows, columns) as well diagram subsets (edge pairs). The corresponding modules are just computing vectorial embeddings. Similarly, there is no reasoning that’s taking place. Reasoning requires tokens and grammar over such tokens which is not there in this case.  The proposed model is non-interpretable. \n\nThe technical writing is loose and hand-wavy. The appendix is a lot of grammatical mistakes.\n\nA few clarifications may be helpful:\n\n- “The reasoning module can also be considered as another graph processing module”?\n\n- “… we use spatial attention to iteratively attend …” – there is no iterative attention. It’s all parallel.\n\n- What do the ‘N’ nodes in each layer correspond to? There are clearly not objects or diagram primitives as they can vary in number in each diagram.\n\n- if interlayer connections are between objects in different layers (diagrams), what is this supposed to capture? Clearly, there may not be any unique correspondence between objects across diagrams.\n\n- What’s a cross-multiplexing gating function? If it’s a known concept, please provide a reference else explain.\n\nFinally, I’m open to revising my score upwards if it turns out that I’m the only one who had difficulty with the writing. The architecture design makes sense for the addressed class of problems (though the proposed network is non-interpretable and doesn’t do any reasoning nor uses graphs or graph based processing in a meaningful way), the results are good and the experimental evaluation sufficient.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper a new method of constructing graph neural networks for the task of reasoning to answer IQ style diagrammatic reasoning, in particular including Raven Progressive Matrices. The model first learns an object representation for parts of the  image and then tries to combine them together to represent relations between different objects of the image. Using this model they achieve SOTA results (ignoring a parallely submitted paper) on the PGM and Raven datasets. The improvement in SOTA is subtantial.

Most of the critique made for the paper is on writing style and presentation. The authors seem to have fixed several of these concerns in the newly uploaded version of the paper. I will further request the authors to revise the paper for readability. However, since the paper presents both an interesting modeling and improved empirical results, I recommend acceptance.
Decision: Accept (Poster)