Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper the authors solve for the task of Raven Progressive Matrices (RPM) reasoning. They do so by considering multiplexed graph networks. They present an architecture for the same. The basic premise is a combination of object level representation that is obtained by a method similar to region proposal and combining them with graph network. The approach uses gated graph networks that also uses an aggregation function. These are combined and result in node embeddings. Detailed analysis of the network is provided. This provides improved results over earlier WREN method. However, the performance is slightly lesser than another paper simultaneously submitted that achieves similar results. That approach uses transformer network for spatial attention while here the spatial attention is just based on object level representation.

Over all while the contribution is useful, not much analysis is provided on the interpretability of the results. For instance, the statistics in terms of the search space reduction as to how many subsets get pruned. Further, there may be subsets of graphs that could span across rows and columns. The decision in terms of restricting the reduction to span specific rows or columns may result in pertinent nodes also being pruned. Certain aspects that relate to object level representation are not very clear. I am not fully aware about results in this specific area and that may also be a reason for the same.

To conclude, I believe this paper provides a useful contribution by modeling the diagrammatic abstract reasoning as a graph based reasoning approach. The multiplex graph network could be a useful component that is also relevant for other problems. The paper provides sufficient analysis to convince us regarding the claims.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a novel, feedforward, end-to-end trainable, deep, neural network for abstract diagrammatic reasoning with significant improvements over the state of the art. The proposed model architecture is reasonable and is designed to exploit the information present at multiple granularities – at the level of objects in the diagram, their relations across diagrams, and diagram subsets. As a multimodule neural pipeline, it seems a reasonable design. Further, it shows significant performance gains over the state of the art.  

However, the writing quality is poor and is the primary reason for my giving it a low score. The paper is difficult to read and it’s hard to figure out the terminology and it’s grounding in the problem; the high-level abstract design and design choices that address the nature of the problem from the low level details, etc. 

The paper uses terminology without explaining the reason for it - for example, why is the approach called ‘Multiplex Graph Networks’? What information is being multiplexed and how? Graphs are conceptual in the proposed approach – there doesn’t seem to be any graph algorithms or graph based processing. Once the module is run for search space reduction, the set of edges or relations (node pairs) become well-defined (in adjacent rows, columns) as well diagram subsets (edge pairs). The corresponding modules are just computing vectorial embeddings. Similarly, there is no reasoning that’s taking place. Reasoning requires tokens and grammar over such tokens which is not there in this case.  The proposed model is non-interpretable. 

The technical writing is loose and hand-wavy. The appendix is a lot of grammatical mistakes.

A few clarifications may be helpful:

- “The reasoning module can also be considered as another graph processing module”?

- “… we use spatial attention to iteratively attend …” – there is no iterative attention. It’s all parallel.

- What do the ‘N’ nodes in each layer correspond to? There are clearly not objects or diagram primitives as they can vary in number in each diagram.

- if interlayer connections are between objects in different layers (diagrams), what is this supposed to capture? Clearly, there may not be any unique correspondence between objects across diagrams.

- What’s a cross-multiplexing gating function? If it’s a known concept, please provide a reference else explain.

Finally, I’m open to revising my score upwards if it turns out that I’m the only one who had difficulty with the writing. The architecture design makes sense for the addressed class of problems (though the proposed network is non-interpretable and doesn’t do any reasoning nor uses graphs or graph based processing in a meaningful way), the results are good and the experimental evaluation sufficient.

review_title: Official Blind Review #1

final_score: rating: null
review_content: It's perhaps useful to compare this paper with Paper 1456, which also shows results on PGM and Raven.

This paper gets slightly better numbers.

A public comment there faults the paper for not showing results on generalization settings. I think this flaw is true for this paper as well.
review_title: Another ICLR 2020 submission on PGM

final_score: rating: 6: Weak Accept
review_content: This paper proposes using a new version of graph networks – multiplex graph networks – which do object representation followed by some form of graph processing and reasoning to answer "IQ test" style diagrammatic reasoning, in particular including Raven Progressive Matrices that have been previously studied (a little).

The paper shows very strong results on multiple datasets, much stronger than previous results (from strong groups) on these datasets. On these grounds, I believe the paper should be accepted. 

However, the structure and writing of the paper was very frustrating to me. The paper just didn't make much of an attempt to explain and then motivate/analyze the model used. I mean, if I were writing the paper, I would have considered and done many things, such as:

 - shortening the introduction
 - shortening the related work
 - making the presentation of the datasets more succinct
 - having only one figure that covers most of what is currently in figures 1 and 2
 - putting details of what seem more ancillary details like the treatment of background lines objects in an appendix
 - remove Figure 3, which didn't convey much to me in the absence of more careful explanation of the model.

so that I could motivate, carefully explain, and evaluate the main model in the paper. But here, all these things fill the main text, and we're told that we have to read the appendices to understand the model....  And the presentation in the appendix is more a dump-all-the-facts presentation than a careful development of the design.

Nevertheless, the general direction of the architecture seems sound, and the results look very strong, and there are even some useful ablations in the appendix.

review_title: Official Blind Review #3

Meta-review: This paper a new method of constructing graph neural networks for the task of reasoning to answer IQ style diagrammatic reasoning, in particular including Raven Progressive Matrices. The model first learns an object representation for parts of the  image and then tries to combine them together to represent relations between different objects of the image. Using this model they achieve SOTA results (ignoring a parallely submitted paper) on the PGM and Raven datasets. The improvement in SOTA is subtantial.

Most of the critique made for the paper is on writing style and presentation. The authors seem to have fixed several of these concerns in the newly uploaded version of the paper. I will further request the authors to revise the paper for readability. However, since the paper presents both an interesting modeling and improved empirical results, I recommend acceptance.
Decision: Accept (Poster)