Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This work proposes a new architecture for abstract visual reasoning, based on Transformer-style soft attention and relation networks. The authors test their network on the PGM dataset, and demonstrate a non-trivial improvement over previously reported baselines. 

In general, abstract reasoning is an important field of current study in neural network-based machine learning, as it is an area that has notoriously eluded these types of models historically. The paper is reasonably well put together, and I have no reason to question the various technical aspects of the work.

Unfortunately, I think there are significant shortcomings. Firstly, the PGM dataset was designed to stress out-of-distribution generalization, and performance on the Neutral split was not proposed as a particularly interesting challenge on its own. This is because, as the name implies, abstract reasoning requires the ability to identify abstract conceptual features of the data and compose them in novel ways at test time, which is *not* a feature of the neutral split.  The authors are encouraged to run their model on these other generalization splits. 

Second, there seems to be little value to the field overall for research involving minor architectural improvements for single datasets. If the authors believe in this method, they are encouraged to demonstrate its effectiveness on a wide variety of data types. On this note, I should add that the authors are incorrect to state that this is the first work to use self-attention for abstract reasoning (please see Zambaldi, 2018 for one example of many papers that have incorporated self-attention into convolutional architectures). 

So to sum up, while this work broaches an interesting subject and is technically fine, it does not surpass the threshold for acceptance because it fails to demonstrate the usefulness of the method on the task at hand, as well as broad utility of the proposed method.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper describes a somewhat novel approach to abstract visual reasoning using transformers in the so-called "Attention Relation Network" (ARNe), which the authors show to improve on the "Wild Relation Network" (WReN). The Transformer is motivated by the role that attention may play in Human information processing - which sounds plausible, but the paper does not expand on this theme.

The paper is well written and makes an interesting contribution, but I feel the results are not quite yet ready for publication. The authors are writing that they are still working on baseline results on the full dataset, which would provide interesting comparisons, and some details on the implementation (number of parameters, etc) are missing - or maybe I missed them.

The learning curve in Figure 3 (sample efficiency, test accuracy) suggests that the ARNe training is not fully stable - why would the model deteriorate when going from ~40% of the training data to ~60%? Is the model potentially overfitting, and how does the size of the proposed model compare to the size of the baseline model(s)? It seems that the field is also moving towards the RAVEN dataset, which presents a more complex structure; it would be more convincing to present results on both datasets, to show that attention can indeed also improve results on more complex setups.

The text in the "Acknowledgments" section should be removed for the camera ready version!

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work introduced an attention-based model to solve the RPM cognitive tasks. The model is based on the transformer network, which performs relational reasoning through its self-attention mechanisms.

Technical novelty:
The method seems to be a straightforward application of the transformer network to the PGM task. The technical novelty of the proposed approach is unclear. I’d love to hear what the authors have to say about the technical contributions of the proposed ARNe model in comparison to prior work.

Supervision with meta-targets:
It also seems that the meta-targets are crucial for attaining a good performance with the ARNe model. According to Table 4, the model without meta-target training (beta=0) only achieved 12% accuracy in train/val/test sets. However, prior work [Santoro* et al. 2018] has demonstrated that even without training on meta-targets, WReN still achieves a performance of over 60% accuracy (Table 1). This result suggests that the proposed ARNe model does not work well when training with weaker supervision without meta-targets. The results could be a lot stronger if the authors show ARNe outperforms the prior work when beta is set to 0.

Ablation studies:
This model is only tested in the neutral PGM dataset. The evaluation would be strengthed with the generalization results of this model in different generalization regimes (see Table 1, Santoro* et al. 2018) and comparing its performance with prior works.
review_title: Official Blind Review #2

Meta-review: This work proposes a new architecture for abstract visual reasoning called "Attention Relation Network" (ARNe), based on Transformer-style soft attention and relation networks, which the authors show to improve on the "Wild Relation Network" (WReN). The authors test their network on the PGM dataset, and demonstrate a non-trivial improvement over previously reported baselines. 

The paper is well written and makes an interesting contribution, but the reviewers expressed some criticisms, including technical novelty, unfinished experiments (and lack of experimental details), and somewhat weak experimental results, which suggest that the proposed ARNe model does not work well when training with weaker supervision without meta-targets. Even though the authors addressed some concerns in their revised version (namely, they added new experiments in the extrapolation split of PGM and experiments on the new RAVEN dataset), I feel the paper is not yet ready for publication at ICLR. 

Decision: Reject