Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose ARII, a new framework to solve the RPM datasets. The main idea is to try to learn a reusable rule representation from an instance of the task, such that it can be applied to other tasks when applicable. They achieve this by mapping individually learned rule vectors to a discrete representation using vector quantization and then reusing those rules to infer some of the already present panels in the RPM instance. Overall, they show that ARII is able to perform better, specifically on larger grids and more difficult settings.
strengths_and_weaknesses: Strengths:
* The paper is nicely motivated in general. The design of each module is well described and easy to follow through
* Final results on both performance and interoperability are quite impressive

Weakness:
* The motivation for the design choices is not well supported in the ablation. I'm specifically not convinced about the design choices of the internal inferences (Section 3.5). Why does the process need to be generative instead of classification? For negative answer choices, you can fall back on the existing options in the instance. The correct answer is the particular image you mask out. The advantage of this is you can potentially reuse the reasoner module for this part as well. This can at least be an additional idea to try for ablations, if not the main approach.
questions: Writing suggestions:
* It would help to have some background on the details of the vector quantization steps involved in the discretization of the rule embeddings. Maybe a motivating figure to explain with an example what q_l, etc., mean can make that section more accessible.
limitations: Please add a section to describe the limitations of the work. I believe the major limitation is deciding the size of the vector quantization you do to obtain unique rules. Potentially, there can be new rules for each instance of the dataset, where your basic hypothesis breaks. Try to add some discussion along these or other relevant points.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the important issue of abstract reasoning, in particular the ability of neural networks at performing extrapolation.

The paper's main goal is summarised by the authors as to "ameliorate the reasoning generalizability by learning a robust rule representation". This needs to be reconciled with the goal of addressing extrapolation, since generalizability is something else. 

The authors claim to learn rule representations and to perform reasoning. Many ML systems learn rules. It is important to note that the rule representations are embeddings and that the reasoning is by similarity only. Looking at the paper from that perspective makes the claim made around visualization and interpretation of the rules seem rather vague. 

Having said that, the task at hand being considered in the paper is very interesting from a reasoning point of view. And the improved results obtained are valid and relevant and should not be ignored. 

A main claim of the paper is that other approaches may "require additional prior information, which is not a general way to address the reasoning problem". 
strengths_and_weaknesses: The paper is well-written and well-organized, although the captions of the figures could have been more informative, e.g. Figs 3 and 4 are difficult to grasp. 

The task at hand being considered in the paper is very interesting from a reasoning point of view. And the improved results obtained are valid and relevant.

The related work section is generally limited to work on the specific datasets, e.g. RAVEN and variations thereof.

A number of acronyms is used in the paper without stating first what they stand for, sometimes even without providing a reference, e.g. PGM. 

Robustness should also be defined (ideally). The most problematic part of the paper is the lack of a definition of "reasoning". Reasoning in neural networks is an important endeavour. It is nowadays common for neural net papers to refer to reasoning without providing any definition of what kind of reasoning or language that is being considered (and there are many). These are all formally well-defined in the area of knowledge representation in AI. The area of neurosymbolic AI has taken those definitions on board. If the field is to develop as needed in this direction then it is important to consider the work that goes on since the 1990's in knowledge representation and neurosymbolic AI. I'd refer the authors at least to:

https://link.springer.com/book/10.1007/978-3-540-73246-4
https://ebooks.iospress.nl/volume/neuro-symbolic-artificial-intelligence-the-state-of-the-art

And specifically on adding knowledge to neural nets:
https://www.ijcai.org/proceedings/2017/221
https://www.researchgate.net/publication/283897473_Semantic-based_regularization_for_learning_and_inference
https://arxiv.org/abs/1605.06523
  
And for a recent survey:
https://arxiv.org/abs/2012.05876 
questions: It is not easy to see the relationship between Fig 1 and the rule encoder, the reasoner, and the internal inferrer. Is this done on purpose?

A main claim of the paper is that other approaches may "require additional prior information, which is not a general way to address the reasoning problem". Other approaches may provide a general mechanism for reasoning. When it comes to a fair comparison, how can you be sure that the "internal inference" process isn't a way of providing additional information?

The "internal inference" process seems to function in such a way as to turn propositional rules into rules with variables. Is this a fair description of the purpose of the proposed process? You state that "we randomly mask one panel of the first two rows and ask the referrer to infer the masked panel based on the rule representation. We perform the internal inference for multiple times to make the rule representation invariant to specific instances." To answer my question above, this process needs to be specified well and analysed from the perspective of reasoning capabilities.

How do we know that the results derive from better reasoning rather than e.g. the combinatorial benefit of using r1&2 with r1&3 and r2&3? Did the ablation study conclude anything about this? How do we check whether this internal inference is indeed producing the expected "more abstract" rules (with variables)?

When one says that one evaluates "reasoning ability", one would wish to evaluate what kinds of reasoning take place, not simply evalute improvements in test set accuracy. Have you evaluated what happens in a situation where the reasoning required to obtain the correct answer is generally seen as harder than the normal type of reasoning?
limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel framework Abstract Reasoning via Internal Inferences (ARII) for the task of solving abstract visual reasoning problems. In particular, the ARII model is designed to solve the task of selecting the correct candidate panel for a Raven's Progressive Matrices (RPM) reasoning problem. The proposed method consists of three components:  

1. A rule encoder network that takes the first two context rows of the RPM matrix, and generates vector quantized codes representing the abstract rule encoded in the images. The encoder works by generating CNN embeddings for each panel, which are then split into groups and concatenated based on their position. These image encodings are once again split into positions for each of the six image panels and then re-combined together to generate a continuous rule representation. The continuous rule representation is then discretized into the Ke X D dimensional code book, which is trained together with the encoder.
2. A reasoner which takes each candidate panel as a potential rule with the context panels in the third row, calculates the rule encoding of this candidate rule with rows 1 and 2, and then selects the candidate rule (panel) which has the highest similarity with the rule encoded by rows 1 and 2.  
3. A generative internal inferrer, that masks out each image sequentially out of the six context images in the first two rows, and aims to generate the original rule encoding with one context image missing by minimizing the distance between the original rule and the rules extracted with a panel missing.  

ARII outperforms existing methods when it comes to generalizing to novel test domains on the i-RAVEN and PGM datasets. Ablation studies performed by the authors demonstrate the utility of both the discretized code book as well as the internal inference mechanism. 

strengths_and_weaknesses: Strength:

1. The paper is extremely well written: the RPM problem is presented succinctly, the ARII model is explained clearly and each component is motivated well, and the training procedure, results and ablation studies are easy to follow.

2. The idea of utilizing a learnable vector quantized codebook for representing abstract rules in PGM problems is quiet novel, and works well in practice as seen from the generalization results of ARII.  

3. Under the scenario where auxiliary information such as symbolic rule embeddings are not used for training, it demonstrates state-of-the-art performance on four generalization splits of the i-RAVEN dataset, and five generalization splits of the PGM dataset.

Weakness:

1. In equation 1, the authors make an implicit assumption that the RPM rule is encoded row-wise in the image panels. While this holds true for the i-RAVEN dataset, it is not necessarily true for PGM. The authors work around this limitation by adding the transposed PGM matrices for training. Several other models compared in the paper do not explicitly rely on this assumption, and hence it would only be fair to compare these models with this extra training step included.   

2. In the ablation study, the authors show that the codes E-358 and E-181 occur in the top three codes for each possible relation in the visual domain of [., line, color] rules. This appears counter-productive to the argument that the ARII model has learned instance-invariant representations, since if it were indeed instance-invariant then the visual domain should have little to no impact on the rule representation. A better way to demonstrate this would have been to take the codes for [AND, ., .] across multiple visual domains and show that the codes are consistent with various object and attribute values.
questions: Could the authors further validate their claims that the codes learned by ARII are instance-invariant? In abstract reasoning terms this would involve that the codes learned for one particular rule (such as AND) will be similar irrespective of the visual domain. The ablation studies done by the authors do not demonstrate this. Please see the limitations highlighted in the review for one possible suggestion to demonstrate this.

**Post-Rebuttal Edit:** See response to author's rebuttal below. Score increased to a 7 post-rebuttal.
limitations: The authors did a good job in explaining several assumptions in their modeling process, such as space of rules is not large. Several of the ideas utilized in the ARII model build upon previous research (such as splitting and grouping panel embeddings like the Scattering Composition Learner, row wise embeddings relying on feature locations like Multiplex Graph Network, etc.) While many of these papers are referenced, the authors fail to draw connections and provide reasonable credit to the authors of these works in their Related Work section. It would be great if the authors can highlight these connections, as it will help future research in abstract visual reasoning and would be only fair to the authors of previous works.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel method for visual abstract reasoning tasks, focused on the RPM-like benchmarks PGM and RAVEN. Some of the key features of the new method are the use of a learnable discrete representation to encode abstract rules, and a self-supervised objective that requires the model to use the encoded rule to fill in masked panels. The model achieves competitive results on both benchmarks, and shows particularly strong performance in the OOD regimes of the PGM dataset. There are also some interesting analyses of the rule representations learned by the model.
strengths_and_weaknesses: ## Strengths
- The model achieves excellent results, surpassing or competitive with the state-of-the-art on both benchmarks, and with particularly strong performance on the OOD regimes of PGM, which are arguably the most important part of these benchmarks.
- The evaluation is very thorough, evaluating the model on all formats and generalization regimes for the two major benchmarks in this area.
- The inductive biases introduced in this method are quite interesting. Using discrete encodings of the rules likely forces them to be more abstract, which presumably contributes to the OOD performance. The self-supervised objective ('internal inference') is an interesting way to capture the kind of 'hypothesis testing' that characterizes the way human reasoners solve these problems (internally generating proposals for the abstract rules, and then checking to see if they explain the presented panels).
- The ablations nicely demonstrate the contributions of these two components.
- The authors have included some code in the supplemental material, and plan to release all code upon acceptance.

## Weaknesses
- The primary weakness is that there are a number of important details missing that need to be included, perhaps in an appendix, e.g. training details (number of epochs, batch size, learning rate etc), model hyperparameters, and the computing machinery that was used to train the models. These should be included in the final version of the paper.

I also have a few other minor notes:
- I like the rule interpretability analysis, but it would be more informative if it were more comprehensive, rather than only focusing on rules applied to line color.
- It would be informative to also test the proposed method on this recently released variation on RAVEN: https://arxiv.org/abs/2206.14187
- It would be good to explicitly mention the connection between the 'internal inference' objective and self-supervised learning.
- The object encoding method employed here is very inflexible, relying on the rigid spatial structure of these problems. I think it's ok in the context of this work, but it would be good to mention this as a limitation and a potential avenue for future work.
questions: I don't have any questions, other than regarding some of the missing details about hyperparameters and training.
limitations: I don't envision any potential negative societal impact from this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: A model called ARII is proposed in this work to solve abstract reasoning problems. The ARII model is composed of a rule encoder, a reasoner, and an internal inferrer. The rule encoder adopts a vector quantization method and turns continuous rule representation from the first two roles into a discrete token in the codebook. The reasoner uses a similarity-based measure to select the answer: the candidate which, when filled in, generates rule features most similar to the first two rows will be treated as the answer. In addition, the internal inferrer reconstructs a masked panel like in BERT. In experiments, the authors conduct evaluation on I-RAVEN and PGM. It's found that the model achieves best performance on I-RAVEN and improves on the generalization regimes on PGM.
strengths_and_weaknesses: The paper is well written with clear explanation and a streamlined flow. 

There are a few things that are done right which I believe lead to the performance boost in this work. (1) image features are extracted from a group level: considering the grid-like layout in I-RAVEN and PGM, the position-based grouping could help better capture the statistics in the images. (2) Quantization in rules: as there are only a finite number of rules, quantization seems a reasonable choice, which is unfortunately missing in existing works. (3) BERT-like masked reconstruction. This might be the part that contributes to the generalization improvement as hinted in existing language modeling works.

I'm wondering, as the masked reconstruction part can be plugged and played, whether performance in generalization could be consistently improved in different models. It would be nice to see some results regarding this point. 

Also I wonder how reasoning is conducted. It occurs to me that the work still applies statistical learning but lacking in explaining how a model performs induction and reasoning. Some statistics may be captured using the method, but as shown in generalization results (which are indeed improved), it is still far below fair expectation. I would be interested in knowing how the authors view the perspective: does reasoning emerge from statistical learning and if not what is your opinion.

Finally, I notice a few neuro-symbolic methods [1, 2] are missing, which when added should better complete the full picture of the area. 

[1] Zhang, Chi, et al. "Abstract spatial-temporal reasoning via probabilistic abduction and execution." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.  
[2] Zhang, Chi, et al. "Learning algebraic representation for systematic generalization in abstract reasoning." arXiv preprint arXiv:2111.12990 (2021).
questions: 1. It would be nice to see results of existing models trained with the masked modeling loss. 
2. How the reasoning capability emerges.
limitations: No. Limitations are not adequately addressed, though there is not ethical issues.
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. The paper presents a method for rule representation learning that can be transferred accross tasks. All reviewers unanimously agree that this paper's strengths outweigh its weaknesses. In particular, reviewers found the method to be well motivated [fEdf], general [fEdf], novel [ky2S], tackling an interesting task [bvvT], achieving strong empirical results [4Dou] and the paper to be well written [bvvT,ky2S,xDma]. Thus, I am recommending acceptance of the paper and encourage the authors to further improve their paper based on the reviewer feedback.
confidence: Certain
Decision: Accept