Learning Robust Rule Representations for Abstract Reasoning via Internal Inferences
Authors: Wenbo Zhang
Likai Tang
Site Mo
Xianggen Liu
Sen Song
Venue: NeurIPS 2022 Accept (2022)
Keywords: visual reasoning
abstract reasoning

Abstract: Abstract reasoning, as one of the hallmarks of human intelligence, involves collecting information, identifying abstract rules, and applying the rules to solve new problems. Although neural networks have achieved human-level performances in several tasks, the abstract reasoning techniques still far lag behind due to the complexity of learning and applying the logic rules, especially in an unsupervised manner. In this work, we propose a novel framework, ARII, that learns rule representations for Abstract Reasoning via Internal Inferences. The key idea is to repeatedly apply a rule to different instances in hope of having a comprehensive understanding (i.e., representations) of the rule. Specifically, ARII consists of a rule encoder, a reasoner, and an internal referrer. Based on the representations produced by the rule encoder, the reasoner draws the conclusion while the referrer performs internal inferences to regularize rule representations to be robust and generalizable. We evaluate ARII on two benchmark datasets, including PGM and I-RAVEN. We observe that ARII achieves new state-of-the-art records on the majority of the reasoning tasks, including most of the generalization tests in PGM. Our codes are available at https://github.com/Zhangwenbo0324/ARII.

Learning Robust Rule Representations for Abstract Reasoning via Internal Inferences
Wenbo Zhang .2,3
Likai Tang22 2.3 Site Mo*
Sen Song
Xianggen Liu
College of Computer Science, Sichuan University
2 Department of Biomedical Engineering. Tsinghua University
3 LLaboratory of Brain and Intelligence, Tsinghua University
*College of Electrical Engineering, Sichuan University
emailss. (zhangwb19, tanglk20) @mails tsingguu....
eew.gghuaaeuu..... edu. en
{mosite, xianggenliu] @scu edu. cn, songson@tsinghua. edu. cn
Abstract
Abstract reasoning, as one of the hallmarks of human intelligence, involves col- lecting information, identifying abstract rules, and applying the rules to solve new problems. Although neural networks have achieved human-level performances several tasks, the abstract reasoning techniques still far lag behind due the complexity of learning and applying the logic rules, especially in unsupervised manner. In this work, we propose novel framework, ARII, that learns rule rep- resentations for Abstraci Reasoning via Internal Inferences. The key idea is to repeatedly apply rule to different instances in hope of having comprehensive understanding (i.e., representations) of the rule. Specifically, ARII consists of rule encoder, a reasoner, and internal referrer. Based on the representations produced by the rule encoder, the reasoner draws the conclusion while the referrer performs internal inferences to regularize rule representations to robust and generalizable. We evaluate ARII on two benchmark datasets, including PGM and I-RAVEN, We observe that ARII achieves new state-of-thi of-tthe-attt records on the majority of the reasoning tasks, including most of the generalization tests in PGM. Our codes available at https::/ :/gt thub. com/ 2hangggeebo033///RRI1
Introduction
Abstract reasoning, the ability to extract patterns and rules from concrete instances and apply them to solve new problems, is one of the hallmarks of human intelligence. It is a critical topic to endow neural networks with the capacity of abstract reasoning on the road from artificial intelligence (AI) to human-like intelligence. As "thinking in pictures" is one of the most effortless and natural ways for humans to perform inference, Raven's ' Progressive Matrices (RPM) test I is widely accepted task to evaluate Al's reasoning ability. As shown in Fig [[ an RPM problem contains incomplete 3 x matrix where the bottom-right entry is missing. The subjects are aware that the rows in the matrix implicitly share identical rules and are asked pick the suitable answer from the choices that would best complete the missing entry. Considering that the RPM test shown to be highly correlated with real intelligence 2 in cognitive and psychological science, Al community recently presents huge 7
interest in this task to explore the model's capacity of abstract reasoning 3
In the early years, computational models for RPM usually assume access the symbolic representa- tions of images and thus solve RPM with heuristics. Recently, various neural networks are proposed
Xianggen Liu is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Color Progresson
Postion XOR
Rule
Luducing rules
Position XOR Color Progression
A Applynng ing rules
Context
?
Answer set
R Reasoning
Internal inference the context
Figure 1: An example of RPM problem and the general solution of ARII. ARII repeatedly applies the rule internal inference in hope of deep "understanding" of the rule.
to accomplish abstract reasoning with the help of abundant reasoning data (such Procedurally Gen- crated Matrices dataset (PGM) [5]. Relational and Analogical Visual rEasoNing dataset (RAVEN) [ and SapartiialRAVEL dataset (I-RAVEN) ( These models mainly treat the RPM problem classification task based on the rules hidden in rows or columns. For example, Contrastive Perceptual Inference Network (CoPINet) 9 utilizes contrast effect improve the feature extraction capacity (7]. Scattering Composition Learner (SCL) designs compositions of neural networks according to the specific characteristics in RPM (e.g., the number of the attributes) 10]
Although existing models have achieved impressive performance on abstract reasoning, the critical capacity, i.c., generalization to different environments remains unsolved [5]
[4] In
particular, Matkiński [15) reports that, the performances of the current models would significantly decrease even if the test images changes slightly [5 [ [ For example, one of the generalization regimes (i.e., an evaluation task) in the PGM dataset is extrapolation, where the model is trained on the images with color size restricted to the lower half of the value set and tested images taking these values from the upper half. Multi-Layer Relation Network (MLRN) [3] achieves nearly perfect accuracy in the normal test of the PGM dataset, but fails completely in the extrapolation regime, yielding only slightly above chance. This reveals that these models are just overfitted to the specific data rather than mastering the reasoning essence, also reflected by low generalization to distracting features [16]. This paper provides novel framework that ameliorates the reasoning generalizability by learning robust rule representations. To achieve it, we make two assumptions of the rule representation for abstract reasoning. On the one hand, we assume the space of the abstract rules is finite and usually not large. Thus, we propose rule encoder that produces discrete rule representations by vector quantisation [17. On the other hand, we assume the rule representations should be invariant to specific instances or distracting features. In human reasoning, people usually apply an abstract rule to different instances several times to deepen the understanding of the rule. We mimic this cognitive process by introducing an internal inference process based on the learned rule representations. Specifically, we randomly mask one panel of the first two rows and ask the referrer to infer the masked panel based on the rule representation. We perform the internal inference multiple times to make the rule representation invariant to specific instances. Based on these instance-invariant representations, conclusions can be made according to the specific context images. Therefore, we call our model ARII, the abbreviation of "abstract reasoning via internal inferences."
We first conduct comprehensive empirical experiments on the I-RAVEN 8 datasets. Compared with the current state-of-the art models (e.g.. SCL [10] and CoPINet 9 experiments show that ARII achieves new records on four of the seven single training tasks in the I-RAVEN datasets. As the generalization tests can better reflect the real reasoming ability of model, we also evaluate ARII on seven generalization regimes the PGM dataset and observe that ARII outperforms the other competing methods in five regimes. To summarize, our contributions include:
We propose a novel framework that learns rule representations for abstract reasoning by internal inferences. In particular, we introduce a rule encoder to produce discrete rule representations by vector quantisation. We design an internal inference process to learn an instance-invarin and robust rule representation.

---

Our model achieves superior results to state-of of- oftt--o-thheart the-art methods on the majority of tasks (four out of seven) on I-RAVEN and better generalization performance on most PGM regimes.
Visualization results on the rule representations indicate that the learned representations can be automatically clustered with respect to the rule categories. Deep analysis shows that we can identify some features in the discrete representations for a specific reasoning rule, validating that the representations are interpretable and instance-ivariiat.
2
Related Work
RPM problems are firstly investigated in the cognitive science community to better understand intelligence, and many computational models are proposed to address this test since 1990s. For example, Carpenter et al. [2] develop production system that takes hand-coded textual descriptions of problems as input and predicts the answer. Overall, these cognitive models mainly solve specific problem sets with only limited number of hand-craft instances in cognitive science [9 20 21].
Recently, RPM problems have gained a lot of attention in the Al field to explore abstract reasoning capability. The PGM [5] together with the RAVEN dataset 77 expands the size of RPM instances through automatic generation algorithms, serving as the benchmark datasets for deep learning network to study abstract reasoning. However, Hu et al. S found severe defects (a short cut for predictions) existing in the RAVEN dataset and created an unbiased version called I-RAVEN dataset to solve Therefore, we evaluate our method on the PGM and I-RAVEN rather than RAVEN.
Based on the above datasets, various end-to-end approaches have been proposed to study the abstract reasoning ability. The typical architectures in computer vision, that simply use CNN as visual feature extractor followed by MLP 3 or LSTM [
process the features from all the image panels, are shown to lack the reasoning ability and struggle in the RPM test. To improve the reasoning capacity, Relation Network (23) equipped and extended in many models, including Wild Relation Network [3] and Multi-scale Relation Network (24).
22
Although the above models obtain elegant performance RAVEN and I-RAVEN, they still fail to achieve satisfactory generalization results on PGM, reflecting the overfitting issue of these reasoning models. Auxiliary labels 23 [T] [6] are therefore utilized develop representations that amenable to generalization by informing the meta-targets (i.c., labels which encode relevant relations and attributes). However, requires additional prior information, which is not a general way to address the reasoning problem.
Our work is closely related to several advanced abstract reasoners, such the SCL [00 and MXGNet [II] multi-layer multiplex graph neural net. In particular, ARII uses the splitting and grouping embeddings and row-wise embeddings in SCL and MXGNet, respectively. Our work is
also connected with neuro-symbolic methods for logical reasoning (26 27 28
29
30
31
32 For
instance, Logic Tensor Networks integrates neural networks with first-order fuzzy logic to perform reasoning with logical formulas. Different from these studies, ARII explicitly maintains a discrete rule representation for abstract reasoning and introduces a novel internal inference process to enforce the rule representation to be robust. The internal inference can be viewed as one type of self-supervieed learning algorithm for the rule representation learning, where we mask parts of the context matrix and require the model to reconstruct it. But the traditional self superviee learning models are usually trained first and finetuned for the downstream tasks afterwards 33 34 35). Different from them, we explore an interesting strategy that performs the internal inference and the supervised learning simultaneously.
3 Methodology
3.1 Problem formulation
The RPM test is conducted on a 3x3 matrix where the rows comply with an identical rule, Subjects presented with an incomplete matrix where entry on the third row is missing, and are asked to select a candidate answer from the choice set to complete the matrix.
Standard Progressive Matrices [8 contains 60 problems
3

---

Formally speaking, a RPM problem contains 16 image panels X = { )== divided into 8 context images X. {2:);-) and candidate answer images X, = { The 8 context images are placed in 3 x 3 matrix with the last element blank. For each particular data point, the three rows in the matrix implicitly share the identical abstract rule. Given the X, the machine asked to select candidate answer image from X. to best complete the matrix (i.e. adhering to the underlying abstract rule context images). The correct answer is denoted by x", where x E Xa.
3.2
The ARII model
ARII consists of three modules: the rule encoder, the reasoner, and the internal inferrer. The rule encoder induces rule representations from the context matrix. The inferrer performs internal inferences to regularize the rule representations to be robust and instance invariant. The reasoner draws the reasoning conclusion based on the rule representations. The rest of this section elaborates on these components and the training process.
3.3
Rule encoder
To tackle this reasoning problem, an ideal reasoning process is to figure out the underlying rule based on the given context images. To achieve this, we aim to learn the rule representations using an effective encoder. In principle, the rule can be extracted from particular row of images. But the above extraction has the potential to make mistakes since single row images may satisfy multiple rules. Therefore, to guarantee the adequate information for rule extraction, propose encode the first two rows of images to represent the underlying rule, given by
T1&2 (ul;)nn(((x;)).1))
(I)
where F1g: stands for the representation of the rule underlying the first two rows the given matrix. Rule-Enc stands for the rule encoder in the ARII model. In the rest of this section, first obtain the image encodings and then build rule representations based on the encodings of the involved images.
Image encodings. As each row contains three separated images, we adopt convolutional neural network (CNN) blocks to get the feature map (f) for each image separately.
f. = CNN(z,)) i= 1,.,...66.
(2)
Once we get the feature map. we further fuse the features of different positions in the feature map. Intuitively, the rule of RPM is highly correlated with visuospatial ability [36] and usually lies in the same position of different panels. Therefore, we first divide the feature map into multiple groups; each group can be considered to contain information from a certain position. Then we apply a fully connected layer (FC) to each group to further extract group features, and concatenate the group features into output features, computed by
ImageFinc(z:) = ll (FC(F)), i 1-
where || stands for the concatenation operation of the vectors. f is k-th group features after splitting original fi into K₂ groups. Similar to the scattering transformation in SCL (10), the group features } are divided evenly from the feature map f.. Finally, the K₂ new groups of the features are concatenated (denoted by l| ) obtain the i-th image encodings 2,.
Continuous rule representations. Next, the rule encoder fuses the image encodings and extracts the rule representations. Similar to the processing of images, extract the relation corresponding the different positions of the six image encodings as Fig 2 Specifically, the encodings of six images are first divided into K, groups independently. The groups the same location in six image encodings are combined together and fed into a fully connected layer to further extract the relation feature. Finally, the multiple relation features merged to form rule features follows:
(4) where z! is the k-th group of the i-th image encodings 2;- | 2 means six le-th group features from encodings are concatenated, and fed into a fully connected layer (FC) to get the k-th relation features. We further merge the K, relation features and obtain the continuous rule representations re-
T- K [FC(I[=_=*))
Discrete rule representations. Note that the above rule representations ra are continuous and dependent on the specific instances. In this work, we assume the space of the abstract rules is finite

---

Mask
Flatten
Image Encoder
Inferrer
Concat
Internal inference
Ez
Image Encoder
FC
X4
X
Ege
Rule encoder
Tc
E
q
1182
Rule Encoder
Sime, :)
1/2
T182
Rule Encoder
Similarity Score
220.3
X7
Rule Encoder
Sim(; :)
1/2
K10
K11
Sg
S10
S11
S12
Prediction
113
F14
K15
X16
Reasoner
$13 515
S16
Figure 2: The architecture of ARII ARII comprises three modules, namely, the rule encoder, the reasoner and the internal inferrer.
and usually not large. Based on this assumption, the discrete and finite representations are more suitable to describe abstract rules.
Therefore, propose to discretize the latent representations of the rules via vector quantisation (VQ) [7]. The general idea of VQ is to use a finite number of vectors (denoted as codebook) to build the rule space. It outputs the closest vector from the codebook for the given input (Appendix Fig 1). In particular, the rule encoder maintains a discrete codebook F. E K, xD Ke is the number of the discrete code, and D is the dimensionality of each discrete code Fgg. We measure the 1.2 distance between the continuous representation Te and code vectors in the codebook. For each vector r in Te x where K, is the number of the continuous vector and the dimensionality each continuous vector r the code vector that yields the minimum distance is taken to obtain the discrete rule representation qi. The detailed computations are
qi Ex, where k ===
arg min jE(1....K.)
||r!
F2l12
where qi is the closest quantized vector for the continuous vector r!. Via vector quantisation, we can find the closest code for the given continuous rule representations, i.e.,
T1&2 (al:))nc(((;){))
[q))
To make the above vector quantisation work, we need to train both the neural networks and the codebook the rule encoder towards the minimum of the distance between the continuous and discrete representations. The objective function of vector quantisation for particular data point is
(X) ||sg(r.) - ri4-2113 2 ||r. sg(r1
where sg(.) is gradient operation during computation. In this way, we can derive the discrete rule representations, which is expected be interpretable and instance-invariant.
3.4 Reasoner
The reasoner is to draw conclusions (i.e., the selection of answer images) based on the extracted rule and the specific context (i.e., the images on the third row). That is, the reasoner applies the extracted rule the third and figures out which candidate answer could best complete the missing entry.
According to this standard, we therefore extract the rule that the third row implicates. Since we have 8 candidate answer images (i.e., X.). there are 8 rules for the third row with the different answer images in the missing entry. However, the rule encoder is designed take two rows of images as inputs and yield the rule representations. Therefore, combine the third row with the first row to

---

derive the rule that these two rows implicitly share. computed by
1&33 = RuleEnc(faa = 1, 2, 3, 7,8,))),
. 16
(8)
where ri&g3 denotes the rule representations of the rule underlying the first and the third row when the third row is completed with the j-th answer image. Similarly, we can obtain the rule representations of the rule underlying the second and third row, given by
T2& RuleEnc( {z,li 5, 5,,,7.
j= ......... 16
Based on the derived rule representations for the third row, the reasoner next measures the similarity between these rule representations and the reference rule (i.e., T1&.2) and selects the most similar one as the prediction. In particular, the reasoner measures the similarity of two rules by calculating the inner product of the corresponding representations. For example, the similarity between the rule r1&3 and the reference rule is calculated by
sim(ri F1822)
== 99...... 16,
(10)
Σj=g e" T1a2"
where sim(-) is the similarity function of two rule representations. Similarly, the reasoner derives the similarity between the rule 12x33 and the reference rule by
22 T1&2
sim s 2233 P182)
j 9. ..... 16.
(11)
Σj=g "ass Tike
Based on the similarities between individual rules, the reasoner then predicts the candidate answer
arg max (E......116)
'sim (sim(ri T182) + sim(r T11=2]]
(12)
For the optimization of the reasoner, adopt the cross entropy loss to guide the reasoner to select the correct answer, given by
1 log sim(rimss F182) + log sim T182)]
(") = 2
(13)
3.5 Internal inferences
Human usually applies an abstract rule to different instances for several times deepen the under- standing of the rule. We suppose that the underlying rules independent of specific instances and can reused in different problems that share the same rules. Therefore we mimic this cognitive process by performing internal inferences based on the rule representations, In the internal inference, we introduce another reasoner to address the reasoning task on the two rows of images when the rule representations available. In this way, method ARII could also better "understand" the rule and have more robust rule representations. robust rule representation insensitive to the sample noises and thus benefits the overall reasoning ability (37)
In particular, the first and second rows are used in internal inference process. We first extract the rule representations T1&2 of these rows and then randomly mask one of the six images {a by a white blank image. We coin the blank image 2... where is the index of the masked images. Thus, the masked two rows can be expressed by
(14) where (1,2,3,4,) 5, 6 stands for a set excluding the index m and we insert blank image &m in the original m-th position. Next, introduce another neural network named inferrer that takes the rule representations and the encodings of the images in the masked two rows as inputs.
Im {zgli E (1,2,3, 4,5, 6) U {2m}
(15) where Inferrer is another CNN model. å., stands for the predicted image for the incomplete rows, and ImageEnc(Tn)) derives the encoding of I., by Equation 2 and 3 To avoid using additional data in our internal inference, we formulate the above prediction as to the generation process. That is to say, the inferrer aims generate the masked image Xmm the pixel level.
i Inferrer Inferrer(ImageEnng (Imm))Enc(Imm)) F1&:2),
In addition, we repeat the above internal inferences for times by masking each image to learn more robust rule representations. We adopt mean squared error (MSE) as the objective function to optimize

---

Table 1: Test accuracy of individual models on I-RAVEN.
Test Accuracy (8)
Model
Average
Center
2x2 Grid
3x3 Grid
L-R
U-D
O-IG
12.5
12.3
12.8
10.3
WReN
17.8
23.3
18.1
17.4
16.5
15.2
16.8
17.3
CNN+MLP
5
12.9
12.9
13.2
12.7
11.5
13.5
12.9
13.7
Resnet-18
14.5
20.8
12.9
14.3
13.4
13.8
12.9
LEN 16
28.4
42.5
19,9
27.6
28.1
32.9
27.0
CoPINel 9
38.6
50.4
30.9
28.5
40.0
40.8
42.7
36.9
SCL. 10
85.4
99.8
72.4
64.2
99.5
99.4
98.6
64.2
98.9
78.9
85.1
the inferrer as well as the rule encoder. In this way, we obtain the overall objective function J by collecting the predictions of these internal inference processes, given by
6 = Σ m=1
(xx)
am 2m
(16)
3.6 Optimization of ARII
As introduced in the section on problem formulation, the major goal of method ARII is to select candidate answer image for the incomplete matrix as accurately as possible (as shown in Equation 13). In the meanwhile, we assume the rule representations are discrete and can reused in internal inferences. That is, apart from the major goal, we have two additional objectives, i.e. the objective of the vector quantisation process and the objective the internal inferences. Therefore, combine these three objectives together for the optimization of the individual components of ARII, given by
TX) + >irTT"
(17)
XED
where the context images X. is parts of the given data point X and D is the dataset. Ares Aug and Xif are the three aagee-errmeeerss that coordinates the importance of the components ARII.
Experiment
4.1 Datasets
PGM. Each matrix in PGM is governed by abstract rules, which are sampled from rule set o O. a E A}, where S, O, are primitive sets of relations, objects and attributes, respectively. S = {progression, XOR, OR, AND, consistent union}. O = (shape, line). A (size, type, color, position, number). There are eight regimes PGM. The easiest one called neutral where the training and test set are sampled from the same distribution, and the others are generalization regimes where the training and test data differ in the pre-defined ways. The generalization regimes consists of seven variants: interpolation, extrapolation, held-out triples HH.O.Triples) held-out attribute pairs (H.O.A.P.). held-out pairs of triples (H.O.T.P.), held-out shape- colour (H.O.S-C) and held-out line-type (H.O.L-T). More details can be found in Barrett et al. [5).
R {[s, : S,
RAVEN and I-RAVEN. The RAVEN and I-RAVEN datasets are extensions to PGM. They share an identical set of attributes but differ in the following attributes: progression, constant and arithmetic. I-RAVEN is adapted from RAVEN dataset for its biased answer sets which may lead to shortcut solution that only candidate answers can yield accurate answers [ In our experiments, we first evaluate the models' reasoning ability on I-RAVEN since it is easier than PGM.
4.2 Competing methods and implementation details
compare our model with several sate-of-the-ar models, including CNN + LSTM 5), ResNet-
based 388 image classifier, Wild ResNet [5], Wild Relation Network (WReN) (5]
Contrastive
Perceptual Inference network (CoPINet) (9). Logic Embedding Network (LEN) 16
Scattering
Compositional Learner (SCL) [10], Stratified Rule-Aware Network (SRAN) 8 Multi-scale Relation
7

---

Table Test accuracy different models on PGM
'Tss Accuracy (%)
Model
Neutral
Interpolation
H.O.A.P.
H.O.T.P.
H.O.Ts
H.O.L.T H.O.CCC Extrapolation
CNN MLP
33.0
CNN LSTM
35.8
ResNet-50 5
42.0
Wild-ResNet
CoPiNet 9
56.4
WReN /]
3
62.6
64.4
27.2
419
19.0
14.4
12.5
17.2
VAE-WReN
64.2
43.6
24.6
MXGNer
11
66.7
65.4
33.6
43.3
19.9
16.7
16.6
18.9
LEN J
0
16
68.1
DCNet
68.6
59.7
17.8
T-LEN
70.3
SRAN
71.3
Rel-H Base
40
85.5
22.
SCL
88.9
MRNet 2
93.4
68.1
38.4
55.3
25.9
30.1
16.9
19.2
MIRN
98.0
57.8
14.9
Ours
72.0
64.1
32.1
16.0
29.0
Table Ablation study on two regimes in PGM
Regime w/o discretization
w/o internal inference
w/o both
ARII
H.O.A.P
47.5
32,0
36,6 51.1
50.0
Interpolation
69.5
54,2
72.0
Network (MRNet) 24]. and Multi-Layer Relation Network (MLRN) [3] Note that model is trained end to end and needs solely the ground truth answer label without auxiliary information (i.e., rule labels). We report the result of the baseline models trained without auxiliary labels for fair comparison. Please refer to Appendix A for more implementation details of ARII.
4.3 Results
Table[]]sowws the test performance of individual models on I-RAVEN dataset. We observe that the typical architectures in computer vision, that simply use CNN as visual feature extractor followed by MLP LSTM, fail to yield satisfactory reasoning results. More advanced models for abstract reasoning such as LEN and SCL significantly improve the performance. In particular, the previous state-of-the-art model SCI, achieves nearly 100% accuracy on the configurations of Center, L-R, U-D, and O-IC, but SCL could not make similar success the other configurations. We conjecture that the reasoning complexity of these four configurations is relatively low and casy. As for our method, ARII performs similarly to SCL the above four configurations but largely outperforms it on the other three more difficult configurations, leading to a notably higher average score on I-RAVEN.
In addition, Table 2 presents the performance on the PGM dataset. For the neutral regime, our model achieves a decent result that is better than majority of baseline models, although does not outperform state-of-the-arr models such as MRNet and MLRN. Note that MLRN achieves near perfect accuracy the neutral regime due the sophisticated design which could not be applied to other problems. More importantly, the neutral regime is not our main focus of metric, since it could hardly indicate the reasoning ability and gencralization towards real intelligence.
PGM also allows us to investigate the generalization capacity of the models under various regimes. We observe from Table 2 that although MLRN achieves superior performance on neutral, it is prone to overfitting and performs poorly on the interpolation and extrapolation regimes, clearly indicating MLRN is overfitted to the neutral setting. On the contrary, the proposed internal inference in ARII serves as the regularities of the rule representations, making the rule representations robust and instance-invariant. As a result, our model outperforms other methods on five of the seven generalization regimes in PGM dataset. In particular, the accuracy of ARII increases by sizable margin over the baselines in the H.O.A.P. and H.O.T.P. regimes (around 10%), further validating the superior generalization ability of abstract reasoning.

---

[progreat wssionn shappe slow)
[cons stant urion, shaps, sio)
DO0, line, color]
lime.
[XD shape, color)
[coms stard, union, shaps, color]
[ORt shape, color)
[on, calor]
(ANG) shaps, sizel
(ADR, shape, siee]
color]
Eprogression, shape. color) FOR, shape, sioel
Figure 3: Visualization of rule embeddings extracted from rule encoder. Each point stands for the rule representation from a data point in the test set.
Code Index
Code Index
Code Index
Frequency
1.00
493
0
-7
(A) [AND, line, color]
(XOR, line, color]
(c) [consistent union, line, color]
2000
400
Prequency
1590
203
190
11
<
(4) Ipcogression, color]
color]
(0 [prugression, slape, size]
Figure 4: The selected frequencies codes in rule representations. We take the rules related to the line object and the color attribute as examples. Similar results for other rules are in Appendix Fig 2.
4.4 Ablation study
We analyze ARII in more detail to validate the effectiveness of the proposed components on two regimes in the PGM dataset (Table In the H.O.A.P. regime, the performance of ARII without dis- cretization decreases by 2.5%, which shows that discrete representation plays its role in generalization of abstract reasoning. The performance is also largely declined by 18% when the internal inference is removed, suggesting that the internal inference vital to learn a more robust rule representation for abstract reasoning. Similar results can be found in the interpolation regime.
We further conduct ablation studies to investigate the impact of generation in internal inference, the combinatorial benefit of F143 and 12&3 in the reasoner, and effectiveness of the internal inference to other models. Please see Appendix B-D for more discussion.
4.5 Visualization of rule representations
Then, we try to validate whether ARII has learned the instance-invarian representations. We first obtain all the rule representations in the test set of the interpolation regime of PGM, and use the representations derived from the problems which contain only single rule (i.c., triple {[s, o, al)). The rule representations are projected to 2D space by t-distributed stochastic neighbor embedding clustered according individual
(t-SNE) 41 in Fig We observe that the rule representations rule triples. Note that these clusters for each rule are formed without supervision of the rule labels. And the representations from the training set. The clustered representations show that these rule representations are universal and independent to the specific instances, even the out-of-distribution brand new instances.
extracted from the generalization test which has a different distribution
Taking one step further, we investigate whether the discrete rule representation interpretable. In particular, we want to see the composition of the representations for each specific rule. Here, take

---

the rules related to the line object and the color attribute (totally five rules, i.e., [s E S, line, color], denoted by the I,line, color] rules) as an example. We first collect the rule representations from all the instances governed by these rules, which are sampled from the codebook F. Then we calculate the selected frequencies of the codes the 30-th position the rule representations and visualize their distributions. Fig 4 (a-e) shows that all the five rules share the same codes F358 and E1st in their top-3 codes, indicating that the codes Bass and E1st are very relevant to the [,line, color] rules. In addition, these codes are also specific to these rules, since they rarely appear in the dissimilar rules such as the rule iprogression, shape, size] (Fig (D). These results reveal that the rule representations learned by ARII are moderately interpretable. Besides, we conduct classification experiment to whether the rules can classified from the rule representations. The classification accuracy based on the rule representations from ARII is 72.1 %6, largely higher than baseline model SRAN [8] that yields 59.4%. This indicates the superiority of ARII which can capture the underlying rules and learn more robust rule representations.
5 Limitations
Abstract reasoning is the ability that consciously applies logic from premises to conclusion 42 For neural networks, they use different layers of the distributed neurons to implicitly represent the premises, the logic and the temporal results, and formulate the final conclusion-making process as the classification generation task. Therefore, neural networks lack in explaining how they perform induction and reasoning. Although our method takes one step further to build interpretable rule representations, is still far from the goal of fully interpretable, robust, and accurate reasoner for general reasoning. Another limitation of this work is the image encoder, which relies on the rigid spatial structure and is difficult for learning new rules in other abstract reasoning problems. The general object detector pre-traine trained on natural images is a potential solution.
6 Conclusion
In this work, we propose novel architecture ARII that learns robust rule representations for abstract reasoning via internal inferences. Experiments on benchmark datasets (I-RAVEN and PGM) show that ARII outperforms previous state-of-the-art methods without using auxiliary annotations on the majority of the reasoning tasks. Moreover, the rule representations of ARII present meaningful and interpretable characteristics, facilitating the exploration of the composition of rules in future work.
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant 61836004 and 62206192, and grant from Institute Guo Qiang, Tsinghua University.
References
[1] John C Raven and JH Court. Ruven's 's progressive matrices. Western Psychological Services Los Angeles, CA, 1938.
[2] Patricia Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
[3] Dokhyam Hoshen and Michael Werman. Iq of neural networks. arXiv:1710.01692, 2017.
arXiv preprint
[4] Ke Wang and Zhendong Su. Automatic generation of raven's progressive matrices. In IJCAI, 2015.
[5] David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Mcasuring abstract reasoning in neural networks. In ICML, pages 511-520, 2018.
[6] Jacek Mandziuk and Adam Zychowski. Deepiq: A human-inspired system for solving iq test problems. In IJCNN, pages 1-8, 2019.
10

---

[7] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, pages 5317-5327, 2019.
[8] Sheng Hu, Yuqing Ma. Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In AAAI, pages 1567-1574, 2021.
[9] Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu, Learning perceptual inference by contrasting. NeurIPS, 32, 2019.
[10] Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The seattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning, arXiv preprint 2rXiv:2007.04212, 2020.
[11] Duo Wang. Mateja Jamnik, and Pietro Lio. Abstract diagrammatic reasoning with multiplex graph networks. In ICLR, 2020.
[12] Tao Zhuo and Mohan Kankanhalli. Effective abstract reasoning with dual-contrast network. In ICL, 2021.
[13] Marius Jahrens and Thomas Martinetz. Solving raven's progressive matrices with multi-layer relation networks. In IJCN,, pages 1-6, 2020.
[14] Victor Vikram Odouard and Melanie Mitchell. Evaluating understanding on conceptual abstrac- tion benchmarks. arXiv preprint arXiv:2206.1410 2022.
[15] Mikotaj Malkiński and Jacek Maúdziuk. Deep learning methods for abstract visual reasoning: survey on raven's 's progressive matrices. arXiv preprint arXiv:2201.12882 2022.
[16] Kecheng Zheng. Zheng-Jun Zha, and Wei Wei. Abstract reasoning with distracting features. NeurIPS, 32, 2019.
[17] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurlPS, 30, 2017.
[18] John C Raven, John Hugh Court, and John Earle Raven. Standard progressive matrices. Australian Council for Educational Research Limited, 1989.
[19] Andrew Lovett, Kenneth Forbus, and Jeffrey Usher. A structure-mapping model of raven's progressive matrices. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 32, 2010.
[20] Maithilee Kunda, Keith McCireggor and Ashok K Goel. A computational model for solving problems from the raven's progressive matrices intelligence test using iconic visual representa- tions. Cognitive Systems Research, 22:47-66, 2013.
[21] Claes Strannegård, Simone Cirillo, and Victor Ström. An anthropomorphic method for progres- sive matrix problems. Cognitive Systems Research, 22:35 46, 2013.
[22] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1835:1735 1997.
[23] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. NeuriPS, 30, 2017.
[24] Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In CVPR, pages 12557-12565. 2021.
[25] Mikolaj Malkiński and Jacek Maúdziuk. Multi-label contrastive learning for abstract visual reasoning, arXiv preprint arXiv:2012.01944 2020.
[26] Donadello, I.. Serafini, and D. Garcez. Logic tensor networks for semantic image interpretation. In IJCAI, pages 1596-1602,
2017. 2017.

---

[27] C. Zhang, B. Jia, S. C. Zhu, and Y. Zhu. Abstract ppatial-tempone reasoning via probabilisticc abduction and execution. In CVPR, 2021.
[28] Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chum Zhu, and Yixin Zhu. Learning algebraic representation for systematic generalization in abstract reasoning. arXiv preprint arXiv:2111. 12990, 2021.
[29] Artur d' Avila Garcez and Luis C Lamb. Neurosymbolic ai: the 3rd wave. arXiv preprint arXiv:20/2.05876, 2020.
[30] Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. Reasoning visual dialogs with structural and partial observations. In CVPR, pages 66669-6678, 2019.
[31] Xianggen Liu, Wenqiang Lei, Jiancheng Lv, and Jizhe Zhou. Abstract rule learning for paraphrase generation. In IJCAI, pages 4273-4279, 2022.
[32] Xianggen Liu, Lili Mou, Haotian Cui, Zhengdong Lu, and Sen Song, Finding decision jumps in text classification, Neurocomputing, 371:177-187, 2020.
[33] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural net- works: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(11):4037- 4058, 2020.
[34] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. In CVPR, June 2020.
[35] Xianggen Liu, Yunan Luo, Pengyong Li, Sen Song, and Jian Peng, Deep geometric representa- tions for modeling effects of mutations on protein-proted binding affinity. PLoS computational biology, 17(8)=e1002244 2021.
[36] Nicolette A Waschl, Ted Nettelbeck, and Nicholas R Burns. The role of visuospatial ability in the raven's 's progressive matrices. Journal of Individual Differences, 2017.
[37] Huan Xu and Shie Mannor. Robustness and generalization, Machine learning, 86(3):391-44 2012.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[39] Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generaliza- tion for abstract reasoning tasks using disentangled feature representations, arXiv preprint arXiv:1811.04784. 2018
[40] Steven Spratley, Krista Ehinger, and Tim Miller. A closer look at generalisation in raven. In ECCV, pages 601-616, 2020.
[41] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1 1), 2008
[42] Michael Proudfoot and Alan Robert Lacey. The Routledge dictionary of philosophy. Routledge, 2009.
[43] Artur SD' Avila Garcez, Luis Lamb, and Dov M Gabbay. Newral-symbolic cognite cognifivveeesonnnng Springer Science & Business Media, 2008
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Our model has improved the generalization performance by a notable margin on majority of tests in PGM, but further research is needed to reach the human level of generalization.
12

---

(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [N/A]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] We have uploaded the codes.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The datasets used in this work are cited in (5) and [8].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "nlgju2MKWd", "reviewer": "Reviewer_bvvT", "summary": "The paper addresses the important issue of abstract reasoning, in particular the ability of neural networks at performing extrapolation.\n\nThe paper's main goal is summarised by the authors as to \"ameliorate the reasoning generalizability by learning a robust rule representation\". This needs to be reconciled with the goal of addressing extrapolation, since generalizability is something else. \n\nThe authors claim to learn rule representations and to perform reasoning. Many ML systems learn rules. It is important to note that the rule representations are embeddings and that the reasoning is by similarity only. Looking at the paper from that perspective makes the claim made around visualization and interpretation of the rules seem rather vague. \n\nHaving said that, the task at hand being considered in the paper is very interesting from a reasoning point of view. And the improved results obtained are valid and relevant and should not be ignored. \n\nA main claim of the paper is that other approaches may \"require additional prior information, which is not a general way to address the reasoning problem\". ", "questions": "It is not easy to see the relationship between Fig 1 and the rule encoder, the reasoner, and the internal inferrer. Is this done on purpose?\n\nA main claim of the paper is that other approaches may \"require additional prior information, which is not a general way to address the reasoning problem\". Other approaches may provide a general mechanism for reasoning. When it comes to a fair comparison, how can you be sure that the \"internal inference\" process isn't a way of providing additional information?\n\nThe \"internal inference\" process seems to function in such a way as to turn propositional rules into rules with variables. Is this a fair description of the purpose of the proposed process? You state that \"we randomly mask one panel of the first two rows and ask the referrer to infer the masked panel based on the rule representation. We perform the internal inference for multiple times to make the rule representation invariant to specific instances.\" To answer my question above, this process needs to be specified well and analysed from the perspective of reasoning capabilities.\n\nHow do we know that the results derive from better reasoning rather than e.g. the combinatorial benefit of using r1&2 with r1&3 and r2&3? Did the ablation study conclude anything about this? How do we check whether this internal inference is indeed producing the expected \"more abstract\" rules (with variables)?\n\nWhen one says that one evaluates \"reasoning ability\", one would wish to evaluate what kinds of reasoning take place, not simply evalute improvements in test set accuracy. Have you evaluated what happens in a situation where the reasoning required to obtain the correct answer is generally seen as harder than the normal type of reasoning?", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and well-organized, although the captions of the figures could have been more informative, e.g. Figs 3 and 4 are difficult to grasp. \n\nThe task at hand being considered in the paper is very interesting from a reasoning point of view. And the improved results obtained are valid and relevant.\n\nThe related work section is generally limited to work on the specific datasets, e.g. RAVEN and variations thereof.\n\nA number of acronyms is used in the paper without stating first what they stand for, sometimes even without providing a reference, e.g. PGM. \n\nRobustness should also be defined (ideally). The most problematic part of the paper is the lack of a definition of \"reasoning\". Reasoning in neural networks is an important endeavour. It is nowadays common for neural net papers to refer to reasoning without providing any definition of what kind of reasoning or language that is being considered (and there are many). These are all formally well-defined in the area of knowledge representation in AI. The area of neurosymbolic AI has taken those definitions on board. If the field is to develop as needed in this direction then it is important to consider the work that goes on since the 1990's in knowledge representation and neurosymbolic AI. I'd refer the authors at least to:\n\nhttps://link.springer.com/book/10.1007/978-3-540-73246-4\nhttps://ebooks.iospress.nl/volume/neuro-symbolic-artificial-intelligence-the-state-of-the-art\n\nAnd specifically on adding knowledge to neural nets:\nhttps://www.ijcai.org/proceedings/2017/221\nhttps://www.researchgate.net/publication/283897473_Semantic-based_regularization_for_learning_and_inference\nhttps://arxiv.org/abs/1605.06523\n  \nAnd for a recent survey:\nhttps://arxiv.org/abs/2012.05876 ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m6PLFniDhEh", "reviewer": "Reviewer_ky2S", "summary": "The authors propose a novel framework Abstract Reasoning via Internal Inferences (ARII) for the task of solving abstract visual reasoning problems. In particular, the ARII model is designed to solve the task of selecting the correct candidate panel for a Raven's Progressive Matrices (RPM) reasoning problem. The proposed method consists of three components:  \n\n1. A rule encoder network that takes the first two context rows of the RPM matrix, and generates vector quantized codes representing the abstract rule encoded in the images. The encoder works by generating CNN embeddings for each panel, which are then split into groups and concatenated based on their position. These image encodings are once again split into positions for each of the six image panels and then re-combined together to generate a continuous rule representation. The continuous rule representation is then discretized into the Ke X D dimensional code book, which is trained together with the encoder.\n2. A reasoner which takes each candidate panel as a potential rule with the context panels in the third row, calculates the rule encoding of this candidate rule with rows 1 and 2, and then selects the candidate rule (panel) which has the highest similarity with the rule encoded by rows 1 and 2.  \n3. A generative internal inferrer, that masks out each image sequentially out of the six context images in the first two rows, and aims to generate the original rule encoding with one context image missing by minimizing the distance between the original rule and the rules extracted with a panel missing.  \n\nARII outperforms existing methods when it comes to generalizing to novel test domains on the i-RAVEN and PGM datasets. Ablation studies performed by the authors demonstrate the utility of both the discretized code book as well as the internal inference mechanism. \n", "questions": "Could the authors further validate their claims that the codes learned by ARII are instance-invariant? In abstract reasoning terms this would involve that the codes learned for one particular rule (such as AND) will be similar irrespective of the visual domain. The ablation studies done by the authors do not demonstrate this. Please see the limitations highlighted in the review for one possible suggestion to demonstrate this.\n\n**Post-Rebuttal Edit:** See response to author's rebuttal below. Score increased to a 7 post-rebuttal.", "limitations": "The authors did a good job in explaining several assumptions in their modeling process, such as space of rules is not large. Several of the ideas utilized in the ARII model build upon previous research (such as splitting and grouping panel embeddings like the Scattering Composition Learner, row wise embeddings relying on feature locations like Multiplex Graph Network, etc.) While many of these papers are referenced, the authors fail to draw connections and provide reasonable credit to the authors of these works in their Related Work section. It would be great if the authors can highlight these connections, as it will help future research in abstract visual reasoning and would be only fair to the authors of previous works.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper is extremely well written: the RPM problem is presented succinctly, the ARII model is explained clearly and each component is motivated well, and the training procedure, results and ablation studies are easy to follow.\n\n2. The idea of utilizing a learnable vector quantized codebook for representing abstract rules in PGM problems is quiet novel, and works well in practice as seen from the generalization results of ARII.  \n\n3. Under the scenario where auxiliary information such as symbolic rule embeddings are not used for training, it demonstrates state-of-the-art performance on four generalization splits of the i-RAVEN dataset, and five generalization splits of the PGM dataset.\n\nWeakness:\n\n1. In equation 1, the authors make an implicit assumption that the RPM rule is encoded row-wise in the image panels. While this holds true for the i-RAVEN dataset, it is not necessarily true for PGM. The authors work around this limitation by adding the transposed PGM matrices for training. Several other models compared in the paper do not explicitly rely on this assumption, and hence it would only be fair to compare these models with this extra training step included.   \n\n2. In the ablation study, the authors show that the codes E-358 and E-181 occur in the top three codes for each possible relation in the visual domain of [., line, color] rules. This appears counter-productive to the argument that the ARII model has learned instance-invariant representations, since if it were indeed instance-invariant then the visual domain should have little to no impact on the rule representation. A better way to demonstrate this would have been to take the codes for [AND, ., .] across multiple visual domains and show that the codes are consistent with various object and attribute values.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RTYPNrJAed", "reviewer": "Reviewer_fEdf", "summary": "The authors propose ARII, a new framework to solve the RPM datasets. The main idea is to try to learn a reusable rule representation from an instance of the task, such that it can be applied to other tasks when applicable. They achieve this by mapping individually learned rule vectors to a discrete representation using vector quantization and then reusing those rules to infer some of the already present panels in the RPM instance. Overall, they show that ARII is able to perform better, specifically on larger grids and more difficult settings.", "questions": "Writing suggestions:\n* It would help to have some background on the details of the vector quantization steps involved in the discretization of the rule embeddings. Maybe a motivating figure to explain with an example what q_l, etc., mean can make that section more accessible.", "limitations": "Please add a section to describe the limitations of the work. I believe the major limitation is deciding the size of the vector quantization you do to obtain unique rules. Potentially, there can be new rules for each instance of the dataset, where your basic hypothesis breaks. Try to add some discussion along these or other relevant points.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is nicely motivated in general. The design of each module is well described and easy to follow through\n* Final results on both performance and interoperability are quite impressive\n\nWeakness:\n* The motivation for the design choices is not well supported in the ablation. I'm specifically not convinced about the design choices of the internal inferences (Section 3.5). Why does the process need to be generative instead of classification? For negative answer choices, you can fall back on the existing options in the instance. The correct answer is the particular image you mask out. The advantage of this is you can potentially reuse the reasoner module for this part as well. This can at least be an additional idea to try for ablations, if not the main approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5_7g21t617c", "reviewer": "Reviewer_4Dou", "summary": "This paper proposes a novel method for visual abstract reasoning tasks, focused on the RPM-like benchmarks PGM and RAVEN. Some of the key features of the new method are the use of a learnable discrete representation to encode abstract rules, and a self-supervised objective that requires the model to use the encoded rule to fill in masked panels. The model achieves competitive results on both benchmarks, and shows particularly strong performance in the OOD regimes of the PGM dataset. There are also some interesting analyses of the rule representations learned by the model.", "questions": "I don't have any questions, other than regarding some of the missing details about hyperparameters and training.", "limitations": "I don't envision any potential negative societal impact from this work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The model achieves excellent results, surpassing or competitive with the state-of-the-art on both benchmarks, and with particularly strong performance on the OOD regimes of PGM, which are arguably the most important part of these benchmarks.\n- The evaluation is very thorough, evaluating the model on all formats and generalization regimes for the two major benchmarks in this area.\n- The inductive biases introduced in this method are quite interesting. Using discrete encodings of the rules likely forces them to be more abstract, which presumably contributes to the OOD performance. The self-supervised objective ('internal inference') is an interesting way to capture the kind of 'hypothesis testing' that characterizes the way human reasoners solve these problems (internally generating proposals for the abstract rules, and then checking to see if they explain the presented panels).\n- The ablations nicely demonstrate the contributions of these two components.\n- The authors have included some code in the supplemental material, and plan to release all code upon acceptance.\n\n## Weaknesses\n- The primary weakness is that there are a number of important details missing that need to be included, perhaps in an appendix, e.g. training details (number of epochs, batch size, learning rate etc), model hyperparameters, and the computing machinery that was used to train the models. These should be included in the final version of the paper.\n\nI also have a few other minor notes:\n- I like the rule interpretability analysis, but it would be more informative if it were more comprehensive, rather than only focusing on rules applied to line color.\n- It would be informative to also test the proposed method on this recently released variation on RAVEN: https://arxiv.org/abs/2206.14187\n- It would be good to explicitly mention the connection between the 'internal inference' objective and self-supervised learning.\n- The object encoding method employed here is very inflexible, relying on the rigid spatial structure of these problems. I think it's ok in the context of this work, but it would be good to mention this as a limitation and a potential avenue for future work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2ZHUe_4Zczi", "reviewer": "Reviewer_xDma", "summary": "A model called ARII is proposed in this work to solve abstract reasoning problems. The ARII model is composed of a rule encoder, a reasoner, and an internal inferrer. The rule encoder adopts a vector quantization method and turns continuous rule representation from the first two roles into a discrete token in the codebook. The reasoner uses a similarity-based measure to select the answer: the candidate which, when filled in, generates rule features most similar to the first two rows will be treated as the answer. In addition, the internal inferrer reconstructs a masked panel like in BERT. In experiments, the authors conduct evaluation on I-RAVEN and PGM. It's found that the model achieves best performance on I-RAVEN and improves on the generalization regimes on PGM.", "questions": "1. It would be nice to see results of existing models trained with the masked modeling loss. \n2. How the reasoning capability emerges.", "limitations": "No. Limitations are not adequately addressed, though there is not ethical issues.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written with clear explanation and a streamlined flow. \n\nThere are a few things that are done right which I believe lead to the performance boost in this work. (1) image features are extracted from a group level: considering the grid-like layout in I-RAVEN and PGM, the position-based grouping could help better capture the statistics in the images. (2) Quantization in rules: as there are only a finite number of rules, quantization seems a reasonable choice, which is unfortunately missing in existing works. (3) BERT-like masked reconstruction. This might be the part that contributes to the generalization improvement as hinted in existing language modeling works.\n\nI'm wondering, as the masked reconstruction part can be plugged and played, whether performance in generalization could be consistently improved in different models. It would be nice to see some results regarding this point. \n\nAlso I wonder how reasoning is conducted. It occurs to me that the work still applies statistical learning but lacking in explaining how a model performs induction and reasoning. Some statistics may be captured using the method, but as shown in generalization results (which are indeed improved), it is still far below fair expectation. I would be interested in knowing how the authors view the perspective: does reasoning emerge from statistical learning and if not what is your opinion.\n\nFinally, I notice a few neuro-symbolic methods [1, 2] are missing, which when added should better complete the full picture of the area. \n\n[1] Zhang, Chi, et al. \"Abstract spatial-temporal reasoning via probabilistic abduction and execution.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.  \n[2] Zhang, Chi, et al. \"Learning algebraic representation for systematic generalization in abstract reasoning.\" arXiv preprint arXiv:2111.12990 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept