Effective Abstract Reasoning with Dual-Contrast Network
Authors: Tao Zhuo
Mohan Kankanhalli
Venue: ICLR 2021 Poster (2021)
Keywords: abstract reasoning
raven's progressive matrices
deep learning

Abstract: As a step towards improving the abstract reasoning capability of machines, we aim to solve Raven’s Progressive Matrices (RPM) with neural networks, since solving RPM puzzles is highly correlated with human intelligence. Unlike previous methods that use auxiliary annotations or assume hidden rules to produce appropriate feature representation, we only use the ground truth answer of each question for model learning,  aiming for an intelligent agent to have a strong learning capability with a small amount of supervision.  Based on the RPM problem formulation,  the correct answer filled into the missing entry of the third row/column has  to  best  satisfy  the  same  rules  shared  between  the  first  two  rows/columns.Thus  we  design  a  simple  yet  effective  Dual-Contrast  Network  (DCNet)  to  exploit the inherent structure of RPM puzzles.  Specifically, a rule contrast module is  designed  to  compare  the  latent  rules  between  the  filled  row/column  and  the first two rows/columns; a choice contrast module is designed to increase the relative differences between candidate choices.  Experimental results on the RAVEN and  PGM  datasets  show  that  DCNet  outperforms  the  state-of-the-art  methods by a large margin of 5.77%.   Further experiments on few training samples and model generalization also show the effectiveness of DCNet.  Code is available at https://github.com/visiontao/dcnet.

Published as a conference paper at ICLR 2021
EFFECTIVE ABSTRACT REASONING WITH DUAL-CONTRAST NETWORK
Tao Zhuo, Mohan Kankanhalli
School of Computing, National University of Singapore zhuotaoθnuse www.uuu..u...........
edu.. sg, mohan comp. nus edu. .g
ABSTRACT
As a step towards improving the abstract reasoning capability of machines, we aim to solve Raven's Progressive Matrices (RPM) with neural networks, since solving RPM puzzles is highly correlated with human intelligence. Unlike previous meth- ods that use auxiliary annotations or assume hidden rules to produce appropriate feature representation, we only use the ground truth answer of each question for model learning, aiming for intelligent agent to have a strong learning capa- bility with a small amount of supervision. Based on the RPM problem formu- lation, the correct answer filled into the missing entry of the third row/column has to best satisfy the same rules shared between the first two rows/columns. Thus we design simple yet effective Dual-Contrast Network (DCNet) to ex- ploit the inherent structure of RPM puzzles. Specifically, rule contrast module is designed compare the latent rules between the filled row/column and the first two rows/columns; a choice contrast module is designed to increase the rela- tive differences between candidate choices. Experimental results on the RAVEN and PGM datasets show that DCNet outperforms the otate-of-the-art methods by a large margin of 5.77%. Further experiments on few training samples and model generalization also show the effectiveness of DCNet. Code is available at http:///ihub..com//iiiotttt//a/a
INTRODUCTION
Abstract reasoning capability is a critical component of human intelligence, which relates to the ability of understanding and interpreting patterns, and further solving problems. Recently, as step towards improving the abstract reasoning ability of machines, many methods (Santoro et al., 2018; Zhang et al., 2019a;b; Zheng et al., 2019; Zhuo & Kankanhalli, 2020) are developed to solve Raven's 's Progress Matrices (RPM) (Domino widely believed that RPM lies at the heart of human intelligence. As the example shown in Figure 1. given 3 x 3 problem matrix with a final missing piece, the test taker has to find the logical rules shared between the first two choices to best complete the matrix. Since the logical rules hidden in RPM questions are complex and unknown, solving RPM with machines remains challenging task.
Domino, 2006; Raven & Court, 1938), since it is
or columns, and then pick the correct answer from 8 candidate
As described in (Carpenter et al., 1990), the logical rules applied in RPM question are manifested as visual structures. For single image in the question, the logical rules could consist of several basic attributes, e.g., shape, color, size, number, and position. For the images in row or column, the logical rules could be applied row-wise or column-wiss and formulated with an unknown re- lationship, e.g., AND, OR, XOR, and so on (Santoro et al., 2018; Zhang et al., 2019a). If we can extract the explicit rules of each question, the problem can be easily solved by using heuristics- based search method (Zhang et al., 2019a). However, given an arbitrary RPM question, the logical rules are unknown. What's worse even the number of rules is unknown. As a result, an intelligent machine needs to simultaneously learn the representation of these hidden rules and find the correct answer to satisfy all of the applied rules.
With the success of deep learning in computer vision, solving RPM puzzles with neural networks has become popular. Because the learned features might be inconsistent with the logical rules, many supervised learning methods, egg., DRT (Zhang et al., 2019a), WReN (Santoro et al., 2018) and

---

Published as a conference paper at ICLR 2021
LEN (Zheng et al., 2019), MXGNet (Wang et al., 2020) and ACL (Kim et al., 2020), not only use the ground truth answer of each RPM question but also the auxiliary annotations (such as logical rules with shape, size, color, number, AND, OR, XOR) to learn the appropriate feature representa- tion. Although auxiliary annotations provide lots of priors about how the logical rules are applied, noticeable performance improvement is not always obtained on different RPM problems, such as in the results reported in Table 1. Moreover, such a learning strategy requires additional supervision. When auxiliary annotations are not available, it will fail to boost the performance. For example, DRT (Zhang et al., 2019a) cannot be applied to PGM dataset (Santoro et al., 2018) for the lack of structure annotations. To overcome the constraint of using auxiliary annotations, a recent method CoPINet (Zhang et al., 201 2019b) only uses the ground truth answer of each question. Meanwhile, to produce the feature representation of hidden rules, CoPINet assumes there are at most N attributes in each problem, and each of which is subject to the governance of M rules. However, due to N and M being unknown for arbitrary RPM problems, such an assumption is still too strong.
In this work, aim to learn the abstract reasoning model by using only the ground truth answer of each question, and there is not any assumption about the latent rules. According to the RPM problem formulation (Carpenter al., 1990), it can be concluded that finding the correct answer of a RPM puzzle mainly depends on two contrasts of rules: (1) compare the hidden rules between the filled row/column and the first two rows/columns to check whether they the same rules; (2) compare all candidate choices to check which one best satisfies the hidden rules. Unlike making specific assumptions about latent rules that are only valid for particular cases, the above two contrasts are general properties of all RPM problems.
Considering above two contrasts, we propose a simple yet effective Dual-Contrast Network (DC- Net) to solve RPM problems. Specifically, a rule contrast module is used to compute the difference between the filled row/column and the first two rows/columns, which checks the difference between latent rules. Additionally, the second choice contrast module is used to increase the relative differ- ences of all candidate choices, which helps find the correct answer when confusingly similar choices exist. Experiments on two major benchmark datasets demonstrate the effectiveness of our method. In summary, our main contributions are as follows:
We propose new abstract reasoning model on RPM with only ground truth answers, i.e. there are not any assumptions or auxiliary annotations about the latent rules. Compared to previous methods, the problem setting of our method is more challenging, intelligent agent to learn a strong model with a small amount of supervision.
aim for an
We propose a simple yet effective Dual-Contrast Network (DCNet) that consists of a rule contrast module and choice contrast module. By exploiting the inherent structures of each RPM with basic problem formulation, robust feature representation can be learned.
Experimental results on RAVEN (Zhang et al., 2019a) and PGM (Santoro al., 2018) datasets show that our DCNet significantly improves the average accuracy by large mar- gin of 5.77%. Moreover, from the perspective of few-shot learning, DCNet outperforms the state-of-the the-ooftthhtrrrr art method CoPINet (Zhang et al., 2019b) by a noticeable margin when few training samples are provided, see Table and 4. Further experiments on model gen- eralization also show the effectiveness of our method, see Table 5 and 6.
2 RELATED WORK
2.1 VISUAL. REASONING
One of the most popular visual reasoning tasks is Visual Question Answering (VQA) (Antol al., 2015; Zellers et al., 2019; Fan et al., 2020). To aid in the diagnostic evaluation of VQA systems, Johnson et al. (2017) designed CLEVR dataset by minimizing bias and providing rich ground-truth representations for both images and questions. It is expected that rich diagnostics could help better understand the visual reasoning capabilities of VQA systems. Recently, to understand the human actions in videos, Zhou et al. (2018) proposed temporal relational reasoning network to learn and reason about temporal dependencies between video frames at multiple time seales. Besides, for explainable video action reasoning, Zhuo et al. (2019) proposed explain performed actions by recognizing the semantic-level state changes from a spatio-tempott video graph with pre-defined rules. Different from these visual tasks, solving RPM puzzles depends sophisticated logical rules
2

---

Published as a conference paper at ICLR 2021
acquiring human intelligence (Raven & Court, 1938; Santoro al., 2018: Zhang et al., 2019a). Therefore, it is expected that solving RPM problems with machines could help better understand and perhaps improve the abstract reasoning ability of contemporary computer vision systems.
2.2 COMPUTATIONAL MODELS ON RPM
The early methods RPM (McGreggor& & Goel, 2014; McGreggor et al., 2014; Mekik al., 2018) often compute the feature similarity of images with a hand-crafted representation. Besides, struc- tural affinity (Shegheva & Goel, 2018) with graphical models also used in RPM problems. Due to the lack of large-scale dataset, the early methods are only evaluated on a small dataset. Recently, with the introduction of deep learning in pattern recognition (Deng et al., 2009), solving RPM with deep neural networks has becomes the main approach. For automatic RPM generation, Wang & Su (2015) introduced an abstract representation method for RPM by using the first-order logic for- mulae, and they applied three categories of relations (i.e. unary, binary, and ternary). Hoshen & Werman (2017) first trained CNN model measure the of of neural networks a simplistic eval- uation environment. Santoro et al. (2018) () measured the abstract reasoning capability of machines with several popular neural networks, such as LSTM, ResNet and Wild-ResNet. Besides, Zhuo & Kankanhalli (2020) studied the effect of ImageNet pre-training (Deng al., 2009) for RPM prob- lems and proposed a novel unsupervised abstract reasoning method by designing a pseudo target.
To further improve the feature representation with auxiliary annotations, Santoro et al. (2018) pro- posed a WReN network that formulates pair-wise relations between the problem matrix and each individual choice in embedding space, independent of the other choices. Zhang al. (2019a) generated a RAVEN dataset with the structure annotations for each RPM problem and proposed a Dynamic Residual Tree (DRT) method. In addition, Zheng al. (2019) proposed Logic Embed- ding Network (LEN) with distracting features, which also uses the auxiliary annotations to boost the performance and designed teacher model to control the learning trajectories. MXGNet (Wang et al., 2020) is a multi-layer graph neural network for multi-panel diagrammatic reasoning tasks. For better performance, MXGNet also uses auxiliary annotations for model training. ACL (Kim al., 2020) uses symbolic labels to generate new RPMs with the same problem semantics but with differ- ent attributes (e.g., shapes and colors). Discarding the use auxiliary annotations, CoPINet (Zhang et al., 2019b) only utilizes the ground truth answer of cach RPM question. By combining con- trasting, perceptual inference. and permutation invariance, CoPINet achieves the state-of-the-ar performance. However, it assumes the maximum number of attributes and rules is known. Dif- ferent from the "objective-level contrast" in (Zhang et al., 2019b) that treats the complete problem object for comparison, our model consists of
matrix (9 images, including filling choice) rule contrast module and a choice contrast module and we only compare the features of different rows/columns (3 images). Thus our method is more effective exploit the inherent structure of RPM on rows/columns. Moreover, no assumptions about attributes and number of rules are made,
3 DUAL-CONTRAST NETWORK
A typical RPM problem usually consists of 16 images, including 3 x 3 problem matrix with final missing piece and 8 candidate answer choices. Solving a RPM problem is to find the correct answer from 8 candidate choices to best complete the problem matrix. Given N RPM questions X' = { vi),,... (#N, JN)], where in each question (æ,, 34) (i E NJ). æ; denotes 16 images aim learning an abstract reasoning model on X
and y; represents the ground truth answer, that finds the correct answer. Different from previous methods (Santoro et al., 2018; Zheng et al., 2019) that use auxiliary annotations on hidden rules, we only use the ground truth answer of each question. Because of less information use for supervision, the problem setting of our method is more challenging.
Based on the RPM problem formulation (Carpenter et al., 1990), given RPM problem with a set of rules applied either row-wise or column-wise, the correct answer filled into the third row/column has to best satisfy the same rules shared between the first two rows/columns. Thus, it follows that the correct answer of a RPM question has to satisfy two contrasts of hidden rules: (1) the third row/column filled with choice whether shares the same rules that of the first two rows/columns; (2) the choice is the best one to complete the matrix when compared to the other candidate choices. Inspired by these two inherent contrasts of RPM problem formulation, propose a dual-contrast.
3

---

Published as a conference paper at ICLR 2021
model for abstract reasoning. Specifically, rule contrast module is designed to compare the hid- den rules between filled row/column and the first two rows/columns; a choice contrast module is designed to increase the relative differences of candidate choices.
3.1 RULE CONTRAST MODULE
As described in (Carpenter et al., 1990), the logical rules in a RPM problem could be applied either row-wise column-wise. However, for arbitrary RPM problem, which logical rules have been applied is unknown. Thus, we equally consider the row-wise and column-wise features to represent the hidden rules and we process them in the same way. In order to simplify the description, we only take the row-wise feature as example in next.
Given a RPM problem 2; we iteratively fill each candidate choice into the missing piece of the problem matrix, forming a new problem of 10 rows/columns. Specifically, Tú, denotes the j-th row of rs. j E {1.... 10), #1j+6 represents (j + 6) image of æ;, If the row ray" share the same latent rules with r;, and T1,2, it will be regarded as the correct answer. Thus, by comparing the features among the last eight rows and the first two rows, the correct answer j can found.
Suppose f(-) is the embedding feature representation of the latent rules of row, and then the row-wise feature of rt,j is denoted as f(r:g). Due to the logical rules applied in different RPM problems often different, which would lead to inaccurate feature representation. To alleviate this issue, we explicitly check how the latent rules are satisfied for each row. Based on the fact that fra,1), (rs.2). and fr.gz- share the same logical rules, they can be clustered into the same group while the features of other rows in another group. Therefore, by subtracting the clustering centroid of ((:;1) and (rr,,), the rule contrast feature g(ri.) on j-th row be extracted compare the relative rule difference of different rows as:
g(r:,)) f(ris) 0.5 * (f(r..)) f(r:2))
(1)
where j 2 3. Since the proposed rule contrast module explicitly compares the latent rules of each row, which matches the inherent structure of RPM problems. Hence, our model adapt to the various logical rules, helping improve the model generalization.
3.2 CHOICE CONTRAST MODULE
The correct answer in a RPM problem satisfies not only the same rules shared between the first two rows but also better fits the third row when compared to all other candidate choices. Given a set of candidate choices for a RPM question, the designed model has to distinguish which one the correct answer. However, to make RPM questions difficult, there are often some confusing choices that are similar to the correct answer (Zhang et al., 2019a; Santoro et al., 2018). Morcover, the confusing similarities can be different for different problems. Therefore, directly learning the model without comparing all of the candidate choices would lead to poor performance.
In order to distinguish the correct answer from similar candidate choices, module to increase the relative differences among the candidate rows (i.e. filling the third row with candidate choice). By reducing the centroid of all candidate choices in a RPM problem, the choice contrast module is formulated as:
choice contrast
((rr))= g(ra.)) oi 1 8
Σ s(rrz))
where h(rr,)) denotes the relative feature of the third row with a candidate choice, j 2 3 (corre- sponding to eight candidate rows), o(-) is an adaptive feature block that consists of a CNN layer and batch normalization layer (loffe & Szegedy, 2015).
Notice that the latent rules have be checked with the rule contrast, the choice contrast is only applied on the last eight rows (f.e. removing the first two rows). Based on such an explicit choice contrast module, the proposed model is able to handle various confusing similarities of candidate choices in different RPM problems.
4

---

Published as a conference paper at ICLR 2021
Probiem Marrk
123
fna
CNN
4.58
TreeBook
789
firus)
g(
16
f(n,0)
sigmoid
output
Chokes
rOWB
rule contrast
choice contrast
Figure I: The architecture of our Dual-Contrast Network (DCNet)). The number in the boxes denotes the corresponding image and the encoder module consists of a CNN layer (with ReLU and batch normalization (loffe & Szegedy, 2015)), a maxpooling layer and a residual block (He et al., 2016). 6). Besides. the two streams DCNet share the same parameters and they are used to extract the row- wise and column-wise features, respectively. Then these two features are added and further fed into a two-layer MLP module for the answer prediction. The correct answer for this problem is 13.
3.3 INFERENCE
Since the logical rules applicd row-wise or column-wise are unknown, we equally consider both conditions. After obtaining the dual-contrast features of the rows and columns, the final answer can be predicted with a classification module. By directly feeding the sum of the row-wise feature h(ray) and the column-wise feature h(c;,)) into two-layer MLP (two fully-connected layers with a ReLU and a dropout layer (Srivastava et al., 2014)), the probability Øt.j of each row/column is estimated
J..- (h(r.,)) + h(cr,))
where j E (3.... 10} corresponds to 8 candidate choices, v-)) represents the projection function of using the MI. module. The choice with the highest score selected as the correct answer.
3.4 TRAINING
The correct answer of RPM question is obtained by comparing the probabilities of all candidate choices, and thus finding the correct answer can be considered label to the correct answer while set the label 0 to the other choices. Besides, in order further increase the difference between the correct answer and the other candidate choices, the estimated probability Hi. of each choice are concatenated and sigmoid function 8(.) is employed for nor- malization. In addition, a Binary Cross Entropy (BCE) loss function L is employed for the model training. Given a dataset X' that consists N questions, the training loss computed as:
ranking task. Hence, we set
N
8
Wi. log(6((r.))) + (1 3i.s) log(1 - 8(1s,)))
(4)
where the index E (1. starts from for the sake of clarity.
3.5 NETWORK ARCHITECTURE
Based on the rule contrast and the choice contrast modules, we propose a Dual-Contrast Network (DCNet) for abstract reasoning on RPM. The overview of the proposed DCNet architecture is de- picted in Figure 1, which consists of two streams and an MLP fusion layer. As the row-wise and column-wise features are equally considered for the final answer prediction, these two streams share the same parameters. Given a RPM problem, by iteratively filling each choice to the missing piece, 10 rows and 10 columns are formed, respectively. Then an encoder of DCNet is used to extract the row-wise and column-wist features, representing latent rules of each row/column. This is fol- lowed by a dual-contrast module to compare the latent rules and increase the relative differences of candidate choices. Finally, two-layer MIP module is used for the correct answer prediction.
5

---

Published as a conference paper at ICLR 2021
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Similar to the previous works (Zhang et al., 2019b; Zheng et al., 2019; Wang et al., 2020), we con- duct experiments on the RAVEN (Zhang et al., 2019a) and PGM (Santoro et al., 2018). All images in our experiments are resized to fixed size of 96 x 96. During the training phase, a mini-batee size of 32 with Adam optimizer (Kingma& & Ba, 2015) is employed to learn the network parameters, and the learning rate is set to 0.001 and fixed. In all experiments, the network architecture and training parameters are kept the same to demonstrate the effectiveness of DCNet. In addition, all models are trained and evaluated on a single GPU of NVIDIA GeForce 1080 Ti with 11 GB memory.
4.2 DATASETS
RAVEN dataset (Zhang et al., 2019a) consists of 70K RPM problems, equally distributed in 7 distinct figure configurations: Center, 2*2Grid, 3*3Grid, Left-Right (L-R), Up-Down (U-D), Out- InCenter (O-IC) and Out-InGrid (O-IG). In each configuration, the dataset is randomly split into three parts, 6 folds for training, 2 for validation, and the remaining 2 for testing. In addition, there is an average of 6.29 rules for each problem, and those rules applied only row-wise.
PGM dataset (Santoro et al., 2018) consists of 8 subdatasets. Compared to RAVEN dataset that consists of shapes only, PGM dataset is constructed with both lines and shapes. Besides, some lines or shapes might be manifested as background for hidden rules in PGM. In our experiment, we adopt the same experimental setup of CoPINet (Zhang et al., 2019b) and we mainly report the testing accuracy of the neutral regime in PGM, since it corresponds most closely to traditional supervised regimes. In total, the neural regime subset consists of 1.42M RPM problems, with 1.2M for training, 20K for validation and 200K for testing. There is an average of 1.37 rules for each problem, and those rules are applied either row- or column-wise.
4.3 BASELINE MODELS
We compare the proposed method with several available results of the state-of-the-arr approaches, including LSTM (Shi al., 2015), CNN (Hoshen & Werman, 2017), ResNet-50 (He et al., 2016), WReN (Santoro et al., 2018), Wild-ResNet (Santoro et al., 2018), ResNet-18+DRT (Zhang et al., 2019a), LEN (Zheng et al., 2019), CoPINet (Zhang et al., 2019b), MXGNet (Wang et al., 2020), and ACL (Kim et al., 2020). In Zheng et al. (2019), there are several models based on different training configurations. For the sake of fair comparison, we only report the performance of LEN with or without auxiliary annotations, due to the fact that T-LEN and teacher model in (Zheng et al., 2019) require external supervision (except using auxiliary annotations in training loss) for model training.
4.4 PERFORMANCE ANALYSIS
Overall Performance. In Table 1, it can be seen that DCNet significantly improves the average accuracy by large margin of absolute 5.77% when compared to MXGNet, i.e. 81.08% vs. 75.31% %. Moreover, without auxiliary amnotations on RAVEN, DCNet achieves the best performance and it outperforms CoPINet by 2.16%, i.e. 93.58% vs. 91.42%. MLRN (Jahrens & Martinetz, 2020) is elaborately designed for PGM and it achieves an accuracy of 98.03%. However, MLRN performs WReN, LEN and
poorly on RAVEN with the same network architecture and MXGNet greatly boost the performance with auxiliary annotations and they achieve better accuracy than other models, but they require additional supervision. Removing the auxiliary annotations for fair comparison, DCNet slightly outperforms LEN by 0.47%.
Performance on RAVEN Dataset. Table shows the testing accuracy of each model on the distinct figure configurations of RAVEN dataset. The performance of the popular CNN and LSTM models is quite poor on the RAVEN dataset, which is less than 40%. Although WReN obtains double accuracy with auxiliary annotations, but is still poor (33.97%). For another method LEN, auxiliary annotations make the performance even worse, L.e. from 72.90% to 53.40%. Such a performance decrease is also observed in (Zhang et al., 2019a). In contrast, without any auxiliary annotations for

---

Published as a conference paper at ICLR 2021
Table I: Testing accuracy of different models on RAVEN and neutral regime of PGM. Aux means auxiliary annotations, Avg represents the average accuracy, DCNet-RC denotes removing the rule contrast module and DCNet-CC denotes removing choice contrast module in our method. "_" indi- cates the results are not reported in the published papers. "#" represents the result is obtained from Zhuo Kankanhalli (2020). "**" represents the result is obtained by running the released code of
Jahrens & Martinetz (2020) with its default parameters.
Method
Aux
Avg
RAVEN
PGM
ResNet-1&+DRT Zhang et (2019a)
59.56
WReN+Aux Santoro et al. (2018)
55.44 70.85
33.97
76,90
LEN+Aux Zheng al. (2019)
59.40
82.30
MXGNet+Aux Wang et al. (2020)
89.60
ACL Kim et al. (2020)
93.71
LSTM Zhang et al. (2019b)
24.44
13.07
35,80
CNN Zhang el al. (2019b)
34.99
36.97
33.00
WReN Santoro al. (2018)
40.10
17.94
62.60
Wild-ResNet Santoro et al. (2018)
48.00
ResNet-50) Santoro et al. (2018)
64.13
86.26 #
42.00
MLRN Jahrens & Martinetz (2020)
55.33
12.50*
98.03
LEN Zheng et al. (2019)
70.50
72.90
68.10
CoPINet Zhang et al. (2019b)
73.90
91.42
56.37
MXGNet Wang et al. (2020)
75.31
83.91
66.70
DCNet-RC
78.10
92.74
63.45
DCNet-CC
47.12
36.47
57.76
DCNet
81.08
93.58
68.57
Table Testing accuracy of different models on different figure configurations in RAVEN dataset.
Method
Aux
Avg
Center
2*2Grid
3*3Grid
L-R
U-D
O-IC
0-IG
WReN+Aux Santoro (2018)
33.97
58.38
38.89
37.70
21.58
19.74
38.84
22.57
LEX+Aux Zheng al. (2019)
59.40
71.10
45,90
40.10
63.90
62.70
67.30
65.20
ResNet-18-DRT Zhang al. (2019a)
59.56
58.08
46.53
50.40
65.82
67.11
69.09
60.11
LSTM Zhang et al. (2019b)
13.07
13.19
14.13
13.69
12.84
12.35
12.15
12.99
WReN Santoro et ul. (2018)
17.94
15.38
29.81
32.94
11.06
10.96
11.06
14.54
CNN Zhang ct al. (2019b)
36.97
35.58
30.30
33.53
39.43
41.26
43.20
37.54
ResNet-50 He al. (2016)
53.43
52.82
41.86
44.29
58.77
60.16
63.19
53.12
LEN Zheng al. (2019)
80.20
$7.50
62.10
73.50
81.20
71.50
CoPINet 7hang al. (2019h)
91.42
95.05
77.45
78.85
99.10
99.65
98.50
91.35
DCNet
93.58
97.80
81.70
86.65
99.75
99.75
98,95
91.45
Human
95.45
81.82
79.55
86.36
86.36
81.81
additional supervision, DCNet performs better than CoPINet and it achieves the best performance among all 7 figure configurations.
Performance PGM Dataset. We employ the same experimental setup of CoPINet (Zhang al., 2019b) that only reports the testing accuracy of the neutral regime of the PGM dataset, as it corre- sponds most closely to traditional supervised regimes. Due the fact that the PGM dataset does not provide information about the figure configurations, we only report the average accuracy of each model. As the results reported in Table the most popular network architectures fail achieve good performance on the PGM dataset, including LSTM, CNN. and ResNet-50. In contrast, the abstract reasoning models, i.e. Nesd-ResNer, WReN, LEN, and CoPINet, achieve better aceuracy. Moreover, with auxiliary annotations about how the logical rules are applicd, the performance of WReN and LEN is greatly improved, i.e. from 62.60% to 76.90% and 68. 10% to 82.30%, respec- tively. However, such learning strategy requires additional supervisory signals.
Performance on Small Data Sizes. In practice, supervised learning methods often require a large amount of well-labeled data for training, while humans able to solve RPM problems without such massive training efforts. In order to evaluate the model generalization and learning efficiency of the models abstract reasoning, further report the testing accuracy of DCNet when the

---

Published as a conference paper at ICLR 2021
training set size shrinks on both RAVEN and PGM datasets. In this experiment, the same experimental setup of CoPINet (Zhang et al., 2019b) and randomly choose different subsets of the full training set and evaluate the average accuracy on the full test set. Besides, compare with CoPINet to demonstrate the effectiveness of our method.'
Table 3: Model performance under different
Table 4:
Model performance under different training set sizes on PGM dataset.
training set sizes on RAVEN dataset.
Training Set Size
CoPINet
DCNet
Training Set Size
CoPINet
DCNet
658 (1.57%)
44.48
60.09 (+15.61)
293 (0.25%)
14.73
15.94 (+1.21)
3,6 (3.13%)
57.69
71.91 (+14.22)
172 (0. 10%)
15.48
18.76 (43.32)
2, 625 (6.25%)
65.55
79.75 (+14.20)
4, 688 (0.39%)
18.39
25.78 (+7.39)
5, 250 (12.5%)
74.53
84.42 (+9.89)
18. 750 (1.56%)
22.07
34.04 (+11.97)
10, 500 (25,0%)
80.92
87.95 (+7.03)
75, 000 (6.25%)
32.39
43.10 (+10.71)
21, 000 (50.0%)
86.43
91.31 (+4.88)
300, 000 (25%)
43.89
50.26 (+6.37)
42, 000 (100%)
91.42
93.87 (+2.45)
12,000,000 (100%)
56.37
68.57 (+12.20)
Table 3 shows the comparison results of CoPINet and DCNet on the RAVEN dataset. It can be seen that DCNet performs better than CoPINet on all subsets. Especially, when half the training samples provided, DCNet achieves comparable accuracy to CoPINet that trains on the full dataset, i.e. 91.31% % vs. 91.42%. In addition, given 658 training samples, i.e. a 64x smaller dataset, DCNet achieves a testing accuracy of 60.09%, which outperforms CoPINet by a large margin of 15.61%.
Table shows the comparison results of CoPINet and DCNet on the PGM dataset. Similar to the results observed in Table 3, DCNet also performs better than CoPINet all subsets, up 11.97% improvement when trained on 18, 750 samples. In addition, when DCNet is trained on a 64 smaller dataset (75K samples). DCNet outperforms LSTM, CNN and ResNet-50. Besides, DCNet achieves better accuracy than Wild-ResNet when trained on 16x smaller dataset. Thus, the results shown in Table 3 and 4 demonstrate the good performance of DCNet when few training samples are provided.
Generalization Test. To measure the generalization ability of our method, we further conduct experiments on RAVEN show the results on unseen figure configurations, see Table 5. Compared to DRT (Zhang et al., 2019a) that trains the model on one figure configuration and then tests it on similar cases, our experimental setup more challenging, as we evaluate the model on all figure configurations, including very different ones. From Table 5, it can be seen that DCNet outperforms CoPINet on figure configurations and only a bit worse on the other 2 configurations, verifying the advantage of our method.
Table 6: Generalization test PGM. neutral
Table 6 shows the generalization test perfor- mance of our method on PGM. Following the experimental setup in MLRN (Jahrens & Mar- tinetz, 2020), we report the performance the interpolation and extrapolation regimes, details about these two regimes are introduced in (San- toro et al., 2018). Although MLRN performs very well on the neutral regime, it is slightly worse than WReN and DCNet on the interpolation. The worst generalization is observed on the extrapo- lation regime, where all compared methods fail to achieve good performance. Thus, robust abstract reasoning model with strong generalization ability still needs further investigation.
Method
interpolation
extrapolation
WReN MLRN
62.6
64.4
17.2
98.0
57.8
14.9
DCNet
68.6
59.7
17.8
Ablation Study. To verify the effectiveness of each contrast module in our method, we conduct experiments on two datasets for ablation study, the results are reported in Table 1. Without the rule contrast module, the testing accuracy degrades 0.84% on RAVEN and 5.12% on PGM, respectively. With the rule contrast module, the proposed method is able to adapt to various logical rules of different RPM problems and further boost the performance, showing the effectiveness of our method.
I We only compare with CoPINet on small data sizes, the reasons being: (1) DCNet outperforms MXGNet with only 1/8 of the training samples on RAVEN dataset (i.e. 84.42% vs. 83.91%) see Table and Table 3; (2) MLRN (Jahrens & Martinetz, 2020) did not report the accuracy on RAVEN and ACL (Kim et al., 2020) did not report the accuracy PGM. Besides, ACI. requires symbolic labels to increase the number of training samples for few-shot learning while do not use such annotations.
8

---

Published as a conference paper at ICLR 2021
Table Generalization RAVEN. Each model is trained row-wise configuration and tested on column-wise configurations.
Method
Avg
Figure
Center
2*2Grid
3*3Grid
L-R
U-D
0-IC
O-IG
66.71
Center
98.05
53.05
49.81
78.65
87.20
61.40
38.35
68.19
2*2Grid
64.55
71.20
67.65
87.25
83.00
62.25
41.45
63.76
3*3Grid
48.90
57.95
76.85
63.95
51.80
71.70
75.20
CoPINet
69.83
L-R
55.50
51.90
54.55
98.25
92.35
76.40
59.85
73.95
U-D
63.25
56.30
55.75
81.90
67.40
94.70
98.35
68.81
O-IC
51.50
46.30
54.60
88.85
64.35
88.90
87.15
68.74
O-IG
44.00
48.60
54.75
75.45
84.30
82.95
91.10
70.76
Center
97.25
54.10
51.80
84.75
86.40
72.90
48.10
74.09
2*2Grid
69.35
76.60
66.55
86.85
88.60
72.45
58.20
68.75
3*3Grid
49.20
56.45
81.75
78.20
78.60
74.80
62.25
DCNet
73.21
L-R
60.30
53.35
59.30
98.50
91.70
83.85
65.45
75.38
U-D
69.30
54.10
56.75
93.25
98.75
85.65
69.85
68.11
O-IC
46.75
40.35
47.30
84.60
88.00
97.80
72.00
66.36
O-IG
48.25
44.60
51.80
79.50
75.75
77.05
87.55
Without the choice contrast module, the testing accuracy of DCNet sharply degrades by 57.07% on RAVEN and 10.81% on PGM. The performance of DCNet on RAVEN degrades more than that of on PGM. The reason for such difference is mainly the characteristic of dataset generation. As there only one attribute change for the answer generation (Zhang et al., 2019a), the wrong choices in RAVEN are more similar to the correct answer when compared to the images in PGM. Thus, similar choices will degrade the correct answer prediction without the choice contrast.
4.5 DISCUSSION
We discuss some key questions on abstract reasoning as follows. (1) Following the recent abstract reasoning works (Santoro et al., 2018; Zhang et al., 2019a; Zheng al., 2019; Zhang et al., 2019b; Wang et al., 2020), we also use neural networks to solve RPM. Since this is a black-box strategy, it does provide certain rules explain how the correct answer is predicted. Developing plainable models on RPM is an important direction in the future.
(2) Because similar problem structure (question and choices) in RPM and Visual Question Answering (VQA), we hope our work may help to improve the performance of some VQA problems and domain adaptation (Cao et al., 2019) on different visual reasoning tasks. (3) Unlike humans who are good solving multiple ab- stract reasoning tasks, existing models mainly focus a single task, e.g., RPM questions in this work. For artificial general intelligence, benchmark datasets on multiple abstract reasoning tasks and appropriate evaluation metrics need further investigation.
5 CONCLUSION
In this work, we propose a Dual-Contrast Network (DCNet) towards solving RPM problems. Specif- ically, rule contrast module is used to compare the latent rules among different rows/columns, and choice contrast module is used to increase the relative differences of candidate choices. Without using auxiliary annotations and assumptions on hidden rules for model training, extensive experi- ments on the RAVEN and PGM datasets show that the proposed DCNet achieves the state-of-the-ar accuracy. In addition, DCNet shows better performance when few training samples are provided. Experiments on generalization test show that robust abstract reasoning models with strong general- ization ability needs further investigation.
ACKNOWLEDGMENTS
This research is supported by the Agency for Science,Technolee, Technology and Research (A STAR) under its AME Programmatic Funding Scheme (#A18A2b0046), Tao Zhuo's 's research is also partially supported National Natural Science Foundation of China (No. 62002188).

---

Published as a conference paper at ICLR 2021
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Zhangjie Cao, Kaichao You, Mingsheng Long. Jianmin Wang. and Qiang Yang. Learning to transfer examples for partial domain adaptation. In CVPR, pp. 2985-2994, 2019.
Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theo- retical account of the processing the raven progressive matrices test. Psychological review, 97 (3):404, 1990.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
George Domino and Marla L Domino. Psychological testing: An introduction. Cambridge Univer- sity Press, 2006.
Hehe Fan, Linchao Zhu, Yi Yang, and Fei Wu. Recurrent attention network with reinforced gen- erator for visual dialog. ACM Transactions Multimedia Computing, Communications, and Applications TOMM, 16(3):78:1-7811 2020. doi: 10. 14553339990
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dokhyam Hoshen and Michael Werman. Iq of neural networks. In htttpss/urrti.orr///////111111/9 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Marius Jahrens and Thomas Martinetz. Solving raven's progressive matrices with multi-layer rela- tion networks. International Joint Conference on Neural Networks (IJCNN), 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Youngsung Kim, Jinwoo Shin, Eunho Yang, and Sung Ju Hwang. Few-shot visual reasoning with meta-analogical contrastive learning. In NeurlPS, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Keith McGreggor and Ashok Goel. Confident reasoning on raven's progressive matrices tests. In AAAI, 2014.
Keith McGreggor, Maithilee Kunda, and Ashok Goel. Fractals and ravens. Artificial Intelligence, 215, 2014.
Can Serif Mekik, Ron Sun, and David Yun Dai. Similarity-hased reasoning, raven's matrices, and general intelligence. In IJCAI, 2018
John Raven and JH Court. Raven's progressive matrices. Western Psychological Services, 1938.
Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In JCML,, 2018.
Snejana Shegheva and Ashok Goel. The structural affinity method for solving the raven's 's progressive matrices test for intelligence. AAAI, 2018.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Ran Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In NewrIPS, 2015.
10

---

Published as a conference paper at ICLR 2021
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research (JMLR), 15(1):1929-1958 2014.
Duo Wang, Mateja Jamnik, and Pietro Lio. Abstract diagrammatic reasoning with multiplex graph networks. In ICLR, 2020.
Ke Wang and Zhendong Su. Automatic generation of raven's progressive matrices. In I/CAI, 2015.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. CVPR, 2019.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: dataset for relational and analogical visual reasoning. In CVPR, 2019a.
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chul Chug-Chun& Zhu. Learning perceptual inference by contrasting. In NeurlPS, 2019b.
Kecheng Zheng, Zheng-Jun Zha, and Wei Wei. Abstract reasoning with distracting features. In NeurIPS, 2019.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, 2018.
Tao Zhuo and Mohan Kankanhalli. Solving raven's progressive matrices with neural networks. arXiv preprint arXiv:2002.01646, 2020.
Tao Zhuo, Zhiyong Cheng. Peng Zhang, Yongkang Wong, and Mohan Kankanhalli. Explainable video action reasoning via prior knowledge and state transitions. In ACM Multimedia, 2019.

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 ROW AND COLUMN FEATURES
In this work, we consider both the row-wise and column-wisg features, for each RPM problem æ;, the formed rows and columns C₁ can be denoted as:
101,1: 33,3);
Ri,4, j = (Zi,2, 215,, F1s).
Ti (1i,4. Dis,, Ti,6);
2
C.j
2 (5)
(mi,7 Fi,8, Fi,)+
2 3,
( Ti,6 Fij+6),
j 2 3,
where Ta.j denotes the j-th row of ris Cijj denotes the j-th column of e. j. E {1, ... 10). 2i.++ represents (j + 6)-th image of az.
A.2 IMPLEMENTATION DETAILS
Similar to the experimental setup of previous works (Zhang et al., 2019b; Zheng et al., 2019; Wang al., 2020), for general performance evaluation, we train the models on the training set, tune the parameters on the validation set, and report the accuracy on the testing set.
In the network architecture, the input dimension N C input channel C' = 3 (as there are 3 gray-scale images in a row/column), image size H 96 and W 96. The output dimension of the feature encoder 128 feature dimension of MLP layer is 512.
W, where batch size 32,
4 4 Besides, the input H W.
A.3 EXAMPLES AND RESULTS ON TWO DATASETS
Figure shows the our testing accuracy of different training epochs, the proposed method achieves an accuracy 83.08% on RAVEN with only one epoch, which has outperformed many state-of- the-art methods, e.g., ResNet18+DDD (Zhang et al., 2019a) (59.56%) and LEN (Zheng et al., 2019) (72.90%), see Table Compared to the most recent method MXGNet (Wang al., 2020) (83.91% % with more than 10 training epochs), our model is more efficient, since we exploit the inherent struc- tures of the RPM problems with dual-contrast.
Figure 4 presents two examples of the RAVEN dataset (Zhang et al., 2019a) with two different figure configurations. Besides, Figure 5 shows two examples the PGM (Santoro et al., 2018) dataset with different line and shape configurations. To demonstrate the robustness of the proposed method, the network architecture and all training parameters are kept the same on both two datasets.
RAVEN
PGM
100
90
Accuracy (%)
80
70
60
50
40
5
10
15
20
epoch
Figure 3: Testing accuracy of different training epochs on two datasets.
12

---

Published as a conference paper at ICLR 2021
oo
60
Problem Matrix
?
?
00
7
8
Choices
10
12
11
12
13
16
Out-InGrid (O-I)
3r3 Grid
Figure 4: Two examples in RAVEN dataset (Zhang et al., 2019a). The correct answer is 16 (the left panel) and 11 (the right panel).
Problem Matrix
?
?
Choices
11
12
10
12
13
15
16
14
15
16
OR relation lines and size of shape object
Background shape objects
Figure 5: Two examples PGM dataset (Santoro et al., 2018). The correct answer is 16 (the left panel) and 15 (the right panel).
13


Reviews:
[{"review_id": "yBJfOmcrOq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new approach for abstract reasoning and explores it in the context of the RPM task. In contrast to other competing approaches, the authors seek to build into the model as few assumptions as possible to keep the model general and not specific to the specific problem or to particular annotations or supervision signals. The general capability that they seek to incorporate into the model is the ability to effectively compare and contrast candidates in tasks that require choosing the best fit. \n\nThe task is important, presented carefully and is well-motivated and the paper is clear and easy to follow. The related work section covers the necessary backgrounds including both visual reasoning and general and the RPM task in particular. It also presents the existing methods and discuss their disadvantages compared to the new approach - mainly their stronger reliance on supervision to learn good, semantic or disentangled features that will help most in addressing the task, or particular assumptions other approaches make about the specific properties or structure of the RPM task that may not hold in others or more general cases. \n\nThe task is also clearly presented and the authors explain how it consists of two main sub-tasks: (1) identifying the rule that links the existing rows or columns, and (2) comparing the candidate answers to choose the best fit. They propose network modules to solve each of these tasks correspondingly: a rule-contrast module, and a choice-contrast module, to complete each of these sub-tasks. For both  sub-stack they use clustering to find contrast and similarities between elements. This idea is simple, nice and quite novel I believe in these contexts. \n\nExperiments are performed over two datasets of RAVEN and PGM, and achieve 5.77% improvement on average over state-of-the-art, and larger improvement in scarcer-data regimes, which the authors particularly focus on. They provide useful information about the datasets, baselines and implementation details and experiment settings, along with an ablation study to give further insight into the benefits of particular aspects of the model.\n\nA particular comment that I have is that it seems that the model considers each answer by replicating the board k times for the k candidates, and considering each such completion alternative. While working for the RPM case, it won’t scale to problems where there’s a larger number of candidates, potentially even not-bounded. Restructuring the network to be able to preprocess as much as it can about the board so to reduce the amount of answer-dependent computation may be very useful to make the approach more general and efficient. I’m also looking forward to hearing about further applications of this idea as discussed at the end of the paper in the general VQA task or on other abstract reasoning, for instance such as the Abstraction and Reasoning Challenge.\n\nOverall, Great work!\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pD3aiZwoINn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a neural network based approach called Dual-Contrast Network (DCNet) to solve Raven’s Progressive Matrices (RPM). The approach consists of a rule contrast module that compares the latent rules between the unfilled (third) row/column and the filled (first and second) rows/columns, a choice contrast module that helps in picking the correct choice among the given eight choices, and finally uses a 2-layer MLP to predict scores for the choices. Different from previous approaches, the only supervision used in the proposed approach is the ground-truth choice. The approach achieves state-of-the-art performance on RAVEN and PGM datasets.\n\n——————————————————————————————————————————————————————————————\n\n\nStrengths:\n\nS1) The approach achieves state-of-the-art performance.\n\nS2) The approach requires the least amount of supervision — the ground-truth option, unlike previous works that require auxiliary annotations or assumptions.\n\nS3) The paper shows the contribution of the approach’s two main modules using ablation studies.\n\nS4) The approach outperforms previous works when trained with smaller training datasets. \n\n——————————————————————————————————————————————————————————————\n\n\nWeaknesses:\n\nW1) The results lack standard deviation or error bars in the tables. Are the differences from previous works statistically significant?\n\nW2) I think the paper would benefit a lot with further analysis into the results — what type of mistakes do previous methods tend to make which the proposed approach is able to overcome? The paper should contain qualitative examples showing what options previous methods predict, especially when they fail and the option predicted by the proposed approach. \n\nW3) Similarly, the paper should analyze the failure modes of the proposed approach. Is the task solved? If not, what obstacles still remain?\n\nW4) How does the computation complexity of the proposed approach compare with previous works?  \n\nW5) The paper claims that the ability to solve RPM correlates well with human intelligence. But the few examples shown in the paper seem super hard. How well do humans perform on these datasets? I believe it should be much lower than that of the previous works as well as the proposed approach. If that’s the case, why do we want to build better and better approaches for this task?\n\n\n——————————————————————————————————————————————————————————————\n——————————————————————————————————————————————————————————————\n\nUpdate after rebuttal: I thank the authors for their responses to my questions. They satisfactorily answer most of my concerns. Overall, I agree with the concern that the proposed approach is specific to RPM and it's unclear how well it (or parts of it) would generalize to other problems but I think the approach is quite interesting, novel and achieves state-of-the-art results. Hence, I think the contributions of the paper are significant for acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "b-P1pswHCov", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n---\nThis paper proposes a new approach that improves performance on the Raven's\nProgressive Matrices problem without auxiliary annotation by using two\ncomplementary principles to structure inference.\n\n(introduction)\nThe RPM problem is a visual reasoning problem where a 3x3 grid of images is provided\nwith one empty slot and 8 choices to fill in to that slot. The goal is to chose\nthe correct image via reference to the logical structure of the grid.\nTwo structures stand out:\n1) The same rules apply to every row/column.\n2) Many of the incorrect choices obey some, but not all of the rules.\n\n(approach)\nThis paper incorporates both structures into its neural net design.\n(structure 1) Each of the 2 provided rows and the 8 possible completions of the 3rd row\nare embedded into a common space, and each choice is represented as a difference\nbetween that choice's embedding and the embedding of the 2 known rows.\n(structure 2) Next the approach considers the average embedding of the choices\nand represents each choice by its position relative to that centroid.\nThis representation is fed through an MLP-sigmoid configuation to get a score\nfor each choice and perform the final classification.\n\n(experiments)\nExperiments compare to many recent state of the art baselines.\n1. The proposed DCNet outperforms all baselines, without or without auxiliary supervision, on the RAVEN dataset.\n2. The proposed DCNet outperforms all baselines, except those with auxiliary supervision, on the PGM dataset.\n3. Performance degrades when the amount of training data is decreased, but not as much as CoPINet.\n4. Ablations show that both of the structures incorporated into DCNet help improve performance.\n\n\n\nStrengths\n---\n\nThe RPM problem is interesting.\n\nThe dual contrast approach is well motivated.\n\nThe proposed DCNet outperforms relevant baselines on both datasets.\n\nAblations help understand the importance of the two types of structure and how performance improvements are distributed over subsets of the dataset.\n\n\n\nWeaknesses\n---\n\nAll the weaknesses I found in this paper were pretty minor. A few parts weren't as clear as I would have liked and the paper probably won't have much significance outside of the literature on RPM.\n\n\nClarity:\n\n* The notation in the 2nd paragraph of 3.1 is a bit confusing. A figure or more specific example might help clarify. Is r_i,j a single image or all three images from a row concatenated together? Is each input image a different channel or are they concatenated?\n\n* Multiple times the paper mentions that a disadvantage of the CoPINet approach is that it requires knowing the maximum number of possible rules and attributes. It would be nice if the paper elaborated on what kind of conern this is. Is this bad because it's not satisfying? (I agree that it isn't.) Or is this bad because of some practical issue? (Does it have a real effect on performance in the considered datasets?)\n\n* Is one of the two contrasts generally enough to solve the problem in theory? Do these two contrasts just make inference easier, or do both really need to be considered by any system that solves the problem in general?\n\n\nSignificance:\n\n* The correlation of RPM scores with general intelligence in humans makes the problem interesting on its own; more than a toy problem. But the approach is still very specific. I doubt very much from this approach will generalize to other problems.\n\n* I understand what makes the approach different, but I don't have a qualitative assessment of how performance changed. What sorts of problems do other models fail at that this DCNet succeeds at? Maybe there's not an intuitive answer to that question which has been found in investigations so far. If so, I'd find it useful for the paper to say that.\n\n* The performance improvements aren't very large, though I think they are probably significant, and that this approach would be a useful contribution even if it didn't beat all the baselines. Still, it would be useful to get a sense for the variance over random initializations and bootstrapping (i.e., confidence intervals).\n\n\nOther:\n\n* The choice of sigmoid instead of softmax in section 3.4 is a bit odd. Exactly one of the choices is correct, so it seems more natural to normalize them together instead of independently.\n\n\nPreliminary Evaluation\n---\n\nClarity - The paper is pretty clear overall, though some parts could be improved.\nQuality - The approach is well motivated and the experiments include lots of ablations and comparisons to baselines that make the conclusion well supported.\nSignificance - This will likely have some small significance in the RPM literature. Improvements are small and the approach is fairly specific to RPM. However, the improvements are meaningul and the approach is interesting.\nOriginality - Previous approaches to RPM problems have taken advantage of the problem structure, but this specific structure in this way.\n\nAll of the factors above clearly point to acceptance, though general interest in this paper will be limited.\n\n\n\nSuggestions\n---\n\n* The related work specifies that CoPINet also takes advantage of contrast. Since contrast is the central motivation of this paper, I wanted more detail about how exactly they do that. It would be nice to expand the related work to provide a bit more detail about contrast in CoPINet.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUtq7Q4YqjE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an inductive-bias contrast module that improves performance on Raven Progressive Matrices datasets. The proposed module makes use of the prior knowledge that rules can only exist in rows and columns of the diagram matrix. \n\nWhile the paper claims to out perform previous SOTA, I have to point out that the authors are missing citations of the more recent SOTA methods on PGM and RAVEN datasets, such as [1] and [2]. If you take into account these more recent results, the proposed model only achieves slightly better results on RAVEN dataset, while not achieving SOTA results on PGM dataset.\n\nThere is some (but limited) novelty in using the prior knowledge of RPM rules to design inductive-bias module that works for this specific tasks. However the module is tailor designed for RPM tasks, which means it cannot be easily adapted to other types of reasoning tasks. Thus I feel the paper will only have impact within the RPM community, but little impact in the general ML and Reasoning community.\n\nWhile the authors reported scores on PGM and RAVEN datasets, the authors did not report generalisation performances on various generalisation splits of PGM and RAVEN datasets. These missing results are in fact more interesting, as PGM and RAVEN datasets are designed for measuring generalisation capability of neural models.\n\nIn summary, I vote for reject as this paper is not citing the more recent SOTA results, is tailor-designed only for RPM tasks, and lacks experiments on the generalisation split of PGM and RAVEN datasets.\n\nClarity:\nThe paper is clearly written and easily understandable\n\n-----------After Rebuttal--------------\nI decide on up my score by 1 since indeed the authors are only required to cite papers that are peer-reviewed and published before 2nd August. I decide not to up my score any further because my other two concerns remain. I appreciate that the authors performed additional experiments. While the generalization performance is better than MLRN, it does not outperform other baselines such as MXGNet. \n\n\nReference:\n1. Kim, Youngsung, et al. \"Few-shot Visual Reasoning with Meta-analogical Contrastive Learning.\" NeurIPS 2020 (2020).\n2. Wu, Yuhuai, et al. \"The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning.\" arXiv preprint arXiv:2007.04212 (2020).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new architecture for solving Ravens Progressive Matrices (RPM), a well known form of visual reasoning problem. The method relies on an operation that directly compares the final row and column (completed with different candidate answers) with the first two rows and columns. Doing this allows the network to perform better than previous approaches when measured on an in-distribution test set on two RPM datasets, RAVEN and PGM. 

As the reviewers pointed out, a strength of this work is the strong performance on the neutral split of these datasets and the fact that the methods do not (unlike some other approaches) require access to any annotations from the dataset other that knowledge of the structure of the RPM task and access to the candidate answers and the correct answer. 

However, a noted weakness is the fact that the network reflects the structure of the task more directly than other approaches, which means that the insight is specific to the problem of solving RPMs. Another weakness is that the authors focus on the neutral (in-distribution) splits. Reading the PGM paper, it is clear that the neutral split is not really the main focus of that dataset (it accounts for only 1/7 of the dataset), which seems to have been specifically developed as a benchmark for out-of-distribution generalisation. Indeed, the whole point of the RPM task is to measure the ability to induce abstract rules and principles from pixels, but without measuring out-of-distribution generalisation, can we really claim that any model has induced a 'rule'? 

The authors mitigate this issue to a small degree during the rebuttal by adding scores on the 'interpolation' and 'extrapolation' splits of the PGM dataset, but still do not consider the other splits where rule application is most clearly tested. 

I note that the weakness described above also applies to lots of other published work involving PGM and RAVEN datasets. 

In summary, this is a well-executed, neat piece of work that shows a better way to fit a large dataset by incorporating knowledge of the structure of the data into the task. Because it does not consider the full benchmarks, only the in-distribution splits, it falls short of showing that this enables better induction of abstract principles or rules. On the majority opinion of the reviewers, and because there are no scientific flaws in the work, I recommend acceptance with weak confidence pending wider calibration across the program. 
Decision: Accept (Poster)