Abstract Visual Reasoning by Self-supervised Contrastive Learning
Authors: Weiwen Lu
Aihua Yin
Sidong Wang
Hongzhi You
Ru-Yuan Zhang
Dahui Wang
Zonglei Zhen
Xiaohong Wan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Neuro-symbolic models of artificial intelligence (AI) have been recently developed to perform tasks involving abstract visual reasoning that is a hallmark of human intelligence but remains challenging for deep neural network methods. However, most of the current neuro-symbolic models also rely on supervised learning and auxiliary annotations, different from human cognitive processes that are much dependent on the general cognitive abilities of entity and rule recognitions, rather than learning how to solve the specific tasks from examples. In this work, we propose a neuro-symbolic model by self-supervised contrastive learning (NS-SSCL) with unique and invariant representations of entities and rules in the perception and reasoning modules, respectively, to solve Raven’s Progressive Matrices (RPMs) and its variant, a typical type of visual reasoning task used to test human intelligence. The perception module parses each object into invariant representations of attributes. The reasoning module grounds the representations of object attributes to form the latent rule representations also through SSCL. Further, the relationships between the neural representations of object attributes and symbols used for rule reasoning are coherently mapped. Finally, the scene generation engine aggregates all attribute and rule representation distributions to produce a probabilistic representation of the target. NS-SSCL obtains state-of-the-art performance in unsupervised models to solve the RAVEN and V-PROM benchmarks, even better than most of the supervised models. The success of the proposed model suggests that construction of general cognitive abilities like humans may render the AI algorithms to solve complex tasks involving higher-level cognition such as abstract reasoning in a human-like manner.

Under review as a conference paper at ICLR 2023
VISUAL ABSTRACT REASONING SUPERVISED CONTRASTIVE LEARNING
BY SELF-
Anonymous authors
Paper under double-blind review
ABSTRACT
Neuro-symbolic models of artificial intelligence (AI) have been recently devel- oped to perform tasks involving abstract visual reasoning that is hallmark hu- man intelligence but remains challenging for deep neural network methods. How- ever, most the current neuro-symbolic models also rely on supervised learning and auxiliary annotations, different from human cognitive processes that are much dependent on the general cognitive abilities of entity and rule recognitions, rather than learning how to solve the specific tasks from examples. In this work, we pro- pose neuro-symbolic model by self-supervised contrastive learning (NS-SSCL) with unique and invariant representations of entities and rules in the perception and reasoning modules, respectively, to solve Raven's Progressive Matrices (RPMs) and its variant, a typical type of visual reasoning task used to test human intelli- gence. The perception module parses each object into invariant representations of attributes. The reasoning module grounds the representations of object attributes to form the latent rule representations also through SSCL. Further, the relation- ships between the neural representations of object attributes and symbols used for rule reasoning are coherently mapped. Finally, the scene generation engine aggre- gates all attribute and rule representation distributions to produce a probabilistic representation of the target. NS-SSCL obtains state-of-le state-of-the-arr the-art performance in supervised models to solve the RAVEN and V-PROM benchmarks, even better than most of the supervised models. The success of the proposed model suggests that construction of general cognitive abilities like humans may render the Al al- gorithms solve complex tasks involving higher-level cognition such abstract reasoning in human-like manner.
INTRODUCTION
Abstract reasoning is essential for human intelligence. The capability of abstract reasoning in hu- mans domain-general and can be effectively estimated by simple visual reasoning task test, such as Raven's Progressive Matrices (RPMs) (Raven et al. 1938). The premise of RPMs is that it does not rely on fomain-speciffi knowledge or verbal ability, but the test performance is diagnostic of verbal, spatial and mathematical reasoning abilitics (Carpenter et al. 1990, Snow et al., 1984). Hence, it is believed that solving RPMs by artificial intelligence (Al) might htght be cornerstone toward artificial general intelligence (AGI) (Bilker et al.) 2//2angag et al. 2019a). Although deep neural networks by supervised learning have achieved a great success in visual categorizing, such a neural network architecture is versatile for visual reasoning (Hoshen & Werman 2017 Barrett al. 2018). The core feature in visual reasoning tasks (e.g., RPMs) is that the rules governing the or- ganization of entities are semantically defined by the spatiotemporal relations between entities, but rather intrinsic to the entities per se (Lovett et al. (007). Thereby, the semantics of entities and the underlying rules are weakly connected. This leads to end-to-end deep-learning (DL. algorithms are inefficiently to concurrently learn both properties. Although a number of variant DL models have been developed to achieve high performance superior to humans (Zhang et al.) 2019b: Zhuo & Kankanhalli 2020 Mandziuk & Zychowski 2019: Zhuo & Kankanhalli 2021 }. these models are monolithic, lacking clear distinctions between the processes of perception and reasoning like in humans (Mareus & Davis 2020; Fodor & Pylyshyn 1988). To mimic the human-like processes involved in solving abstract reasoning tasks, some neuro-symbolit (NS) methods have been recently proposed combine the deep neural network for the perception module with reasoning module

---

Under review as a conference paper at ICLR 2023
for symbolic logic execution (Yi et al. 2019: Mao et al. 2019 Zhang al. 2021). However, these models also need to learn the connections between the contexts of instances and the supervised answers from scratch as DL models.
In striking contrast, humans who have never previously met the problems can soon solve the ab- stract reasoning tasks. Humans do not rely the task-specific experiences in solving the tasks, but their prior general cognitive abilities, namely, object and rule recognitions. Humans, even at a very early stage of life, can recognize tons of objects and their attributes (Spelke) [990), and soon later recognize the rules governing the world and apply these rules to new situations (Gopnik et al., 2004). For these reasons, the RPM tests are capable of evaluating human's 's general reasoning abil- ities (Marcus & Davis 2020; Fodor & Pylyshyn| 1988). Although it remains open questions about how humans learn and form the object and rule representations, a critical feature of these general cognitive abilities granted for abstract reasoning is that the attribute and rule representations are unique and invariant in the brain (Li & DiCarlo 2008 Mansouri et al., 2020). For instance, we recognize the same color of 'green' from different objects and recognize the latent rule governing the color relationship across a set of objects. Thus far, it remains challenges to design an automatic Al algorithm behaving like humans in these tasks.
To better investigate abstract visual reasoning, RAVEN (Appendix AA) and other RPM-like datasets have been proposed. In RAVEN dataset, each RPM problem consists of 9 panels in form of 3 3 matrix with 8 context panels and a missing panel at the 9th entry (Matzen et al. 2010: Zhang et al.] 2019a). The goal of the task is find out the correct answer from 8 candidate panels that completes the matrix with satisfactions of the latent rules governing the organization of object attributes in the three continuous panels within each row and are consistent across the three rows in the matrix (Figure D. Overall, the task requires two independent cognitive abilities of visual perception and rule reasoning. If visual perception module can perfectly recognize all object attributes, then the process of identifying the latent rules becomes plain and reduced to exhaustive search in the rule space (Matzen al. 2010; Zhhng et al., 2019a). Different from most of visual perception tasks, both the object attributes and rules are required to be identified. This is difficult for deep neural networks, and so far also remains challenges for the NS models. Additionally, the V-PROM task (Appendix [F) is also RPM-like task but with natural images (Teney et al., 2020). This new task increases the difficulty of visual perception.
Inspired by the above-mentionel human cognitive processes in abstract reasoning, here demon- strate a human-like NS model can solve abstract visual reasoning in a human-like manner. To build up non-verbal visual reasoning ability for AI model the basis of the general cognitive abili- ties in object and rule recognitions, we move a step further towards a NS model without supervisions of the answers or annotations, but with a self-supervised contrastive learning (SSCL) method tablish both the object and rule recognition abilities, denoted as NS-SSCL. The motivation of this unsupervised approach is make representations of the same attributes and the same rules are as close as possible across different objects and problems, respectively. Critically, the mapping from the neural embeddings of object attributes and the symbols used for rule logic execution can be fur- ther established by virtue of their stable representations, even though these representations learned by SSCL, are not necessarily aligned well with ground truths the tasks due to the unsupervised nature. Notably, the current model relies on the probability codes of the discrete symbols of object attribute values (Figure [D as similar as used in the probabilistic abduction and execution (PrAE) model (Zhang et al. 2) However, the PrAE model used supervisions of the correct and incor- rect answers from the candidate panels, and also relies on the ground truth of rules contained in each instance auxiliary annotations. In this work, without these ample supervisions, NS-SSCL obtains state-of-the-ar performance accuracy in the unsupervised models on the context of instance, but without the candidates, in solving the RPM-like tasks, even better than most of the previous supervised models.
2 RELATED WORK
Object representations is critical to correctly recognize the exact object attributes in visual reasoning tasks, as the latent rules governing the context are defined by these features. Although deep neural networks are versatile to fit any desired function constrained by the loss function, the neural embeddings of latent object representations are too flexible to fit well with the semantics of
2

---

Under review as a conference paper at ICLR 2023
object attributes used to form the latent rules, such as types, sizes, and colors in the RAVEN dataset. Instead, these object attributes are often mixed in the embeddings. Recent studies have proposed
using variational autoencoder ( (VAE) (Mańdziuk & Zychowski| 2019; Klein & Nabi 2020 Higgins eet al. 2017; Burgess et al. 2018 Pekar et al. 2020) and neural-vectors (Hersche et al.
2022 to
disentangle the blended representations of visual attributes. SSCL has been used to discover better representations by comparing the augmented data from the same or different images (Chen et al.)
2020 He et al. (2019). We here leverage this unsupervised method to shaping the neural embeddings to largely comply with the semantical dimensions of defined object attributes in tasks.
Visual reasoning Most of supervised models designed to solve visual reasoning tasks mainly fo- the visual reasoning process (Hoshen & Werman 2017 Barrett et al.) 2018]] Zhang et al.
2019b Zhuo & Kankanhalli 2020 Mańdziuk & Zychowski, 2019 Zhuo & Kankanhalli (0021),
as the baselines of DL models fail solve these high-level cognitive tasks. A common motiva- tion for visual reasoning models is to learn relational representations of latent rules governing the task context by maximizing similarity between analogical relations and minimizing similarity be-
tween non -onaaoaloii relations (Barrett et al.) 2018 Zhang et al. [2019b] Jabrens & Martinetz. [2200] Malki' nski & Ma ndziuk 2022 Wu al. 2021 Kiat al. 2020; Kim et al, (020). This is achieved by comparing the relational representations of the correct answer with that of the incorrect answers. Instead, it can also directly utilize the relation structures within the context only and with no super- visions. We here leverage the SSCL unsupervised strategy the contexts of instances to training the rule reasoning module. Differing from the pairwise relations discriminator (PRD) model (Kiat et al. 2020), we here further use SSCL to learn the neural representations of object attributes and additionally map these representations with the symbols used for rule reasoning.
Neuro-symbolli models It arrives consensus that the NS architecture might be proper model to provide interpretability and generalizability in solving abstract reasoning tasks (Yi et al.) 2019 Mao al.] 2019; Zhang et al., 2021; Ding et al. 202Tb). Unlike the monolithic DL models. the NS models are composed of perception module and a reasoning module (Zhang et al. 2021 Ding et al. 202Tb). Nonetheless, it remains challenges to train the NS models in an end-to-end supervised form. Thereby, auxiliary annotations of the latent rules are required to constrain the rule representations in the PrAE model (Zhang et al., 2021). In striking contrast to PrAE, we here use SSCL to separately train the neural representations of object attributes and rules separately in the perception and reasoning modules of the NS model.
3
METHODS
The current work here mainly focuses on the learning process, rather than the NS architecture (Fig- ure T (Chen et al., 2020). This proposed model put forward advances in Al solving abstract visual reasoning as follows. First, inspired by human cognitive processes of solving these tasks, we are motivated to train a NS model equipped with the general cognitive abilitics of visual object attribute recognitions and rule recognitions, both of which are readily acquired in humans prior to solving RPMs (Marcus& & Davis 2020 Fodor & Pylyshyn| 1988 Spelke 1990 Gopnik et al. 2004 Li & DiCarlo 2008 Mansouri al. 2020). To acquire the two general cognitive abilities, we leverage SSCL on the context set, but not the candidate set or the answer in each task instance (Figure 2). Second, we do not require the NS model have prior knowledge about either the dimension of object attributes or the underlying rule information, including their formats and dimensions (Zhang al. 2021). This provides the current method with flexibility in solving other visual reasoning tasks. Third, due to the perception and reasoning modules are separately and sequentially trained, the representation-symblee
mapping from the perception module to the reasoning module is easily optimized, even though both are not perfect due to the lack of ground truths. Fourth, the training is not relied on the candidate sets and the selection of answer from candidates is generative from the context. In brief, training the current model on the context of instance consists of four sequential processes (Figure 2 : (1) Pretraining for semantic concepts; (2) Rule Induction Network; (3) Sym- bolic rule knowledge; (4) Fine-tuning, while solving the tasks consists of three processes (Figure []: (1) Object attribute perception; (2) Rule reasoning; (3) Answer generation (see implementation details in Appendix B).
3

---

Under review as a conference paper at ICLR 2023
3.1 PRETRAINING FOR SEMANTIC CONCEPTS
For the RAVEN task (Matzen et al. 2010; Zhang et al. 2019a), we use an object-based convoluted neural network (CNN) parcel the objects into attributes in the perception module (Figure [ The attributes include existence of objects, types, sizes and colors, while the representations of types, sizes, and colors are subject to whether existence of objects is true. We assume that the model has such prior semantic concepts about type, size and color. we thus generate standard forms of objects for types, sizes, and colors, respectively (Figure )). For example, a standard form for the color attribute is circle of standard size which has the same color as the original object. So, if two objects share the same color, they should have similar standard forms of color (Chen et al. 2020)) For each attribute, we train the perception module to have similar representations for objects and their corresponding standard forms. Further, the perception module distinguishes different attribute values. The loss function has a similar form in contrastive learning except that the output of perception module is coded in a probability form:
D(p. 9:. Ta)
D(qs, 9s: Ta)
Σ -log Losso(1) = N
Losss(1) = N 1=1
llog
==1 E/=1
P(p Ta)
N Σイ=1
D(gi qs. Ta)
Loss;, (I) -log| H
Ei 9:
)} with N = =1,, D(,.q, cp.l/T
EN %:
Lossa ALoss, -1 A)Losse()) + ((lc)++Lossss pLossa(T)
(1)
where is the set of inputs in the batch, a certain attribute, Ps. 9: are the probability output of the äth original object in I and corresponding standard form respectively, is the temperature parameter, X. jl are weight parameters and H means entropy. The term Loss, is the contrastive loss between original objects and standard forms, the term Loss, is the contrastive loss with standard forms, and the term Loss;, is to avoid a trivial situation that p; Pj = 1 holds for all inputs.
In addition, the chosen part of image fed to the perception module may contain no object. This would affect the object-based pretraining and later the reasoning. So, we add term for existence to the loss and only inputs with existence of objects to calculate the attribute loss as listed above. The existence loss is calculated by negative tog-likelihood and the labels are obtained by simple image processing methods. The final loss is:
Loss Losstype (Ie) Losssize(TT) + LosS (Tc) + Lossenist (T)
with I, = {i E I| an object exists in i}
For simplicity, the perception module is no longer updated after pretraining. Different objects' attributes within a panel are further integrated to form the panel information.
For the V-PROM task (Teney et al.) 2020), we used ResNet (He et al. 2015) network pretrained on ImageNet (Russakovsky et al. 2014) and further finetuned with supervision. However, the corresponding order of supervised labels of object attributes is randomly shuffled from the ground truths. That is, similar to the perception module in the RAVEN task, the perception module also only knows whether two attribute values are same or not, but do not know the exact attribute value (ground truth).
3.2 RULE INDUCTION NETWORK
We also build model having the ability of distinguishing rules through SSCL (Figure 2). The unique feature of rules governing the RPM-like tasks is that the latent rule for each attribute row-wised and consistent across the rows in the matrix of a context instance. Thereby, the same attribute should share the same rule across the first two rows of each instance (intra-problem rule analogy as the positive examples), but unlikely share the same rule across the rows paired between (inter-problens problem rule non-analogy as the negative examples) (Wu et al. 2021 Kiat et al. 2020). We then use this meta-rule knowledge about the task structure to constraint and induce rules. Differing from previous supervised models, we do not require the model to precisely produce representations of the rules, but only require it to discriminate the rule relationship of given pair
different problems (inter-
4

---

Under review as a conference paper at ICLR 2023
bock Natwork
Fure: Eaggidan
PFageste
Score used coraitanty
Telr
contest panels
Prididir
candidate panels
Figure 1: A schematic of the model architecture. See the main text for details.
of rows. For training, we only use the first rows in each context of instance. Therefore, we have 2N rows for batch of size N, and have 2N x 2N pairs of rows. We concatenate the perception output of each pair as the input of RIN. If the two coupled rows come from the same instance, then the pair is labeled 1, indicating they share the same rule, otherwise, it is O. The RIN is MLP for each attribute and the loss function is weighted
as follows:
Loss
RIN
Σ
Wiish..
[Sa (row. roww!") 8z.01
11-112
where f stands for RIN, í are row indexes, x, y are RPM indexes, Wij,ds. are weight parameters and 8 means the Kronecker delta function.
On the basis of the relationships across the rows, we build up an unweighted and undirected graph network for each attribute where node stands for context of instance and edge indicates shared rule between a pair of instances Operationally, only a subset of samples (e.g., 512) are chosen build up the graph. An edge is added into the graph when pair of the first rows from two different instances have RIN outputs above the threshold 0.5:
Ga =V, Ea)
V C RPMs, E. = (V
y. fa row?"") 0.5 v 1,2]
To classify the rule structures, we use the label propagation method (Raghavan et al. 2007 of community detection to find the communities, clusters, in the graph within which the nodes are more frequently connected to one another and less the rest the network. Notably, the current contrastive learning approach by RIN does not directly result in discriminations of the rules, but the problem identities. The facts that different attribute values with the rule of "distribution three" or "union" do appear in the instance, lead the problems with such rule are isolated with one another the graph (Figure
3.3 SYMBOLIC MAPPING
The model can detect the rule categories of each attribute. But since it does know the order or exact value of each attribute, it cannot specify the exact rule, particularly the progression or
In fact, we can directly use the RIN to solve RPMs with a method similar to pairwise relations discriminator (PRD) (Kiat et al] 2020) proposed in the previous study to compare the similarity of relations between rows, namely compare the rule-consistency between the third row and the first two rows by individually adding each candidate into the third row of the RPM context.

---

Under review as a conference paper at ICLR 2023
arithmetic rules. Generally, rule of each attribute can be viewed of ordered tuples. For example, the arithmetic plus rule can be viewed as:
(5) To identify the exact rules, we further map the series of representations of each attribute in the perception module to a set of tokens or symbols used in the rule reasoning module. To do so, we train MLP map the neural representations to the symbols used for reasoning the rules. Specifically, we run all possible rules with the set of attribute values under the assumed symbolic knowledge, we then calculate the probability of rules by aggregating all probabilities of attribute value distributions. For example, the probability that the panels of row (a, b, c) follows the arithmetic plus rule on certain attribute can be calculated as:
r+ = ((1, 1),, (1.3),
P(r+la,b,c)=
Σ z(a= =
(6)
(..v.))Er+
Hence, what we do is just to properly assign the rules to corresponding problem communities found in the graph by maximizing the probability of the assignments, i.e., the probability that all instances follow corresponding assigned rules (Figure (2). In theory, an improper assignment may cause conflicts and thus lead to a higher loss value. We then train MLP map the representations of object attributes to the corresponding symbols.
3.4 FINETUNING
Further, we use rules predicted by symbolic reasoning as pseudo-labels to finetune the weights of the MLP that maps the neural representations to the symbols (Figure 2)). To predict the rule, we normalize the rule probability and choose the rule with highest normalized probability in PrAE (Zhang et al. 2021). Normalizing rule probability means both to balance the numbers of possible situation among rules and to make sum of rule probability 1.
Pnorm (T) EVEER Q(r') with Q(r) P(r) |r]
Q(r)
where E denotes the set of possible rules. The loss function the negative log-likeihhod of the predicted rules. This approach of self-correction usually further improves performance.
3.5 ANSWER GENERATION
Finally, the model also predicts the rules of each attribute through symbolic reasoning, and accord- ingly generates a predicted probability distribution of attributes. Meanwhile, the probability distri- butions of object attributes of the candidates are also obtained by the model. We then compare the Jensen-Shannon divergence (JSD) between the predicted distributions with those of the candidates. The candidate with smallest divergence is then selected as the answer.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We test NS-SSCL on the RAVEN with the three benchmarks of RAVEN, I-RAVEN and RAVEN-
FAIR (Zhang et al. 2019a Hu et al. 2021 Benny et al., 2020). The three benchmarks share the same contexts, but different candidate sets. I-RAVEN and RAVEN-FAIR are later designed to avoid the drawbacks in the original RAVEN benchmark in that the candidate panels have shorteuts to identify the correct answers even without context information (Hu et 2021). Because our training processes do not access the candidate sets, the training processes had no differences in these three benchmarks. Also, test NS-SSCL portion of V-PROM dataset with object counts as the attribute for rule reasoning (Appendix
4.2 EXPERIMENTAL RESULTS
Evaluations of general performance in I-RAVEN First, evaluate the proposed model's per- formance on I-RAVEN in comparison with some representative supervised and unsupervised mod- els. To the best of our knowledge, the NVSA (Hersche et al. 2022) and SCL. (Wu al.) 2021)

---

Under review as a conference paper at ICLR 2023
A
D
Pre-training semantic concests
Pieudo
Fine-tunting
prediciion
Rule induction module
W
Symbolic rule knowledge
dffrerttu Nit
laarning
(0223) (LSA).. (2L.) (3L22)
Community detection
(LLL)) (222).)
Figure The components of NS-SSCL. (A) Pretraining semantic concepts of objects; (B) Building-up up Rule Induction Network; (C) Constructing symbolic rule knowledge; (D) Fine-tuning.
Figure 3: The representations of object attributes in the perception module. (A) The embeddings of object attributes (t-SNE) (B) The confusion matrix of attributes.
Figure 4: The representations of rules in the rule reasoning module. (A) The graph built by RIN and the communities with pre- dicted rules (B) The confusion matrix of rules.
models the currently two state-of-the-art art supervised models (Malki' nski & Ma ndziuk 2022, We also compare with the PrAE model (Zhang et al.) 202T). as our model shares similar probability codes [2) The unsupervised models used in RAVEN are few, here compare with the PRD model (Kiat et al.] 2020). Table T shows the performance of different models. On average, our unsuper- vised model achieves 95.5% accuracy on I-RAVEN, 39.6% higher than the PRD model. Even in comparison with the supervised models, our unsupervised model achieves performance better than SCL. and PrAE, particularly in 2x2 Grid, 3x3 Grid and O-IG configurations, but NVSA.
Evaluations of performance in each sub- process We then evaluate the importance of each com- ponent in the proposed model. As the four components are sequentially conducted, we can step- wisely add the corresponding component into model and then assess its importance (Appendix C). Table shows the performance improvement on the three RAVEN benchmarks by sequentially adding each component into the model (Table for details). On average, the performance on the
"The PrAE models here are trained separately on each configuration. But since PrAE model is hard to train on 3x3 Grid due to limited computational resources, the 3x3 Grid result shown in this paper is the test accuracy of the model trained on 2x2 Grid as in the original PrAF paper (Zhang et al. 2021 which is starred Table

---

Under review as a conference paper at ICLR 2023
three benchmarks is similar, but consistently lower in RAVEN than I-RAVEN and RAVEN-FAID This is likely due to the candidate panels in RAVEN share more object attributes (Zhagg et al. 2019b; Hu al. 202T), causing the generative answer a little more difficult to distinguish the true one from the other candidates, as similar as in PrAE (Zhang et al. 2021 In details, testing the per- ception module only achieves average test accuracy close to 50% I-RAVEN and RAVEN-FAIR. Second, adding the RIN component into the model improves the accuracies about 20%. Please keep in mind that RIN only has concepts about the rule categories, but not the exact rules. Third, after mapping the representations of object attributes to symbols with a MLP, the accuracies further in- crease over 20%. The two steps of symbolic rule reasoning module increase the accuracies by 40%, indicating their importance in the current model. Finally, finetuning further improves the accuracies.
Table I: Accuracy(%) of different models on I-RAVEN
Supervised Method
Avg
Center
2x2Grid
3x3Grid
L-R
U-D
O-IC
O-IG
PrAE
87.8
100.0
87.5
55.5*
97.6
98.1
98.4
78.0
NVSA
98.8
100.0
99.6
96.7
100.0
100.0
100.0
95.0
SCL
95.0
99.0
96.2
89.5
97.9
97.1
97.6
87.7
Unsupervised Method
Avg
Center
2x2Grid
3x3Grid
L-R
U-D
O-IC
O-IG
PRD
55.9
73.1
39.9
35.3
67.3
67.3
68.1
40.6
NS-SSCL(Ours)
95.5
99.1
95.7
94.5
96.4
97.5
92.4
93.1
Table 2: Average accuracy($)) of levels of models
Dataset
PM
PM+RIN
PM+RIN+RK
PM+RIN+RK+FT
RAVEN
35.0
57.4(+22.4)
83.8(+26.4)
92.3(+8,4)
I-RAVEN
48.3
68.5(420.2)
89.7(+21.1)
95.5(+5.8)
RAVEN-FAIR
53.8
71.3(+17.5)
91.5(+20.22)
96.2(+4.7)
Table 3: Accuracy(%) of models trained on 2x2Grid only
Method
Avg
Center
2x2Grid
3x3Grid
L-R
U-D
O-IC
O-IG
(S-SSCL(OOrs)
68.8
60.3
95.7
66.0
79.7
79.5
57.0
43.5
PrAE
77.0
90.5
85.4
45.6
96.3
97.4
63.5
60.7
Notably, differing from the supervised learning with the auxiliary annotations of object attributes rules, the neural representations of object attributes (Figure 33) and the rule assignments (Figure 4 are not perfect by the current SSCL method without any supervision. Nonetheless, the model can perform quite well the RAVEN visual reasoning task. This largely credited by the symbolic mapping.
Evaluations of generalizability We further evaluate the generalizability of our proposed model in solving I-RAVEN. First, we examine the performance dependence on the training sample size (Appendix D) Figure 5 illustrates that our model is immune the reduction of training sample sizes. Even when the number of training samples is reduced to 300, 5% of the full sample size, the average accuracy remains as high as 80.5%. Although PrAE has a similar robustness of performance, the performance of SCL (Wu et al.) 2021) is considerably sensitive to the training sample sizes, and reduced to the chance level when the training samples are 10% and 5% of the full samples. Second, evaluate coosscooffiguattion generalizability for model (Figure 6). The models trained in the simple configurations of center, L-R, U-D and even 2x2 Grid fairly transfer to solve the problems in the other simple configurations, but the models trained in the 3x3 Grid, O-IC, and O-IG cannot transfer to solve the problems in simple configurations (Appendix Our model in 2 x Grid has roughly similar generalizability as in PrAE (Table 3
Tests on the V-PROM task We further test NS- NS-SSCL on the V-PROM task (Appendix F). We selected to test the feature of object count, as only this part is valid for all the rules in V-PROM.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Accuracy(%) of models object count split of V-PROM
Method
Accuracy
Baseline (RN, ResNet aux. loss)
55.4 85.6
NS-SSCL
Acaray
87.5%
14
5/0
302
12.0%
100M
cae LR
Takking
0/0
010
Figure 5: The test accuracy change with num- ber of samples.
Figure 6: Cross-confiurraton generalizability.
achivees turns out that NS-SSCL achieves much better performance (Table than the baseline method using a relational network (RN) (Teney et al. 2020).
5 CONCLUSION
In this paper, we present NS-SSCL to obtain state-of-the-art performance in unsupervised models on the RPM-like tasks involving abstract visual reasoning. 'he previous state-of-the-ard results were obtained by learning the task-specific knowledge with supervision. By contrast, NS-SSCL. provides approach to establish the general cognitive abilities in solving the visual reasoning task from the contexts tasks, but not the task-specific knowledge (Marcus & Davis 2020 Fodor & Pylyshyn, 1988 Chollet 2019). Thereby, its performance has strong robustness even for small sample size for training and broad generalizability for tross-configuret
tests. The simplicity of this approach, we believe, should afford its broad applications in solving other spatiotemporal reasoning tasks (Chollet 2019 Spratley et al. 2020). Further, although the SSCL method has been extensively used in many domains, these models share common drawback of lacking interpretability. We here SSCL. to train NS model, to provide interpretability and generalizability in solving high-level cognitive tasks, such RPM-like tasks, in a human-lite -iinan-like manner. our opinion, this promising approach towards AGI (Chollet) 2019).
The current model of NS-SSCL also has some important limitations deserved to improved. First, both the perception and reasoning modules can be pretrained by independent situations and tasks to construct its general cognitive abilities in object and rule recognitions. It remains to explore this potential by training independent tasks and testing on other independent tasks, similar to the models used in NLP (Dong et al., 2018 Dosovitskiy et al. 2020). Second, the NS-SSCI, performance might be further improved by jointly optimizing the interactions between the components. Third, more general cognitive abilities and higher-level cognitive function (c.g., metacognition) can be further incorporated into the model provide more versatile intelligent abilities in solving complex tasks. Finally, the potential applications the current model should be extensively tested in other abstract visual reasoning tasks.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A Theoretical Analysis Contrastive Unsupervised Representation Learning. international con- ference machine learning, 2019.

---

Under review as a conference paper at ICLR 2023
David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, and Timothy P. Lillicrap. Measuring abstract reasoning in neural networks. international conference on machine learning. 2018.
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-Localized Abstract Reasoning. arXiv: Artificial Intelligence, 2020.
Warren B. Bilker, John A. Hansen, Colleen M. Brensinger, Jan Richard, Raquel E. Gur, and
Ruben C. Gur. Development of Abbreviated Nine-Item Forms of the Raven's Standard Pro- gressive Matrices Test: Assessment, 2012.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nicholas Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in 3 -AV ArXiv, abs/ 180440384999 2018.
Patricia A. Carpenter, Marcel A. Just, and Peter Shell. What one intelligence test measures: A the- oretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97(3):404 431, 1990. doi: 10. 10.1037/0033- 29333295xx99..4000
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. international conference on machine learn- ing, 2020.
François Chollet. On the Measure of Intelligence. arXiv: Artificial Intelligence, 2019.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matthew Botvinick. Attention over Learned Object Embeddings Enables Complex Visual Reasoning. neural information processing systems, 2021a.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo. Joshua B. Tenenbaum, and Chuang Gan. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language. neural information processing systems, 2021b.
Linhao Dong, Shuang Xu, and Bo Xu. FrecchTrranffommr:: A No-Recurrence Sequence-to- Sequence Model for Speech Recognition. international conference on acoustics, speech, and signal processing. 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Learning, 2020.
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical anal- ysis. Cognition, 28(1-))33-7, 1988. doi: 10.001660100022778))901555
Alison Gopnik, Clark Glymour, David M. Sobel, Laura E. Schulz, Tamar Kushnir, and David Danks. A Theory of Causal Learning in Children: Causal Maps and Bayes Nets. Psychological Review, 111(2):3-3,, 2004. doi: 1037/00/0 2177700033-29x111111 111.1.3.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arXiv: Computer Vision and Pattern Recognition, 2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv: Computer Vision and Pattern Recognition, 2019.
Michael Hersche, Mustafa Zeqiri, Luca Benini, Abu Sebastian, and Abbas Rahimi. A Neuro-vector- symbolic Architecture for Solving Raven's 's Progressive Matrices. ArXiv, abs/2203.04571 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. bAt:-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. Learning. 2017.
Dokhyam Hoshen and Michael Werman. IQ of Neural Networks, arXiv: Learning, 2017.
10

---

Under review as a conference paper at ICLR 2023
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified Rule-Aware Network for Abstract Visual Reasoning. national conference on artificial intelligence, 2021.
Marius Jahrens and Thomas Martinetz. Solving Raven's Progressive Matrices with Multi-Layer Relation Networks. international joint conference on neural network, 2020.
Nicholas Quek Wei Kiat, Duo Wang, and Mateja Jamnik. Pairwise Relations Discriminator for Unsupervised Raven's Progressive Matrices. arXiv: Artificial Intelligence, 2020.
Youngsung Kim, Jinwoo Shin, Eunho Yang, and Sung Ju Hwang. Few-shot Visual Reasoning with Meta-Analogical Contrastive Learning. neural information processing systems, 2020.
Tassilo Klein and Moin Nabi. Contrastive SelerSuuee Learning for Commonsense Reasoning. meeting of the association for computational linguistics, 2020.
Nuo and James DiCarlo. Unsupervised Natural Experience Rapidly Alters Invariant Object Representation in Visual Cortex. Science, 321(5895):5002-5007 2008. doi: 10.1 126/scccc.. 1160028.
Andrew Lovett, Kenneth D. Forbus, and Jeffrey Usher. Analogy with Qualitative Spatial Represen- tations Can Simulate Solving Raven's Progressive Matrices. Proceedings of the Annual Meeting of the Cognitive Science Society, 2007.
Mikolaj Malkinski and Jacek Mandziuk. Multi-Label Contrastive Learning for Abstract Visual Rea- soning. arXiv: Artificial Intelligence, 2020.
Mikolaj Malki' nski and Jacek Ma' ndziuk. Deep Learning Methods for Abstract Visual Reasoning: A Survey Raven's Progressive Matrices. ArXiv, abs/2201. 12382, 2022.
Jacek Mańdziuk and Adam Zychowski. DeepIQ: Human-Inspired AI System for Solving IQ Test Problems. international joint conference on neural network, 2019.
Farshad Alizadeh Mansouri, David J. Freedman, and Mark J. Buckley. Emergence of ab- stract rules in the primate brain. Nature Reviews Neuroscience, 21(11):595-610, 2020. doi: 10.10388/415333000-06666
Jayyan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervi- sion. Learning, 2019.
Gary Marcus and Ernest Davis. Insights for AI from the human mind. Communications of The ACM, 2020.
Laura E. Matzen, Zachary O. Benz, Kevin R. Dixon, Jamie Posey, James K. Kroger, and Ann Speed. Recreating Raven's s: Software for systematically generating large numbers of Raven-like matrix problems with normed properties. Behavior Research Methods, 2010.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Rep- resentations of Words and Phrases and their Compositionality. neural information processing systems, 2013.
Niv Pekar, Yaniv Benny, and Lior Wolf. Generating Correct Answers for Progressive Matrices Intelligence Tests. neural information processing systems, 2020.
Usha Nandini Raghavan, Réka Albert, and Soundar R. T. Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical Review E, 2007.
John C Raven, J. H Court, and Jean Raven. Raven's 's progressive matrices. Oxford Psychologists Press, 1938.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2014.
11

---

Under review as a conference paper at ICLR 2023
R E Snow, Patrick C. Kyllonen, and B Marshalek. The topography of ability and learning correla- tions. In Robert Stembeerg (ed.), Advances in the psychology of human intelligence, pp. 47-103. Psychology Press, 1984.
Elizabeth S. Spelke. Principles of Object Perception. Cognitive Science, 14(1):29-555 56, 1990. doi: 10.1207//15111779coggl
Steven Spratley, Krista A. Ehinger, and Tim Miller. A Closer Look at Generalisation in RAVEN. european conference computer vision, 2020.
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hen- gel. V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices. national conference on artificial intelligence, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. neural information process- ing systems, 2017.
Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The Scattering Compositional Leamer: Discovering Objects, Attributes, Relationships in Analogical Reasoning. arXiv: Learning, 2021.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: CoLlision Events for Video REpresentation and Reasoning. Learning. 2019.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RAVEN: Dataset for Relational and Analogical Visual REasoNing. computer vision and pattern recognition, 2019a.
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song- Chg--Chun& Zhu. Learning Perceptual Inference by Contrasting. neural information processing systems, 2019b.
Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu. Abstract Teappol-Tempora Reasoning via Probabilistic Abduction and Execution. compuler vision and pattern recognition, 2021.
Tao Zhuo and Mohan S. Kankanhalli. Solving Raven's Progressive Matrices with Neural Networks. arXiv: Computer Vision and Pattern Recognition, 2020.
Tao Zhuo and Mohan S. Kankanhalli. Effective Abstract Reasoning with Dual-Contrast Network. Learning, 2021.
APPENDIX
A DETAILS OF RAVEN DATASET
There are 7 configurations in RAVEN dataset as shown in Figure 7 Zhang et al. 2019a). The configurations are determined by the different organizations of a variety of objects in each panel (i.e.. the cell of the 3x3 matrix in the context). For instance, in the configuration of 2x2 Grid, each panel consists of 2x2 cells and in each cell an object may be occupied or not. The number and position attributes are then determined by the compositions of cells in each panel. On the other hand, the configuration of Legh, each panel consists of the left and right parts, but the two parts are independent, thereby, the rules governing each part of the context panels are also independent.
The rules that govern the RAVEN dataset are normatively classified into (I) Constant; (2) Pro- gression; (3) Arithmetic; and (4) Distribute Three. Further, internal parameters are used to diver- sify the above abstract rules into concrete rules. For instance, Arithmetic contains plus or minus, while Progression contains increments or decrements of or 2, and Distribute Three contains left- revolving or reght-revolving. In total, there are possible concrete rules. However, the numbers of possible concrete rules different for different object attributes.
12

---

Under review as a conference paper at ICLR 2023
Center
2x2 Grid
3x3 Grid
?
?
?
LeftRight
Up-Down
Out-inCenter
Out-InGrid
?
?
?
?
Figure Representative RAVEN problems of all configurations
B IMPLEMENTATION DETAILS
General implementation details on RAVEN We separately train the models for the different RAVEN configurations [Center, Left-Right (L-R), Upper-Down (U-D), 2x2 Grid, 3x3 Grid, Out-In Center (O-IC), Out-In Grid (O-IG)). We train model the 6,000 samples in the training dataset and test the model on the 2,000 samples in the testing dataset for each configuration. The models are implemented in PyTorch and run with Intel(R) Xeon(R) Platinum 8272CL CPUs and NVIDIA Geforce RTX 3090 Founders Edition GPUs. The inputs of the object images (originally 160 160 for a panel) are resized to 32. We have referred PrAE (Zhang et al.) (02T) in parts of our implementation. For instance, in the answer generating part, we follow the idea of executor in PrAE.
Implementation details in visual perception module
N log
D(ps, Ta)
N Loss,(T) = N log
D Gi (ay
Losse(1) (T) = N 1=1
Ej=1 D(vi 4j Ta)
Σラ= D(gi, 4js Ta)
EN Qi
LossE (I) -log (
)} with N -11,, D(p, T) p/9
E di
Loss (Looss(()=tt A)Loss + pLosss
(8)
For visual perception pretraining, the batch size of the RAVEN problems N is 32 and only the context panels are used. The number of object images used for pretraining in each batch depends on the configurations. For instance, in configuration of 2x2 Grid, there are 32 x 8 x = 1024 0.6, μ 0.01. The temperature parameter 5
objects in each batch. The weight parameter 0.1 for the attribute of type, and 0.05 for the attributes of size and color. The object-based visual perception modules are trained for 100 epochs. The dimensions of probabilistic representations of object attributes are 20, while the actual categories or values of the type, size and color attribute are 4, and 10, respectively.
Implementation details in rule reasoning module
Loss
Wiitda. (row roww!d) - 62.
(9)
11=12
For training rule induction in RIN, the batch size is Inputs with the same row index, e.g., both the first row, weighted by 0.1. Besides, positive samples, i.e., from the same RAVEN
13

---

Under review as a conference paper at ICLR 2023
problem, are weighted by N - 1. The network is trained for 100 epochs. In search for best rule assignment, the learning rate ADAM optimizer is set to be 0.1, and the model is optimized for 20 steps before evaluating each assignment. Notably, longer training lower learning rate should lead to more precise evaluation but also more compute time. After the best assignment is determined, we reduce the learning rate to be 0.01 and conduct training for 1,000 steps. The number of vertices in the graph network, i.e., the batch size, is 512 in common setting but is limited to min (512, 0.8n) in few-shot setting with n samples in total.
Implementation details in finetuning As for finetuning, the learning rate is 0.01 and gradients are clipped into the range [-1,1]. The batch size 256 and the number epoch 10. For each batch, first predict the rules and then optimize the MLP mapping representation to symbols accordingly for 30 steps. The rule prediction with normalized probability less than 0.9 or absolute probability less than 10 are weighted with O.
Implementation details in panel information integration The panel information integration pro- cess is slightly different for different procedures. The probability of the type, size, color attribute is existence-weightedly averaged for rule induction as follows:
Ener Pa Pexiet
Pa
panel
(10)
Σ obj
Instead, the log probability is existence weightedly averaged for optimizing rule probability:
Eod, log(pe Peasst
log(p (ogge
(11)
While testing, the log probability of objects averaged a probhilltticcmagiiiliiigg-sy/ as similar PrAE (Zhang et al. 2021):
log (og() Jeristori
pa
ponel
Σ exist
Ponist
exp
(12)
Σ Eevistors
where ewist a binary vector describing the existence of objects.
As for the integration of existence, RIN simply concatenates the object-based prediction in each panel. While probabilistic integration used to obtain the position and number information each panel during testing. Notably, for independent components, for instance, in the U-D configuration, the object information is not integrated and is processed independently.
C PERFORMANCE OF THE FOUR LEVELS OF MODELS ON EACH CONFIGURATION
Table 5 details the test accuracy each configuration for levels of models described in the main text. is much appreciated that the accuracy greatly improved by the rule reasoning proce- dures of RIN and rule assignments in each configuration, respectively.
D VARYING THE BATCH SIZE WHILE PRETRAINING THE PERCEPTION MODULE
In SSCL (Chen et al. 2020), the performance image categorization is significantly dependent on the batch size. The larger the batch size, the higher the performance. To test whether proposed model of NS NS-SSCL is also dependent on the batch size used for visual perception module, we train the module with different number of batch sizes. Notably, the batch size is the number of the RAVEN problems, rather than the contained objects. For sake of simplicity, we here merely test on the configuration of Center I-RAVEN On this configuration, there are 8 objects in each RAVEN
14

---

Under review as a conference paper at ICLR 2023
Table 5: Accuracy(%) of levels of models for each configuration
Dataset
Config
PM
PM+RIN
PM+RIN+RK
PM+RIN+RK+FT
Avg
35.0
57.4(+22.4)
83.8(+26.4)
92.3(+8.4)
Center
31.8
60.1 (+28.4)
91.4 (+31.3)
98.6 (+7.2)
2x2Grid
60.4
73.3 (+13.0)
89.5 (+16.2)
92.9(43.44
3x3Grid
48.3
55.5 (+7.2)
90.3 (434.8)
91.7 (+1.5)
RAVEN
L-R
18.0
51.8 (433.9)
76.4.5)
94.0
U-D
22.1
67.1 (+45.0)
78.7
93.9 (+15.2)
O-IC
27.1
36.7 (49.6)
76.5 (+39.9)
86.0
O-IG
37.4
57.5
84.3 (426.8)
88.9 (+4.6)
Avg
48.3
68.5(420.2)
89.7(+21.1)
95.5(+5.8)
Center
43.9
69.9 (+26.0)
95.6 (+25.7)
(43.5)
2x2Grid
63.7
80.1 (+16.4)
93.0
95.7 (+2.8)
3x3Grid
52.0
92.8 (+29.5)
94.5 (+1.8)
I-RAVEN
L-R
38.9
(421.8)
82.7 (+22.0)
96.4
U-D
41.9
82.1 (+40.2)
88.4 (+6.3)
97.5 (+9.2)
O-IC
46.9
85.8 (+33.6)
92.4 (+6.6)
O-IG
51.2
71.7 (+20.5)
89.8 (+18.1)
93.1 (43.4)
Avg
53.8
71.3(+17.5)
91.5(+20.2)
96.2(44.7)
Center
47.1
69.6 (+22.5)
95.6 (+26.0)
99.3
2x2Grid
75.8
84.7
94.5 (+9.8)
(5.40)
3x3Grid
66.2
72.2 (+6.0)
95.7 (+23.6)
96.2 (+0.5)
RAVEN-FAIR
L-R
39.5
64.1 (+24.6)
86.5 (+22.4)
96.9 (+10.5)
U-D
43.0
82.4 (+39.4)
89.3 (+6.9)
97.1 (+7.8)
O-IC
45.5
53.3 (+7.8)
86.5 (+33.2)
93.8 (+7.3)
O-IG
59.9
73.1 (+13.3)
92.8 (+19.7)
95.0 (+2.2)
problem used for training the visual perception module. As matter of facts, the performance is insensitive to the batch sizes used for training (Table 6). This is different from SSCL used visual categorizing tasks. As have shown in Table 5 the high accuracy of the model is much relied the backend rule reasoning module.
Table 6: Accuracy(%) of models with different pretraining batch sizes on Center
Batch size
2
8
16
32
64
Accuracy 95.1
96.4
95.6
97.4
98.9
99.1
98.1
E DETAILS OF CROSS CONFIGRUNIIUN TESTS
In the configurations like Center, there no chance that no object exists. Thus, the models trained on the Center configuration cannot learn such an idea of existence. This would lead to extremely poor performance when the models are used to test on the configurations like 2x2 Grid. For this reason, the cross-configuration tests in this study actually rely on the existence prediction from the model that trained on the corresponding configuration. Notably, significant factor that affects cross-configuration testing performance is the dissimilarity between objects in different configura- tions, partially because of different resolutions. Figure illustrates representative examples of the differences between objects in the configurations of Center and 2x2 Grid. The difference is most significant at the boundaries of objects. Hence, it seems that the models learn the configuration- dependent semantic concepts about the objects.
15

---

Under review as a conference paper at ICLR 2023
Center
2x2 Grid
Figure 8: Example objects in the Center and 2x2 Grid configurations
F DETAILS OF V-PROM DATASET
V-PROM is an RPM-like task in which the visual objects are natural images (Figure 9). In this task, there four types of rules (And, Or. Union, Progression). These rules are applied three dimensions of Object Attribute, Object Category, and Object Count. Only in the dimension of Object Count all of the four rules can be applied. For simplicity, we then test on the Object Count split of the V-PROM benchmark. We further split the data into the training set and testing set as ratio 7
Since natural images are used in the V-PROM benchmark, the SSCL approach used in the RAVEN benchmark cannot be directly applied. Instead, we use supervised but label-shuffled samples to finetune pretrained ResNet model (see 3.1 We also the community detection method to recognize the clusters of different rules (Figure IOp. following part of incorporating symbolic rule knowledge, slight additional supervision of cluster rules is applied to speed up training.
16

---

Under review as a conference paper at ICLR 2023
?
Figure 9: An example of the object count split in the V-PROM benchmark from the original paper Teney 2020). The correct answer is the second candidate.
17

---

Under review as a conference paper at ICLR 2023
USI0 progressoo and
progression 2NO
Figure 10: The graph built by RIN and the communities with predicted rules in the V-PROM bench- mark.
18


Reviews:
[{"review_id": "qWResDE5u43", "reviewer": "Reviewer_Naed", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presented a framework that combines self-supervised contrastive learning and neuro-symbolic method for \"abstract visual reasoning,\" specifically grounded on the task of RAVEN's Progression Matrices (RPM). The main framework has four steps. First, it trains a per-object perception model that maps images to different embedding spaces such as shape, size, and color. Next, it uses self-supervised learning objectives to learn \"possible rules\" from the images, leveraging the definition of RPM matrices. In the third step, the model maps tokens into \"symbolic\" tokens (in particular natural numbers) based on an given set of possible rules in RPM. Finally, the model finetunes the model using \"pseudo-labeling\" for symbols.", "summary_of_the_review": "See my weakness discussions for details.", "strength_and_weaknesses": "The paper combines neuro-symbolic reasoning methods and self-supervised contrastive learning (SSCL), which indicates an important future direction for neuro-symbolic methods. Specifically, a typical issue with neuro-symbolic methods is that they require additional supervision or strong domain-specific knowledge. Leveraging patterns in existing data to automatically discover such prior knowledge and reduce the amount of annotations needed, is an interesting idea. I acknowledge the contribution of this paper, given that the authors have successfully demonstrated this idea in the RPM domain. Although the domain is toy-ish, I still think these results are promising.\n\nHowever, I think the current manuscript is still not ready to be published at ICLR due to the following limitations of the methods.\n\nThe most critical concern I have about this paper is the exact \"supervision\" and/or \"assumptions\" used by the proposed framework. The authors have emphasized a lot in the abstract and intro that a critical issue of existing neuro-symbolic reasoning methods is that \"they rely on supervised learning and auxiliary annotations,\" but the paper has failed to illustrate the advantage of the method in this regard. Specifically,\n\n1. the model indeed requires annotations for the visual recognition modules (size, shape, color) in the first step, because the contrastive loss is computed with respect to \"prototypical objects.\"\n2. actually, compared to most existing frameworks for solving RPM tasks, the paper requires a given object detection model. (This point is minor to my score but I think the authors should be explicit about that)\n3. the model requires very specific domain-specific knowledge when they are learning \"symbol\" grounding. Specifically, they require a set of symbolic set that explains all possible structures of models (step 3).\n\nSecond, I would not consider the SSCL part in step 2 really \"self-supervised\" and \"domain-independent.\" Specifically, this training objective is basically the data generation rule of the RPM dataset. That is, the author is exactly training the model using the task objective on RPM (of course, using only the first two rows). It is very unclear how SSCL can be generalized to other visual reasoning tasks.\n\nThird, steps 3 and step 4 seem a bit unmotivated. For example, why do we want to extract symbolic labels from data? Is it for interoperability? Is it for better performance?\n\nStep 3 seems to be a very hard multiple-instance learning problem. How hard is it in this domain?\n\nAlso this paper misses some strongly relevant works, especially the work on \"perception and grammar\" learning:\n\n- Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning. Li et al. In ICML 2020.\n\nMeanwhile, this paper seems to be conceptually related to the SCL work (Wu et al). For example, they also study how to use neural networks to learn meta-level rules from data. I would like to see a detailed comparison between this work and SCL.\n\nMinor questions:\n\n1. In step 2, how would the model handle ambiguities in progression rules? For example, the observation \"123\" can be interpreted as \"progression\" or \"arith addition.\"\n2. I think the results in Table 1 does not contain the SOTA performance models on these datasets. Also, as I have said, I will not call this model completely \"unsupervised.\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gGzbAoMBH9-", "reviewer": "Reviewer_QQJu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on providing a learning strategy tailored to tackle the problem of Raven’s Progressive Matrices (RPMs). The main idea consists of (i) distinguishing between attribute-level (determining the shape, color, size and presence of an object) and task-level (associating a symbol to each combination of object attributes from a row sequence of images) representations and (ii) devising a multi-stage strategy to learn them. Specifically, the attribute-level representation is learnt in a contrastive fashion by leveraging two pieces of information, namely using object pairs generated through attribute intervention and by comparing attributes between image pairs from same and different RPM tasks. The task level representation is learnt in a supervised fashion to distinguish between the different RPM rules. Experiments on RAVEN benchmarks show the comparable performance of the proposed strategy with state-of-the-art supervised approaches.", "summary_of_the_review": "The paper proposes a multi-stage strategy to tackle abstract visual reasoning tasks based on RPMs. While the idea of using self-supervised contrastive learning is original in this context, the work is preliminary as it requires additional major effort to overcome the highlighted weaknesses.", "strength_and_weaknesses": "Please find a list of strengths and weaknesses and subsequently more detailed comments with actionable points.\n\n**STRENGTHS**\n- Solving RPMs with reduced supervision is a relevant and timely challenge.\n- Introducing the use of contrastive self-supervised learning is novel in the context of RPMs.\n\n**WEAKNESSES**\n- Imprecise/not well supported claims. The claim that the proposed model is unsupervised is imprecise. The authors should clearly state the underlying assumptions on the available supervision and its forms used in the different stages of training. Additionally, the claim that the model is a novel neuro-symbolic framework is imprecise. The authors should clearly specify in which aspect the model can be regarded as a neuro-symbolic approach.\n- The proposed solution is tailored to the specific RAVEN problems and it’s not clear how well the strategy can generalise to other settings. Therefore the scope is limited.\n- Important related work is missing and should be discussed.\n- Experimental comparisons with unsupervised baselines is not fair and the comparisons with supervised ones do not support the superiority of the approach\n\n**COMMENTS and ACTIONABLE POINTS**\n\nThe main claims of the paper are vague and not precise. Regarding the unsupervised aspect, it is unclear to me at which level the authors consider their method as an unsupervised strategy. To be more specific:\n- When learning attribute level representations, supervision is introduced in at least two ways, namely when augmenting data based on attributes (this relies on the assumption that the attributes are known and the user have full control on the attributes during augmentation) and when relating to attributes of objects/images in the same task (indeed this relies on the assumption that images contain objects with similar properties). Clearly, these aspects are an important limitation of the proposed approach and they should be mentioned and discussed.\n- When learning task-level representations, it is unclear which supervised information is used to learn the mapping from attributes to task-specific symbols as the rules seems to be given during training.\nConsequently, I suggest to explicitly mention about the underlying assumptions on supervision and clearly state in which stage/aspect the method is considered unsupervised. One can consider (i) to use a table to highlight the similarities and differences in terms of supervision compared to the competitors (e.g. PRD, PrAE) and (ii) to be more precise when defining the equations (1,2,3,6) as lacking such information.\nRegarding the neuro-symbolic aspect. In which sense, can the proposed solution be considered a neuro-symbolic framework? The work circumvents the problem of integration between a neural and a symbolic module by proposing a multi-stage procedure, without even mentioning it.\n\nThe work seems to be specifically tailored to the experimental setting of RAVEN, which is in stark contrast with the main claims of the paper .  More concretely, it seems that the proposed solution introduces assumptions specific to RAVEN, including the division of image into cells containing a single object, the fact that objects in an image share similar attributes. Furthermore, it seems that several hyper parameters need to be properly tuned, including all the weights for the losses in Eq. 1 and Eq. 2, together with the other hyper parameters required from the different stages. How are these hyper parameters influencing the performance of the proposed approach?\nTherefore, how well can the proposed strategy generalise to other abstract visual reasoning tasks, like for instance the ARC benchmark (from Francois Chollet)? How many changes are required in order to ensure a proper level of generalisation in such case?\n\nImportant related work is missing. The authors should discuss the relation with respect to neuro-symbolic approaches and object-centric representation learning. Please, refer to [1]-[2] for two good starting points.\n\nAs a result of the above-mentioned issues, the experimental comparison with the unsupervised baseline on RPMs is not fair. Additionally, in which sense is the proposed strategy superior to supervised approaches (as mentioned in the abstract and in other parts of the paper)? For instance, by looking at Table 1 in the experimental section, I’m not able to see a convincing improvement over the competitors.\n\n[1] Neural-symbolic learning and reasoning: A survey and interpretation. 2022. Neuro-Symbolic Artificial Intelligence: The State of the Art\n\n[2] Neural Production Systems. 2021 NeurIPS\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_vj-6sns2OD", "reviewer": "Reviewer_FVX7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel method for solving visual reasoning (specifically RPM-like) problems, that combines contrastive losses at both the perceptual and abstract levels, together with a symbolic rule induction module. The method achieves high accuracy on both the RAVEN and V-PROM datasets.", "summary_of_the_review": "The proposed method achieves strong results on the RAVEN benchmark, but the model is misleadingly characterized as 'self-supervised' when in fact it requires stronger supervision than many other models. The method is not evaluated on PGM, presumably because of the difficulty imposed by reliance on such strong supervision.", "strength_and_weaknesses": "# Strengths:\n- The proposed method achieves strong results on the RAVEN dataset, a popular benchmark for evaluating abstract visual reasoning abilities.\n- The authors also evaluate the proposed method on the V-PROM dataset (with favorable results), which extends the RPM problem format to include naturalistic images. This dataset is a welcome introduction to the visual reasoning literature (usually focused on problems composed of abstract geometric forms), and it is good to see the authors adopt it.\n\n# Weaknesses:\n- The primary weakness is that the proposed method, NS-SSCL, which the authors characterize as 'self-supervised', is arguably not actually self-supervised. In fact, the method involves stronger supervision than some of the other methods that are characterized as 'supervised'. In particular, the contrastive learning used to shape perceptual representations in NS-SSCL depends on having groundtruth annotations of the perceptual attributes (e.g. object size, shape, etc.), whereas in many 'supervised' methods (which are supervised at the level of answer choice selection) these perceptual representations are learned end-to-end in the service of the downstream reasoning task. The paper characterizes the training procedure for perceptual encoding as 'self-supervised' because a contrastive loss is employed, similar to the SimCLR method of Chen et al. (2020). However, there is a crucial difference between SimCLR and the contrastive loss employed in the present work. In SimCLR, certain image transformations are employed (e.g. rotation, color distortion, etc.), and the model is trained to generate embeddings that are similar for both an original image and a transformed version of that image. The major advantage of this method is that one does not need image labels at all during training, thus it is *self*-supervised. In NS-SSCL, by contrast, the same kind of contrastive loss is used, however the positive and negative pairs used to compute the loss are defined in terms of groundtruth attributes. For instance, a positive pair for the 'size' attribute might consist of two objects of the same size (but potentially of different shapes, colors, etc.). This is very different than applying generic image transformations, since it actually requires the attribute labels in order to define what constitutes a positive example. Thus, even though the model is not trained explicitly to label the attributes, computing the loss requires the exact same level of supervision as if the model had been trained to generate labels. So, although it is correct to refer to this as a contrastive loss, it is not correct to refer to this as self-supervised learning, since supervision is required. \n    - Note that I am not arguing that supervised learning of perceptual attributes is necessarily unreasonable. I think a reasonable argument can be made that, at least for human reasoners, perceptual representations are not learned in the context of learning to perform visual reasoning tasks, and so can be treated as the product of a separate learning process. However, it is very misleading to call this 'self-supervised' learning, and to contrast it with other 'supervised' methods, which actually require weaker supervision. It is also unclear, if the method requires groundtruth perceptual annotations, what advantage is conferred by using a contrastive approach as opposed to a more straightforward supervised classification method.\n- I had some difficulty understanding how the symbolic reasoning module works, but if I understand correctly, it assumes prior knowledge of the rules employed in these problems, and also employs supervised training to map the perceptual representations (learned through contrastive learning) to the relevant symbols in the rule representations. This is an even stronger form of supervision, and makes it so that the model cannot process any rules other than those built into it by the modeler.\n- The paper focuses on the RAVEN benchmark, but does not evaluate the method on PGM, the other major benchmark of RPM-like problems. This is presumably because PGM, unlike RAVEN, does not contain groundtruth annotations for the perceptual attributes, and so it is not possible to apply NS-SSCL to that benchmark, underscoring the reliance of the model on such strong supervision.\n\n## Other comments:\n- It seems that the method requires groundtruth object segmentations, is this the case? This aspect of the model is not addressed in the paper.\n- The axes for figures 3 and 4 are extremely small and almost impossible to read.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZlCiaYm4Ho", "reviewer": "Reviewer_mdUd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A new model called NS-SSCL, neuro-symbolic self-supervised contrastive learning, is proposed for solving abstract reasoning problems in RAVEN and V-PROM, two abstract reasoning problems used for evaluating machine intelligence. The NS-SSCL method leverages SSCL to learn visual representation for attribute disentanglement, a rule induction network to build community of rule examples self-supervisedly, and finally pseudo labels from the PrAE engine to learn a mapping from representation to symbols. The majority of the model is based on the PrAE workflow and turned into a self-supervised method. In experiments, the authors show that NS-SSCL achieves improved performance compared to certain supervised baselines.", "summary_of_the_review": "I'm very concerned about the novelty of the work and the presentation of the manuscript. See above for details.", "strength_and_weaknesses": "The paper performs thorough evaluation on the chosen problems and is well-motivated. The experimental part details a wide spectrum of the learning landscape and shows different aspects of the model itself compared to others.\n\nMy major concern on the work is novelty. PRD is an established work that shows very good performance even when unsupervised. It also uses contrastive learning on the rule induction network and can directly reach answers. From this perspective, using unsupervised learning for facilitation is not new any longer. Besides, the work largely follows PrAE, but only turns the the original supervised method into a self-supervised one with bells and whistles. It's interesting to see it but considering the two aspects mentioned above, the contribution is only incremental. \n\nThe paper seems to intentionally confuse reviewers. For one thing, PrAE is a supervised method, so using the intermediate results of PrAE should not be considered unsupervised. The authors continually claim state-of-the-art, while on the experiments, the method fares worse than existing ones.\n\nThe manuscript is poorly presented. Can you use more illustrative figures to explain the idea and properly typeset your equations? The equations are so confusing and needs significant rewording to help explain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject