Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection
Authors: Fan Shi
Bin Li
Xiangyang Xue
Venue: ICLR 2024 poster (2024)
Keywords: Deep Latent Variable Models
Generative Models
Raven’s Progressive Matrix
Abstract Visual Reasoning
Primary area: generative models
TL;DR: This paper proposes a novel deep latent variable model to solve generative RPM problems through rule abstraction and selection.

Abstract: Endowing machines with abstract reasoning ability has been a long-term research topic in artificial intelligence. Raven's Progressive Matrix (RPM) is widely used to probe abstract visual reasoning in machine intelligence, where models will analyze the underlying rules and select one image from candidates to complete the image matrix. Participators of RPM tests can show powerful reasoning ability by inferring and combining attribute-changing rules and imagining the missing images at arbitrary positions of a matrix. However, existing solvers can hardly manifest such an ability in realistic RPM tests. In this paper, we propose a deep latent variable model for answer generation problems through Rule AbstractIon and SElection (RAISE). RAISE can encode image attributes into latent concepts and abstract atomic rules that act on the latent concepts. When generating answers, RAISE selects one atomic rule out of the global knowledge set for each latent concept to constitute the underlying rule of an RPM. In the experiments of bottom-right and arbitrary-position answer generation, RAISE outperforms the compared solvers in most configurations of realistic RPM datasets. In the odd-one-out task and two held-out configurations, RAISE can leverage acquired latent concepts and atomic rules to find the rule-breaking image in a matrix and handle problems with unseen combinations of rules and attributes.

Published as a conference paper at ICLR 2024
TOWARDS GENERATIVE ABSTRACT REASONING: COMPLETING RAVEN'S PROGRESSIVE MATRIX VIA RULE ABSTRACTION AND SELECTION
Fan Shi Bin Li" Xiangyang Xue
Shanghai Key Laboratory of Intelligent Information Processing School Computer Science, Fudan University fsh 1220m @m. fudan. edu. cn {libin, xyxue)
ABSTRACT
Endowing machines with abstract reasoning ability has been a long-term research topic artificial intelligence. Raven's 's Progressive Matrix (RPM) is widely used to probe abstract visual reasoning in machine intelligence, where models will an- alyze the underlying rules and select one image from candidates to complete the image matrix. Participators of RPM tests can show powerful reasoning ability by inferring and combining attribute-changing rules and imagining the missing images arbitrary positions of matrix. However, existing solvers hardly manifest such an ability in realistic RPM tests. In this paper, we propose a deep latent variable model for answer generation problems through Rule Abstractlon and SElection (RAISE). RAISE can encode image attributes into latent concepts and abstract atomic rules that act on the latent concepts. When generating an- swers, RAISE selects one atomic rule out of the global knowledge set for each latent concept to constitute the underlying rule of an RPM. the experiments of bottom-right and urbitrary-position answer generation, RAISE outperforms the compared solvers in most configurations of realistic RPM datasets. In the odd- one-out task and two held-out configurations, RAISE can leverage acquired latent concepts and atomic rules to find the rule- breakbna image in a matrix and handle problems with unseen combinations of rules and attributes.
INTRODUCTION
The abstract reasoning ability is pivotal abstracting the underlying rules from observations and quickly adapting to novel situations (Cattell 1963 Zhuo & Kankanhalli [021 Matkiński &
Mandziuk] 2022a), which is the foundation of cognitive processes (Gray & Thompson (004)) like
number sense (Dehaene) 2011). spatial reasoning (Bymn & Johnson-Laird 1989), and physical rea- soning (MCCloskey 1983). Intelligent systems may benefit from human-like abstract reasoning when leveraging acquired skills in unseen tasks (Barrett et al. 2018), for example, generalizing the law object collision in the simulation environment real scenes. Therefore, endowing intelli- gent systems with abstract reasoning ability is the cornerstone of higher-intellignnee systems and long-lasting research topic of artificial intelligence (Chollet 2019 Matkiński & Mańdziuk (022b).
Raven's 's Progressive Matrix (RPM) is classical test of abstract reasoning ability for human and intelligent systems (Matkinski Mandziuk] 2022a)). where participators need to choose one im- age out of eight candidates to fill in the bottom-right position of 3x3 image matrix (Raven & Court] 1998). Previous studies demonstrate that participators can display powerful reasoning ability by directly imagining the missing images (Hua & Kunda 2020 Pekar et al.) 2020), and answer- generation tasks can more accurately reflect the model's understanding of underlying rules than answer-selection ones (Mitchell 202T). For example, some RPM solvers find shorteuts in discrimi- native tasks by selecting answers according to the bias of candidate sets instead of the given context.
To solve answer-selection problems, many solvers fill each candidate to the matrix for score estima-
tion and can hardly imagine answers from the given context (Barrett et al. 2018 Hu et al. 2021
"ooresponding author

---

Published as a conference paper at ICLR 2024
Some generative solvers have been proposed to solve answer-generation tasks (Pekar et al. 2020 Zhang et al. 20) They generate solutions for bottom- right images and select answers by com- paring the solutions and candidates. However, some generative solvers do not parse interpretable attributes and attrbbute-chnnging rules from RPMs (Pekar et al. (020), and usually introduce artifi- cial priors in the processes of representation learning abstract reasoning (Zhang et al. [201bb a) On the other hand, most generative solvers are trained with the aid of candidate sets in training, bringing the potential risk of learning shortcuts (Hu et al. 2021 Benny et al., 20221
Deep latent variable models (DLVMs) (Kingma & Welling 2013 Sohn et al. 2013) can capture
underlying structures of noisy observations via interpretable latent spaces (Edwarch & Storkey 2017 Eslami et al. (2018) Garnelo et al.) 2018 Kim et al.) 2019). Previous work (Shi et al. [202]]
solves generative RPM problems by regarding attributes and atribbue-changing rules as latent con- cepts, which can generate solutions by executing attribute-specific predictive processes. Through conditional uuswer-generatee
processes that consider the underlying structure of RPM panels, the distractors are not necessary to train DLVM-based solvers. Although previous work has achieved answer generation in RPMs with continuous attributes, understanding complex discrete rules and abstracting global rules in realistic datasets is still challenging for DLVMs.
This paper proposes a DLVM for generative RPM problems through Rule Abstraction and SElection (RAISE) [ RAISE encodes image attributes (e.g.. object size and shape) as independent latent con- cepts to bridge high-dimensional images and latent representations of rules. The underlying rules of RPMs are decomposed into subrules in terms of latent concepts and abstracted into atomic rules as a set of learnable parameters shared among RPMs. RAISE picks up proper rules for each latent concept and combines them into the integrated rule of an RPM to generate the answer. The con- ditional generative process of RAISE indicates how to use the global knowledge of atomic rules imagine (generate) target images (answers) interpretably. RAISE can automatically parse latent concepts without meta information of image attributes to reduce artificial priors in the learning pro- cess. RAISE can be trained under semi-supervised settings, requiring only small amount of rule annotations to outperform the compared models in non-grid configurations. By predicting the target images at arbitrary positions, RAISE does not require distractors of candidate sets in training and supports generating missing images at arbitrary and even multiple positions.
RAISE outperforms the compared solvers when generating bottom-right and arbitrary-positte an- swers in most configurations of datasets. We interpolate and visualize the learned latent concepts and apply RAISE in odd-one-out problems to demonstrate its interpretability. The experimental results show that RAISE can detect the rule-breaking image of matrix through interpretable la- tent concepts. Finally, we evaluate RAISE on two distr--distriuttion configurations where RAISE retains relatively higher accuracy when encountering unseen combinations of rules and attributes.
2 RELATED WORK
Generative RPM Solvers. While selective RPM solvers (Zhuo & Kankanhalli 2021 Barrett al.
2018
Wu et al. 2020 Hu et al. 2021; Benny et al.] |021]] Steenbrugge et al. 2018 Hahne et al.
2019
Zhang et al. 2019b; Zheng al. 2019 Wang et al. (0019/20000) Jahrens & Martinetz 2020)
problems, generative solvers predict representations or images at missing positions (Pekar et al. 2020 Zhang et al.) (202Tba). Niv et al. extract image representations through Variational AutoEncoder (VAE) (Kingma & Welling 2013) and design a relation wiie perception process for answer prediction (Pekar et al.) 2020). With interpretable scene representations, ALANS (Zhang et al. 2021b) and PrAE (Zhang et al. 202 adopt algebraic abstract and symbolic logical systems as the reasoning backends. These generative solvers predict answers at the bottom- right position. LGPP (Shi et al.) (021) and CLAP (Shi et al. 2023) learn hierarchical latent variables to capture the underlying rules of RPMs with random functions (Williams & Rasmussen, [0066] Garnelo et al.) 2018), and can generate answers at arbitrary positions on RPMs with continuous attributes. RAISE is a variant of DLVM to realize generative abstract reasoning on realistic RPM datasets with discrete attributes and rules through atomic rule abstraction and selection.
focus on coswer-selectio
Bayesian Inference with Global Latent Variables. DLVMs (Kingma & Welling 2013 Sohn et al., 2013 Sønderby et al. 2016) can capture underlying structures of high high-dimensional data in latent
Cooee available tttp:///w...oooiiaaaorreoooeeeee
2

---

Published as a conference paper at ICLR 2024
Context intent concepls x5
Target latont
conceple x5
Targer Int alzation
Sample
Global Knowledge Set
FEA
PS
I//
Rule Execution
Vxx
Sample
Rule Selection
(a) Graphical model of RAISE
(b) Abstract reasoning process
Figure 1: An overview of RAISE. The graphical model in (a) displays the generative process (solid black lines) and inference process (dashed red lines). Panel (b) shows the computational details of the abstract reasoning process and highlights the rule selection, rule execution, and global knowledge with blue, yellow, and red backgrounds, respectively.
spaces, regard shared concepts as global latent variables, and introduce local latent variables condi- tioned on the shared concepts distinguish each sample. GQN (Eslami al. 2018 captures entire 3D scenes via global latent variables to generate 2D images of unseen perspectives. With object- centrie representations (Yuan et al., 2023), global latent variables can explain layouts of scenes
(Jiang & Ahn 2020) or object appearances for multiview scene generation (Chen et al. [021 Kabra
al. 2021 Yuan et al. 2022 Gao & Li 2023) Yuan et al.) 2024). Global concepts can describe
common features of elements data with exchange invariance like sets (Edwards & Storkey 2017
Hewitt et al. 2018:
Giannone Winther 2021). NP family (Garnelo et [ Kim et 2019
constructs different function spaces through global latent variables. DLVMs can generate answers at arbitrary positions of an RPM by regarding the concept-changing rules as global
Foong al. 2020)
concepts (Shi et al. 2021 2023). RAISE holds similar idea of modeling underlying rules as global
concepts. Unlike previous works, RAISE attempts to abstract the atomic rules shared among RPMs.
3 METHOD
In this paper, an RPM problem is (æs, ΦT) where yy and ær are mutually exclusive sets of images, S indexes the given context images, and T indexes the target images to predict (I' can index multiple images). The objective of RAISE is to maximize the log- lig-likelihood log p(zr/=s)) while learning atomic rules shared among RPMs. In the following sections, we will introduce the generative and inference processes of RAISE that can abstract and select atomic rules in the latent space.
3.1 CONDITIONAL GENERATION
The generative process the foundation of answer generation, including the stages of concept learn- ing, abstract reasoning, and image generation.
Concept Learning. RAISE extracts interpretable image representations for abstract reasoning and image generation in the concept learning stage. Previous studies have emphasized the role of ab- stract object representations in the abstract reasoning of infants (Kahneman et al., 1992, Gordon & Irwin 1996) and the benefit of disentangled representations for RPM solvers ( (Van Steenkiste et al.,
2019) which reflect the compositionality of human cognition (Lake et al., 2011)) RAISE realizes
compositionality by learning latent representations of attributes (Shi et al. 2022 2023). RAISE regards image attributes as latent concepts and decomposes the rules of RPMs into atomic rules based on the latent concepts. Since the description of attributes is not provided in training, the latent concepts learned by RAISE are not exactly the same as the realistic attributes defined in the dataset. RAISE extracts C' context latent concepts 2, {=g]C for each context image æ, (s E S):
98 enc (xx),
E
2 N o
c_ S
The encoder 5eth outputs the mean of context latent concepts. The standard deviation is controlled by hyperparameter o₂ keep training stability. Each context image is processed through ne
3

---

Published as a conference paper at ICLR 2024
independently, making it possible to extract latent concepts for any set of input images. In this stage, the encoder does not consider any relationships between images and focuses on concept learning.
Abstract Reasoning. As illustrated in Figure lb RAISE predicts target latent concepts æT from context latent concepts 25 in the abstract reasoning stage, involving rule abstraction, rule selection, and rule execution processes. To abstract atomic rules and build the global knowledge set, RAISE adopts K global learnable parameters & = (1/x ) each indicating an atomic rule shared among RPMs. In rule selection, we use categorical indicators {r"]a- (r = 1.... K) to select a proper rule out of & for each concept. Inferring the indicators from zş correctly critical rule selection. RAISE creates a 3x3 representation matrix Z for each concept, initializing the representations of context images with the corresponding context latent concepts and those of target images with zero vectors. Then RAISE extracts the row-wise and column wise representations:
p FrOW (1:33)
di
= f Fool Zi:3.)
i = 1,2,3,
c=
(2)
RAISE averages the representations via n (pi P P and qf = (af + +g + 03)/3 3 to obtain integrated representations of row and column rules. We concatenate Pe and q to acquire the probability of selecting atomic rules out of the global knowledge set:
r Categorical (TT..K) क.. - T
find
c=
(3)
We denote the learnable parameters as {δr, d2, Φ3) } for convenience. In rule execution, RAISE selects and executes an atomic rule on each concept to predict the target latent concepts:
μ = h ( 2 NN
c= c=1,..,CC C,
(4)
LET, c=
RAISE instantiates by selecting the - learnable parameters from the global knowledge set &i to convert the zero-initialized target representations in Z into the mean of target latent concepts. As in the concept learning stage, the standard deviation of target latent concepts is controlled by a,. consists of convolution layers aggregate information from neighbor context latent concepts on the matrix and update target latent concepts. Each learnable parameters in ve indicates a type of atomic rule. See Appendix C.I for the detailed description of h.
Image Generation. Finally, RAISE decodes the target latent concepts predicted in the abstract reasoning stage into the mean of target images:
2z oZ A. dee (z):O), E T.
(5)
RAISE generates each target image independently make the decoder focus image reconstruc- tion. We control the noise of target images by setting the standard deviation 0g as a hyperparameter.
According to Figure
la
we decompose the conditional generative process
(h, (h, xr (ss))
pe
II
(2) II
Pi , (1 II (o(([())))
(6)
1ET
where h is the set all latent variables and = {0, de v/ sp) are learnable parameters of RAISE.
3.2 VARIATIONAL INFERENCE
RAISE approximates the untractable posterior with a variational distribution (h|ar,, æs) Kingma & Welling 2013), which consists of the following distributions.
(<<[<=,) = N' OT
8 SES, c= 1=,..., C,
((>][==)= N
c= 1....,C. C, t tET,
/ x
Categorical (Tf.k),
c I,...,C. C.
Since RAISE shares the encoder between the generative and inference processes to reduce the model parameters, compute context latent concepts pi!:C and target latent concepts zi!tC via the same process described Equation I In the inference process, RAISE reformulates the variational dis- tribution of the categorical indicator g q(r%|2g. zf.) o p(2f/r ())((r=(2))) That is, RAISE
4

---

Published as a conference paper at ICLR 2024
predicts the prior probabilities Ti.K of p()))(() from the context latent concepts 23 and compute the likelihood p( 2 Ir", x5) by executing the atomic rule r (r ====1, = 1,... K) on zG.. In this way, estimate the variational distribution Q(r"|=s; 2 by considering both the prior probabilities and the likelihoods of atomic rules, which reduces the risk of model collapse (e.g., always select-
ing one atomic rule from +)). We provide more details of q(r") 22g. 29.) in Appendix A. Letting
I = {0, Φ, } we factorize the variational distribution as
qu (h) 11, xS)
II
(r 27) II
go (0(25| x.)
II
qo (2{{(zz))
(8)
#ES
SET
3.3 PARAMETER LEARNING
We update the parameters of RAISE by maximizing the evidence lower bound (ELBO) of the log-
likelihood log p( æs) (Kingma & Welling 2013). With the generative process Pe and the varia- tional distribution qu defined in Equations and 8 the ELBO (q denotes the variational distribu- tion, and we omit the parameter symbols é and v for convenience)
pe (h, xx)3))
L go (hleeryass
log qu (h))),
log ((29 p 1r x5)
Big log 1(19 27.) p(r 23
(9)
Byg log [ogp(ar;/2=]
Bg
RE
Live
Rpued
Rall
The reconstruction loss Cree measures the quality of the reconstruction images. The concept regu- larizer Reed estimates the distance between the predicted target concepts and the concepts directly encoded from target images. Minimizing Reed will promote RAISE to generate correct predictions in the space of latent concepts. The rule regularizer Rull expects RAISE to select the same rules when given different sets of images an RPM. The variational posterior 12g =f.) conditioned
on the entire matrix and the prior p(r*123 conditioned on the context images are expected to have similar probabilities. The detailed derivation of the ELBO is provided in Appendix A.2
A.22
The abstraction and selection of atomic rules rely the acquired latent concepts. Therefore, RAISE introduces auxiliary rule annotations to improve the quality of latent concepts and stabilize the learn- fea } where A is the number of ground truth
ing process. We denote rule annotations attributes and va indicates the type of rules on the a-th attribute. For example, v = [2, 1, means that the attributes follow the second, first, and third rules respectively. RAISE does the meta-information of attributes in training since the rule annotations only inform the type of rule on each attribute. The meaning of attributes is automatically learned by RAISE for accurate rule abstraction and selection. One key to guiding concept learning with rule annotations is determining the correspondence between latent concepts and attributes. RAISE introduces a A C binary matrix M where Ma. = 1 indicates that the a-th attribute is encoded in the c-th latent concept. Therefore, the rule predicted on the e-th latent concept supervised by the rule annotation U,, and the auxiliary loss measures distances between the predicted and ground truth types of rules:
leverage
A
C
Cxup
Ma.e log (T + TT.))
(10)
12
The auxiliary loss Caup is the log-likelihood of the categorical distributions considering the attribute- concept correspondence M. The binary matrix M is derived by solving the following assignment problem on a batch of RPM samples:
Ee Ma.- = 1,
a= 1,..,, A, c= I,..,, C, a a= A, C.
arg max Coup s.t. M
ΣE M or 1,
(11)
Ma.- 0r
Equation TI allows the existence of redundant latent concepts, which can be solved using the modi- fied Volgenat algorithm (Crouse 2016). In this case, the training objective becomes
arg max Lree B Rpred 3r Rrule + 3y
(12)
5

---

Published as a conference paper at ICLR 2024
Table 1: The accuracy (%) of selecting bottom-right answers on different configurations (i.e., Center, L-R, etc) of RAVEN/I-RVVEN. The table displays the average results of ten trials.
Models
Average
Center
L-R
U-D
O-IC
O-IG
2x2Grid
3x3Gri
GCA-I
12.0/24.1
14.0/30.2
7.9/22.4
7.5/26,9
13.4/32.9
15.5/25.0
11.3/16.3
14.5/15.3
GCA-R
13.8/27.4
16.6/34.5
9.4/26.9
6.9/28.0
17.3/37.8
16.7/26.0
11.7/19.2
18. 18.1/19.3
GCA-C
32.7/41.7
37.3/51.8
26.4/44.6
21.5/42.6
30.2/46.7
33.0/35.6
37.6/38.1
43.0/32.4
ALANS
54.3/62.8
42.7/63.9
42.4/60.9
46.2/65.6
49.5/64.8
53.6/52.0
70,5/66.4
75.1/65.7
PrAE
80.0/85.7
97.3/99.9
96.2/97.9
96.7/97.7
95.8/98.4
68.6/76.5
82.0/84.5
23.2/45.1
LGPP
6.4/16.3
9.2/20.1
4.7/18.9
5.2/21.2
4.0/13.9
3.1/12.3
8.6/13.7
10.4/13.9
ANP
7.3/27.6
9.8/47.4
4.1/20.3
3.5/20.7
5.4/38.2
7.6/36.1
10.0/15.0
10.5/15.6
CLAP
17.5/32.8
30.4/42.9
13.4/35.1
12.2/32.1
16,4/37.5
9.5/26.0
16.0/20.1
24.3/35.8
Transformer
40.1/64.0
98.4/99.2
67.0/91.1
60.9/86.6
14.S/69.9
13.5/57.1
14.7/25.2
11.6/18.6
RAISE
90.0/92.1
99.2/99.8
98.5/99.6
99.3/99.9
97.6/99.6
89.3/96.0
68.2/71.
77.7/78.7
where 81. 82, and B3 are hyperparameters. RAISE also supports semi-supervised training settings. For samples that do not provide rule annotations, RAISE can set f3 = 0 and update parameters the unsupervised part Lie B1 Rped Rmlel
EXPERIMENTS
In the experiments, we compare the performance of RAISE with other generative solvers by gen- erating answers at the bottom right and, more challenging, arbitrary positions. Then conduct experiments to visualize the latent concepts learned from the dataset. Finally, RAISE carries out the odd-one-out task and is tested in held-out configurations to illustrate the benefit of learning latent concepts and atomic rules in generative abstract reasoning.
Datasets. The models in the experiments are evaluated on the RAVEN (Zhang al. 2019a) and I-RAVEN (Hu et al., (202T) datasets having seven image configurations (e.g., scenes with centric objects or object grids) and four basic rules. I-RAVEN follows the same configurations RAVEN
and reduces the bias of candidate sets to resist the shortcut learning of models (Hu et al. 2021 See Appendix B for details of datasets.
Compared Models. In the task of bottom-right answer selection, we compare RAISE with the powerful generative solvers ALANS (Zhang al. 20211), PrAE (Zhang et al.) 2021a), and the model proposed by Niv et al. (called GCA for convenience) (Pekar et al. 2020). RAISE selects the candidate closest the predicted result the latent space as the answer. We apply three strategies of answer selection in GCA: selecting the candidate having the smallest pixel difference to the prediction (GCA-I), having the smallest difference in the representation space (GCA-R), and having the highest panel score (GCA-C). Since these generative solvers cannot generate non-bottom-right answers, we take Transformer (Vawwani et al. 2017). ANP (Kim et al 2019), LGPP (Shi et al.) 2021), and CLAP (Shi et al.) 2023) as baseline models to evaluate the ability to generate answers arbitrary positions. We provide more details in Appendix C
Training and Evaluation Settings. For non-grid layouts, RAISE is trained under semi semi-supervised settings by using 5% rule annotations. RAISE leverages 20% rule annotations on O-IG and full rule annotations 2x2Grid and 3 3Grid.. The powerful generative solvers full rule annotations and are trained and tested on each configuration respectively. We compare RAISE with them to illustrate the acquired bottom-right answer selection ability of RAISE under semi-supervised settings. The baselines can generate answers at arbitrary positions but cannot leverage rule annotations since they do not explicitly model the category of rules. We compare RAISE with the baselines to illustrate the benefit of learning latent concepts and atomic rules for generative abstract reasoning, Since the only
training of RAISE and the baselines do not require the candidate sets, and differ in the distribution of candidates, we train RAISE and the baselines on RAVEN and test them on RAVEN/I-RAVEN directly. See Appendix C for detailed training and evaluation settings.
4.1 BOTTOM-RICH RIGHT ANSWER SELECTION
This experiment conducts classical RPM tests that require models to find the missing bottom-right images in eight candidates. Table T illustrates RAISE's outstanding generative abstract reason-

---

Published as a conference paper at ICLR 2024
Center
LR
U.D
Q.C
0-16
2x20nl
3x30rd
Tandoire
CLAP
CLAP
LLAP
Lopp
Riid
skante
Tantffrme
[T|=2
Figure 2: Selection accuracy at arbitrary positions. The selection accuracy RAISE (purple), Transformer (orange), CLAP (green), ANP (blue), and LGPP (black) in arbitrary positions. The x-axis each plot indicates the number of candidates, and the y-axis is the selection accuracy.
Sample
[T]1
DG
+
#
0
220
300
Figure 3: Answer generation at arbitrary positions. The prediction results on RAVEN are high- lighted (red box) to illustrate the arbitrary-position generation ability. Due to the existence of noise, some predictions may differ from the original sample, but they still follow the correct rules.
ing ability on RAVEN/I-RAVEN By comparing the difference between predictions and candidates, RAISE outperforms the compared generative solvers in most configurations of even the distractors in candidate sets are not used in training. All the powerful generative solvers take full rule annotations for training, while RAISE in non-grid configurations only requires small amount of rule annotations (5% samples) to achieve high selection accuracy. RAISE attains the high- selection accuracy compared to the baselines which generate answers at arbitrary positions. By comparing the results on RAVEN/IRRVEE,, we find that generative solvers are more likely to I-RAVEN, because I-RAVEN generates distractors that less sim- ilar to correct answers to avoid significant biases in candidate sets. For grid-shaped configurations, we found that the noise in datasets will significantly influence the model performance. By removing the noise in object attributes, RAISE achieves high selection accuracy three grid-shaped config- urations using only 20% rule annotations. See Appendix D.I for the detailed experimental results.
have accuracy improvement
4.2 ANSWER SELECTION AT ARBITRARY POSITIONS
The above generative solvers can hardly generate answers at non-bottom-right positions. In this experiment, we probe the ability of RAISE and baselines to generate answers at arbitrary positions. We first generate additional candidate sets in the experiment because RAVEN and I-RAVEN do not provide candidate sets for non- coon-botto--righ images. To this end, we sample a batch of RPMs from the dataset and split the RPMs into target and context images in the way. For each matrix,

---

Published as a conference paper at ICLR 2024
So See Ceta
Center
Original Sample
Canige
Modiiisa Sample
Odd-one-out in RPMs
Calor
Dua
legeed
Conn
Paa
4
Amth
0
V
O:IC Held Out
DM0 Denrouse
(a) Visualization of concepts
Held-out configurations
Figure 4: Panel (a) shows the interpolation results of latent concepts and the correspondence between the concepts and attributes. Panel (b) provides an example of RPM-based odd-one-out tests and displays the prediction deviations in concepts of each image. Panel (c) illustrates the strategy to split rule-attribute combinations in held-out configurations.
use the target images of other Ne samples in the batch as distractors to generate candidate set with No+1 I entries. This strategy can adapt to the missing images at arbitrary and even multiple positions, and we easily control the difficulty of answer selection through the number of distractors
Figure 2 displays the accuracy of RAISE and baselines when generating answers at arbitrary and multiple positions. RAISE maintains high accuracy in all configurations. Although Transformer has higher accuracy than the other three baselines, especially in non-grid scenes, the prediction accuracy drops significantly
2x 2Grid and 3 3Grid.. Figure providees the qualitative prediction results RAVEN. It is difficult for ANP and LGPP to generate clear answers. CLAP can generate answers with partially correct attributes simple cases (c.g.. CLAP generates object with the correct color but the wrong size and shape in the sample of Center). RAISE produces high-quality predictions and can solve RPMs with multiple missing images. By predicting multiple missing images at arbitrary positions, The qualitative results intuitively reveal the in-depth generative abstract reasoning ability models, which the bottom-right answer generation task does not involve.
4.3 LATENT CONCEPTS
4 visualizes the latent concepts learned from Center and O-IC by traversing concept representations of an image in well decomposed, decoding the interpolated concept represen-
Latent concepts bridge atomic rules and nigh-ddimensionll observations. Figure
the latent space. If the concepts tations will change one attribute of the original image. Besides observing visualization results, we find the correspondence between concepts and attributes with the aid of the binary matrix M. As shown in Figure 4p. RAISE can automatically set some redundant concepts when there are more concepts than attributes. (c.g., the first concept of Center). The visualization results illustrate the concept learning ability of RAISE, which is the foundation of abstracting and selecting atomic rules shared among RPMs.
4.4 ODD-ONE-ON OUT IN RPM
In odd-one-out tests, RAISE attempts to find the rule-breaking image in panel. To generate RPM- based odd-one-out problems, we replace the bottom-right image of an RPM with random distractor in the candidate set. Taking Figure 4p as example, change the object color from white to black by replacing the bottom-right image. RAISE takes each image in an RPM as the target, gets the prediction results, and computes the prediction error latent concepts. The right panel of
8

---

Published as a conference paper at ICLR 2024
Table 2: Selection accuracy (%) on two held-out configurations.
OOD Settings
RAISE
PIAE
ALANS
GCA-C
GCA-R
GCA-1
Transformer
ANP
LGPP
CLAP-NP
Center-Held-Oui
99.2
99.8
46,9
35.0
14.4
12.1
12.1
10.6
8.6
19.5
O-IC-Held-Out
56.1
40.5
33.4
10.1
5.3
4.9
15.8
7.5
4.6
8.6
Figure F shows the concept-level prediction errors, and we find that the 7th concept of the bottom- right image deviates the most. According to Figure Fa, the 7th concept on Center represents the attribute Color, which is indeed the attribute modified when constructing the test. The last row has relatively higher concept distances since the incorrect image tends to influence the accuracy of answer generation at the most related positions. Because of the independent latent concepts and concept-specific reasoning processes of RAISE, the high concept distances only appear in the 7th concept. By solving 4PM-based odd-one-out one-out problems, we explain how concept-level predictions improve the interpretability of answer selection. Although RAISE is tasked with generating answers, it can handle answer-selection problems by excluding candidates violating the underlying rules.
4.5 HELD-OUT CONFIGURATIONS
To explore the abstract reasoning ability on diistii-iitiitttinnn (OOD) samples, we construct two held-out configurations based on RAVEN (Barrett et al.) 2018) as illustrated in Figure 4p. (1) Center-Held Helerr- keeps the samples of Center following the attribute-rule tuple (Size, Constant) as test samples, and the remaining constitute the training and validation sets. (2) O-IC-Held-Out keeps the samples of O-IC following the attribute-rule tuples (Type In, Arithmetic), (Size In, Arithmetic), (Color In, Arithmetic), (fyye In, Distribute Three), (Size In, Distribute Three), and (Color In, Dis- tribute Three) as test samples. The results given in Table 2 indicate that RAISE maintains relatively higher selection accuracy when encountering unseen combinations of attributes and rules. RAISE learns interpretable latent concepts to conduct concept-specific reasoning, by which the learning of rules and concepts are decoupled. Thus RAISE can tackle OOD samples via compositional gener- alization. Although RAISE has not ever seen the attribute-rule tuple (Size, Constant) in training, it can still upply the atomic rule Constant learned from other attributes Size in the test phase.
CONCLUSION AND DISCUSSION
This paper proposes generative RPM solver RAISE based on conditional deep latent variable mod- els. RAISE can abstract atomic rules from PRMs, keep them in the global knowledge set, and predict target images by selecting proper rules. As the foundation of rule abstraction and selection, RAISE learns interpretable latent concepts from images to decompose the integrated rules of RPMs into atomic rules. Qualitative and quantitative experiments show that RAISE can generate answers at ar- bitrary positions and outperform baselines, showing outstanding generative abstract reasoning. The odd-one-out task and held-out configurations verify the interpretability of RAISE in concept learn- ing and rule abstraction. By using prediction deviations on concepts, RAISE can find the position and concept that breaks the rules in odd-one-out tasks. By combining the learned latent concepts and atomic rules, RAISE can generate answers on samples with unseen attribute-rule tuples.
Limitations and Discussion. The noise in data is a challenge for the models based on conditional generation. In the experiment, we find that the noise of object attributes in grids will influence the selection accuracy of generative solvers like RAISE and Transformer on 2 2Grdd. The candidate sets can provide clearer supervision in training reduce the impact of noise. Deep latent variable models (DLVMs) can potentially handle noise in RPMs since RAISE works well on Center and O-IC with noisy attributes like Rotation. In future works, exploring appropriate ways to reduce the influence of noise is the key to realizing generative abstract reasoning in more complicated scenes. For generative solvers that do not rely on candidate sets or are completely unsupervised, whether using datasets with large amounts of noise benefits the acquisition of generative abstract reasoning ability is worth exploring since the noise can make a generative problem have numerous solutions (e.g., PGM (Barrett al.) 2018)). )). In Appendices B.2 and D.I we conduct an initial experiment and discussion on the impact of noise, but a more systematic and in-depth study will be carried out in the follow-up works. Some recent neural approaches attempt to solve similar systematic generalization problems (Rahaman et al.] 2021 Lake & Baroni 2023). We provide a discussion on the Bayesian
and neural approaches of concept learning in Appendix

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation China (No.62176060) and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
REFERENCES
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pp. 511-520. PMLR, 2018.
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12557-12565, 2021.
Ruth MJ Byme and Philip N Johnson-Laird. Spatial reasoning. Journal of memory and language, (8(5):6645575 1989.
Raymond B Cattell. Theory of fluid and crystallized intelligence: A critical experiment. Journal of educational psychology, 54(1):1, 1963.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots:
Object-centric repretttttonn and rendering of 3d scenes. The Journal of Machine Learning Research, 22(1): 2021.
François Chollet. On measure of intelligence. arXiv preprint arXiv: 1rX1v:1911.01547,200
David F Crouse. On implementing 2d rectangular assignment algorithms. IEEE Transactions Aerospace and Electronic Systems, 52(4):1679-166 1696, 2016.
Stanislas Dehaene. The number sense: How the mind creates mathematics. OUP USA. 2011.
Harrison Edwards and Amos Storkey. Towards neural statistician. In International Conference Learning Representations, 2017.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 36046994) 2018.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-leamiing stationary stochastic process prediction with convolutional neural pro- cesses. Advances in Neural Information Processing Systems, 33:8284 8295, 2020.
Chengmin Gao and Bin Li. Time-conditioned generative modeling of object-centric representations for video decomposition and prediction. In Proceedings of the Conference Uncertainty in Artificial Intelligence, pp. 613-623, 2023.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. In ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
Giorgio Giannone and Ole Winther. Hierarchical few-shot generative models. In Fifth Workshop Meta-Learming at the Conference on Neural Information Processing Systems, 2021.
Robert Gordon and David Irwin. What's in an object file? evidence from priming studies.
Perception & Psychophysics, 1868):1200-1277, 1996.
Jeremy R Gray and Paul M. Thompson. Neurobiology of intelligence: science and ethics. Nature Reviews Neuroscience, 5(6):471-45) 2004.
Lukas Hahne, Timo Lüddecke, Florentin Wörgötter, and David Kappel. Attention on abstract visual reasoning. arXiv preprint arXiv: 191 2019.
10

---

Published as a conference paper at ICLR 2024
Luke B Hewitt, Maxwell I Nye, Andreea Gane, Tommi S Jaakkola, and Joshua B Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. In Conference Uncertainty in Artificial Intelligence. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp- 1567-1574, 2021.
Tianyu Hua and Maithilee Kunda. Modeling gestalt visual reasoning on raven's progressive matrices using generative image inpainting techniques. In CogSci, volume 2, pp. 2020.
Marius Jahrens and Thomas Martinetz. Solving raven's progressive matrices with multi-layer lation networks. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-6.
IEEE, 2020.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. Advances in Neural Informa- fion Processing Systems, 13:12572-12582, 2020.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerehner, and Chris Burgess. Simone: View-invariant, temporally-abstractedd object representations via unsupervised video decomposition. Advances in Neural Information Process-
ing Systems, 34:20146-20133 2021.
Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing of object files: Object-specific integration of information. Cognitive psychology, 24(2):175-219, 1992.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
Diederik P Kingma and Max Welling. arXiv:1312.6114.220 2013.
Auto-encoding variational bayes.
arXiv preprint
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Brenden M Lake and Marco Baroni. Human-like systematic generalization through a meta-learning neural network. Nature, 123-79,5):115-12,, 2023.
Mikolaj Malkiński and Jacek Maádziuk. Deep learning methods for abstract visual reasoning: A survey on raven's progressive matrices. arXiv preprint arXiv:2201.12382 2022a.
Mikolaj Malkiáíki and Jacek Mańdziuk. review of emerging research directions in abstract visual reasoning. arXiv preprint arXiv:2202. 10284, 2022b.
Michael McCloskey. Intuitive physics. Scientific american, 248(4): 1983.
Melanie Mitchell. Abstraction and Academy of Sciences, 1505(1):79-101,
in artificial intelligence. Annals of the New York
2021. 2021.
Niv Pekar, Yaniv Benny, and Lior Wolf. Generating correct answers for progressive matrices intel- ligence tests. arXiv preprint arXiv:2011.00496 2020.
Nasim Rahaman, Muhammad Walced Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. Advances in Neural Information Processing Systems, 34: 10985, 2021.
John Raven and John Hugh Court. Raven's 's progressive matrices and vocabulary scales, volume 759. Oxford pyschologists Press Oxford, 1998.
Fan Shi, Bin Li, and Xiangyang Xue. Raven's 's progressive matrices completion with latent gaussian process priors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9612-9620, 2021.
11

---

Published as a conference paper at ICLR 2024
Fan Shi, Bin Li, and Xiangyang Xue. Compositional law parsing with latent random functions. In International Conference on Learning Representations, 2023.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in newral information processing systems, 28, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29:3738-3746,
2016.
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generaliza- tion for abstract reasoning tasks using disentangled feature representations. arXiv preprint arXiv:1811.04784, 2018.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Duo Wang, Mateja Jamnik, and Pietro Lio. Abstract diagrammatic reasoning with multiplex graph networks. In International Conference on Learning Representations, 2019.
Duo Wang, Mateja Jamnik, and Pietro Lio. Abstract diagrammatic reasoning with multiplex graph networks. arXiv preprint arXiv:2006. / /97, 2020.
Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships analogical reasoning. arXiv preprint arXiv:2007.04212. 12, 2020.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Unsupervised learning of compositional scene represen- tations from multiple unspecified viewpoints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8971-8979, 2022.
Jinyang Yuan, Tonglin Chen, Bin Li, and Xiangyang Xue. Compositional scene representation learn- ing via reconstruction: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence, 45(10)111500-11500 2023.
Jinyang Yuan, Tonglin Chen, Zhimeng Shen, Bin Li, and Xiangyang Xue. Unsupervised object- centric learning from multiple unspecified viewpoints. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2024.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chur Zhu, Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5317-5327, 2019a.
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu. Learning perceptual inference by contrasting. arXiv preprint 1oXiv:/9/2.000 2019b.
Chi Zhang, Baoxiong Jía, Song- Zhu, and Yixin Zhu. Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9736-9746, 2021a.
Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chun Zhu, and Yixin Zhu. Learn- ing algebraic representation for systematic generalization in abstract reasoning. arXiv preprint arXiv:2111.12990, 2021b.
Kecheng Zheng, Zue Zha, and Wei Wei. Abstract reasoning with distracting features. Ad- vances in Neural Information Processing Systems, 32, 2019.
Tao Zhuo and Mohan Kankanhalli. Effectivee abstract reasoning with dual-contrast network. In International Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2024
A PROOFS AND DERIVATIONS
A.1 REFORMULATION OF THE POSTERIOR DISTRIBUTION
According to Bayes' theorem, the posterior distribution of rule indicators g(r] 24) is the product of the conditional prior (z())))) and the likelihood p(zjj p(2j/rr zG):
p p (5) (zg)
( S z7)
x plz (
(13)
p(z9 r k 2g) = k((z3)
Considering that p(29)11 rr =5) is an isotropic Gaussian, NV (h (/r)) . Equation
13 becomes
1
27 - h (Z (z) 1 p(r =$}
Q( &
exp
D(=5)
20
2TT=
(14)
1
11:) p(r p(r))2))
x exp
2
27
(Z% (Z
where D(z)) is the size of zÿ. In practice, RAISE predicts unnormalized logits Üi. instead of the probabilities #f. Therefore, we use the logarithmic version of Equation 14
1 20
h (2 18 ) | log p(r)(())))) + C 2f)
log (r") 27)
(15)
in Equation 15 will not influence the results of normalization, RAISE ignores the constant term and predicts the unnormalized logits via
Since the constant C(2g, zf.
2T h (2"; V&) - 1(r = =k(z))
20
(16)
2
29 h (Z=; vv))
log The 1,... K.
2o?
Finally, the variational distribution g5 z4.) is parameterized by
exp
(1 x4-
Categorical (Tfx),, where TE
for 1,... K.
(17)
ΣK ==
exp 16
A.2 DERIVATION OF THE ELBO
With the variational distribution qu (h|æT,, æs), the ELBO C is (Shhn et al. 2015
pe (h, xr xs L. qu (h|Ww xt))
log Pe (ær(æs) 2 Egg (hlar,as)
log
(18)
Considering the generative and inference processes
Pe Pe (
II TET
Po (a, ze)
II
(r)
II Po po(25(zz))
c=l
SE.
(19)
qy (h|2T,,555) 2s)
1s (r - 24)
go (=< (x.)
II qo (2 (xt))
8ET
13

---

Published as a conference paper at ICLR 2024
Equation 18 is further decomposed by
C = Egu (klerwss)
--Eg
log II (z))
LET
lo. (1 z7) ThES do (z
(x,)
Ieer go (=(ff10))
Eg (hlæv,as))
log
Pe Ps. 28)pp (r 1 (2)) Ilee Po (2g a:s))
(20)
log qo (z (TT)
Σ tET
Egg (hlarr,a as))
[log (20e
Σ
Be (blaras))
Pvi (27 rr; =58)
log (o Zg.27. x7)
log qo [2 xz,)
gu gu (hlar.as)
2o (h my ans)
ps
(r 25
pal po (zā vv.)
SES
Since the encoder is shared between the generative and inference processes, have q8 (=[[[[x,) (2f(xx)) and
Lo (0 ats)) po (z (x.))
Eg (h ass)
log
= O.
(21)
sES
Therefore, the ELBO is
log 90(29 x.
L Σ E (h loy ms)
Eg (h ber ms)
pe (2 re, x²3)
ET
Lres
Rpred
(22)
log (0 (2, x5)
go (hlwr
Poo (ro(r0(2))
Rule
A.3 MONTE CARLO ESTIMATOR THE ELBO
For given RPM problem (æs, .T), sample the latent variables F, ēs. and By from the variatonal posterior qu (h/ (arr,, æ.s) to compute the ELBO:
z N (F. o
8 S S = 1, 1,... C
2 ~ N (Ri, o?!)
LE T,
== 1 I,..C C,
(23)
F ~ Categorical (ff.k),
c= I,...CC. C.
ã.:C = go (x. and p. p.c.C. = g6 (x;) are means of latent concepts computed by the encoder. πं given by 17 and the indicator 7 is sampled through the Gumbel-Softmax distribution (Jang al.
2016). Using the Monte Carlo estimator, L can be approximated by the sampled latent variables.
A.3.1 RECONSTRUCTION LOss
Lree log (2,12,)
Σ
Dt
A, 1|2 + Cres
where A, gdte (21:0)
(24)
20 x 1ET
tET
14

---

Published as a conference paper at ICLR 2024
A.3.2 CONCEPT REGULARIZER
log 90 (z (2)))r)
peed
(e5 (x))
B (oo les)
T ye (molag
Pvi pv / mr 25)
(g0 (es)
Ep. Reo((gi
log
9o (0(2)/20)
(2. 25)
(25)
E (x les)
[Bp.(r=(e)) DRI.
(2) =3))]]
cml
DEL (g0(2) (20(((((TTT) pa 23))
uT u Cpred-
20
c=1
Trained with rule annotations, RAISE can quickly approach the real distribution p. (r*|(z) provided in rule annotations after the early learning stage. Therefore, we regard the real distribution as the predicted rule distribution, which is related the matrix rather than conditional on the latent concepts, That is, we assume that samples from (r)(z) are similarly distributed to those from la.0 (r 2 after few learning epochs. By replacing the ge (r E4-) to px (r we move the inner expectation r to the front. In this way, the inner expectation becomes the KI. divergence between Gaussians and has a closed-form solution, reducing the additional noise in the sampling process.
A.3.3 RULE REGULARIZER
go (#r) Beo(sglus) (r log 20.0 (r') x7)
Ryuk
Po(
Σ 20(551100) (xo(aglas) [DXL (ga.v(1 (8) (
(26)
# log #
DEL (1o. (1
A.3.4 ELBO
Ignoring the constant terms Cree and Cpred- the approximation of ELBO is
K
#
L R
Dt
A
P
log
20
20
2
(27)
Lixe
R Reood
Reule
B DATASETS
B. RAVEN AND I-RAVEN
Figure displays seven image configurations of RAVEN Zhang et al. 2019a). The image attributes include Number/Position, Type. Size, and Color, which can follow the rules Constant, Progress, Arithmetic, and Distribution Three. Each configuration contains 6000 training samples, 2000 vali- dation samples, and 2000 test samples. RAVEN provides eight candidate images and attribute-level rule annotations for each RPM problem. Previous work pointed out the existence bias in candi- date sets of RAVEN (Hu et al. 2021). which allows models to find shortcuts for answer selection. I-RAVEN uses Attribute Bisection Tree (ABT) to generate candidate sets to resist shortcut learning (uu et al. 202T The experiment shows that the models trained with only the candidate sets of IRAVEN have selection accuracy close to the random guesses, which evidences the effectiveness of the candidate generation strategy.
15

---

Published as a conference paper at ICLR 2024
(a) Center
(b) L-R
(c) U-D
(d) O-C
Candidate Set
Candidate Set
Candidate Set
Candidate Set
Rules
Rules
Type Left: Constant
Type Up: Progression
Rules
Size Left: Constant
Size Up: Arithmetic
Type Out: Corstant
Rules
Color Left: Progression
Color Up: Progression
Size Out: Progression
Type: Progression
Type Right: Distribute
Type
Type Distribute Size in: Distribute Three Color Progression
Size: Distribute Three
Size Right: Progression
Size Down: Arithmetic
Color: Arithmetic
Color Right: Constant
Color Progression
(e) O4G
(f) 2x2Gri
(g) 3Grid
Candidate Set
Candidate Set
Candidate Set
Rules Type Out: Progression Num/Pos Distribute Three Type In: Distribute Three Arithmetic
Rules Num/Pos: Constant
Rules
Num/Pos: Distribute Three
Type: Progression
Type: Distribute Three
Size: Progression
Size: Distribute Three
Color in: Arithmetic
Color: Arithmetic
Color: Arithmetic
Figure 5: Different configurations of RAVEN. In each figure, the top panel is an RPM where the target images highlighted in red boxes; the middle panel is candidate with eight candidate images; and the bottom panel shows the attribute-changing rules in the RPM.
B.2 ATTRIBUTE NOISE OF RAVEN AND I-RAVEN
RAVEN and I-RAVEN introduce noise some attributes to increase the complexity of problems. In Center, L-R, U-D, and O-IC, the rotation of objects is the noise attribute. We can keep objects unchanged in rows or make random rotations. Figure displays the noise object grids on IG, 2 2x2Grid, and 3x3Grid, including the noise of object attributes (i.e., objects in Figure 6 can have different colors and rotations), and the noise object positions (Figure 6d). The candidate set ensures that only candidate image is the correct answer. To explore the influence of noise on selection accuracy, we remove the noise object attributes from object grids, keep the noise of object positions, and generate three configurations O-IG-Uni, 2x 2Grid Uni, and 3 3x3Grid-Umm
16

---

Published as a conference paper at ICLR 2024
(a) Problem Panel
(c) Noise of Object Attributes
(b) Candidate Set
(d) Noise of Object Positions
Figure 6: The illustration of attribute noise. (a) is an RPM from 2x2Grid; (b) is the candidate set; (c) and (d) visualize two possible types of noise in the RPM. In this case, the image is correct as long there are two pentagons of the correct size. The color, rotation, and position of objects will not influence the correctness
image.
C MODELS
C.I RAISE
This section introduces the architectures and hyperparameters of RAISE. The network architectures introduced in the order of 90 frow f Joy h. and g
enc RAISE used a convolutional neural network to downsample images and extract the mean of latent concepts. Denoting the number and size of latent concepts C' and d,, the encoder is
4 x 4 Cony, stride 2, padding 1, 64 BatchNorm, ReLU
4 X 4 Conv, stride 2. padding 128 BatchNorm, ReLU
4 4 Cony, stride 2, padding 256 BatchNorm, ReLU
X 4 Conv, stride 2, padding 512 BatchNorm, ReLU
4 X 4 Conv, 512 BatchNorm. ReLU
ReshaapeBlcck, 512
-Fully Connected, C d,
The ReshapeBlock flattens the feature map of the shape (512, 1, to the vector with 512 dimen- sions, which is projected and split into the mean of latent concepts.
frow and fool The two networks have the same architecture to extract the row and column representations from RPMs:
Fully Connected, 512 ReLU
Fully Connected, 512 ReLU
Fully Connected, 64
where the input size is d, and the size of output row and column representations 64.
find This network converts the overall and column representations of an RPM the logits of selection probabilities for atomic rule selection:
Fully Connected, 64 ReLU
Fully Connected, 64 ReLU Fully Connected, K
where K is the number of atomic rules. Since the row and column representations are concate- nated as the input, the input size of the network is 128.
iz::; thx). This network a fully convolutional network, which predicts the means of target latent concepts from the representation matrix Zo
x 3 Conv, stride I, padding 1, 128 ReLU
3 x 3 Cony, stride 1, padding 128 ReL.U
17

---

Published as a conference paper at ICLR 2024
3 x 3 Conv, stride I, padding d,
h adopts convolutional layers with 3x3 kernels, stride I, and padding to keep the shape of the 3x3 representation matrix. The global knowledge set V1:K stores K learnable parameters of h, which represents K atomic rule respectively.
doc The decoder accepts all latent concepts of an image as input and outputs the mean of the 99
pixel values for image reconstruction. The architecture
ReshapeBlock, (C 1,1)
X 1 Deconv, 256 BatchNorm, LeakyReLU
4 X 4 Deconv, 128 BatchNorm, LeakyReLU
X 4 Deconv, stride 2, padding 64 BatchNorm, LeakyReLU
4 X 4 Deconv, stride 2. padding 32 BatchNorm, LeakyReLU
X 4 Deconv, stride 2, padding 32 BatchNorm, LeakyReLU
4 X 4 Deconv, stride 2, padding 1, Sigmoid
where the negative slope of LeakyReLU is 0.02. Since the images of RAVEN and I-RAVEN are grayscaled, the decoder output only one image channel and uses the Sigmoid activation function to scale the range of pixel values to (0, 1).
For all configurations of RAVEN, we set learning rate as 10 batch size as 512, K 4, 0x 0.1, 0= 0.1, C 8, d_ 8, 31 5, 32 20, and B3 10. RAISE is insensitive when increasing C since it can generate redundant latent concepts. When C is too small to encode all attributes, the selection accuracy will decline significantly. We set large C and reduce it until the number of redundant latent concepts is reasonable. In general, we choose K by directly counting the number of unique labels in rule annotations. RAISE updates the parameters through the RMSprop optimizer (Hinton et al. 2012). To select the best model, we watch the performance on the validation set after each training epoch and save the model with the highest accuracy.
C.2 POWERFUT. GENERATIVE SOLVERS
ALANS (Zhang et al. 2021b) We train ALANS on the codebase released by the authors setting the learning rate 0.95 10 and the coefficient of the auxiliary loss 1.0. Since the model can hardly converge from the initialized parameters, we initialize the parameters of ALANS with the pretrained checkpoint provided by the authors. More details can be seen in the repository.
For PrAE, we use the commended hyperparameters that the learning rate is 0.95 10 and the weight of auxiliary loss is 1.0. The implementation of PrAE is based on
PrAE (Zhang et al. 2021a)
the official repository
GCA (Pekar et al. 2020) The official code of GCA only implements the auxiliary loss on the PGM dataset (Barrett et al. 2018). Therefore, we modify the output size of the auxiliary network to the size one-hot rule annotations RAVEN/I-RAVEN, We set the latent size in GCA as 64 and
the learning rate 2 x 10-4
C.3 BASELINES
Transformer (Vaswani et al., (017) To improve the model capability, we first apply the encoder and decoder to project images into low-dimensional representations and then predict the targets in the representation space via Transformer. Transformer uses the same encoder and decoder struc- tures as RAISE. The hyperparameters of Transformer are chosen through grid search. We set the learning rate as x 10 from (5 10 - x 10-4, 5 x 10-5), the representation size as 256 from (512, 256, 128}. and the number of Transformer blocks as 4 from {2, 4, 6}. In addition, the number of attention heads is 4, the hidden size of feedforward networks is 1024, and the dropout is 0.1. All parameters updated by the Adam (Kingma & Ba 2015) optimizer.
18

---

Published as a conference paper at ICLR 2024
Table Learning rates of ANP on RAVENII-RAEEN
Center
L-R
U-D
O-IC
O-IG
2x2Grid
3x3Grid
5 x 10
x 10-6
x 10-5
x 10
5 x 10-6
3 10-5
3 10-5
Table 4: Hyperparameters of CLAP. We give the number of concepts, weights in the ELBO (8,, 8y, and Brc), and standard deviation ds on RAVEN/I-RAVEN.
Hyperparameters
Center
L-R
U-D
O-IC
O-IG
2x2Grid
3x3Grid
#Concepts
10
Be
100
50
50
30
30
30
80
B1
100
50
50
60
30
30
80
Brc
100
50
50
50
30
30
80
aa
0.1
0.3
For configurations, we set the size of the global latent as 1024 and the + shows the confiuuraionn-speciii learning rates. Other hyperparameters and the model architecture remain the same as the 2D regression configuration the original paper (Kim al. 2019).
ANP (Kim et al. 2019
batch size 512. Table
LGPP (Si et al. 2021 In the experiments, we use the official code of LGPP 5 by setting the learning rate
TO and the batch size 256. In terms of model architecture, axis latent variables as 4, the size of axis representations as 4, and the input size of the RBF kernel 8. The network that converts axis latent variables to axis representations has hidden sizes [64, 64]. The network to extract the features for RBF kernels has hidden sizes [128, 128, 128, 128]. The hyperparameter 33 that promotes disentanglement LGPP is set 10. For the configuration Center, the number of concepts is 5, while the others use 10 concepts.
set the size of
CLAP (Shi et al., 2023) Here we adopt the model architecture the CRPM configuration in the official repository and adjust the learning rate to 5 x 10-1 the batch size to 256, and the concept size Other hyperparameters are displayed Table 4
C.4 COMPUTATIONAL RESOURCE
All the models are trained on the server with Intel(R) Xeon(R) Platinum 8375C CPUs, 24GB NVIDIA GeForce RTX 3090 GPUs, 5 512GB RAM, and Ubuntu 18.04.6 LTS. RAISE is implemented with PyTorch (Paszke et al. 2019).
D ADDITIONAL EXPERIMENTAL RESULTS
D. BOTTOM-RIGHT ANSWER SELECTION
We generate new configurations by removing the noise in object attributes to analyze the influence of noise attributes. As shown in Table 5) RAISE achieves the highest accuracy on all three config- urations. When introduce rules will increase. In this case, the provided candidate set with one correct answer and seven dis- tractors
noise to RPMs, the number of solutions that follow the correct
clear supervision in model training. Without the assistance of candidate sets in training, it is challenging to catch rules from noisy RPMs with multiple potential solutions. There- fore, RAISE and Transformer have significant accuracy improvements on configurations with less noise attributes. Overall, the experimental results show that reducing noise can bring significant improvements for the models trained without distractors in candidate (such as Transformer and RAISE). RAISE only requires 20% rule annotations to learn atomic rules from low-noise samples.
'htts://gituu..om/uuuan hwtp://www.cccccoaaaaaa/aaaaaatn
19

---

Published as a conference paper at ICLR 2024
Table 5: The accuracy (%) of selecting bottom-right answers on O-IG-Uni, 2x2Grid-Uni and 3 3Grid-Uni.
Models
O-IG-Umi
2x2Grid-Umi
3x3Grid-Umi
GCA-I
21.2/36.7
19.5/23.3
20.6/21.4 66
GCA-R
20.7/36.3
21.9/28.1
25.9/25.2
GCA-C
53.8/37.7
58.8/35.6
67.0/27.5
PrAE
29.1/45.1
85,4/85.6
26.8/47.2
ALANS
29.7/41.5
66.2/55.3
84.0/73.3
LGPP
3.4/12.3
4.1/13.0
4.0/13.1
ANP
31,5/34.0
10.0/15.6
12,0/16.3
CLAP
14.4/31.7
22.5/39.1
12.1/32.9
Transformer
70,6/57.9
73.3/73.0
34.2/37.0
RAISE
95.8/99.0
87.6/97.9
95.3/93.2
Table 6: The accuracy (%) of selecting bottom-right answers different configurations (i.e.,
Center, L-R, etc) of RAVEN/I-RAVEN In this table, RAISE is trained without the supervision of rule annotations (-aux) illustrate the abstract reasoning ability in the unsupervised training setting. The table displays the average results of ten trials.
Models
Average
Center
L-R
U-D
O-IC
O-IG
2x2Grid
3x3Grid
LGPP
6.4/16.3
9.2/20.1
4.7/18.9
5.2/21.2
4.0/13.9
3.1/12.3
8.6/13.7
10.4/13.9
ANP
7.3/27.6
9.8/47.4
4.1/20.3
3.5/20.7
5,4/38.2
7.6/36.1
10.0/15.0
10.5/15.6
CLAP
17.5/32.8
30.4/42.9
13.4/35.1
12.2/32.1
16.4/37.5
9.5/26.0
16.0/20.1
24.3/35.8
Transformer RAISE (-aux)
40.1/64.0
98.4/99.2
67.0/91.1
60.9/86.6
14.5/69.9
13.5/57.1
14.7/25.2
11.6/18.6
54.5/67.7
30.2/56.6
47.9/80.8
87.0/94.9
96.9/99.2
56.9/83.9
30.4/30.5
32.0/27.8
We also provide the selection accuracy of unsupervised RAISE in Table The average accuracy of unsupervised RAISE lies between the unsupervised urbitrary-generann baselines (i.c., LGPP, ANP, CLAP, and Transformer) and the powerful generative RPM solvers trained with full rule annotations (i.e., GCA, ALANS, and PrAE).
D.2 ANSWER SELECTION ARBITRARY POSITION
In this section, we give additional results for urbitrary-position answer generation. Figure provides the detailed results of for example, the prediction results when [T] (Figure 8b) and [T] visualization results, RAISE can generate high-quality predictions when [T] 1 and T = 2. The performance of Transformer varies significantly among different configurations. Transformer predicts accurate answers Center, while the predictions 3 3Grid deviate significantly from the ground truth images. In most cases, ANP, LGPP, and CLAP tend to generate incorrect images. provides the selection accuracy I-RAVEN with different numbers of target images ([T) 1,2) and different numbers of distractors candidate sets (N₂ = 1, 7, 15). We can make further analysis through the selection accuracy with test errors in Tables X and where RAISE outperforms other baseline models on all image configurations of RAVEN and I-RAVEN.
answer generation for all seven configurations of RAVEN, (Figure KD). In the
ProuidePr
D.3 LATENT CONCEPTS
As mentioned the main text, concept learning is an important component of RAISE. This section shows the interpolation results of latent concepts on all image configurations and the correspon- dences between latent concepts and real attributes RAISE can learn independent latent concepts and the binary matrix M that accurately reflects the concept-atrribute correspondences. RAISE does not assign the latent concepts encoding object rota- tions to any attribute since the noise attributes are not included in rule annotations. This experiment illustrates the interpretability the acquired latent concepts, which benefits the prediction correct answers and the following experiment of odd-one-out.
Figures and 10 In most configurations,
20

---

Published as a conference paper at ICLR 2024
Center
U-D
Q-IC
C-15
2Grd
3Grid
Barsc
RAST
BAISE
BASST
RAESE
RAJSE
Tranzo
Trnssforer
Varstome
CLAP
APP
CLAP
CLAP
AdI
CLPP
Tonstamicr
Tharatorm
LGPP
32
LGPP
LGGs
RMISE
RAISE
BAyS!
WiLt
mun
Tansforma
ststiiee
Farsffrmet
rardfamcc
[T~2
CLAP
CUP
CCAP
CLAP
MP
Ponstomee
37
Figure 7: Selection accuracy at arbitrary positions on I-RAVEN. Each plot contains the selection accuracy of RAISE (purple), Transformer (orange), CLAP (green), ANP (bluc), and LGPP (black). The x-axis is the number of candidates, and the y-axis is the selection accuracy.
Table 7: The accuracy (%) using different strategies of answer selection.
Models
Average
Center
L-R
U-D
O-IC
O-IG
2x2Grid
3x3Grid
RAISE-latent 90.0/92.1
99.2/99.8
98.5/99.6 90.6/95.8
99.3/99.9
97.6/99.6
89.3/96.0
68,2/71.3
77.7/78.7
RAISE-pixel
72.9/77.8
95.2/96.8
96.6/98.5
80.4/90.6
69.1/81.1
40.1/42.6
1/39/
D.4 ODD-ONE-OUT IN RPM
In this experiment, we provide the additional results of odd-one-out on different configurations where RAISE picks out nule-breaking images interpretably via prediction errors on latent concepts. Figure 11 visualizes the experimental results of odd-one-out. RAISE will display larger prediction errors at odd concepts, which important evidence when solving odd-onc-out problems. It should be pointed out that forming such concept-level prediction errors requires the model to parse inde- pendent latent concepts and conduct concept-specific abstract reasoning correctly. RAISE can apply the atomic rules in the global knowledge set to tasks like out-one-out and has interpretability in generative abstract reasoning.
D.5 STRATEGY OF ANSWER SELECTION
In this experiment, we evaluate RAISE with two strategies of answer selection: comparing candi- dates and predictions in pixel space (RAISE-pixel) and latent space (RAISE-latent) reporss higher accuracy when candidates and predictions are compared latent space. Due to the noise in attributes, there can be multiple solutions to a generative RPM problem. Assume that the answer to an RPM the image having two triangles, the answer images may significantly differ from each other in the pixel space by generating two triangles in various positions. However, they still point to the same concepts Number=2 and Shape=Triangle the latent space. Therefore, selecting answers by comparing candidates and predictions in the latent space be more accurate than comparing the pixel space.
E DISCUSSION ON BAYESIAN AND NEURAL CONCEPT LEARNING
The learning objective. A recent neural approach MLC (Lake & Baroni 2023) uses meta-learning objectives to solve systematic generalization problems. Grant al. (Grant al. 2018 g have reported connection between meta-learning and hierarchical Bayesian models. The discussion section of MLC has also mentioned that the hierarchical Bayesian modeling can be explained from the view of meta-learning. In this perspective, the global atomic rules in RAISE act as global latent variables in hierarchical modeling. Although RAISE and MLC have different motivations for model design, there are potential connections and similarities between their learning objectives if we explain the reasoning process RAISE from the perspective of hierarchical Bayesian modeling and meta- learning.
21

---

Published as a conference paper at ICLR 2024
Interprefability of latent variables. Both Bayesian and neural approaches can define basic mod- ules in the learning processes, e.g., Functions in Neural Interpreters (Rahaman et al., 2021 and atomic rules in RAISE. Bayesian approaches usually design interpretable latent variables in gener- ative processes, e.g., RAISE uses categorical random variables to indicate the types of the selected rules explicitly. While Neural Interpreters route inputs different Functions by calculating specific scores. DLVM provides a powerful learning framework to learn interpretable latent structures from data, e.g., RAISE defines latent concepts capture image attributes. In this way, visual scenes are decomposed into a simple set of latent variables, which may reduce the complexity of abstract reasoning and enable systematic generalization on attribute-rule combinations.
Solving multi-solution problems. There can be multiple solutions for one generative reasoning problem due to the noise data. DLVMS can handle multi-solution problems by stochastic sampling from the generative and inference processes. For example, RAISE can produce results different from the original sample but still follow the correct rules. Instead of making deterministic predictions, DLVMs attempt to provide probabilities of generating specific answers and capture randomness and uncertainty in abstract reasoning.
REFERENCES
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference machine learning, pp. 511-520. PMLR, 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical bayes. arXiv preprint 1801.08800.89000 2018.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Sheng Hu, Yuqing Ma. Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1567-1574, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv: 1. 01 144, 2016.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference Learning Representations, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Brenden M Lake and Marco Baroni. Human-like systematic generalization through a meta-learning neural network. Nature, (23(7985):115-11) 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Niv Pekar, Yaniv Benny, and Lior Wolf. Generating correct answers for progressive matrices intel- ligence tests. arXiv preprint axX1v:2011.00496, 2020.
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. Advances in Neural Information Processing Systems, 34: 10985-19999 2021.
Fan Shi, Bin Li, and Xiangyang Xue. Raven's 's progressive matrices completion with latent gaussian process priors. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9612-9620, 2021.
Fan Shi, Bin Li, and Xiangyang Xue. Compositional law parsing with latent random functions. arXiv preprint 20X 2022.
22

---

Published as a conference paper at ICLR 2024
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Chng-CCunn& Zhu. Raven: A dataset for relational and analogical visual reasoning. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu. Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings the Vision and Pattern Recognition, pp. 9736-9746, 2021a.
the IEEE/CVF Conference on Computer
Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chun Zhu, and Yixin Zhu. Learn ing algebraic representation for systematic generalization in abstract reasoning. arXiv preprint 11X192211..12990, 2021b.
23

---

Published as a conference paper at ICLR 2024
Table 8: Answer generation at arbitrary positions RAVEN. We provide the average accuracy (%) and test errors (%) of ten trials on RAVEN for RAISE and baselines.
Center (TT) =
Center (IT) = 2)
Model
N₂ =
=
=
N, 15
N: =
N: =o=0 3
No =
N₂ 15
LGPP
2.5
30.8 1.9
17.1 2.0
9.1 1.2
53.8 # 1.4
28.9 1.6
15.4 + 1.2
8.3
ANP
6.7
0.7
0.9
14.5 0.7
58.3
34.7
20.5
12.2
CLAP
0.75
80.1
53.8
80.3
Transformer
99.1 0.2
98.5
97.3
97.2
91.1
88.0 4.0
90.2
RAISE
99.9 0.1
99.6 + 0.2
99.1 + 0.2
98.1 + 0.3
99.5 0.2
98.7 + 0.3
97.5 + 0.7
96.5 + 0.5
L-R ([7)
1)
L-R ((7)
2)
Model
=1=1
N: 3
N. R==7
N₂ 15
N: =1
N: 3
N₂ N==7
N₂ 15
LGPP
2.3
32.7 + 3.2
18.7 +: 2.0
9.6 1.3
574 2.1
31.9 + 1.9
18.4 + 2.0
9.4 1.1
ANP
5..0
34.6 +
20.6+
+1.7+00.5
60.5
1.2
36.3 +
21.7 + 0.9
12.6 0.7
CLAP
0:
60.7
80.3
62.6
02.62:6
46.4
33.9
Transformer
0.2
98.8
0.3
98.1 1 0.4
97.1 0.3
95.8
90.5
2.3
87.2 1: 2.8
8144 4.9
RAISE
99.9
99.9
0.0
99.9 + 0.0
99.9 + 0.1
99.9 0.1
99.7
0.2
99.3 + 0.4
98.8 + 0.7
U-D ([T)
1)
U-D ([T)
2)
Model
=
N: =3=3 3
N: =
N, 15
N:
N: 3
N =
N 15
LGPP
2.3
32.8
2.0
19.7 +
10.3 1.3
57.6 1.5
32.5 4 1.5
18.0 4: 1.2
10.2
+ 1.1
ANP
1.1
34.3
0.6
1944 0.8
10.7
59.6
0.6
35.6
1.4
20.8 0.5
11.9
0.8
CLAP
78.8
59.1
43.1 1.3
784 1.6
2.8
31.5
Transformer
0.2
0.3
96.5
94.8
1.7
85.2 2
75.6
3.1
70.6 1.9
RAISE
99.9 0.0
99.9 0.0
99.9 1 0.0
99.9 + 0.0
99.6 0.2
99.1 = 0.3
98.2 - 0.5
97.1 + 111
O-IC (IT)
1)
O-IC ([T)
2)
Model
-
N. 3
N. 7
N, 15
N:
N. 3
N. 7
N, 15
LGPP
50.5 1.3
0.5
13.2
6.6 + 0.5
49.8 1.3
25.7 2 1.1
12.8 0.5
6.7 0.4
ANP
62.0
0.7
26.5
17.1 0.6
6166
38.6 1.3
24.3 1.2
15.2
CLAP
9.1
81
68.1
$4.1
9..
814
2.4
68.8
575
Transformer
0.4
0.6
90.1
82.3
96.7
92.1 3.3
90.2
8.02
RAISE
9.0
9.0
9..9
99.8 0.1
99.9
99.9 0.1
99.9 0.2
99.8 + 0.1
04G ([T) =1)
O4IG ([T)
Model
=1=1
= 3
N. N==7
N, 15
=
N 3
N. =
N₂
LGPP
49.9
25.0 1.2
12.0 0.6
6.1 0.3
5..0
25.1
1.2
12.1
6.4:0.5
ANP
1.1
4.11
30.1 2.0
2..2
66.5
44.0
1.4
28.5
18.0
CLAP
1.5
584
4..2+
30.5 1.0
80.5
63.1
47.6 322
35.2+
Transformer
9..9
0.5
0.9
8.8
9.7
1.6
3.8
RAISE
99.9 0.0
9..9 0.1
99.7
99.5 0.3
99.9 0.0
99.9 0.0
99.9 + 0.1
99.9 + 0.1
2x20rid
2x20nd ([T)
2)
Model
N.=1
N
N.=7
N,
N.
N. 3
N.=1
N₂
LGPP
51.3
26.7
13.6
6.9106
52.6
27.0
13.8
7.2 0.6
ANP
4:.8
0.7
18.2
9.5 9.5
55.5 1.0
31.7 + 0.8
18.4 0.8
10.5 0.6
CLAP
6.1
39.
24.5
15.0 1,
64.9
1.
41.8
25.2
16.9 1.4
Transformer
1.231112
44.0
30.3
21.6
63.1
43.3
28.8
20.9
RAISE
0.3
0.7
89.8 0.6
85.9 L1
96.5 0.4
92.1
87.5 21
83.2 1 1.7
3x3Grid ([T)
1)
3x3Gri ([T)
2)
Model
=1
N.=3
N. N==7
N
N1=1
N
N. T==7
N
LGPP
53.2 1.3
0.9
14.8
8.1 1.0
52.8
27.9
14.8
7.8
ANP
53.9
29.7
16.7 0.2
9.4
55.0
313 1.4
17.9
1..4
CLAP
1..2
1..2
564
43. 1.2
8.1
72
60.9 3.4
47.1 4.0
Transformer
5.84
37.8
2.
16.2
5..5
3..
23.6
16.4
RAISE
9.2
9.2
97.0
95.1
984
97.2
95
93.6
24

---

Published as a conference paper at ICLR 2024
Table 9: Answer generation at arbitrary positions I-RAVEN. We provide the average accuracy (%) and test errors (%) of ten trials on I-RAVEN for RAISE and baselines.
Center (TT)
1)
Center (IT) = 2)
Model
N: =
= 3
No =
N, 15
N: =
N: =o=3 3
No =
N₂ = 15
LGPP
30.0 2.9
16.8 + 1.6
9.0
54.3
28.7 1.5
15.7 + 1.4
8.4 + 0.8
ANP
77.1++1.2
63.1
53.0 1.1
45.3 +.3
6
42.3
1.0
28.0 0.8
18.1 0.8
CLAP
79.6
67.1
53.4
90.8
$1.7 4.2
Transformer
9.1
98.9 9.2
97.8 07.8
9.2
93.1 +.1+422
87.6
85.8
6.1
RAISE
99.9 0.1
99.7 + 0.1
99.3 + 0.2
98.3 + 0.3
99.5 0.2
98.8 0.4
97.8 + 0.4
96.1 + 1.3
L-R ([7)
1)
L-R ([T)
2)
Model
=1
N: 3
N: W2=7
N₂ 15
N: =
N: 3
N₂ R==7
N₂ 15
LGPP
2.9
31.9 3.3
19.4 + 2.3
11.2 1.1
56.7 + 1.9
32.5 + 2.2
18.7 + 1.7
11.0 1.0
ANP
71.4
49.8
24.0 + 1.3
68.7
1.3
46.3
1.0
30.8
+.8+008 0.8
20.4
CLAP
8.6
61.5 1.3
45.9 1.7
33.3
1.3
80.5
63.2
47.4
2.8
35.1
Transformer
0.1
9.4
99.0
98.8
1 0.3
96.4
93.1
89.3 1 2.5
86.9 6.1
RAISE
99.9 0.0
99.9 + 0.0
99.9 + 0.0
99.9 + 0.0
99.9
0.1
99.7
0.1
99.4 + 0.3
99.3 + 0.5
U-D ([T)
1)
U-D ([T)
2)
Model
=
N: =3
N: =
N₂ 15
N:
N: 3
N₂ =
N 15
LGPP
58.5
33.3
2.0
19.2 +: 3.8
10.8 + 1.5
56.1 2.6
32.6
2.2
19.6 2.6
10.7 + 1.5
ANP
69.5
49.1
1.2
33.8 - 0.9
23.0 + 1.3
66.7
44.1 + 1.3
29.2 1.1
18.9
0.6
CLAP
0.9
60.4
32.3
1.4
79.8
2.0
59.9
32.0
Transformer
0.11
0.3
98.5
97.7
94.8
87.6
82.0 3.1
76.5
RAISE
0.0
99.9
0.0
99.9 + 0.0
99.9
1 0.0
99.6
0.2
99.1 0.3
98.1 + 0.5
96.9 + 0.8
0-IC (IT)
O-IC ([T)
2)
Model
- 1
N 3
N. -
N, 15
N:
N. 3
N.
N, 15
LGPP
31.4100
0.74
12.9
6.7 + 0.5
50.5
1.3
25.8 0.7
13.1 0.7
6.6 0.5
ANP
0.7
1.0
5.9
$1.+1.11
71.6
1.2
51.5 1.3
37.9 1.6
26.9 2.0
CLAP
9.
814
68.6223
478
915 1.6
2.7
72.1
57.1 37.13.7
Transformer
0.21
0.3
0.4
92.9
97.9
96.6
94.2
90.0 10.0552
RAISE
9.0
0.0
0.1
99.9 0.1
99.9
99.9 0.0
99.9
99.8 + 0.1
04G ([T) =I)
O4IG ([T)
Model
=1=1
= 3
N
N,
=
N: 3
N. =
N₂
LGPP
50.0
24.8 1.0
12.6
6.2
49.7 0.7
24.9 0.8
12.4
6.7 0.4
ANP
0.7
7.22
60.3
0.9
75.9
57.5 1.2
12.1224
29.9 1.5
CLAP
7..0
6.12+
4..2+1.11
32.2 1.9
81.0
64.2+1.3
49.0
36.4
2.6
Transformer
0.3
0.
0.4
9.7
0.9
9.
87.9
83.5 1.8
RAISE
99.9
9..9
99.9
99.8 +
99.9 0.0
9..
99.9 - 0.1
99.9 + 0.1
2x2Grid (IT)
2x20rd ([7)
2)
Model
=
N.=3
N.=7
N,
N. =1
N.=3
N.=7
N
LGPP
5.8
26.6
13.9
7.1 0.5
51.9
26.4 0.6
13.7
4 0.4
6.8
ANP
54.9 1.0
31.4 1.0
+
9.9 0.7
55.6 + 0.7
18.4
+ 1.0
10.5
1.0
CLAP
63.9 1.4
4.5
25.4 12
14.8
1.0
64.8 +:
64.8
42.5
26.6
+ 18
16.1 2.1
Transformer
1.4
45.3
32.1
1
24.2
1
0.8
64.2
1.9
1
22.7
RAISE
9.
0.6
911107 0.7
87.0
0.7
9641
93.2
89.0 1
85.5 2.3
3x3Grid (IT)
1)
3x3Grid ([T)
2)
Model
=1=1
N.=3
N. =
N.
N1=1
N.=3
N. N==7
N
LGPP
52.4
27.2 1.3
14.9 1
8.0
52.4
2.22
15.0
8.1
ANP
1.2
29.5 1.0
0.8
10.1
54.5
3.76
17.411
1..2
CLAP
85.9.11..2
1..7
56.6
42.6
85.6
7..
60.0
46.4
Transformer
1.37
3..7088
2.70
1.5
1.0
1..4
24.5
1..0
RAISE
99.6
98.8 0.2
9.3
95.5
98.8
9.
94.8 1.5
92.2.2.5
25

---

Published as a conference paper at ICLR 2024
+
A
Transformet
(a) The results of answer generation when [
Center
L-R
UD
O-IC
O-IG
2x2Grid
3x30nl
Transformer
(b) The results of answer generation when [
2
Figure 8: Answer generation at arbitrary positions. The predictions are given in red boxes to illustrate the ability of (a) arbitrary-position and (b) multiple-position answer generation.
26

---

Published as a conference paper at ICLR 2024
Type Left
Size Left
Color Left
Type Right
Size Color Right
Concept Interpolation L-R
Right
1
Concept Interpolation U-D
Type
Size
Color
Down Down Down
Type
Size
Color
o
1
o
0
0
0
Figure 9: Concept learning on RAVEN. The table shows the interpolation results of latent concepts and the binary matrices indicating the correspondence between concepts and attributes on L-R and U-D.
27

---

Published as a conference paper at ICLR 2024
Num/ Pos
Type Out
Type
Size
Color
Concept Interpolation O-IG
In
In
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
0
0
#
#2
#
S2
22
sB
98
0
0
88
88
88
28
8B
8B
26
1
0
88
88
88
88
88
28
28
88
38
88
AB
Num/
Concept Interpolation on 2x2Grid
Type
Size
Color
1
0
Num/ Pos
Concept Interpolation on 3x3Grid
Type
Size
Color
0
Figure 10: Concept learning on RAVEN. The table shows the interpolation results of latent con- cepts and the binary matrices indicating the correspondence between concepts and attributes on O-IG, 2 2Grid, and 3x3dr..
28

---

Published as a conference paper at ICLR 2024
L-R
U-
0
Original
Original Sample
O
Prediction destations
Sample
Fredition deviltions
O-C
Q-IG
Origgia Sample
Origincl Sampic
/
Sample
Prddction destations in inten concepts
Sammpe
3318122
2x2Grd
3x:3Gno
Original
Original Semple
Sample
Srediction destadinn lntent concepts
Figure 11: Odd-one-out based on RPMs. The plots display how construct odd-one-out from different configurations of RPMs and how to find the odd image according to the prediction emrrr on latent concepts.
29


Reviews:
[{"review_id": "Q4xfHI1M1B", "reviewer": "Reviewer_5rkm", "summary": "The authors propose a novel method (RAISE) for solving Raven's Progressive Matrices (RPM) abstract visual reasoning problems. RAISE is a deep latent variable model, that learns the underlying rules in an RPM problem as latent variables and then uses the learned rules to conditionally (based on the context rows/columns present in one problem) generate the target image. The authors show that this leads to strong results for standard RPM reasoning tasks, generalizes better to more esoteric tasks (like answer selection at random locations). The visualization of latent concepts also shows that the latent variables are able to reasonably capture the atomic rules underlying the reasoning problem.", "questions": "I don't have particular questions regarding the paper: I think it is a good paper, and the authors achieve what they set out to do in the introduction. My main concern is whether this is the right thing to do, especially for more \"real-world\" reasoning problems where the cardinality of the rule set is exponentially larger. I think the paper would benefit from the authors providing a more comprehensive discussion on Bayesian vs Neural concept learning since their current motivation on building a DLVM is not very convincing. \n\nMinor question: In Sec 5. \"Too much noise will make a problem have a large number of solutions (e.g., PGM (Barrett et al., 2018)), such data may not be proper for validating the generative reasoning ability of models\" I am not exactly sure what the authors mean by \"a problem have a large number of solutions\" - does this refer to potentially many rule compositions satisfying the bottom-right answer selction in RPMs.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The motivation of building a generative solver versus a selective one to avoid shortcuts in discriminative reasoning task is very valid and has been highlighted in prior literature [1, 2]. I commend the authors for trying to solve the harder problem in RPM reasoning: mapping the underlying data generative process.  \n* The RAISE model is very well thought our and describe in the paper. The modelling choices make sense for the few-shot style reasoning tasks in RPM problems. Also see weaknesses regarding the motivation and usability in a broader context.\n\n\n**References** \n\n1. Hu, S., Ma, Y., Liu, X., Wei, Y. and Bai, S., 2021, May. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 2, pp. 1567-1574).\n2. Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M. and Wichmann, F.A., 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), pp.665-673.", "weaknesses": "* In Sec 1. Introduction: \"It has been suggested that Bayesian inference with shared latent concepts can explain the emergence of in-context learning in LLMs, which is the foundation of few-shot reasoning in tasks like RPM tests.\" This premise has several logical leaps which the authors don't explain. For example, Chan et al [1] showed that the nature of training data for natural language enables few-shot reasoning from in-context learning. This does not hold for RPM reasoning tasks - the data distribution does not have the same long tailed and co-occurence properties as natural language. Similarly, Bayesian inference of shared latent concepts being able to explain in-content learning is just one lens of explainability for this empirical phenomenon, and there can be other lenses which are purely frequentist (e.g. the one showed by Chan et al). Overall, the motivation derived for RAISE from this statement is very loose and falls apart on deeper inspection.\n\n* My chief concern witht this paper, and the overall idea behind structuring latent representations and re-combining them at inference is how scalable is it beyond toy problems like RPMs? The bottleneck with DLVMs like RAISE is two-fold: learnability (e.g. authors re-use encoder parameters between generative and inference process) and reliance on human knowledge to design the graphical model. RPMs have a very limited rule set, and the visual abstraction process is vastly simpler than human scenes. These reasons why existing concept learning methods that rely on similar modelling and inference have not been able to scale to harder visual reasoning problems are yet to be solved.  \n\n* In Sec 3.1 \"Previous studies have emphasized the role of abstract object representations in the abstract reasoning of infants, which is similar to the idea of object-centric representation learners that decompose complex scenes into object representations. Both views reflect the compositionality of human cognition (Lake et al., 2011).\" I don't know if the human analogy to concept learning should drive artificial concept learning agents, particularly because the underlying constraints (inductive biases?) on human and artificial systems are vastly different. A much more recent work from Lake et al [2] shows that neural network based reasoning combined with a meta learning objective can solve similar systematic generalization problems. I would also refer the authors to the discussion section of the paper for a much more nuanced discussion on Bayesian vs Neural approaches to learning systematicity. For a similar treatment of systematic reasoning by learning the underlying rule structure and routing at inference but from a purely neural treatment, I would refer the authors to [3] which also reports strong results on RPM style reasoning tasks.\n\n**References** \n\n1. Chan S, Santoro A, Lampinen A, Wang J, Singh A, Richemond P, McClelland J, Hill F. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems. 2022 Dec 6;35:18878-91.\n2. Lake, B.M. and Baroni, M., 2023. Human-like systematic generalization through a meta-learning neural network. Nature, pp.1-7.\n3. Rahaman, N., Gondal, M.W., Joshi, S., Gehler, P., Bengio, Y., Locatello, F. and Schölkopf, B., 2021. Dynamic inference with neural interpreters. Advances in Neural Information Processing Systems, 34, pp.10985-10998.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VahTwBgmZp", "reviewer": "Reviewer_w266", "summary": "This paper introduces a conditional generative model for Raven progressive matrices. It is trained from images, encoding each image into a set of C Gaussian latents, which can be decoded back to images, while also learning a set of K rules governing how a target image is composed (given its position in the grid and what the inferred progressive rule is). It is trained through an ELBO loss with some partial auxiliary rule supervision.\n\nIt shows clear advantages compared to existing baselines, both in terms of actual accuracy, but also in flexibility, as it can generate images in arbitrary cells of the RPM.\n\nOverall, the generative model is rather straightforwardly designed, albeit feels rather specific to RPM problems, hence this is addressing a very specific problem and significance might be limited. I also have a few reservations about some baselines (in particular the Transformer one).", "questions": "1. Do you have any suggestions for what one can learn from your model that can be generalized away from the RPM setting? Any specific insights / technical novelty compared to previous works?\n2. Most decisions were extremely clear, but it was not that well explained how candidate answer selection was performed (Section 4.1 and 4.2)\n   1. Do you generate a sample x_t and compare to the x_candidates in pixel space?\n   2. Do you generate a z_t and compare g^enc(x_candidates)?\n   3. Do you instead do a likelihood test?\n3. Finding values for C and K required going into the Appendix (C=8, K=4), and their choice wasn’t discussed.\n   1. How sensitive is the model to varying C or K?\n   2. How were these chosen? How adapted to the number of attributes and real rule numbers do they have to be? I’m aware they are different, but how sensitive is it e.g. can you use K=100?\n4. The Transformer baseline lacked details, even in Appendix C.3.\n   1. Which encoder did you use? Was it a discrete representation? Transformers behave much better on VQ-like latents.\n   2. I somehow expected it to do better, e.g. if attributes were provided instead of pixels, isn’t a Transformer a pretty strong baseline?\n5. How needed was the rule supervision?\n   1. Do you have numbers when you drop this to 0?\n6. Nits/typos:\n   1. It would have helped to explicitly write that r^c takes values between 1 and K.\n   2. \\mu_T^c -> \\mu_t^c in (4)\n   3. z_T^c -> z_t^c in (6)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is very clear, presents the problem well, and the math is clear and easy to follow. Figure 1 is very helpful to unpack the generative process, and overall I feel like it made all its choices clear. I could find nearly all details I needed about the implementational details (see questions below)\n2. Experiments are comprehensive and well executed, with a good choice of baselines (I am not an expert in RPM however)\n3. From results shown in Figure 2, I think it’s fair to say that it “solves” PGM quite effectively, but an expert might be able to comment better on the complexity of this problem for current SOTA.", "weaknesses": "1. This is a generative model designed specifically for RPM, and it is unclear how one would leverage this work or its findings in any other context.\n2. It is equally a “straightforward” application of an ELBO-based generative process. It is well executed, but not surprising and I did not find particularly interesting pieces of technical/insightful choices throughout the paper.\n3. It is unfortunate that some amount of rule annotation is still used. It oscillates between amounts (5% for non-grids, 20% for O_IG and 100% for 2x2 and 3x3 grids), but I was wondering what the performance would have been with 0% supervision, as this seems like the “optimal” solution target for RPMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FW1h5KPaJj", "reviewer": "Reviewer_qkwH", "summary": "A new model -- RAISE has been proposed by the authors for RAVENs. The model contains several components, such as an image encoder, two variation autoencoder, and a global knowledge set. This model first extracted image features and then used these features to generate answers by a conditional generative process. The proposed model was evaluated through experiments on RAVEN and I-RAVEN datasets and showed better performance than other generative-based methods. Ablation studies showed the proposed RAISE can better handle the selection in arbitrary positions.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and the main idea is easy to follow.\n\n2. The authors proposed a new generative model -- RAISE that can explicitly encode rule-related information. It will be more useful to construct interpretable machine learning algorithms in this community.\n\n3. The authors showed that their RAISE can perform better than previous generative methods, and is also on par with some of the existing selection-based methods.\n\n4. The authors also analyze the proposed models with different experiments.", "weaknesses": "1. The authors fail to convince me, why we need to generate answers in arbitrary positions ? I am eager to see what is the advantage of generating an arbitrary position over generating the right-bottom answer, not only the final performance, but also the rationale or the motivation behind this design.\n\n2. Whether the model shown in App is used in all configurations or not?\n\n3. Too many hyper-parameters should be tuned.\n\n4. Experiments are not enough. I suggest using PGM-Neutral, PGM-Interpolation and PGM-Extrapolaton to confirm the effectiveness of RAISE.\n\n5. The authors also should clearly state why to use rule annotations, but not the answer images. Which indeed violates the RPM question. \n\nOverall, it is a borderline paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EUmAsCpJ4e", "reviewer": "Reviewer_TA5q", "summary": "The paper proposes RAISE, Rule AbstractIon and SElection model for generative abstract reasoning. In particular, the model is evaluated on Raven's Progressive Matrices. To solve a Raven problem, the model first encodes the context images and samples the latent rule for different latent concepts and generates the answer. On both RAVEN and I-RAVEN, the model shows improved performance while being generative, and also the model enables arbitrary panel generation and odd-one-out problem testing.", "questions": "Check above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The best generative solution I'm aware of for solving the abstract reasoning problem, while problem structure has been taken into the design. To the best of my knowledge, previously similar performance is only attained by discriminative models and this is the first of the generative model to achieve similar performance. The authors have also tested the model on different setups, and the results are good enough.\n\nThe authors have shown sufficient experiments to show the meaningfulness of the latents: by varying the latents, they could get desired image results.\n\nWhile the problem formulation follows the conventional approach, the problem decomposition makes sense and is intuitive.", "weaknesses": "The formulation for conditional generation is rather standard. While it is not exactly the same as conditional VAE, the derivation follows the same principles and the tweaks are only made due to the structured inference prior employed in modeling.\n\nIn general, I don't think the comparison is completely fair compared to other baselines, as some of the baselines only use the ground truth answers. While RAISE only uses the rule annotations, grounding of rules to corresponding hidden concepts is also implicitly encoded in the matrix. So supervisory signals could actually be backpropagated to the attribute / concept part. Besides, in evaluation, I do note that PrAE and ALANS are only trained on a specific split whereas RAISE are trained on more than one, and that is at least twice the data.\n\nOne thing I'm not particularly sure is how is the answer selected in RAISE. When you generate the answer, how do you pick the candidate from the given set? PrAE and ALANS actually only generate the hidden latents and compare in the latent space. Do you compare in the pixel space? Do you think comparing in the hidden space would help further improve performance of RAISE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)