GenVP: Generating Visual Puzzles with Contrastive Hierarchical VAEs
Authors: Kalliopi Basioti
Pritish Sahu
Tony Qingze Liu
Zihao Xu
Hao Wang
Vladimir Pavlovic
Venue: ICLR 2025 Poster (2025)
Keywords: abstract visual reasoning; contrastive learning; variational autoencoders; out of distribution generalization
Primary area: generative models
TL;DR: We propose Generative Visual Puzzles (GenVP), a generative RPM solver exceeding SOTA accuracy and capable of generate new RPM matrix instances different from original ones while following the same rules.

Abstract: Raven’s Progressive Matrices (RPMs) is an established benchmark to examine
the ability to perform high-level abstract visual reasoning (AVR). Despite the current success of algorithms that solve this task, humans can generalize beyond a given puzzle and create new puzzles given a set of rules, whereas machines remain locked in solving a fixed puzzle from a curated choice list. We propose Generative Visual Puzzles (GenVP), a framework to model the entire RPM generation process, a substantially more challenging task. Our model’s capability spans from generating multiple solutions for one specific problem prompt to creating complete new puzzles out of the desired set of rules. Experiments on five different datasets indicate that GenVP achieves state-of-the-art (SOTA) performance both in puzzle-solving accuracy and out-of-distribution (OOD) generalization in 22 out
of 24 OOD scenarios. Further, compared to SOTA generative approaches, which struggle to solve RPMs when the feasible solution space increases, GenVP efficiently generalizes to these challenging scenarios. Moreover, our model demonstrates the ability to produce a wide range of complete RPMs given a set of abstract rules by effectively capturing the relationships between abstract rules and visual object properties.

Published as a conference paper at ICLR 2025
GENVP: GENERATING VISUAL PUZZLES WITH CON- TRASTIVE HIERARCHICAL VAES
Kalliopi Basioti', {Pritish Sahu", Tony Liu'), }, Zihao Xu', Hao Wang', Vladimir Pavlovic'
'Rutgers University, SRI International
(kalliopi abasiott tony .1i,, zihao. xu, vladimi rly www.rrrrururreeee.ooorrr edu pritish.
sahu@sri. pritish.sahussr.
com, hw4880cs. rutgers.ed.. .ddu
ABSTRACT
Raven's Progressive Matrices (RPMs) is established benchmark examine the ability to perform high-level abstract visual reasoning (AVR). Despite the cur- rent success of algorithms that solve this task. humans can generalize beyond a given puzzle and create new puzzles given a set of rules, whereas machines re- main locked in solving fixed puzzle from a curated choice list. We propose Generative Visual Puzzles (Gen a framework to model the entire RPM gener- ation process, a substantially more challenging task. Our model's s capability spans from generating multiple solutions for one specific problem prompt to creating complete new puzzles out of the desired set of rules. Experiments on five dif- ferent datasets indicate that Gen VP achieves state-of-the-as (SOTA) performance both in puzzle-solving accuracy and disttr-diitiibuttin (OOD) generalization in 22 OOD scenarios. Compared to SOTA generative approaches, which struggle solve RPMs when the feasible solution space increases, Gen VP efficiently gener- alizes these challenging setups. Moreover, our model demonstrates the ability produce a wide range of complete RPMs given a set of abstract rules by effectively capturing the relationships between abstract rules and visual object properties.
INTRODUCTION
Human reasoning involves decision-making, concluding and learning from experiences, knowledge, and sensory input such as visual cues. Various intelligence tests, notably Raven Progressive Matrices (RPM) (Raven& & Court 1998), assess this capability. Originally for humans, RPMs now also test AT's abstract visual reasoning capabilities. Standard RPM puzzles include two parts, a context matrix, and a choice list. The context matrix presents a problem prompt for an incomplete 3 x 3 puzzle with a missing bottom-right image, and the choice list provides eight potential solutions.
Recent developments in learning-basse based RPM solvers comprised two categories: generative. Discriminative approaches discern the correct answer among the list of choices using visual features extracted from the RPM puzzle (Zhang et al., 2019a bj Wu et al. 20200 Benny al.
discriminative and
2021
Hu et al. 202 Sabu et al. 2022b Malkiński & Mańdziuk] 2022: Yang et al. 2023 Xu
al. 2023 Mondal et al.] 2023 Yang et al.] 2023 Zhao et al. (024). Although high in accu- racy, discriminative solvers are limited by the choice list, fixed in size and limited in search space, often leading to short-cut learning (Zhuo & Kankanhalli 2020 Zhang al. 2019b). Generative approaches (Pekar et al. 2020 Sahu et al. (2022a) alleviate these issues by generating a proposed solution and retrieving the elosest matching element from the choice list. Despite good performance and desirable properties, generative approaches (Pekar et al. 2020 Zhang et al. [202]] Sahu et al.) 2022al Shi et al.] (024) are sensitive to noisy and distractive puzzle attributes, lacking the abil- ity to distinguish relevant image features. Furthermore, these solvers are restricted to image-level generation, demonstrating limited high-level abstraction capability.
We propose Generative Visual Puzzles (GenVP), a generative RPM approach that models the entire RPM generation process. Ge achieves this through a graphical model that contains hierarchi- cal inference and generative pipeline which are trained simultaneously. Ge graphical structure
Authors named inside {) equally contributed.

---

Published as a conference paper at ICLR 2025
Observations
GenVP Graphical Model
MoE Prediction of RPM Rules
X 11
X33]
R
Z
R(z,)
Z,
RZ,,)
Zo
MoE R
N0 Rule
Const:
Distr-3
Prog
Arithin.
Zt
Zo
R R
Color
Image level: M²
Size
RPM level: D
Zo
R R(Z
Shape
Angle
0
Inference path: Generation path:
given complete, valid puzzle X and our task is to infer all intermediate random variables Z, Zo, Zo, Z, and predict its rules R. using our Mixture of Experts (MoE) strategy. During generation, given an abstract set of rules R. we generate a complete puzzle X following the desired rules. We note that during generation, we don't 't need the MoE module, but we directly generate Z, from the given set of rules.
Figure 1: Generative and Inference Graphical Models of GenVP. During inference,
introduces a powerful Mixture of Experts (MoE) mechanism for puzzle rule prediction, improv- ing the model's resilience to noisy features and shorteut learning. Our approach can also generate complete RPM matrices, demonstrating deep understanding of the puzzle rules. We robustify our generative model training by designing novel contrastive learning scheme, leading to cross-puzzle and cross-candidate comparisons. In summary, our contributions are:
We propose GenVP, novel approach for solving and creating visual puzzles. GenVP is the first AVR model to learn a generative path from abstract rules to complete visual puzzles. Our model promotes AI creativity by producing new, complete puzzles, which also serve high- level, human-readable visual explanations of the learned concepts. This is because we can assess the model's understanding of rule by prompting it to create complete puzzle based that rule.
We propose novel cross-puzzle and cross-candidate contrastive loss for AVR. Additionally, we introduce a MoE mechanism to learn robust representations that can accurately deduce the rules of visual puzzles and subsequently solve RPMs.
Our experiments on five different AVR datasets show that GenVP outperforms state-of-the-an (SOTA) RPM solvers. Furthermore, GenVP significantly surpasses SOTA in out of our 24 examined out-of nut-of-distribution (OOD) evaluations.
Our qualitative analysis indicates that Gen VP exhibits a high-level capability for rule abstraction by generating new RPMs that differ from those in the original datasets while following the same set of rules.
RELATED WORK
Generative approaches. GCA (Pekar al. 2020) pioneered generative approach for RPM solving, integrating a VAE-based (Doersch) 2016) image reconstructor, used for feature extraction, with a discriminative RPM classifier. was trained using image reconstruction, rule classification, and contrastive learning for the latent representations of the choice list images; however, the model struggles with low puzzle solution accuracy. DAREN (Sahu al.) (022a)) improved the accuracy by adopting separate training for generative and discriminative components by employing pre- trained VAE to extract robust image features for the solver module. While DAREN has shown high accuracy, is limited the constant attribute rule, making it unsuitable for more complex rules such as arithmetic, progression, and distribute-there
Shi al. 2024) aim to generate the missing puzzle image in RPMs. Specifically, PRAE (Zhang et al. 2022) and ALANS (Zhang et al.) (202T) employ neural perception module to produce probabilistie scene representation alongside symbolic log-
Later approaches (hhng al. 2021 2022
2

---

Published as a conference paper at ICLR 2025
ical reasoning component. These methods improve the performance of RPM solving using strong inductive biases, exploiting specific configuration layouts, or hidden relations within RPMs in their model design. However, the high inductive biases limit generalization capability to novel puzzle setups, such as different layouts, and the models have high computational overhead and scalabil- ity issues within their architecture. RAISE (Shi et al., 2024) also generates the missing image but differs from previous works in that it employs special rule-cxecutors to generate images adhering to specific rules predicted from the context matrix. Each non-random rule (constant, progression, arithmetic, and distribute-3) has its dedicated executor module that generates latent features for fi- nal image decoding. Despite the SOTA RPM solving accuracy, RAISE suffers from the following drawbacks: a) error propagation: RAISE will use a faulty rule executor when the rule prediction is incorrect; b) RAISE struggles to adapt in puzzle regimes with a large number of feasible solutions, leading to poor puzzle-solving performance in noisy, large solution space puzzles. As the solution space expands, RAISE fails to capture it, leading to an inability to even distinguish the correct solu- tion from the incorrect ones in the choice list. Our experiments also show that RAISE suffers from training instability and poor OOD generalization.
Compared to prior SOTA generative approaches, GenVP learns to focus relevant factors for the rules and minimizes the impact of distracting attributes on rule prediction. Further, our model employs a general rather than dedicated module for image generation. Existing approaches can only reconstruct the missing panels of the context matrix, whereas Ge has broader generative capabilities, including generating multiple solutions for given puzzle and creating novel complete puzzles from just a set of abstract rules. GenVP learns robust representations by proposing a novel contrasting scheme for the rules of valid and invalid puzzles (the latter are formed by completing the context matrix with incorrect candidates). Moreover, our approach utilizes robust cluster of rule predictors (MoE) by 1) exploiting various combinations of puzzle images and 2) employing low- and high-level latent variables. Note that each Gen predictor has to predict all present rules in puzzle compared. Whereas, in RAISE, the latent space is broken down into concept-specific dimensions, where single rule predictors are applied. Therefore, our MoE does not strong inductive biases (i.e.. prior knowledge of the number of attributes), while at the same time, it increases our model robustness by using partially observed information (a subset of puzzle images) and coarse to fine representations. Finally, GenVP exceeds SOTA generative approaches in puzzle-solving accuracy for both in-distribution and in 22 out of 24 examined OOD cases. The OOD scenarios include novel attribute values, rules, domain transfer and compositional generalization.
Contrastive Learning in RPMs. CoPINet (Zhang et al., 2019b), DCNet (Zhuo & Kankanhalli 2020) and MICL (Malkinski & Mandziuk, 2022) were the first to adopt contrastive learning (Chen) al. 2020 Liu et al. (022) for discriminative RPM solvers. CoPINet and DCNet use contrastive
learning for the choice list images but suffer from short-cut learning issues. MLCL utilizes SimCLR
(Chen al. 2020) contrastive loss by creating positive and negative batches through examination of RPM rules (if two puzzles share common rules, they are marked as positive or otherwise negative).
However, MLCL uses contrasting only as separate pre-training step, where the main training re- lies on standard cross-entropy objective. In our work, we create novel contrasting terms for rule representations to accompany the generative learning of Ge We adopt dual-level contrasting, comparing both global (cross-puzzlo and local cendss-candidate) RPMs, which leads to robust rule estimation.
3 METHOD
Problem Setting. The Raven Progression Matrix (RPM) consists of M x N images (usually M N 3) denoted as X = [xyhee(M),JM)) where each X.j has dimensions H and W. The RPM adheres to a set of rules encoded in R E {0, KRxxNN with rows representing KR attributes and columns representing one-hot rule encodings. The RPM puzzle includes a context matrix, derived by masking the image XMN and a choice list containing this masked image plus the negative examples set A {a. a, E R H'x 1 where A is the number of negative images (typically A = 7 in RPM datasets). The objective is to identify the correct X in the choice list to complete the context matrix.
3

---

Published as a conference paper at ICLR 2025
3.1 GENVP LATENT VARIABLES
Latent Variable Notations. GenVP includes image-level and RPM/puzzle-level latent variables. For image-level, Z (2ij)EE[M represents the compact latent representation of RPM images X. Z is split into Zo (Zo,ij JE( Zo,ij E K2, for relevant RPM attributes, and Z5 {za.1) 1M
=i Z0.i5 E R2 for irrelevant attributes, as shown in Fig. I (left). Due to insufficient knowledge about full puzzle the image level, puzzle-level variables Z, and R. introduced to bridge this gap and aid in decoupling of Z, with R. denoting RPM rules and Z, capturing the relationships among the M images per r, depicted in Fig. T (middle).
3.2 GENERATIVE AND INFERENCE MODELING GENVP
In this section, we outline the generative and inference pipeline of GenVP, using previously defined variables. We denote categorical distributions by C(.), Gaussian distributions by N/.), and the set index by [n] (1,2..... n/n E Nj.
3.2.1 INFERENCE MODULE
Ve inference module is divided into two steps, image-level and RPM-level inference. For the simplicity of notation, we denote all learnable parameters for the inference pipeline as Φ.
Image Level Inference. Image level inference processes individual images from the context matrix infer variables Z, Zo, Zo as shown Fig. [ conditional Gaussian distribution. The means and variances are parameterized by neural networks with learnable parameter Φ, and the functional forms are detailed the Appendix A
(middle). The posterior for these variables follow
RPM-Level Inference. The RPM level representations consist of the rule matrix R and the rows representation Z,.. We first infer Z, N' (Z Φ). oz inferred Z. Using Zr, we proceed to predict the rules R. for RPM X. We employ ensemble of predictors to establish a robust rule estimator, assuming that X's rules are derivable from the following combinations of latent variables:
Zr.q (Zo: Φ)) conditioned on previously
The M image latent representations Z, = (No With posterior
g ( (Z) C CR: (Zo: x))
(1)
Any MN - context latent representations Z Z, Zo. where h E [M], E [N] are the missing image coordinates. With posterior
I (R Z C (R: PR Zota :)))
(2)
Any MN - N latent representations Zerow Zo Bo with k E [M] the row-index of the excluded row (partial rows). With posterior
g R Zgrow) C (R:PRa. Zg Zorow x))
(3)
The M image row latent representations Z, {
With posterior
ge (R Iz)) = C (R: ,R,. (Z,
(4)
Our final puzzle rules estimation results from mixture of the aforementioned experts (MoB):
1 (R Zo Z,) C (R:; 9 (Z. Φ))
For the MoE estimator shown in Fig. D (right), we investigated parametric (neural network approxi- mations) and non-pane parametric solutions. In Appendix CC6], we provide an ablation study for different mixture functions using parametric (ncural networks) and non-parametric approaches. The MoE module also ensures that Z, and Z, learn to capture essential information for rule prediction dur- ing training. The model is tasked with identifying the most probable rule set without full visibility of the M x N puzzle. To enhance the stability of contrastive learning (Section 3.3 separate NN approximators are employed for each predictor in Eqs. to
RPM Solution Selection. We observed that the most likely RPM puzzle solution, from a choice of A + 1 images (A negatives and one correct), is the one that forms the most rules with the context
4

---

Published as a conference paper at ICLR 2025
matrix. Using this observation, select the RPM answer by first creating eight complete candidate puzzles, filling the bottom-right empty spot with choice list images. For each candidate we use Gen inference pipeline to find their MoE rule matrix prediction, then choose the having the largest set of active rules (constant, progression, distribute-three, arithmetic) and return it as the final answer.
3.2.2 GENERATIVE MODULE
The generative pipeline inputs a rule matrix R and irrelevant attributes Zo producing com- plete RPM instances as shown in Fig. limiddle) with red solid edges. The puzzle rule variable p(R) = I. C(rR) comprises independent variables r& for each attribute k, each following cate- gorical distribution C(rx). Irrelevant attributes Z5 are sampled from a standard Gaussian distribution (O, 1). The remaining random variables follow conditional multivariate Gaussian distributions, whose conditional relations are illustrated in Fig. [ (middle), with means and covariances approx- imated by neural network decoders parameterized by 0, with the functional definitions detailed in the Appendix A
3.3 THE OBJECTIVE FUNCTION
Our Ge is trained with the combination of standard Hierarchical VAEs (HVAE) optimiza- tion by maximizing the evidence lower bound (ELBO) (Doersch) 2016), and contrastive learning between different RPM matrices composed of the set of provided negative answers A.
ELBO. To learn the encoders and decoders model parameters, we maximize the ELBO of GenVP:
p(R)
p (Z,))
ELBO0, (X, (X,00,,,X,,,, R) B1 E, [log po X 2) B.
log Is (R ZZ, Z,)
+.9g log
Is (Za
Lyze
ER log
Lzo (Z)Z.. Zo)
Pe Zo, Zn log
po (Z R)
+
+Bg log
(6)
9 Z
aΦ (Z, Z,)
9 (Z
L.z.
Cz.
Ez
We list the description for each individual term as follows:
free trains the reconstruction image X using image embedding Zij E Z.
Ln trains the rule-relevant latent representations, Z, and Z,, decipher the puzzle rules.
Lz. ensures the posterior distribution Zg follows standard normal distribution.
Lz., Lz,, Lz regularize the latent representation and encourage learning of informative latent representation for the three variables.
When rule annotations accessible during training, then the prior p(R) becomes data-dependent. In particular, for pair of complete puzzle and rule observations (X,, R,) we will use prior pR that is highly concentrated around R, by approaching a Dirac Delta distribution RR) 5(R - R,). For all loss terms, we use the Kullhack-Leibles (KL.) divergence (Joyce) 2011) closed form for Gaussian variables, except for the free and LR. which reduce to reconstruction losses between the puzzle and rule predictions and their corresponding annotations.
Contrastive Learning Overview. We design contrastive terms using the property boundedness of R on the [0, 1] interval, with all row sums equal to one. Since the rule variable R is a leaf node in our graphical model, the contrastive gradients are able to backpropagate and influence all latent variables of Gen VP. Our contrastive loss contains two terms, global masked contrastive loss and local contrastive loss, which we discuss next.
Global Masked Contrastive Loss. We design a global contrastive loss with the aim of learning accurate rule estimations by comparing globally across different puzzles. For simplicity, we contrast two puzzles P. (X, and P (X,, R2) at time, knowing that the following holds:
1. R R2: puzzles follow the same rules.
2. R, 7 R2, R1 * R2 rows Vi 1,... ... K puzzles follow entirely different rules.
5

---

Published as a conference paper at ICLR 2025
3. R1 + Rg: puzzles share the same rule for some attributes |C] KR-
Consequently, we choose to contrast pairs of puzzles from category 2 and to contrast pairs from category only for attributes that have different rules. To achieve this, design the following masked contrastive loss between pairs of RPM puzzles:
(1 if R1 # R2,ú 00 otherwise
ge (2) | (Ro(X.) - R2)ll || (Ro(X1) R2)11 M,
where Ro (X are the predictions of the rule matrix of the puzzle X; using the encoders from Eqs. (1) to ( We use O for deement-wise multiplication by masks M and M. The mask is designed to have the opposite effect of M; it is equal to one when the two rows follow the same rules and zero otherwise.
1, if R1.1 R2,35 and R2
M2
0, otherwise.
We can generalize Eq. 7 to batch with size B for each puzzle Xs, b E [B] by
1
1 B-1
(P,, PB)
(P P.) iE
Σ
(9)
B
Local Masked Contrastive Loss. ELBO objective and global contrastive loss promote learning from correct puzzles. However, these objectives lack contrast with invalid/incorrect ones. To make the model aware of the invalid puzzles, we introduce local masked contrastive loss between the valid puzzle X and set of invalid puzzles X (X; 1/-1 obtained by replacing masked XMI with each negative image a, from the choice list. Since the choice list is generated by modified rule-relevant attributes, we are able to adjust the rule matrix R to R, by setting the rule value for modified attributes 'random/no rule'. We then form tuple of negative puzzle and perturbed rule matrix P, (X,-. R, and establish the local masked contrastive loss between valid puzzle P and invalid one P, follows:
LP PT) = Σ Po RE[M]
(10)
Training. During training, we start with a 'warm-up" phase that maximizes only ELBO, Eq. (6 Afterward, continue training with ELBO and the global-local masked contrastive terms:
1 Σ B
BG B
EL (X,,, Luup +
Σ (P Pr) iEB/b
PT)
0.
where Sa, Br, are scalar hyper-parameters tuning the effect of the contrastive terms.
Data augmentation has been shown to play crucial role in the success of contrastive learning (Chen al. 2020 Liu et al. 202T). For this reason, adapted rule-invariant puzzle augmentations from
Zhang et al. (011b)); Malkiñski & Mandziuk] (2022); Zhao et al. (2024) during training. Types
of such augmentations include performing image transformations like horizontal or vertical flips, rolling, and grid shuffling.
4 EXPERIMENTAL SETUP
Data. We assessed GenVP with the RAVEN-based (RAVEN (Zhang al. 2019a). I-RAVEN (Hu et al., 2021) and RAVEN-FAIR (Benny et al. 2021) and the VAD (Hill et al. 2019) and PGM (Brrrett al. 2018). Developing efficient generative approaches for the last two datasets poses significant challenge due to the vast number of feasible solutions for each puzzle (Shi et al. 2024). For more dataset details see Appendix

---

Published as a conference paper at ICLR 2025
Table 1: RPM solving accuracy (%) for RAVENI- datasets. RAISE*: reproduced results
MODEL
C:5
U-D
2x2
3x3
GCA
32.7/41.7
37.3/518
26.4/44.6
21.5/42.6
30.2/46.7
33.0/35.6
37.6/38.1
43.0/32.4
ALANS
54.3/62.8
42.7/63.9
42.4/60.9
46.2/65.6
49.5164.8
53.6/52.0
70.5/66.4
75.1/65.7
PRAE
0/85.7
97.3/99.9
96.2/97.9
96.7/97.7
95.8/98.4
82.0/84.5
23.2/45.1
RAISE
90.0/92.1
99.2/99.8
98.5/99.6
99.3/99.9
97.6/99.6
89.3/96.0
68.2/71.3
77.7/78.7
RAISE
39.0/92.3
98.7/100
99.1/99.2
97.7/99.0
94.5/97.7
79.1/85.6
73.4/83.5
79.4/82.3
GENVP 94.7/96.1
100/100
99.7/99.8
99.9/100
99,9/100
86.0/90.1
93.3/94.6
84.1/88.1
Table 2: OOD RPM solving accuracy (%) for puzzles with unseen attribute values. -1 indicated inter- polating values and -E for extrapolating values. IN-DIST the accuracy for the original RAVEN/I- RAVEN testing sets (without unseen attributes). RAISE*; reproduced results, original paper did include this evaluation.
00D
MODEL
AYG
C-S
L-R
U-D
O-IC
D-IG
2x2
3x3
RAISE
89.0/92.3 3
98.7/10K
99.1/99.2
97.7/99.0
94.5/97.7
79.1/85.6
73.4/83.5
79.4/82.3 93.3/94.6
IN-DIST
94.7/96.1
99.7/99.8
RAISE*
62.4/73.8
78.6/85.2
68. 68.4/84.0
69.5/85.5
45.8/65.2
44.3/57.7
$7.5/69.5
70.2/69.6
ANGLE-1
GENVP
90.2/91.7
100/100
87.3/88.4
87.8/93.7
99.3/99.8
80.8/80.7
92.9/94.1
83.2/85.1
68.0/71.8
99,7/99.9
97.7/98.7
98,4/99.3
17.
7.6/17.4
74.1/80.6
81.4/78.6
COLOR-1
GENVP
69.2/74.5
94,9/98.2 22
93.5/97.6
93.8/98.
18.0/33.9
10.8/17.3
91.6/91.4
82.0/84.6
RAISE*
44.7/64.4
50.2/58.5
34.1/57.7
41.5/62.1
31.6/62.3
48.7/48.6
45.8/58.5
60.1/61.0
$12F-1
GENVP
83.1/87.5
97.2/97.5
70.9/84.3
67.2/87.5
85.7/89.6
85.2/87.2
93.4/93.2
82.1/83.1
RAISE*
28.5/43.6
39.1/52.5
24. 24.5/45.4
26.5/47.7
19.5/35.7
17.2/31.6
33. 1/47.7
39.4/44.7
SIZE-E
45.0165.5
72.3/83.3
48.1/62.2
35.8/53.9
35.7/51.0
78.4/74.2
68.3/67.1
Baselines. We compare our model puzzle-solving accuracy with the SOTA generative RPM ap- proaches, including GCA (Pekar et al. 2020), ALANS (Zhang et al., 2022). PRAE (Zhang et al. 2021, and RAISE (Shi 2024). We report their accuracy from Shi et al. (2024); for the RAISE model, we also reproduce their results for additional analysis and comparisons.
Implementation Details. We trained our model using the correct puzzle X, the negative images
A. and the associated set of rules R. We 'warm up" Gen VP parameters by maximizing ELBO Eq. (6). and then continue their training combining ELBO and the contrastive terms in Eq. [T). In both cases, we used the AdamW algorithm (Loshchilov & Hutter] (017) with a learning rate 10 Additional information for our experimental setup (i.e., hyperparameter values) can be found in Appendix B Scalability and efficiency analysis of Gen VP and baseline models in Appendix D
4.1 RPM SOLVING
In this section, we evaluate the Gen RPM solving performance. Given a context matrix and the choice list (candidates), we ask the model to identify the best fit for the missing bottom-right images. We assess performance on both in-distribution and out-of-distribution examples.
4.1.1 IN-DISTRIBUTION PERFORMANCE
Table shows shows RPM solving accuracy for our Gen VP and baseline models on RAVEN and I-RAVEN datasets. GenVP surpasses other methods across all layouts without needing specific image encoders or detailed knowledge of attribute relations, such as progression dynamics or arithmetic operations (e.g., addition, subtraction) as in PRAE and ALANS. GenVP's MoE module for rule prediction, its powerful contrastive scheme" and the modeling of the complete RPM reasoning process enables our approach to learn robust representations and reduce inference noise. This is contrast to RAISE's structure, which is overly simplistic and sensitive noise (i.c., error propagation due to faulty rule predictions). Gc also significantly improves performance in complex grid configurations (0-1G, 2 x 2, 3 x 3), dealing with smaller objects and more intricate rules regarding number and position attributes. Finally, we compute the zero-shot performance of the RAVEN-traine models for the
In Appendix C.5 we provide ablation study comparing the training of Ge VP with and without con- trastive training, in order to understand the effect of our contrastive scheme.
7

---

Published as a conference paper at ICLR 2025
RAVEN-FAIR dataset (see Appendix C.2 accuracy.
GenVP achieves a state-of-the-an 97.3% suzzle-solving
4.1.2
GENERALIZATION
We evaluate models' OOD performance by creating new training sets using the following two pro- cedures.
Unseen Attribute Values. We create new values for size, color and angle attributes via interpo- lation and extrapolation. The interpolation creates unseen intermediate states, and the extrapolation
creates attribute states outside the original range. We provide the exact OOD values for each at- tribute in the Appendix [.T
Rule Exclusion. We develop two OOD training sets following Shi et al. (2024), each omitting spe- cific rules. The first, CS-HELD-OUT, excludes the (Constant, Type) rule from the C-S configuration. The second, O-IC-HELD-OUT, omits following rules from the O-IC configuration: (Type In, Arith- metic), (Size In, Arithmetic), (Color In, Arithmetic), (Type In, Distribute Three), (Size In, Distribute Three), and (Color In, Distribute Three). Models are trained on these sets and assessed using the original test set, which retained the excluded rules.
Table 2 shows the OOD performance on unseen attribute values for GenVP comparing with the SOTA RAISE model. Ge outperforms RAISE in all setups, especially in OOD angle scenarios, where angle serves as a distracting attribute in RAVEN/I-RVEN. By separating the latent space into relevant, Zo, and distracting, Zo. components, Ge minimizes the impact of such distractors. Conversely, RAISE struggles with puzzles that include unseen angles.
For the RPM-relevant attribute, such as color, OOD performance closely matches in-distribution across most configurations (C-S, L-R, U-D, 2 2,3 x 3), demonstrating strong model generalization. However, in the O-IC, O-IG configurations, we observe significant performance drop. This drop due to poor color generalization in the outer objects of O-IC, O-IG layouts, which were consistently white during training but are replaced by gray tones in the OOD dataset. This deterministic training behavior restricts generalization in O-IC, O-IG, unlike in configurations like center-single, where object colors vary, However, our model still outperforms RAISE in O-IC and competes effectively in O-IG.
Our model generalizes well for interpolated values (SIZE-1) of the size attribute but shows poor generalization during size extrapolation (SIZE-E), as indicated by the worst performance in all OOD scenarios in Table[] This is due to the inclusion of significantly smaller objects in size extrapolation compared to the in-distribution set, leading to reduced performance in O- IC, O-IG, and 3 x 3 layouts where Ge encoders struggle extract features from extremely small visual objects. How- ever, compared to the RAISE model, Ge still achieves superior performance across the board.
Table 3: OOD RAVEN/I-RAVEN
puzzle solving accuracy (%) for un- rules RAISE*; reproduced sults
OOD
GENVP
RAISE"
CS-HELD-OUT
100/100
97.0/98.8
H-ID-HELD-OUT
47.5/65.7
47.0/62.9
Table presents results on OOD performance for unseen attribute rules for our model and the best- performing baseline model RAISE. Our model achieves a perfect generalization for the CS-HELD- OUT case, with the removal of the constant size rule from the training set. For the more challenging setup, O-IC-IIELD-OUT, where a total of five rules are being removed, our VP still exceeds the SOTA performance.
4.1.3 SOLVING VISUAL PUZZLES WITH LARGE SPACES OF FEASIBLE SOLUTIONS
In 4 we present the puzzle-solving performance of generative approache for the challenging PGM and VAD having significantly larger solution spaces. We observe that RAISE is able to adapt to the large solution space datasets PGM and VAD (especially for PGM) as it was discussed Shi al. (2024). In contrast, our approach, Gen demonstrates efficient generalization and
?PRAE and ALANS models depend on grid grid-structured LAVEN-based panels to crop individual objects. However, in PGM and VAD, there is no prior information on the location of each object. Consequently, PRAE and ALANS are not scalable for these datasets. GCA was evaluated in |Pekar et al. (2020) on the PGM Neutral (N) and Interpolation (I) training sets, achieving reported performance of 82.9% and 61.7%, respectively.
8

---

Published as a conference paper at ICLR 2025
C-S
L-R
U-D
O-IC
O-IG
x
3
Original
GenVP
Color
x
x
x
X
Type
X
X
Angle
x
Num.
X
x
X
Pos.
x
x
Aritha.
Prog.
Distr.
Coust.
No Rule
Arithm.
Prog.
Distr.3
Const.
No Rule
Arithm.
Prog
Distr.
Cousi.
No Rule
Arithm.
Prog.
Distr.3
Const.
Arithm.
Prog.
Distr.3
Const.
No Rule
Arithm.
Prog.
Distr-3
Const.
No Rule
Arithm.
Prog
Distr-3
Const.
No Rule
Rule
Leh-Ohjeet
Up Object
Out-Chjjct
Oun-Ohjoc
x
Righi-Ohjeet
X
Down Dows.Object
x
ln-Uijjcct
x
In-Dbject
Figure 2: Generated RPM matrices by Ge.VP. Top: Original RPM matrices; Middle: Gen VP gen-
erations; Bottom: Rules
achieves SOTA performance in almost all cases. This is a result of our robust mechanisms in both modeling and learning.
In particular, our MoE rule pre- dictor extracts rule information from various puzzle views and multi-level features, encompass- ing low-level image features and high-level puzzle features, which increases the model robustness by reducing the prediction noise since do not rely single predictor like prior work. Another crucial aspect of Ve is its capability to filter out rule-irrelevant factors representations during the rule reasoning process, minimizing the impact of distractors and noise in the puzzle-solving process. In terms of learning robustness, cross-puzzle and cross-candidate contrastive learning en- courages the model learn potent features by comparing different valid valid vs. invalid puzzles. This is critical for learning representations that can accommodate the big solution space present in PGM and VAD datasets. Refer Appendix C.3 C.3 for more details.
Table Puzzle-solving accuracy for the large solution space datasets PGM and VAD.
DATASET CONFIG.
VAD
PGM
TD-SC
N
75.7
63.8
22.4
GENVP
96.3
94.9
80.1
76.3
79.6
85.5
65.6
16.2
17.6
12.8
73.2
15.5 78.6
4.2 RPM GENERATION
this section, we evaluate the RPM generation capability of GenVP. generate new RPM matri- ces X using GenVP, given R as input, i.e., R GenVP, XX, and then perform quantitative and qualitative analysis that examines the coherence and quality of the generation. Note that this generation process is different from the generation offered by RAISE, which focuses on the generation of the missing panels alone context of an incomplete RPM puzzle.
4.2.1 QUANTITATIVE ANALYSIS
Experiment Setup. We evaluate the coherence of Gen generated RPM samples by assessing the accuracy of the predicted attribute rules R compared to the input R. We used the GenVP inference
module to extract R, i.e., GenVP, X GenVP-1 R We report the percentage of attributes consistent with the input rule across all generated RPMs for each layout configuration.
The Gen VP inference model demonstrates a high level accuracy in extracting attribute relations, seen in Table 5 for the complete puzzles in the original test set (TEST ACC). This highlights

---

Published as a conference paper at ICLR 2025
the model's ability to accurately deduce the rules of valid puzzle, hence reasonable choice for measuring coherence for the generated samples.
Quantitative Results. The quantitative results in Table show that GenVP generates RPM samples with reasonable coherence, adhering to the rule matrix input. For simpler layouts such as C-S, L-R, U-D, and O-IC, GenVP produces highly coherent samples. Even with complex layouts involving more shapes, such as O-IG, 2 x 2, and 3 X 3, it still scores relatively high. Overall, Gen effectively understands and ap- plies rules to generate coherent RPM instances. In Appendix C.4 we include additional quantitative analysis, including the performance for each specific rules with format (attribute, rule).
Table GenVP inference model testing accuracy
(row 1) and generated RPM coherence (row 2).
AVG
2
99.5 99.8 99.9 99.9 99.0 97.0
GEN. COHERENCE 78.0 83.4 76.1 80.1 82.6 72.5 75.9 75.4
4.2.2 QUALITATIVE ANALYSIS
Fig. 2 presents examples created by Gen accompanied by the original RPM matrices for each layout. Ge successfully generates novel RPM ma- trices that are distinet from the origi- nal ones while largely adhering to the same set of rules. Our model excels at forming shapes of various sizes, types, and colors, particularly in simpler lay- outs like C-S. Additionally, Gen VP is capable of generating diverse number of shape entities, as seen in the 3 x 3 configuration. Despite occasional chal- lenges in generating high-quality out- puts for smaller shapes, Gen VP exhibits exceptional comprehension of rules, evidenced by its ability to generalize them to novel RPM matrices. In Fig. we investigate Ge ability to gener- multiple correct answers for spe- cific RPM panel, such as the bottom-right image. GenVP can generate various feasible answers by sampling different realizations of latent representations Zo, Zg. In Fig. we first present the solution provided from the dataset (image with green block), and seven Gen VP generated answers for the bottom-right image. We include additional qualitative examples in Appendix HH
Pulle
Comet
Dettr 3
Krithm.
No
Conte.
Detr.
Anthm.
Color
Rule
Piog.
Anthm.
Rule
Const
Detr3
Aithhm.
Typn
Color
Angla
Type Angle
Num.
Type Angle
Type
Pos.
Angle
?
Figure 3: Multiple solutions for the bottom-right panel. Top: RPM rules, Middle: Context Matrix, Bottom: So- lutions. Green-marked image is the dataset solution, fol- lowed by GenVP answers.
CONCLUSION
In this paper, we present GenVP, an advanced HVAE- based generative RPM solver. GenVP ex- ceeds the SOTA benchmarks in terms of puzzle-solving accuracy in both the in-distribution and out-of-distribution scenarios within RAVEN-based datasets and efficiently generalizes to PGM and VAD characterized by enlarged feasible solution space. Furthermore, our model is also the first model to exhibit high-level rule-completion capability as demonstrated by the generation of novel RPM matrices. Despite surpassing the SOTA benchmarks, GenVP and similar RPM solvers face challenges with complex configurations like O-IG, 2 x 2, AND 3 x 3, as well as intricate arithmetic rules. Improving visual comprehension of numerical relationships is an ongoing challenge, and we reserved this for future work.
ACKNOWLEDGMENT
We thank the reviewers/AC for the constructive comments improve the paper. HW is partially supported by Microsoft Research Al & Society Fellowship. NSF Grant IIS-2127918, NSF CAREER Award ITS-234012 25, NIH Grant IROICA297832, and an Amazon Faculty Research Award.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning neural networks. In International conference on machine learning, pp. 511-520. PMLR, 2018.
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 12557-12565, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv: 059000 2016.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1990000.199.,,.,. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Felix Hill, Adam Santoro, David GT Barrett, Ari S Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. arXiv preprint 2019.
Sheng Hu, Yuqing Ma. Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 1567-1574, 2021.
James M. Joyce. Kullback-Leible Divergence, pp. 720-722. Springer Berlin Heidelberg, Berlin,
Heidelberg, 2011. ISBN 978-3-642-04898-2
doi: 10.1007/978-3-66 0210//977----6622088-----2332230 URL https://dot: wwogg//10 1007/978 3-642-04898 327
Yaakov Kraftmakher. Noise reduction by signal accumulation. The physics teacher, 44(8):528-530, 2006.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engi- neering, 35(1):857-876, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1111.05101, 2017.
Mikolaj Malkiński and Jacek Mańdziuk. Multi-label contrastive learning for abstract visual reason- ing. IEEE Transactions Neural Networks and Learning Systems, 2022.
Shanka Subhra Mondal, Taylor Webb, and Jonathan D Cohen. Learning to reason over visual ob- jects. arXiv preprint arXiv:2303.02266 2023.
Niv Pekar, Yaniv Benny, and Lior Wolf. Generating correct answers for progressive matrices intel-
ligence tests. Advances in Neural Information Processing Systems, 33:7390-7400, 2020.
John C Raven and John Hugh Court. Raven's 's progressive matrices and vocabulary scales. Oxford Psychologists Press Oxford, 1998.
Pritish Sahu, Kalliopi Basioti, and Vladimir Pavlovic. Daren: A collaborative approach towards visual reasoning and disentangling. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 4448 4455. IEEF, 2022a.

---

Published as a conference paper at ICLR 2025
Pritish Sahu, Kalliopi Basioti, and Vladimir Pavlovic. Savir-t: Spatially attentive visual reasoning with transformers. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 460-476. Springer, 2022b.
Fan Shi, Bin Li, and Xiangyang Xue. Towards generative abstract reasoning: Completing raven's progressive matrix via rule abstraction and selection. In The Twelfth International Conference on Learning Reprexentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL htttt:://pppeeeeeeeeeeeeee
net ott.ppawwwwwweeoo...orrree.eeeea
Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint
24X1v:2007.04222 2020.
Jingyi Xu, Tushar Vaidya, Yufei Wu, Saket Chandra, Zhangsheng Lai, and Kai Fong Emest Chong. Abstract visual reasoning: An algebraic approach for solving raven's progressive matrices. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6715-
6724, 2023.
Yuan Yang, Deepayan Sanyal, James Ainooson, Joel Michelson, Effat Farhana, and Maithilee Kunda. A oognitivel-iinspiee neural architecture for visual abstract reasoning using contrastive perceptual and conceptual processing. arXiv preprint arXiv:2309. 10532, 2023.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Chuu Zhu. Raven: A dataset for relational and analogical visual reasoning. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 5317-5327, 2019a.
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu. Learning perceptual inference by contrasting. Advances in neural information processing systems, 32, 2019b.
Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu, Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9736-9746, 2021.
Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chur Zhu, and Yixin Zhu. Learning alge- braic representation for systematic generalization in abstract reasoning. In European Conference on Computer Vision, pp. 692-709. Springer, 2022.
Kai Zhao, Chang Xu, and Bailu Si. Learning visual abstract reasoning through dual-stream net- works. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16979- 16988, 2024.
Tao Zhuo and Mohan Kankanhalli. Effective abstract reasoning with dual-contrast network. In International Conference Learning Representations, 2020.
A GENVP INFERENCE AND GENERATIVE MODEL
Inference Pipeline Variable Functional Form. Starting from the puzzle-level representations
Z, Zo, Zo posteriors, we model them as conditionals Gaussian distributions:
do (Z.)
II M
II
N (Mrsso b), o 2
(X:si +)))
(12)
26 ZZ (Z)
II
N (UZ.. (Zizi Φ), oz. (Ziz; ())
(13)
MEIM
JE(M]
Φ ZZ)
N (pzag (Ziyi ④) O2.0 (Ziji Φ) 1).
(14)
iE[M] JE(M)
In implementation, similarly to the generation model, reduce the model complexity by signing the first K2, dimensions of Z to Z,, and the remaining ones to Zz. Due to this modeling decision, do not need to learn additional parameters for Eqs. (13 and (14
12

---

Published as a conference paper at ICLR 2025
Generative Pipeline Variable Functional Form.
Pe (Z, R) = ' (uz, (R; (R:0))
(15)
Po [Z,] = N (pz, (Z,; 0). oz (2)
(16)
Po Zo: Zo)=
II
II N (Lo 20.5l E
(17)
M JE[M]
po (x
II ie JE[M]
II
N (Uxyg (Zzz 0) 0),
(18)
We simplify Eq. (17) by concatenating representations of active and inactive attributes for the mean vector and forming a block diagonal matrix (-() for variance. For the image decoder in Eq. (88)
we also simplify by setting the variance ozyy to zero.
B EXPERIMENTAL SETUP
Data. We assessed Gen VP with the RAVEN-based datasets (RAVEN (Zhagg et al. 2019a). I-
RAVEN (Hu al., 202T and RAVEN-FAIR (Benny et al.) 202T), ) featuring seven layout types: C-S (centered single object), L-R (objects left and right), U-D (objects up and down), 0-IC (nested single objects), O-IG (nested 2 x 2 mesh), and g x 9 (objects in g x g mesh for g = 2,3). In valid puzzles, active object attributes include number, position, type, size, and color, with possible relations including constant, progression, arithmetic, and distribute-three. object orientation/angle and uniformity should be treated as distractors during testing. In the RAVEN-
datasets, the completed puzzles have dimensions M = N = 3. and the choice list has 8 images (one correct and seven incorrect). These datasets differ primarily in their choice list generation. The RAVEN dataset presents a more challenging choice list, with hard negative images differing from the target by only one attribute, yet shorteut solution often exists through analysis of these images, hence the need for the I-RAVEN and RAVEN-FAIR to correct such a shortcut.
We conducted experiments using the challenging PGM and VAD datasets (Barrett al., 2018 Hill jet al. 2001)). The PGM dataset includes seven different training and testing sets, with one for in-distribution and six for various -iitt---iitiitttoon scenarios. The in-distribution set is labeled as Neutral (N), while the Interpolation (I) set uses -iitti--iitiitttion attribute values for testing by interpolating values of the training set regime. Similarly, the Extrapolation (E) set uses extrapolating attribute values for testing. Additionally, the datasets Attribute Shape Color (SC) and Line Type (LT) withhold the corresponding object attribute rules from the training dataset and test the out-of- distribution performance puzzles with rules shape color and line type.
The Held-Out Triples (Attr Rels (AR)) test involves holding out triples of (attribute, object, specific values) in the training set. The Held-Out Pairs of triples (Attr Rel Pairs (ARP)) and the Held- Out Attribute Pairs (Attr Pairs (AP)) are tests for compositional generalization. The ARP test set contains puzzles with the triples (attribute #1, object #1, value #1) and (attribute #2, object #2, value #2), whereas in training, there are no puzzles with this combination of triples. Similarly, the AP test set is designed to assess compositional generalization, where puzzles with the pairs (attribute #1, object #1) and (attribute #2, object #2) do not occur in the training set.
Each training set consists of 1.2 million puzzles, and each testing set consists of 200,000 puzzles. The complete puzzle dimensions are with M = N = 3 and the choice list has A = 7 incorrect images. Unlike RAVEN-based testing in PGM, all attributes can follow a rule or not. This means that an attribute for a puzzle can be distractor, and for another, an attribute for which an active rule has to be inferred. This design choice results in a much larger puzzle solution space since the distractor attributes can have any value they want, and all images with different distractor values are possible solutions to the puzzle.
VAD puzzles focus on making analogies, which are accomplished by puzzles with two rows (M- = N = 3). The first row follows a rule of the type (rule_type #1, attribute #1, object #1, value #1), and the second row should also follow the same rule_type #1 but with different attributes objects. Each puzzle has only one rule_type. which means that most of the object attributes are distractors. As result, VAD has much larger solution space than RAVEN- based datasets. However, it has a smaller choice list, with only four candidates (one correct and three incorrect (A = 3) generated by the learning by contrasting approach). VAD includes five different training and testing sets to
13

---

Published as a conference paper at ICLR 2025
examine various types of OOD mnalogy-makingg The training set consists of 600K examples, and the testing set consists of 100K. Similar to PGM, it has Interpolation (I) and Extrapolation (E) sets for OOD attribute value generalization. It also includes a Domain Transfer (DT) set, which measures the model's ability to solve VAD puzzles with unfamiliar source domain (first puzzle row) to target domain (second puzzle row) transfers. Finally, similar to PGM, it evaluates generalization unseen target domains for shape color (DT-SC) and line type (DT-LT).
Model Architecture. For the encoder from pixel space images to the latent dimension Ziy (En- coderZ) and the reverse decoder from Zij to pixel space (DecoderX), we use the same architecture as those in RAISE. For the remaining encoders and decoders, we use fully connected neural net- works. For our PGM and VAD models for the Zo to Zr mapping (encoder) we use instead of the sampled Z, its attended representation using a transformer block with MN tokens (for and PGM M = N = 3 and for VAD M = 2, N 3), where each token is the Zo,ij latent representation for each image of the complete puzzle. In particular, for PGM, we have nine tokens, and in VAD, six. Our transformer has five heads and two layers.
Regarding the remaining dimensions of our variables, we set H = W 64 for the image dimen- sions as in RAISE; for the intermediate latent variables, we have K 64, Kz. 54, Kz. 10, Kz = 192 = 3 64, KR = 12, NR = 5. For the 3 hyperparameters we set them to 31 = = 0, 1, =, = 1, 1, 1. 250, BG 20, 3r 20. Notice that in our implementation. when rule annotations are available, we set 82 = 0 to simplify our optimization objective since the reconstruction loss between rule annotations and predictions is enough to learn to extract the rules of a given complete puzzle. For PGM and VAD we use the following dimensions for the intermediate latent variables K 102, z, 100, z, 2, KR 10, .R 6. We also perform decoupling for the puzzle level latent variable Z, (similarly to the image level decoupling), which has 300 rule relevant dimensions, 600 rule irrelevant for PGM, 200 rule relevant, and 400 rule irrelevant dimensions for VAD.
Experimental Setting/Details. We trained our models using the correct puzzle X. the negative
images set A available in the datasets (A 7 for RAVEN-based and PGM and A = 3 for VAD), and the associated set of rules R. We 'warm up" GenVP parameters by maximizing ELBO Eq. (6) and then continue their training combining ELBO and the contrastive terms in Eq. (1). We set the
batch size to B- = (RAVEN-based, 100, PGM: 400, VAD: 400} RPM puzzles, which means that
B valid puzzles for ELBO and global contrasting and batch size of A (RAVEN-based 7, PGM: 7, VAD: 3} for the local contrasting loss. In both cases, we used the Adam W algorithm Loshchilov & Hutter 2017) with learning rate 10
Experiments Compute Resources. All the models are trained on a server with 24GB NVIDIA RTX A5000 GPUs, 512GM RAM, and Ubuntu 20.04. For the efficiency and scalability evaluations, we used a server with characteristics of 48GB NVIDIA RTX A6000 GPUs and Dual AMD EPYC 7352 @ 2.3GHz 48 cores, 96 vCores CPU. GP implemented with PyTorch.
C EXPERIMENTS
C.I OUT-OF-DISTRIBEN GENERALIZATION DETAILS FOR RAVEN-BASED DATASETS
Here, we include the details for reproducing the OOD at- tribute values in RAVEN and I-RAVEN datasets. The angle distractor initial values are [-135, 900 -45, 0, 45, 90, 135, 180), for the OOD angle interpolating values we used the set [-157_ 112, -67, 222, 22, 67, 112, 157]. For rule-relevant color attribute the in-distribution val- ues are [255, 224, 196, 168, 140, 112, 84, 56, 28, 0] and for our OOD analysis we used the in- terpolated values [238, 210, 182, 154, 126, 98, 70, 42, 14). Finally, for the size attribute the in- distribution set is [0.4, 0.5, 0.7,0.8,0.9). our OOD values for interpolation analysis are [0.45, 0.55,, 0.75, 0.85] and for extrapolation [0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95).
OOD attribute values.
C.2 RAVEN-FAIR DATASET ZERO-SHOT PERFORMANCE
In Table 6. present the RAVEN-FAIR dataset (Benny et al. 2021 zero-shot performance using RAVEN dataset trained models. We notice that the proposed model, GenVP, achieves the best average puzzle-solving accuracy for the RAVEN-FAIR dataset. We could not train PRAE for the
14

---

Published as a conference paper at ICLR 2025
Table 6: RAVEN-FAIR (Benny al. 2021 zero-shot puzzle-solving accuracy using models trained on RAVEN dataset, For PRAE, we do not report the AVG performance since we were not able to run the model for the 3 configuration because of PRAE's poor scalability, out-of-memory (OOM), even with batch size equal to one.
MODEL
AVG
LR
O1C
OIG
x
PRAE
100,0
100.0
98.9
97.3
71.2
91.5
0OM
RAISE
95.6
100.0
99.9
99.8
99.2
94.5
86.0
90.0
GENVP (OURS)
97.3
100.0
100.0
100.0
100.0
95.7
96.1
89.2
3 x 3 configuration because of its scalability problems (the model could not fit in server even with a batch size equal to one).
C.3 PUZZLE-SOLVING PERFORMANCE FOR DATASETS WITH LARGE SOLUTION SPACE
In (Shi et al. 2024), the authors discuss the challenges of generative approaches in abstract visual reasoning, particularly in puzzles with large solution spaces. These datasets differ from RAVEN- based datasets in the following ways: PGM and VAD have more complex panel layouts consisting of both geometric shapes and lines. Furthermore, the configuration layout not predefined (i.e., single object per image or four objects in 2 2 grid), and objects can overlap. However, the most critical difference between these two types of datasets is the number of rules per puzzle. In RAVEN-based datasets, attributes like color, shape, and scale will always follow one of the active rules, so we know beforehand that these attributes follow a rule, and the quest is to find what the type of the rule. In PGM and VAD datasets, few attributes follow rule (usually one two), the model should also understand which attributes follow a rule and which are distractors. After that, should reason about the rule type. Note that, the distractor attributes if the shape is a distractor, it does not matter if the solution to the puzzle is a triangle, rectangle, pentagon, etc.), which drastically increases the solution space of RPM puzzle.
have random values (i.e.,
In Table we observe that RAISE is not able to adapt to the large solution space datasets PGM and VAD (especially for PGM) as was discussed in (Shi al., 2024). In contrast, our approach, Ve demonstrates efficient generalization to the challenging RPM and VAD datasets. This is result of our robust mechanisms in both modeling and learning.
The MoE rule predictors extract rule information from various puzzle views and multi-level features, encompassing low-level image features and high-level puzzle features which increase the model robustness by reducing the prediction noise since we do not rely on a single predictor. Another crucial aspect of Ge its capability to filter out rule-irrelevant factors the image-level (Z) and puzzle-level (Z,) representations during the rule reasoning process, minimizing the impact of distractors and noise in the puzzle-solving process.
In terms of learning robustness, cross-puzzie and cross-candidate contrastive learning encourages the model to learn potent features by comparing different valid or invalid puzzles. This is critical for learning representations that accommodate the big solution space present in PGM and VAD datasets.
Table 7: Ge inference model Rule Estimation Testing Accuracy (row 1), Generated RPM coher- ence (row 2), and Multiple Solutions Generation coherence (row 3) in percentage (%) for RAVEN dataset.
AVG
C-S
L--R
U-D
O-IC
O-IG
2 x 2
3 x 3
TEST ACC
99.8
99.5
99.8
99.9
99.9
99.0
97.4
97.0
RPM GEN. COHERENCE
78.0
83.4
76.1
80.1
82.6
72.5
75.9
75.4
MULT. GEN. COHERENCE
93.0
94.3
97.5
97.4
95.7
87.5
92.9
85.4
C.4 RPM GENERATION PERFORMANCE
In this section, provide additional quantitative results regarding the RPM generation performance of our model. In Table / we present the accuracy of rule prediction on the testing set for the follow- ing scenarios: a) ground truth complete puzzles (where add the correct image in the bottom-right
15

---

Published as a conference paper at ICLR 2025
Contures La
6/13
KMI
299
CHORAT
LAA
PoyroM
138
D000
C20
0.0
Contutte Matre for DK
Co Condasia Natte
$:4.
832
6.0
301
Figure 4: Rule Prediction performance for different (rule, attribute) pairs for RPM generated puz- zles by GenVP trained on RAVEN-base, datasets. The RPM puzzles using the Ve generative graphical model (from rules to complete RPM puzzles).
position) in row 1 (labeled as TEST ACC); b) the RPMs generated from our model's generative graphical model (from rule matrix RPM) in row 2 (labeled as RPM GEN. COHERENCE).
We notice that GenVP has good rule prediction for the testing set with ground truth complete puz- zles, demonstrating that it correctly identifies the rules of a given complete RPM. Further, our gen- erative model is also able to efficiently generate novel answers for all images in the RPM. Finally, for the challenging task of generating complete puzzles from abstract rule matrix Ge VP demon- strates effective generation capabilities by achieving an overall 78% rule accuracy prediction.
To gain better understanding of the types of rule-attribute abstract relations that the model effi- ciently represents in the raw-pixel space, we analyze the overall rule prediction accuracy for differ- ent (attribute, rule) pairs. In Fig. [ we can observe the performance analysis for generating RPMs from an abstract rule matrix for each layout and each rule relations. We find that our model faces the greatest challenge when understanding RPM puzzles with progression, arithmetic, and 3 rules for number and position in grid configurations (2 x 2 (D4) and 3 3 (D9)). However, for the 3 configuration, the performance of the number attribute column improves compared the 2 x 2 configuration, which is expected due to the increased number of objects, leading to wider variety rule instantiations. This helps the model better understand the concept rules (progression, arithmetic, and distribute-3) 3) related to number.
C.5 TRAINING WITH ELBO AND CONTRASTIVE TERMS
we present the result GenVP when trained only with ELBO Eq. 6 and when trained with ELBO and the contrasting terms Eq. (1). The performance of Ge improves dras-
In Tables & and 9
16

---

Published as a conference paper at ICLR 2025
Table RPM solving accuracy (%) for RAVEN dataset. Training Gen VP without contrasting terms (only ELBO maximization) (row I) and with contrasting training (row 2).
MODEL
AVG
C-S
L-R
U-D 0-IC 0-1G 2x2 3x3
GENVP (NO CONTRAST)
50.5
54.5
48.7
48.9
49.3
46.0 59.0 46.9
GENVP
94.7
100
99.7
99.9
99.9 86.0 93.3 84.1
Table RPM solving accuracy (%) for VAD and PGM datasets. Training GenVP without contrast- ing terms (only ELBO maximization) (row 1) and with contrasting training (row 2).
MODEL
DT
TD-LT TD-SC
N
1
E
LT
SC
AP
AR
ARP
GENVP (NO CONTRAST)
67.0
68.0
55.4
54.7
58.5
47.4
30.3
14.3
17.8 12.7 27.4
16.1 26.6 17.0 72.2
GENVP
95.8 94.4
81.4
74.5
78.7
79.5
58.9
15.6
17.6
12.7
65.2
tically when we introduce contrastive learning. This is expected since ELBO only maximizes the likelihood of full puzzles, which are completed with the correct answer from the choice list; it is the Eq. (TT) which introduces the negative candidates and helps GenVP distinguish the solution from hard negative choice list images.
Table 10: RAVEN-FAIR (Benny al.
2021
puzzle-solving accuracy for different MoR estimators
using GenVP individual rule predictions.
MIXTURE TYPE
OIC
OIG
2
3 x 3
WEIGHTED AVG
100
100
100
100
95.7
96.1
89.2
AVG
100
100
100
100
95.2
95.1
88.7
ARGMAX AVG
99.9
97.4
97.5
99.4
85.2
75.8
68.8
PROD
97.5
96.2
97.1
99.1
91.7
81.2
83.3
ARGMAX PROD
97.6
93.4
91.5
95.4
84.9
68.2
68.3
NN
100
99.7
99.9
99.9
94.4
96.0
90.2
C.6 ABLATION STUDY FOR DIFFERENT TYPES OF MIXTURE EXPERTS FOR RULE PREDICTION
When solving an RPM puzzle, we aim to exploit signals from all learned rule predictions (Eq. (.) Eq. @ Eq. (3), Eq. (4) to improve GenVPs' robustness. In our experiments, we analyzed the performance of the following mixture functions:
1. Non-Parametric:
(a) WEIGHTED AVG: Our pest-performing non-parametric function is the weighted av- erage estimator, benefiting from the noise reduction properties of averaging (Kraft- makker 2006):
PMoE,g 1
PR (
(oo)
Σ
PRI.g (Zo \ Zboloo +
4
MN
kE[M))E[[]]
(19)
Σ
PR (Zo 2ko.o) + PRo.q (Z,)
M
kE[M]
(b) AVG: All terms contribute the same:
PMoE,y
PR.,((Z))
> pR.. (Z.) k.!
+ (Z))zz.,)) + +PRs (Zr))
NMoE
(20)
where NooB is equal to the total number individual predictors. (c) ARGMAX AVG: First, calculate the simple average of the individual rule predictors, as described in Eq. (20). Next, we replace each row of the return matrix with the one- hot encoding of the column index that has the maximum value.
17

---

Published as a conference paper at ICLR 2025
Table 11: PGM (Barrett al. 2018)
and VAD (Hill et al. 2019) puzzle-solving accuracy for para-
metric (NN) and non-parameinc (WEIGHTED AVG) MoE estimators using GenVP individual rule predictions.
VAD
PGM
MIXTURE TYPE
DT
E
TD-LT
TD-SC
LT
SC
AR
ARP
WEIGHTED AVG
95.8
94.4
81,4
74.5
78.7
79.5
58.9
15.6
17.6
12.7
65.2
17.0
72.2
NN
96.3
94.9
80.1
76.3
79.6
85.5
65.6
16.2
17.6
12.8
73.2
15.5
78.7
(d) PROD: The final rule matrix is the product of all rule predictors.
(e) ARGMAX PROD: We begin by calculating the PROD mixture. Next, we replace each row of the return matrix with the one-hot encoding of the column index that has the maximum value.
2. Parametric (neural network approximators (NN)): We use neural network to determine the best data-driven mixture of rule predictions, maximizing the puzzle-solving accuracy. The NN takes as input all rule predictions from the pretrained and frozen GenVP, which are arranged channels of CNN layer. Our objective is to use the rule predictions from the completed puzzles (using the choice list) to find the solution. We use standard cross- entropy loss to train NN.
Table TO presents the puzzle-solving accuracy of RAVEN-FAIR using the aforementioned mixture functions. In almost all cases, WEIGHTED AVG achieves the best performance, followed by AVG and SC. In Table II we compute the puzzle-solving performance using the best non-parametric estimator (WEIGHTED AVG) and the parametric NN.
D MODEL EFFICIENCY AND SCALABILITY EVALUATION
In this section, we measured generative models' efficiency and scalability abilities for abstract visual reasoning. For fair comparison, we evaluate all models on the same server for the same batch size (set to 100). The server characteristics are 48GB NVIDIA RTX A6000 GPUs and Dual AMD EPYC 7352 @ 2.3GHz 48 cores, 96 vCores CPU.
Table 12: Time and Memory requirements of GenVP and the baseline models RAISE (Si et al., 2024], PRAE (Zhang al., 2019b), and GCA (Pekar et al. 2020). All models were evaluated on the same server for batch size of 100, except PRAE3 3, which could not fit even with batch size one on our server (marked as out-of-memory (OOM)). Time per iteration complexity is computed across an average of 100 training iterations. The models that have confiuraationindepennne (therefore, architecture does not change for different configurations) are marked as N/A the row CONFIG.
architecture
MODEL.
GENVP
RAISE
PRAE
GCA
TRAINING PHASE
ELBO ELBO+C
(FOR GENVP)
CONFIG
N/A
N/A
N/A
CS
LR
UD
OIG 245,555
OIC
2 2 3 3
N/A
# OF PARAMETERS
8,879,877
9,346, 703
23,619,591
BATCH SIZE
100
100
100
100
100
100
100
100
100
100
GPU MEM. (GB)
2.8
5.17
4.2
1.12
1.76
1.76
1.76
3.64
3.03
00M
15.3
TIME/ITER. (MSEC)
208.2
240.4
210.6
68.1
113.2
114.9
128.2
189.6
131.6
1082.7
D.1 EFFICIENCY EVALUATION OF DIFFERENT ABSTRACT VISUAL REASONING MODELS
To evaluate the model's efficiency, we measure the average duration of a training iteration (time/iter. in msec). We compute the average 100 training steps. We also record the GPU memory (in GB) requirements and the required number of parameters of each model. In Table 12 present the efficiency analysis results. We note that all experiments were performed on a single GPU using the RAVEN training data. We notice that approach, GenVP, compared to
18

---

Published as a conference paper at ICLR 2025
GCA: Gen requires ~ 1/3 of the model parameters, 1/3 of GPU memory, and 1/5 the iteration duration.
PRAE: Although PRAE has significantly fewer model parameters than other generative models, due to the strong inductive biases used in its reasoning (PRAR uses knowledge of the rule formulas rather than learning them from the data), observed that the model struggles to scale in complex scenarios. In particular, for the RAVEN configuration 3 x 3, PRAE could not even fit our server with batch size of one.
RAISE: GenVP has 500, 000 fewer parameters from RAISE. The two models have sim- ilar GPU memory and average time per iteration requirements. Specifically, during the ELBO pretraining phase, Gen VP occupies slightly less GPU and runs slightly faster. Dur- ing our joint ELBO and contrasting training (ELBO+C), we observe opposite: RAISE is slightly lighter and faster.
Table 13: Scalability evaluation for different Gen VP neural network architectures and puzzle image
sizes. All models were evaluated on the same server for a batch size of 100. Time per iteration complexity is computed across average of 100 training iterations. We also report each model's accuracy on the RAVEN dataset (we train all models using all RAVEN configurations)
IMAGE SIZE ARCHITECTURE
x CNN-S
x 64 64 X
128 X 128
224 224
221 224
221 x 224
CNN-L
RESNET VIT 6,070,469
RESNET
RESNET 25,918,758
RESNET
RESNET
# OF PARAMETERS
2, 2,122,253
6,323,973
8,879,877
9,945.028
25,918,758
25,918,758
100
100
100
100
# OF GPUS
1
1
1
2 28.7/GP 729.0
4 14.9/GPU 390.3
GPI MEM. (GR)
5.17
TIME//TER.
209.6
268.1
240.4
505.
1263.9
ACCURACY
94.9
89.5
94.2
94.4
94.8
90.6
D.2 GENVP SCALABILITY EVALUATION
In this section, we investigate the scalability properties of our model using different neural network architectures and sizes of the puzzle images. In particular, we perform experiments by training a single model for all RAVEN configurations for the following different architectures and puzzle image input sizes:
For 64 64 image size
CNN-S: we use much smaller CNN-based (LeCun et al. 1995) image encoder.
- CNN-L: the architecture we used for training our RAVENT-RAVEN models in the
main paper.
ResNet: we use a ResNet-based (He et al. 2016) image encoder.
VIT: we ViT-based (Dosovitskiy 2020) image encoder.
For 128 x 128 image size, we use a Ressettbased encoder architecture.
For 224 x 224 image size, we use ResNet-based encoder architecture.
Naturally, the time and memory complexity increase as we increase the model size and image reso- lution as shown in Table Despite this, we found that even our heaviest version (with 224 x 224 image) can still run on single GPU on our server. Additionally, in order to reduce time complexity, we parallelized Ge to run on multiple GPUs. This allowed our model to train faster on our server with smaller memory requirements per GPU at a time per iteration, which is comparable to our lowest resolution (64 64) models. Regarding puzzle-solving performance, we noticed that our approach can be generalized to different architectures and achieve state-of-the-arl performance with almost setups having above 90% accuracy.
E PERFORMANCE OF INDIVIDUAL RULE PREDICTORS VS. MIXTURE OF EXPERTS (MoE)
In this section, we compare GenVP's performance using individual rule predictors against its per- formance using the MoE (all predictors) for the RAVEN dataset. The results summarized in Table 14 We observe the following:
State-of-the-Art Performance with MoE: GenVP achieves the best performance (95.0% accuracy) when combining rule predictors through the MoE mechanism, demonstrating its effectiveness.
19

---

Published as a conference paper at ICLR 2025
Table 14: In the MoE column, we present the puzzle-solving accuracy using all rule predictors. In the remaining columns, we show the accuracy when using individual rule predictors R(Z), where Z denotes the input used for the individual rule prediction. For notation simplicity in the Table, for the rule predictions using only two rows, we use the notation 2 meaning that we use the two rows i and j. for j.j 1.2.3. For the context-based rule predictions (one image is missing from the puzzle), we use the notation z meaning that the i-th image of the puzzle is missing for i == ......99 9. The cases _2 and g are excluded since these instances do not contain the choice list/candidate images placed in the bottom-right position of the RPM matrix.
MoE I() R(Z,)) I(2(3")) 15
R(221)
R(("))) I((23)) R(Z R(22)) R(ZC))) R(Z2F) R(ZZE)
95.0
48.7
93.2
87.0
87.1
81.7
82.8
80.6
82.0
82.9
81.7
16.3
16.7
Performance of Individual Predictors: Among individual predictors, the puzzle-level representation predictor R(Z,) achieves the highest performance, followed by predictors using two puzzle rows (R(Zi3")) R(Z25"") ")) and context-based predictors (R(ZZT), R( .... R)) Predictors using image-level representations (R(Z,) show com- paratively lower performance.
The high performance of R(Z,) indicates that the puzzle-level latent variables exclude most the information irrelevant to the rules (noise), making them robust for the puzzle- solving task.
Additionally, we observe that the rule predictors focusing on only two rows outperform R(Zo), which observes all three rows. By restricting these predictors to derive the rules from just two rows instead of three, the problem becomes more challenging, leading to greater robustness. This restriction prevents these predictors from exploiting shortcuts for rule prediction. For example, in the case of R(Z,), the model might predict rules by focusing only on the first two rows of the puzzle, which would fail in scenarios where the third row contains an imposter image.
Challenging Scenarios: The context predictors (R(Z=*)) R(Zz)) perform poorly when the missing context matrix image lies in the same row as the negative candidate. This occurs because, in such cases, even though the last row contains an imposter image, the absence of the 7th or 8th image allows GenVP to imagine plausible missing image that aligns with the puzzle rule.
For example, consider puzzle governed by arithmetic rule based the number of objects. The correct puzzle might have a number of objects (1,2,3; 2,2,4; 3,2,5), where the third image in each row is the sum of the first two. Now, suppose the last row contains an incorrect image, such as (1,2,3; 2,2,4; 3,2,4). This row violates the arithmetic rule, and Ge should identify this as a puzzle with an incorrect candidate.
However. if we consider the R (ZZzz) predictor and observe partially complete matrix, such as (1,2,3; 2,2,4; 3,,,4), this puzzle could still be valid under the arithmetic rule if the missing image were (1,2.3; 2,2,4; 3,1,4). Thus, when the 7th or 8th image is removed from a puzzle containing an imposter image the 9th panel, GenVP can imagine a plau- sible image that corrects the rule violation, leading to performance drop for these context predictors.
Table 15: Gen VP Puzzle Solving Accuracy for the RAVEN dataset. We train four different versions, using different hyperparameter values 3r Ba, Br, 3x- for 1, 1,..,,6, 6, 2
GENVP HYPERPARAMETERS
ACCURACY
81
BRI
33
90
AB
31
1
250
20
20
94.7
1
1
1
20
20
95.0
1
1
1
92.1
1
1
40
94.9
20

---

Published as a conference paper at ICLR 2025
F GENVP PERFORMANCE VS. HYPERPARAMETERS VALUES
F.1 ABLATION ON DIFFERENT OPTIMIZATION OBJECTIVE HYPERPARAMETERS
We conduct additional ablation studies to evaluate the influence of hyperparameters on Gen VP's performance. Our results, in Table show that GenVP performance remains robust across range of configurations. particular:
Rule Matrix Weight (Bn=): In our primary experiments, assigned higher weight (3R- = 250) to the rule matrix prediction term compared to other ELBO terms (3, = 1, + 2). To test sensitivity, we set all 3, and 8R- i 1.2..... 6, + 2 one, effectively removing the emphasis on the rule matrix. This resulted in an improvement in perfor- mance (94.7% to 95.0%) demonstrating that the model is robust even when the weighting is altered.
Contrastive Loss Weights (5c, 3z): In our main experiments, 3G and 3r. were set to 20. In the additional experiments, we evaluated three configurations (9g. 8g = 1, 20, 40). Results showed that very small weights (Bg. B2 1) caused modest degradation (92. % accuracy), but increasing the weights (Be, Br, = 20, 40) stabilized performance 95%. This suggests that Ge benefits from reasonable tuning of contrastive terms but does not require fine-gn tineggrinee optimization.
Performance Consistency Across Hyperparameters: Across all tested hyperparameter configurations. Ge maintained strong puzzle-solving accuracy. The observed varia- tions in performance were within an acceptable range, indicating that the model's core de- sign, alongside its generative and contrastive training schemes is robust to hyperparameter changes.
In summary, while adjustment of certain hyperparameters, such BR., fa. and Br., can optimize performance, Ge does not rely heavily on precise tuming. Its stability across wide range of settings underscores its adaptability and reliability in RPM puzzle solving tasks.
Table 16: GP puzzle solving accuracy for the RAVEN dataset. We trained five different versions of the model, using varying hyperparameter values K, Kz Kz Kz,, KR. and NR, which control the size of Gen VP latent variables.
# PARAMETERS
z,
K2,
Kz,
KR
NR
ACCURACY
8,250.313
32
20
12
60
12
5
91.6
8.912.077
54
162
95.3
8,957,157
64
54
10
162
12
15
92.4
9.758,185
128
100
28
300
12
5
94.8
20,208,245
128
126
2
378
12
5
94.9
F.2 ABLATION ON VARYING LATENT VARIABLES DIMENSIONALITY
In this ablation study, we conducted additional experiments to explore the impact of latent variable size Gen performance. As observed in Table 16 GenVP consistently achieves high accuracy across a range of hyperaaraeeter configurations. Notably, the latent variable dimensions and total number of model parameters complexity), we expect Gen performance to be affected. However, although representing each pixel-level image of RAVEN with latent vector Zij. i,j E 3 of dimensionality K 32 is re- strictive (prior work, i.e. RAISE, uses 64-dimensional image encodings in the latent space), Ge manages to achieve an accuracy of 91.6% (first row in Table I6.. As the model total parameters and the capacity of the latent space vectors increase, GenVP's accuracy improves and stabilizes at 95%. This indicates that Gen VP is robust to architectural hyperparameter choices, though models with large enough latent vectors generally yield better results.
restrictive (do not have enough capacity capture the puzzle
Table 17: GP performance when trained without rule annotations for the RAVEN and I-RAVEN datasets.
DATASET
RAVEN
I-RAVEN
RAISE
54.5
67.7
GENVP
70.3
73.8
21

---

Published as a conference paper at ICLR 2025
G GENVP WITHOUT RULE ANNOTATIONS
We investigated the performance of GenVP without any rule annotations, using only the puzzle im- ages and their correct candidate label. The results in Table 17 highlight GenVP's ability to perform reasonably well without explicit rule annotations, achieving significantly higher accuracy than in prior work. This indicates that Gen captures useful latent representations even without direct supervision of the rules.
H RPM GENERATION QUALITATIVE EXAMPLES
In Figs. Ligs..5aa to 8 we include additional generated RPMs from GenVP. The generation process starts by
selecting desired set of rules and then performing ancestral sampling for the rule-invariant latent
representation Zg. In each case, we present three different generated RPM samples for specific set
of rules (located in the top part of the figures).
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr.-3
Arithm.
Prog.
Prog.
Prog.
Prog.
Color
Color
Color
Color
Size
Size
Size
Size
Type
Type
Type
Type
Angle
Angle
Angle
Angle
Num.
Num.
Num.
Num.
Pos.
Pos.
Pos.
Pos.
0 00
Figure 5: Generated RPM matrices by Gen VP for distribute nine (3 x 3) configuration. Top: Rules; Bottom: Three different sampled generated puzzles.
22

---

Published as a conference paper at ICLR 2025
No Rule
Const.
D
Arithm.
No No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
D
Arithm.
Prog.
Prog.
Prog.
Prog
Color
Color
Color
x
Color
Size
Size
x
Size
Size
Type
Type
Type
X
Type
Angle
x
Angle
Angle
Angle
Figure 6: Generated RPM matrices by GenVP for L-R configuration. Top: Rules; Bottom: different sampled generated puzzles.
Three
23

---

Published as a conference paper at ICLR 2025
No Rule
Const.
Distr.-3
Prog. Arithm.
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr--3
Arithm.
Rule
Const.
Distr.-3
Arithm.
Prog.
Prog.
Prog.
Color
Color
Color
Color
Size
Size
Size
Size
Type
Type
Type
Type
Angle
Angle
Angle
Angle
Figure 7: Generated RPM matrices by GenVP for center single (c-s) configuration. Top: Rules; Bottom: Three different sampled generated puzzles.
24

---

Published as a conference paper at ICLR 2025
No Rule
Const.
Distr.-3
Prog. Arithm.
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr.-3
Arithm.
No Rule
Const.
Distr.-3
Arithm.
Prog.
Prog.
Prog.
Color
Color
Color
Color
Size
Size
Size
Size
Type
Type
Type
Type
Angle
Angle
Angle
Angle
Num.
Num.
Num.
Num.
Pos.
Pos.
Pos.
Pos.
Figure 8: Generated RPM matrices by Ve for distribute four (2 x 2) configuration. Top: Rules; Bottom: Three different sampled generated puzzles.
25


Reviews:
[{"review_id": "1akMp1G4O9", "reviewer": "Reviewer_YBT2", "summary": "Briefly, this paper presents a framework to solve and create complete new puzzles out of the desired set of rules for Raven’s Progressive Matrices (RPM) by introducing the contrastive learning scheme (i.e., cross-puzzle and cross-candidate contrastive loss) and MoE mechanism for puzzle rule prediction. The experimental results are strong.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The paper is well-written and easy to follow. The idea of the proposed method is quite interesting.\n+ Extensive and comprehensive experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "1) The technical novelty of the proposed method seems to be marginal since the authors directly employ the existing techniques (e.g., contrastive learning scheme and MoE). More detailed discussions and analyses are required to demonstrate the contribution of the proposed method. \n\n2) More ablation studies are required to demonstrate the contribution of the main component of the proposed method in the main paper. For instance, the mentioned rule estimators for the RPM-Level inference, the performance gain of the introduced MoE mechanism, and the contrastive learning scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iYO4aL88BS", "reviewer": "Reviewer_YfAj", "summary": "The authors proposed a novel deep latent variable model GenVP for RPM problems. GenVP extracts image features and then uses these features to form relevant and irrelevant representations for later inference. The proposed MoE estimator can infer categories of attribute-level rules used to generate RPM panels. The proposed model was evaluated through experiments on RAVEN/I-RAVEN, VAD and PGM, showing better performance than other generative-based methods.", "questions": "1.\tCould the authors explain the process of answer selection in detail?\n2.\tWill the selection of hyperparameters heavily influence the performance of GenVP? The RPM datasets contain just small sets of object attributes and rules, and the selection of hyperparameters can probably matter a lot in this case. The authors should discuss this as well.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.\tThis paper is generally well-written and the main idea is easy to follow. The authors proposed a new generative model GenVP to encode rule-related information. It will be more useful to construct interpretable machine learning algorithms for abstract visual reasoning.\n2.\tCompared to the previous generative approach RAISE, GenVP is unconditional and therefore can generate novel RPMs without given context.\n3.\tThe decomposition of relevant and irrelevant representations improves the robustness of GenVP when there is too much noise or resolution in puzzles.\n4.\tThe authors showed that their GenVP can perform better than previous generative methods.", "weaknesses": "1.\tMany important factors in GenVP should be carefully tuned. For example, and in the sampling process should be carefully handcrafted.\n2.\tThe proposed model seems to rely heavily on ground truth rule annotations. GenVP leverages rule annotations of each RPM sample in the training process. This limits the applicability of this method to unlabelled visual reasoning datasets. The authors should investigate why the models cannot learn the rules well without annotations.\n3.\tUsing VAE-based generative solvers or contrastive loss is not novel in RPMs. Is there any novel technical or insightful design in GenVP compared to previous approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AfxObhEfu3", "reviewer": "Reviewer_GBgR", "summary": "This paper presents a latent variable model named GenVP for RPM-style visual puzzles. GenVP is trained from RPM images and the corresponding categorical rule matrices by inferring hierarchical Gaussian latent representations, which can be decoded back to images, through an ELBO loss and a global-local masked contrastive loss. GenVP learns a set of underlying rules controlling how an RPM is composed. The experiment results show that GenVP outperforms SOTA generative approaches in both out-of-distribution generalization and large solution space scenarios.", "questions": "1.\tIn the graphical model of GenVP in Figure 1, $R$ is a shaded circle which means observation; however, from the context of model description it seems that $R$ is inferred. \n2.\tCould the authors provide results on any other visual puzzles or abstract visual reasoning tasks? My main concern is whether GenVP can be applied to more realistic reasoning problems with a unspecific rule set.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper defines delicate generation and inference models for RPMs, and the details are easy to follow. The choice and hierarchy of latent variables are reasonable, e.g., decomposing relevant and irrelevant attributes from an image.\n2.\tExperiments are comprehensive, with a focus on OOD configurations and some challenging datasets. According to the testing and visualization results, the proposed GenVP outperforms SOTA generative approaches.", "weaknesses": "1.\tLimitation of the application scenarios. The design of the generative model seems rather tailored to RPM problems. This focuses on a very specific problem hence its significance might be limited. It is unclear how to apply GenVP in other abstract visual reasoning tasks.\n2.\tRequirement of annotation of rules. Can a GenVP be trained without supervision of rules? For human beings, one does not require to learn rules through supervision, the rules can be discovered by the test subject.\n3.\tThe interpretability of answer selection. GenVP infers MoE rule matrix predictions for candidates and chooses the one with the largest set of active rules as the final answer. Why not select answers by comparing the generated answers to candidates? Maybe it is a more human-like way of answer selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kXErx1LpCu", "reviewer": "Reviewer_npYP", "summary": "This article introduces a new framework called Generative Visual Puzzles (GenVP), which aims to simulate the entire generation process of Raven's Progressive Matrices (RPMs). It not only performs excellently in solving existing puzzles but also demonstrates strong capabilities in creating new puzzles and generalizing to new, unseen puzzle scenarios.", "questions": "1.Although the author has done a lot of mathematical derivations, I am still somewhat confused about the process of generating puzzles. Could the author please describe in more detail how to generate puzzles from the rules?\n2.It seems that the author did not compare the visualization of image generation with previous methods. Is it because previous methods couldn't generate such images?\nI will consider to raise my score according to the rebuttal and discussion with  other reviewers", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The authors propose GenVP, a novel approach for solving and creating visual puzzles.\n2.The authors design a new cross-puzzle and cross-candidate contrastive loss for AVR. The proposed GenVP is robust to noise.\n3.The authors conducted extensive experiments to demonstrate the superior performance of the GenVP method.\n4.GenVP can generate a large number of new puzzles beyond the original source dataset.", "weaknesses": "1.Although the author's experiments are very detailed, the rules of the puzzle generation task in this article are relatively simple, and the task's search space is relatively small.\n2.According to Figure 2, it can be seen that the image quality of the puzzles generated by this method is still not good enough; the edges of the puzzle elements are relatively blurry, and there are artifacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)