GenVP: Generating Visual Puzzles with Contrastive Hierarchical VAEs

Authors: Kalliopi Basioti, Pritish Sahu, Tony Qingze Liu, Zihao Xu, Hao Wang, Vladimir Pavlovic
Primary area: generative models
Keywords: abstract visual reasoning; contrastive learning; variational autoencoders; out of distribution generalization

Abstract:
Raven’s Progressive Matrices (RPMs) is an established benchmark to examine
the ability to perform high-level abstract visual reasoning (AVR). Despite the current success of algorithms that solve this task, humans can generalize beyond a given puzzle and create new puzzles given a set of rules, whereas machines remain locked in solving a fixed puzzle from a curated choice list. We propose Generative Visual Puzzles (GenVP), a framework to model the entire RPM generation process, a substantially more challenging task. Our model’s capability spans from generating multiple solutions for one specific problem prompt to creating complete new puzzles out of the desired set of rules. Experiments on five different datasets indicate that GenVP achieves state-of-the-art (SOTA) performance both in puzzle-solving accuracy and out-of-distribution (OOD) generalization in 22 out
of 24 OOD scenarios. Further, compared to SOTA generative approaches, which struggle to solve RPMs when the feasible solution space increases, GenVP efficiently generalizes to these challenging scenarios. Moreover, our model demonstrates the ability to produce a wide range of complete RPMs given a set of abstract rules by effectively capturing the relationships between abstract rules and visual object properties.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730468234778

thread_type:
official_review

<thread_object>
creator: reviewer_npyp

<content>
summary:
This article introduces a new framework called Generative Visual Puzzles (GenVP), which aims to simulate the entire generation process of Raven's Progressive Matrices (RPMs). It not only performs excellently in solving existing puzzles but also demonstrates strong capabilities in creating new puzzles and generalizing to new, unseen puzzle scenarios.

strengths:
1.The authors propose GenVP, a novel approach for solving and creating visual puzzles.
2.The authors design a new cross-puzzle and cross-candidate contrastive loss for AVR. The proposed GenVP is robust to noise.
3.The authors conducted extensive experiments to demonstrate the superior performance of the GenVP method.
4.GenVP can generate a large number of new puzzles beyond the original source dataset.

weaknesses:
1.Although the author's experiments are very detailed, the rules of the puzzle generation task in this article are relatively simple, and the task's search space is relatively small.
2.According to Figure 2, it can be seen that the image quality of the puzzles generated by this method is still not good enough; the edges of the puzzle elements are relatively blurry, and there are artifacts.

questions:
1.Although the author has done a lot of mathematical derivations, I am still somewhat confused about the process of generating puzzles. Could the author please describe in more detail how to generate puzzles from the rules?
2.It seems that the author did not compare the visualization of image generation with previous methods. Is it because previous methods couldn't generate such images?
I will consider to raise my score according to the rebuttal and discussion with  other reviewers
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528945989

thread_type:
official_review

<thread_object>
creator: reviewer_gbgr

<content>
summary:
This paper presents a latent variable model named GenVP for RPM-style visual puzzles. GenVP is trained from RPM images and the corresponding categorical rule matrices by inferring hierarchical Gaussian latent representations, which can be decoded back to images, through an ELBO loss and a global-local masked contrastive loss. GenVP learns a set of underlying rules controlling how an RPM is composed. The experiment results show that GenVP outperforms SOTA generative approaches in both out-of-distribution generalization and large solution space scenarios.

strengths:
1.	This paper defines delicate generation and inference models for RPMs, and the details are easy to follow. The choice and hierarchy of latent variables are reasonable, e.g., decomposing relevant and irrelevant attributes from an image.
2.	Experiments are comprehensive, with a focus on OOD configurations and some challenging datasets. According to the testing and visualization results, the proposed GenVP outperforms SOTA generative approaches.

weaknesses:
1.	Limitation of the application scenarios. The design of the generative model seems rather tailored to RPM problems. This focuses on a very specific problem hence its significance might be limited. It is unclear how to apply GenVP in other abstract visual reasoning tasks.
2.	Requirement of annotation of rules. Can a GenVP be trained without supervision of rules? For human beings, one does not require to learn rules through supervision, the rules can be discovered by the test subject.
3.	The interpretability of answer selection. GenVP infers MoE rule matrix predictions for candidates and chooses the one with the largest set of active rules as the final answer. Why not select answers by comparing the generated answers to candidates? Maybe it is a more human-like way of answer selection.

questions:
1.	In the graphical model of GenVP in Figure 1, $R$ is a shaded circle which means observation; however, from the context of model description it seems that $R$ is inferred. 
2.	Could the authors provide results on any other visual puzzles or abstract visual reasoning tasks? My main concern is whether GenVP can be applied to more realistic reasoning problems with a unspecific rule set.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561956739

thread_type:
official_review

<thread_object>
creator: reviewer_yfaj

<content>
summary:
The authors proposed a novel deep latent variable model GenVP for RPM problems. GenVP extracts image features and then uses these features to form relevant and irrelevant representations for later inference. The proposed MoE estimator can infer categories of attribute-level rules used to generate RPM panels. The proposed model was evaluated through experiments on RAVEN/I-RAVEN, VAD and PGM, showing better performance than other generative-based methods.

strengths:
1.	This paper is generally well-written and the main idea is easy to follow. The authors proposed a new generative model GenVP to encode rule-related information. It will be more useful to construct interpretable machine learning algorithms for abstract visual reasoning.
2.	Compared to the previous generative approach RAISE, GenVP is unconditional and therefore can generate novel RPMs without given context.
3.	The decomposition of relevant and irrelevant representations improves the robustness of GenVP when there is too much noise or resolution in puzzles.
4.	The authors showed that their GenVP can perform better than previous generative methods.

weaknesses:
1.	Many important factors in GenVP should be carefully tuned. For example, and in the sampling process should be carefully handcrafted.
2.	The proposed model seems to rely heavily on ground truth rule annotations. GenVP leverages rule annotations of each RPM sample in the training process. This limits the applicability of this method to unlabelled visual reasoning datasets. The authors should investigate why the models cannot learn the rules well without annotations.
3.	Using VAE-based generative solvers or contrastive loss is not novel in RPMs. Is there any novel technical or insightful design in GenVP compared to previous approaches.

questions:
1.	Could the authors explain the process of answer selection in detail?
2.	Will the selection of hyperparameters heavily influence the performance of GenVP? The RPM datasets contain just small sets of object attributes and rules, and the selection of hyperparameters can probably matter a lot in this case. The authors should discuss this as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730791601432

thread_type:
official_review

<thread_object>
creator: reviewer_ybt2

<content>
summary:
Briefly, this paper presents a framework to solve and create complete new puzzles out of the desired set of rules for Raven’s Progressive Matrices (RPM) by introducing the contrastive learning scheme (i.e., cross-puzzle and cross-candidate contrastive loss) and MoE mechanism for puzzle rule prediction. The experimental results are strong.

strengths:
+ The paper is well-written and easy to follow. The idea of the proposed method is quite interesting.
+ Extensive and comprehensive experiments demonstrate the effectiveness of the proposed method.

weaknesses:
1) The technical novelty of the proposed method seems to be marginal since the authors directly employ the existing techniques (e.g., contrastive learning scheme and MoE). More detailed discussions and analyses are required to demonstrate the contribution of the proposed method. 

2) More ablation studies are required to demonstrate the contribution of the main component of the proposed method in the main paper. For instance, the mentioned rule estimators for the RPM-Level inference, the performance gain of the introduced MoE mechanism, and the contrastive learning scheme.

questions:
Please refer to the Weaknesses section.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes GenVP, a generative framework for solving and creating Raven’s Progressive Matrices (RPM) puzzles by leveraging contrastive hierarchical VAEs. The model introduces novel disentangled representations for rule-relevant and irrelevant factors, a Mixture of Experts (MoE) module for rule prediction, and a contrastive learning scheme to enhance generalization. Experiments show state-of-the-art (SOTA) performance in RPM-solving accuracy and out-of-distribution (OOD) generalization, as well as the capability to generate novel RPM puzzles.

Reviewer GBgR maintained a negative stance, primarily due to concerns about the limited generalizability of GenVP beyond RPM tasks, the reliance on annotated rules for training, and perceived limitations in interpretability during answer selection. Reviewer YBT2 criticized the incremental nature of the contributions, particularly the use of existing techniques like VAEs and contrastive loss. The authors provided detailed rebuttals and additional experiments addressing these concerns. For GBgR’s generalization concerns, they demonstrated GenVP's scalability to diverse datasets and problem settings, including rule-annotation-free training, which showed significant performance gains over prior models. Regarding the reliance on annotated rules, the authors emphasized consistency with prior methods and conducted unsupervised training experiments to showcase potential for broader applicability. For YBT2’s novelty concerns, the authors highlighted several distinct contributions, including disentangled latent spaces, high-level rule reasoning, and puzzle-wide contrastive loss, which differentiate GenVP from prior approaches.

Considering the detailed rebuttal, good empirical results, and improvements over prior methods, the AC recommends acceptance. The authors have addressed major concerns with additional evidence and clarified the unique contributions of their approach. However, future work should focus on demonstrating broader applicability and minimizing reliance on annotations to further solidify the framework's impact.