Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Authors: Fan Shi
Bin Li
Xiangyang Xue
Venue: ICML 2025 poster (2025)
Keywords: Abstract Visual Reasoning
Conditional Generative Model
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A unified conditional generative solvers to solve different abstract visual reasoning tasks

Abstract: Abstract visual reasoning (AVR) enables humans to quickly discover and generalize abstract rules to new scenarios. Designing intelligent systems with human-like AVR abilities has been a long-standing topic in the artificial intelligence community. Deep AVR solvers have recently achieved remarkable success in various AVR tasks. However, they usually use task-specific designs or parameters in different tasks. In such a paradigm, solving new tasks often means retraining the model, and sometimes retuning the model architectures, which increases the cost of solving AVR problems. In contrast to task-specific approaches, this paper proposes a novel Unified Conditional Generative Solver (UCGS), aiming to address multiple AVR tasks in a unified framework. First, we prove that some well-known AVR tasks can be reformulated as the problem of estimating the predictability of target images in problem panels. Then, we illustrate that, under the proposed framework, training one conditional generative model can solve various AVR tasks. The experiments show that with a single round of multi-task training, UCGS demonstrates abstract reasoning ability across various AVR tasks. Especially, UCGS exhibits the ability of zero-shot reasoning, enabling it to perform abstract reasoning on problems from unseen AVR tasks in the testing phase.

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Fan Shi
Bin Li
Xiangyang Xue
Abstract
2021; Malkiński & Mandziuk, 2022a). This ability is pivotal in cognitive activities like decision visual analogy and spatial reasoning (Gray& & Thompson, 2004; McCloskey, 1983). Many psychological tests are designed to assess the AVR ability of humans, which often requires the subject to discover specific rules that connect different visual concepts (Raven & Court, 1938; McMillen, 2007; Nie et al., 2020). For example, Raven's Progressive Matrix (RPM) (Raven & Court, 1938) focuses assessing generalization ability, as the tests include unseen combinations of visual concepts and rules, which is a classical intelligence quotient test.
Abstract visual reasoning (AVR) enables humans to quickly discover and generalize abstract rules to new scenarios. Designing intelligent systems with numan-like AVR abilities has been long- standing topic in the artificial intelligence commu- nity. Deep AVR solvers have recently achieved re- markable success in various AVR tasks. However, they usually use task-specific designs or parame- ters in different tasks. In such a paradigm, solv- ing new tasks often means retraining the model, and sometimes retuning the model architectures, which increases the cost of solving AVR problems. In contrast task-specific approaches, this paper proposes a novel Unified Conditional Generative Solver (UCGS), aiming to address multiple AVR tasks in a unified framework. First, we prove that some well-known AVR tasks can be reformulated as the problem of estimating the predictability of target images in problem panels. Then, we illus- trate that, under the proposed framework, training one conditional generative model can solve vari- ous AVR tasks. The experiments show that with a single round of multi-task training, UCGS demon- strates abstract reasoning ability across various AVR tasks. Especially, UCGS exhibits the ability of zero-shot reasoning, enabling it to perform ab- stract reasoning on problems from unseen AVR tasks in the testing phase.
Developing models that reveal human-like AVR ability re- mains a significant challenge in artificial intelligence re- search (Zheng al., 2019; Chollet, 2019; Malkiński & Mańdziuk, 2022b). Many AVR benchmarks have been pro- posed to evaluate intelligent systems in way that mirrors human-like AVR ability (Barrett et al., 2018; Hill et al., 2019; Zhang et al., 2019; Nie et al., 2020). These bench- marks usually adopt the problem structure of traditional psy- chological tests and provide programs to generate a large number of problems automatically, e.g., the RAVEN dataset generates RPM problems by combining predefined visual concepts and abstract rules (Zhang et al., 2019). In recent years, deep AVR solvers have achieved remarkable progress on the proposed AVR benchmarks (Barrett et al., 2018; Zhuo & Kankanhalli, 2021; Mondal et al., 2023). However, they often rely on task specifei inductive biases of model archi- tecture and hyperparameters, and solving different tasks typically means training different instances of models. This task-specific paradigm arouses interest in exploring more unified framework for AVR solvers (Webb et al., 2024).
1. Introduction
AVR tasks take different structures of problem panels to as- sess the subject's ability to understand abstract rules. RPM- style tasks (Barrett et al., 2018; Zhang et al., 2019) require discovering abstract rules of 3 x 3 image matrix and select- ing answers from a candidate panel to complete the matrix via analogy. Odd-one-out problems (McMillen, 2007) test the ability to identify the image that violates the abstract rules of a sequence. In addition, AVR tasks can be distin- guished as generative tasks and selective tasks. Generative tasks require the synthesizing or reconstruction of images (Bar et al., 2022; Dedhia al., 2023; Bai et al., 2023), while
Abstract visual reasoning (AVR) a fundamental cognitive ability that enables humans to identify visual concepts, infer underlying abstract rules, and generalize the rules new scenarios effectively (Cattell, 1963; Zhuo & Kankanhalli,
'Shanghai Key Laboratory of Intelligent Information Process- ing, School of Computer Science, Fudan University. Correspon- dence to: Bin Li <libin @ .dd.n.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
selective tasks provide predefined options for the selection of answers (Barrett et al., 2018; Hill et al., 2019; Nie al., 2020). These AVR tasks have different structures and goals, posing challenges in proposing a unified problem-solving framework.
such as hierarchical features (Zheng et al., 2019; Benny et al., 2021; Hu et al., 2021), disentangled representations (Van Steenkiste et al., 2019; Wu et al., 2020), and object- centric representations (Mondal et al., 2023; Webb et al., 2024), have been introduced as cues for abstract reasoning. Some approaches solve RPMs based on neuro-symbolic architectures (Hersche et al., 2023). On the other hand, some approaches focus on solving generative AVR tasks (Pekar et al., 2020; Shi et al., 2021: Zhang et al., 2021a;b; Shi et al., 2023; 2024). They directly complete problem panels from the given contexts and select answers by comparing the generated results with candidates. Inspired by language- prompted LLMs (Brown et al., 2020), visual in-context learners have been proposed to solve analogical reasoning problems in visual data (Bar et al., 2022: Dedhia et al., 2023; Bai et al., 2023; Zhao et al., 2023). Compared with the previous solvers, UCGS focuses on solving AVR tasks in a unified conditional generation framework.
Conditional generative models can discover the latent struc- ture of observations, capture complex conditional dependen- cies between latent factors, and generate outputs conditioned on specific input variables context (Mirza, 2014; Sohn et al., 2015; Chen et al., 2016). This ability is beneficial for AVR tasks that require understanding the abstract rules underlying the problem panels, where the latent structures can represent the abstract rules like shape transformations and spatial arrangements. Conditional generative models offer a powerful tool for solving AVR problems. Previous works have shown that both generative and selective AVR tasks can be tackled via conditional generation (Pekar et al., 2020; Shi et al., 2021; 2024). The solvers can generate the missing part of the panel and then compare it with the can- didates to select the correct answer of an RPM test, which sheds light on unifying selective and generative AVR tasks.
Deep Conditional Generative Models. Deep generative models learn distributions of entire datasets, then they can generate novel samples that do not appear in the training stage (Kingma & Welling, 2013; Goodfellow al., 2014; Du & Mordatch, 2019; Rombach et al., 2022). Deep condi- tional generative models (DCGMs) estimate the conditional probability distribution of the output given certain input (Mirza, 2014; Sohn et al., 2015; Chen et al., 2016). Con- ditional Generative Adversarial Network (CGAN) (Mirza, 2014) is a classical type of DCGM, where a generator pro- duces data conditioned on the input, and a discriminator dis- tinguishes between the real and generated data. Conditional Variational Autoencoder (CVAE) (Sohn et al., 2015) intro- duces conditional inputs to conventional VAE (Kingma & Welling, 2013), where conditional inputs are encoded with input data into a latent space, and the latent representation is decoded to generate new samples in terms of the conditional imputs. Conditional generative models reveal a deep under- standing of complex structures behind conditional inputs and data. Some DCGMs discover the underlying structure behind complex data by predicting the missing part of the data from the given context. Masked autoencoder (MAE) (He et al., 2022) is given an input where some portions are masked, and the task is to generate the missing pieces. The NP family (Garnelo et al., 2018; Kim et al., 2019; Dutordoir et al., 2023) can simulate a continuous function based on given points. In this paper, UCGS solves different AVR tasks with a unified conditional generative model, which extends the application of DCGMs in learning underlying structures from data.
We propose a novel framework, Unified Conditional Gener- ative Solver (UCGS), to address multiple AVR tasks within unified framework. UCGS frames various AVR tasks as a conditional generation process, which does not need repeated model training or task-specific hyperparameters when solving different AVR tasks. We prove that some classical AVR tasks, including RPM (Zhang et al., 2019; Barrett et al., 2018), Visual Analogy Problem (VAP) (Hill al., 2019), Odd-One-Oul (O3) (Mańdziuk & Zyychowsk,, 2019) and Synthetic Visual Reasoning Task (SVRT) (Fleuret et al., 201 1), can be solved by estimating the probability of generating the target images conditioned on the remaining images within a problem panel. We design a conditional generative network to instantiate the proposed framework, which can infer visual concepts from image patches and conduct abstract visual reasoning in terms of the concepts. The experiments demonstrate that UCGS exhibits not only the abstract reasoning ability on in-distribution tasks, but also the zero-shot reasoning ability to handle unseen tasks during the test phase.
2. Related Work
Abstract Visual Reasoning. Abstract visual reasoning (AVR) tasks are taken to measure the ability of abstract rule learning and problem-solving through analogical reasoning. Some early models (Lovett et al., 2010; Little et al., 2012) relied on artificially designed features to perform abstract visual reasoning. In recent years, a large number of models based on deep features have emerged (Barrett et al., 2018: Wang et al., 2020; Yun et al., 2020; Zhuo & Kankanhalli, 2021; Depeweg et al., 2024). Different abstract features,
3. Methods
In this section, we illustrate how to solve commonly used AVR tasks through conditional generation and leverage
2

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
(a) Raven's 's Progressive Matrix (RPM)
(b) Visual Analogy Problem (VAP)
(e) Odd-One- (03) Problem
(d) Synthetic Visual Reasoning
Task (SVRT)
o
A
1
7
1
?
10
Left Problem Panel 1
Right Problem Ponel I'
Problem Panel
B
Query Panel 1
Problem Pane
Condidates 1*
Condidatees
Problem Ponel PP
Figure An illustration abstract visual reasoning (AVR) tasks. This paper involves four AVR tasks, despite their differences in the predefined visual concepts (e.g., object shape and color) and abstract rules (e.g., progressive change and logical rule), all assess the ability to infer abstract rules from visual stimuli. (a) Raven's 's Progressive Matrices (RPMs) are visual puzzles where participants are given problem panel with one missing piece. The task is to select the correct image (e.g., the 6th candidate image in the example) from the candidate panel that completes the problem panel. (b) Visual Analogy Problems (VAPs) are similar to RPMs. The participant must choose candidate images that complete the analogy and fit the missing piece problem panels. (c) an Odd-One-Our (O3) problem, solvers are given a problem panel find the odd image that breaks the abstract rule (e.g., the 2nd image the example panel). (d) A problem of Synthetic Visual Reasoning Task (SVRT) consists of two problem panels following different abstract rules. The task deduce the rule that differentiates the problem panels, and solvers required categorize the query image into the left right panel.
unified conditional generative network for abstract visual reasoning. We then introduce an instance of UCGS, where the conditional generative network can generate missing panels from context across various tasks.
candidate panels (Zhang et al., 2019). Let Pi be the panel created by replacing the i-th image of the panel 1 to æ. Using Definitions 3.1 and 3.2, UCGS can solve the following selective tasks by estimating the predictability.
Proposition 3.3. Given the problem panel I and candidate panel I' of RPM test VAP, where N 11%, the correct answer xx argmax æEI ((x)((..))
3.1. Unified Conditional Generative Solver
As shown in Figure 1, the AVR tasks define different visual concepts and the abstract rules the concepts, which test the understanding of abstract rules in different ways of ob- taining answers. We will introduce how to solve different AVR tasks in unified way as follows.
Proof. The context panel I" y is created by removing the last image from I". The modified panels {I" N- je r'} are created by placing the candidate images to the removed position. The correctness I N
I is
Definition 3.1. I" (rpli I,...,N .N is problem panel where E is the i-th panel image of I". The correctness of I' is the joint probability p(r"). 1" is rule-compliant if p( PP) is large, and rule-violating if p(1²) + 0.
p (( p(x)
p(=
(I)
The correct answer xt make up the modified panel with the largest correctness, given by argmax øe p(x|1? N)) O
By modeling the correctness of a problem panel joint probability over panel images, we can define the conditional probability of predicting specific image in the panel.
Proposition 3.4. Given the panel IP of 03 problem, the index the odd image it argmin 1=1,-.,, PP.)) where N P.
Definition 3.2. For panel IP, the predictability p(æ] p(=: indicates the probability that the i-th panel image is given the context 1h where 1² {1))
Proof. We create context panels { 1, .N} by removing each image from I The correctness of 1",
Estimating and sampling results from the predictability can solve generative RPM problems, since the problems require solvers to generate answers from the context (Pekar et al., 2020). But the predictability cannot directly solve selective RPM problems where the solvers must pick answers from
P(IP) & pl P( = P(P)IE P(
(2)
If I is the odd image, p(I".) will be large since the rule- breaking image is removed, otherwise 1" is still rule-
3

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
violating and p( x) + 0. Hence the index of the odd image is 8" argmin px
3.2.1. IMAGE ENCODER AND DECODER
M
Figure 2b depicts the architecture of the image encoder and decoder. UCGS-T learns patch representations of the in- put image by mapping continuous representations into finite set of discrete codes, which is then decoded back into the original data space (Van Den Oord et al., 2017; Razavi et al., 2019). The image encoder extracts a feature map with M visual features from I using a CNN-based neural network, which are projected to the nearest vectors in fixed- size codebook (e), 62, ..... et.) to obtain M quantized patch representations Z; = [Z Zi ............ Zt,M): ). The image decoder takes the quantized patch representations as imput and reconstructs the image from the discrete rep- resentation space. The input images are compressed into quantized patch representations to learn more abstract and general representations for reasoning. We can estimate the over the high-dimensional images through pl (Z on the patch representations reduce the complexity of reasoning.
Proposition 3.5. Given a left problem panel I', a right problem panel I' and a query panel I of SVRT, where N = [ = I"), the query image x. E I" belongs to the panel r" argmax PeA r p 1P).
Proof. By appending the query image x E I to the end of the problem panels I' and ," we get the extended panels 142 and l+ The correctness of the extended panels is
- p(( p(F+a)= = p(
(3)
If x belongs to I", the extended panel F. remains rule- compliant because the panel rule is not changed. Therefore, P. has the largest correctness. Consider that I' and I' are sampled from the dataset 'D uniformly without overlap, in Equation 3 we have p( [') = p(I")== = 1/|D]. Therefore, æ belongs to the panel I' argmax ( p(x)) I.). IT
predictability ((T))
3.2.2. PATCH ENCODER
The quantized patch representations focus on local regions of the image, potentially corresponding to a specific geomet- ric shape or pattern learned by the codebook. As shown in Figure 2c, the patch encoder computes the relation between the local patch representations to understand the role of each patch in the overall image and extract the image-level visual concepts (e.g., the number of entities in the image). We add M learnable positional embeddings to the patches of each panel image, which is projected to the high-dimensional rep- resentations Z by linear network to encode spatial position information, where Zi, the position-augeen augmented repre- sentations of Zi, for i = 1,...,. N and 1,..., M. 2, are passed through a Transformer decoder capture global dependencies and relationships between the patches. In addition to the patch representations, we introduce class tokens {CLS),.... CLSx as learnable slots to aggregate global information from the patches and obtain the visual concepts S = (S = 1,..., K} where
Propositions 3.3, 3.4, and 3.5 demonstrate that although the answers to AVR problems have different forms, they can be converted into the estimation of predictability on panel images. In this way, answer selection and generation are uni- fied. If the i-th image of an RPM problem panel is missing, we can generate answers by sampling from p(æ: I" ()), as well as select answers by filling in the candidate x: and com- puting the panel correctness p(T)..)) = p(æ= ( (2,))(("r)) (T..). With a unified predictability estimator, UCGS can solve different AVR problems after a round of multi-task train- ing, preventing repeated training or tuning of AVR solvers. The core of UCGS is a shared predictability estimator and training-free judgment functions. The predictability estima- tor is conditional generative network trained different AVR tasks to estimate the predictability of panel images. As shown in Propositions 3.3, 3.4, and 3.5, the judgment functions transform outputs of the predictability estimator to obtain the final results, which are task- specifcc.... In the next sections, we provide an instance of UCGS and introduce the architecture of the conditional generative network in detail.
S.A = TransformerDecoodr((CC)
2,).
(4)
The independent slots capture K visual concepts from the patches. The Transformer decoder updates the slots based on attention mechanisms, where the slots act as queries and the patch representations 2, are keys and values. The final output of the patch encoder is the visual concepts S; that consider relationships across the patches of the entire image.
3.2. Instantiation of UCGS
We instantiate UCGS with a Transformer-based conditional generative network, called UCGS-T. Figure 2a illustrates the architecture of UCGS-T via an RPM problem. The problem panel of RPM contains N = 9 images, where we leave one as the prediction target and the others as the context. We denote the index of the target image as t, and the indices of the context images as a set C. The objective of UCGS-T is to estimate the predictability p(I 18). UCGS-T consists of five modules, which will be introduced as follows.
3.2.3. CONCEPT ENCODER
The concept encoder combines the visual concepts of the context images (i.e., context concepts) Sc = {S.]i E C). understanding the overall layout and abstract rule within the
4

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
(a) Overview
Problom Powel
Concept Encoder
Patch Encoder
Patch Encoder
Patch Encoder
Patch Decoder
Legend
Image Encoder
Image Encoder
Image Encoder
Image Decoder
Posisonal Enibediings
Corioe Slots
Torget
(b) Image Encoder and Decoder
(c) Patch Encoder
(d) Concept Encoder
(e) Patch Decoder
Image Encoder
Transformer Decoder
Transformer Doceder
Prediction Head
Codebook
Quantization
Autorogressive Decoder
Linear Projection
Concept-hasee Grouping
++
+
Image
Decoder
Dio.
Figure 2: An Overview of UCGS-T. The image encoder extracts high-level features from each context panel image, which are mapped into discrete context patch representations via vector quantization. The patch encoder captures image-level visual concepts from context patches that encode local information. The panel encoder integrates visual concepts of the context images, understanding abstract rules on the panel, and predicting visual concepts for the target image. The patch decoder generates patch representations of the target image from the target visual concepts predicted by the panel encoder. The image decoder reconstructs the target image from the target patches.
panel, and predicting the visual concepts of the target image (ie., target concepts) S, Figure 2d displays the architecture of the concept encoder. First, the input context concepts Se are reorganized into K groups [G1. Gk} in terms of the concepts, where the k-th group G1. ( C). We group the context concepts since the abstract rules typically global and shared among different visual con- cepts (Wu et al., 2020; Shi et al., 2024), e.g., the tuples (Triangle, Square, Pentagon) and (Small, Middle, Large) point to the same abstract rule Increase. To encode the position of concepts within group Gx, learnable po- sitional embeddings are added to the visual concepts in Gk. followed by linear projection produce the position- augmented representations GG. Each group is processed independently by shared Transformer decoder to capture concept-specific abstract rules and predict the target visual concepts S, = {St, = 1,...,K] K} where
3.2.4. PATCH DECODER
The patch decoder in Figure 2e generates target patches from the target visual concepts S, in an autoregressive manner (Yan et al., 2021). The predictability is factorized by
M
II
p pZ Ze Zc)
(6)
where a learnable token Zr,o is used to indicate the begin- ning of the autoregressive decoding process. The process generates the patch sequence one step at time, where the m-th step depends on the previously generated patches Ze,cm and the target concepts S, computed from Zo. The m-th step p( (Z Zt.cm= Zc) is parameterized by Trans- former decoder with prediction head:
Zi,cm <m
PostionalEmbeodding (Ze,m))
S Transformerreecodr((PP,,,, G2).
(5)
H TransformerDecoode [2,cm, St
PE, is a learnable target positional embedding. The output S, is used to reconstruct the patches of the target image.
Softmax (Linear m)))
Z t. [[cm], where Cm ~ Categorical (TT.).
5

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
The output of the Transformer decoder is passed through a prediction head, which maps the hidden representations into the probabilities over the discrete codes in the codebook. The final prediction of the target patch I.. is given by querying the Gm Gm-th vector el [ccm] in the codebook.
datasets are designed evaluate the ability to reason on unseen AVR tasks that share in-distribution visual concepts and abstract rules with the training data. See Appendix A for more details on the dataset construction.
Metric. We compute the selection accuracy, a standard met- ric that evaluates model performance on AVR tasks (Barrett et al., 2018). This metric is applicable to both selective and generative solvers. For selective solvers, which predict the index of answers, selection accuracy can be calculated straightforwardly. Although candidate panels guarantee a unique correct answer, many AVR problems admit multiple valid solutions, making it challenging to enumerate all possi- ble correct answers when evaluating generative solvers (Shi et al., 2024). We consider the output of a generative solver to be correct if the closest match to the ground truth answer among the candidate panels. Accordingly, both se- lective and generative solvers are evaluated using selection accuracy in our experiments.
3.2.5. MODEL TRAINING
The total loss of the model consists of an image reconstruc- tion loss Lreson and patch prediction loss ppedd We adopt the training objective of VQVAE (Van Den Oord et al., 2017) as the image reconstruction loss to ensure that the in- put image can be correctly reconstructed from its quantized discrete representations. freson is defined as the distance between the input images and the reconstructions with a commitment loss to ensure the continuous representations are not too far from the codebook vectors. The patch pre- diction loss evaluates the difference between the predicted target patches (discrete codes) and the target patches en- coded from the image encoder. Cpred is defined as
Ablation Baselines. We employ three backbone architec- tures and two conditional generative solvers to construct a set of ablation baselines. One of the backbones is the patch- based backbone (Patch) used in UCGS-T. Given that recent tusk-specific solvers (Mondal et al., 2023; Webb et al., 2024) have demonstrated strong performance using object-centric representations, we introduce an object-centrie backbone (OCL) (Locatello et al., 2020; Yuan et al., 2023), which extracts slot representations to capture individual objects in panel images. We also incorporate monolithic backbone (Mono) that encodes each panel image as a single represen- tation. We introduce two baseline conditional generative solvers for predictability estimation. The Transformer-bseed solver (TF) maps context images to target images using a Transformer encoder-decoder architecture. The ANP-based solver (ANP) (Kim et al., 2019) models the distribution over image panels using stochastic functions where context images are used to infer the functions, and target images are sampled from fixed locations of the functions. By combin- ing the backbones and conditional generative solvers, we construct six ablation baselines. Further details of UCGS-T and these baselines are provided in Appendix B. UCGS-T and the ablation baselines are trained under a multi-task setting using all seven configurations of RAVEN and the neutral configuration of PGM. The models are evaluated on VAP, Gl-set, and SVRT without retraining or fine-tuning to assess zero-shot reasoning ability. We conduct qualitative experiments by visualizing the target images generated by the models to illustrate and compare the answer-generation capabilities of UCGS-T and the ablation baselines.
M
Cpred
Σ m=
log Ze.m Ze. Ze).
(8)
By combining the image reconstruction loss and patch pre- diction loss, the total loss is given as
Lixall
pred
Frecom
where λ is hyperparameter that balances the losses. The image encoder and decoder are pretrained before training the remaining modules. We can set À > 0 to finetune the image encoder and decoder in the following training stage. But we find that freezing their parameters is the best choice.
4. Experiments
In this section, we compare UCGS-T with ablation base- lines, task-specific solvers and Multimodal LLMs (MILMs) on four classical AVR tasks shown in Figure 1. We also con- duct qualitative experiments to illustrate the performance of UCGS-T in answer generation tasks by visualizing the generation results of RAVEN and PGM.
Datasets. The models are evaluated on RAVEN (Zhang et al., 2019) and PGM (Barrett et al., 2018) to assess their reasoning abilities on RPM tasks. The ability to reason over 03 problems is tested using the Gl-set dataset (Mańdziuk & Zychowski, 2019). Performance on the remaining AVR tasks is evaluated using the VAP (Hill et al., 2019) and SVRT (Fleuret et al., 2011) datasets. RAVEN and PGM are used for both training and testing, while G1-set, VAP, and SVRT are used only during testing to assess the zero- shot reasoning capabilities of the models. In addition to the public datasets, we construct three new datasets, i.e., 03-ID, VAP-ID, and SVRT-ID, based on RAVEN. These
Task-specific Solvers and MLLMs. Since UCGS-T is de- signed to handle both selective and generative tasks, we compare it against several task-specific solvers that sup- port both answer selection and generation: PrAE (Zhang
6

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Table I: Selection accuracy (%) on AVR tasks. ID tasks have only in-distribution visual concepts and abstract rules, while OOD tasks contain cittof-distributiinn visual concepts and abstract rules. The problems of zero-shot tasks only appear in the testing stage. The models marked with * are evaluated on subsets of the datasets.
ID TASKS
ID AND ZERO-SHOT TASKS
OOD AND ZERO-SHOT TASKS
MODEL
RAVEN
PGM
03-1D
VAP-ID
SVRT-ID
GI-SET
VAP
SVRT
ANP MONO
6.0
14.1
11.9
5.8
51.9
32.9
30.7
48.4
ANP OCL
8.3
12.3
13.3
8.5
49.4
29.3
30.4
49.8
ANP PATCH
10.5
12.3
11.7
10.9
49.8
30.6
30.9
51.0
TF + MONO
11.2
14.1
13.6
6.5
56.4
32.7
30.2
50.8
TF + OCL
7.4
12.3
12.7
7.1
49.6
33.7
30.6
50.0
TF PATCH
21.0
15.6
35.8
15.5
71.8
30.5
27.1
51.4
UCGS-T
64.6
38.1
33.6
35.8
84.6
30.4
28.8
52.8
GPT-40"
12.6
21.4
30.4
12.6
40.8
52.9
29.3
64.2
UCGS-T"
60.9
37.8
32.7
34.0
85.9
30.4
30.1
52.8
RANDOM GUESS
12.5
12.5
11.1
12.5
50.0
22.5
25.0
50.0
et al., 2021a), NVSA (Hersche al., 2023), GCA (Pekar et al., 2020), ALANS (Zhang et al., 2021b), and RAISE (Shi et al., 2024). To ensure consistency with the experimen- tal setup used for UCGS-T and the ablation baselines, all spec-fpe solvers are trained and tested on each dataset separately, without supervision from rule or attribute an- notations. In addition, include GPT-4o (Achiam al., 2023), a powerful general-purpose multimodal language model (MLLM), as reference point for comparison. Task- specific prompts for GPT-4o are designed based on the prior benchmark study (Cao et al., 2024).
generated using the same sets of abstract rules and visual concepts, the rules and concepts are composed differently between the splits. The purpose of the ID tasks is to evaluate the ability to reason over the abstract rules and visual con- cepts that have been learned during training. As shown in Table 1, Transformer- based baselines outperform their ANP- based counterparts, with TF + Patch achieving the strongest results among them. UCGS-T further achieves 64.6% curacy RAVEN and 38.1 1% on PGM, demonstrating the abstract reasoning capabilities on ID tasks. In contrast, ANP-based baselines perform poorly, with accuracies close to random guessing. This suggests that ANP-based models struggle with compositional reasoning in AVR tasks such as RAVEN and PGM.
4.1. Performance on AVR Tasks
The experiments evaluate model performance across four AVR tasks. We categorize the evaluation settings into the following three groups. In Di-Distribution (ID) Tasks include problems that are seen during training, whose abstract rules and visual concepts are also encountered during training. In in-Distribution Zero-Shot (ID-ZS) Tasky training, but their abstract rules and visual concepts are present in the training data. Out-of-Diss Distributin Zero-Shot (OOD-ZS) Tasks are the most challenging tasks, which are entirely unseen during training and involve novel abstract rules or visual concepts. We compare UCGS-T with both ANP-based ablation baselines (ANP+ Mono, ANP OCL, ANP Patch) and Trransformrrbbssed baselines (TF Mono, TF + OCL, TF Patch), well as GPT-4o. For reference, we also include a random guess baseline.
4.1.2. IN-DISTRIBUTION ZERO-SHOT TASKS
We introduce ID-ZS tasks to evaluate the ability of models to generalize their reasoning capabilities to novel tasks. In this setting, all models are trained on RPM tasks and evalu- ated on other types of AVR tasks. They are required to solve problems from zero-shot tasks that contain in in-distribution abstract rules and visual concepts. We construct three ID- ZS datasets, Le., O3-ID, VAP-ID, and SVRT-ID, which fol- low the forms of O3, VAP, and SVRT, respectively. These datasets are built on the abstract rules and visual concepts of RAVEN. Across all ID-ZS tasks, models exhibit a noticeable drop in performance compared to ID tasks, highlighting the difficulty of transferring learned reasoning abilities to new task formats. Transformer-base baselines continue out- perform their ANP-based counterparts. UCGS-T achieves the best results overall, with accuracies of 33.6% O3-ID, 35.8% on VAP-ID, and 84.6% on SVRT-ID. These results suggest that UCGS-T is more effective at generalizing rea-
used during
4.1.1. IN-DISTRIBUTION TASKS
UCGS-T and the baselines are trained on RAVEN and PGM, whose test splits are treated as in-distribution (ID) tasks. While the training and testing samples of RAVEN and PGM

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
soning abilities from the training data novel tasks. For example, UCGS-T can perform abstract reasoning VAPs with six-image panels, despite having been trained solely on RPMs with nine-image panels.
Table 2: Selection accuracy ( (%) of UCGS-T and the task- specifie solvers on RAVEN and PGM. The task-specific solvers are trained and tested separately on each dataset. They are trained without additional annotations to keep consistent with the experimental setup of UCGS-T and base- lines. PrAE and ALANS only define the architecture to solve RAVEN. The codebase of NVSA is not applicable to PGM, and the reported accuracy on PGM is obtained by using additional annotation information. We only report their performance on RAVEN here.
4.1.3,
ZERO-SHOT TASKS
OOD-ZS tasks evaluate models on novel abstract rules and visual concepts that are entirely unseen during training. with test problems presented in forms that differ from those used in the training phase. We use the Gl-set, VAP, and SVRT datasets to assess the distif-distributtion and zero-shot rea- soning capabilities of models. As shown in Table 1, both UCGS-T and the baselines experience a substantial drop in performance compared to the ID tasks. On Gl-set and VAP, UCGS-T T and the baselines achieve accuracies around 30%, which is 5-8% higher than random guessing. This indi- cates that the UCGS framework enables a certain degree of generalization in abstract reasoning. UCGS-T slightly out- performs the other baselines on SVRT, task that requires holistic pattern recognition based on spatial relationships among randomly generated abstract shapes. This require- ment is similar to image-level visual concept recognition in datasets such as PGM and RAVEN (e.g., recognizing the number of objects in scene). The results SVRT suggest that UCGS-T may benefit from its panel encoder, which is specifically designed to extract high-level image abstractions. Furthermore, T achieves a large accu- racy improvement over the baselines in ID tasks, while the performance gains are relatively smaller in OOD tasks. We suppose that the unseen visual concepts and abstract rules in OOD tasks lead to significant drop in accuracy for both UCGS-T and the baselines, thereby narrowing the perfor- mance gap that arises from different model architectures.
MODEL
RAVEN
PGM
PRAB (ZHANG ET AL., A021A)
13.6
NVSA (HERSCHE ET AL., 2023)
11.5
GCA (PEKAR ET AL., 2020)
37.3
31.7
ALANS (ZHANG ET AL., 2021B)
50.1
RAISE (SHI ET AL., 2024)
54.5
14.0
UCGS-T
64.6
38.1
RANDOM GUESS
12.5
12.5
incorporate prior knowledge in the prompts, such as stat- ing that an RPM problem is a 3 x 3 matrix, which is not provided to UCGS-T during either training or testing. Prior work (Cao et al., 2024) has shown that GPT-4o achieves notable performance boost (approximately 20%) when provided with language descriptions of candidate images. Although GPT-4o is powerful general- purpose model, it is not specifically designed for AVR tasks. Its performance on RPM tasks may benefit from more advanced prompt engineering and thain-of-thougle design. Appendix B. provides the prompts used in this experiment.
4.1.5. COMPARISON WITH TASK-SPECIFIC SOLVERS
4.1.4. COMPARISON WITH GPT-40
This experiment compares UCGS-T with classic generative task-specific RPM solvers. Table 2 shows the experimental results where UCGS-T outperforms the task-specific solvers under the same experimental setup. The experimental re- sults reveal that, without additional annotation information, the performance of generative task-specific solvers drops significantly. For example, PrAF achieves an accuracy of 65.0% with rule supervision during training (Zhang et al., 2021a), but its accuracy drops to 13.6% in our experiment. The reasoning process of GCA is specifically designed for predicting the bottom-right image in an RPM matrix, which limits its ability to generalize to other AVR tasks. Addi- tionally, models like PrAE, ALANS, and NVSA rely on predefined rules and explicit representations tailored spe- cific datasets, making it difficult for them to handle unseen or undefined visual concepts and abstract rules. UCGS-T does not rely on manually designed concepts or rules. It performs an independent reasoning process over each visual concept, inferring both the underlying visual concepts and abstract rules without the need for additional annotations.
Table 1 also presents the results of the tem GPT-4o. We randomly sampled subsets from the test sets of RAVEN, PGM, and VAP to evaluate both GPT-4o and UCGS-T. The results show that UCGS-T outperforms GPT-4o significantly ID and ID-ZS tasks, while GPT-4o achieves better performance on OOD-ZS tasks. This may be attributed to GPT-4o's ability to recognize unseen visual con- cepts or abstract rules. However, this does not necessarily imply that GPT-4o possesses stronger zero-shot reasoning ability, as difficult to determine whether similar visual concepts or rules were encountered during its training. In contrast, UCGS-T is trained on clearly defined and con- trolled set of samples, making its evaluation setting more transparent. Notably, GPT-4o shows weaker generalization on ID-ZS tasks, suggesting that UCGS-T is more effective at transferring its reasoning capabilities to novel task formats. In this experiment, GPT--4o provided with ssec-fpecfiic prompts, following the setup in (Cao et al., 2024). We also
sys-
8

---

Beyond Task-Specifie Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Sample
UCGS-T
Sample
UCGS-T
V
TP-Patch
TF-OCL
TF-MoRo
TF-Parch
TP-OCI,
TF-MoRu
0
0
000
ANP-PPotck
ANP-OCL
ANP-Mono
ANP-Patch
ANP-OC
ANP-Mono
RAVEN
(h) PGM
Figure 3: Comparison of generation results on RAVEN (left) and PGM (right). In the visualization result of each model, the bottom-right position the prediction and the remaining context images are ground truths. The figure illustrates qualitative differences between models, with errors and artifacts appearing in some results predicted by baselines. Note that there noise in the problem panel, therefore the generated result of UCGS-T PGM is correct though it is little different from the ground truth.
4.2. Answer Generation on RAVEN and PGM
5. Conclusion and Limitations
Figure visualizes the generated answers produced by UCGS-T and the baselines. UCGS-T can generate rule- compliant answers from the context RAVEN and PGM. While TF-Patch is capable of generating clear and reason- able outputs on the PGM example, it fails to produce the correct answer for the RAVEN problem. Other baselines often generate images containing incorrect visual concepts. For instance, the solutions predicted by TF-Mono and ANP- Mono on RAVEN display scenes with four objects, whereas the correct using the object-centric backbone generally struggle to gen- erate high-quality outputs where the mismatch between pre- dicted target slots and the true scene structure significantly degrades the pixel reconstruction quality. We observe that UCGS-T and TF-Patch are better at capturing diversity in answer generation. Since RAVEN and PGM introduce degree randomness noise data generation, UCGS- T's outputs may differ slightly from the ground truth but still conform to the task rules. For example, in Figure 3a, the answer should contain three objects, but their positions within the panel may vary without affecting correctness.
We propose UCGS, a unified AVR framework to solve multi- ple AVR tasks using a single conditional generative network. We instantiate UCGS with model that infers visual con- cepts from image patches to perform reasoning on the prob- lem panel. Experimental results demonstrate that UCGS not only exhibits abstract reasoning ability ID tasks but also zero-shot generalization, enabling it to solve unseen tasks during testing, even when those tasks involve OOD visual concepts and abstract rules.
should contain three objects. Models
Limitations. While UCGS-T outperforms the generative task-specific RPM solvers, its accuracy is still lower than state-of-the the-thh-art selective solvers. Reducing the performance gap remains important direction for future work. We believe this paper provides a unified perspective for solv- ing AVR tasks. This work does not address AVR tasks in- volving real-world scenes. Real-world problems introduce more complex visual concepts and abstract rules, which may require more powerful image tokenizers and decoders. Effectively discovering and utilizing such rules in realistic settings remains challenge for future investigation.

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Impact Statement
learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
This paper presents work whose goal to construct uni- fied abstract visual reasoning solvers. We feel that there are no potential societal impacts that must be specifically highlighted and discussed here.
Chollet, F. On the measure of intelligence. arXiv preprint arX1v:1911.01547, 2019.
Dedhia, B., Chang, M., Snell, J. C., Griffiths, T. L., and Jha,
Acknowledgements
N. Im-promptu: In-context composition from image prompts. arXiv preprint arXiv:2305.1726. 2023.
This work was supported by the National Natural Sci- ence Foundation of China (No.62176060) and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
Depeweg, S., Rothkopf, C. A., and Jäkel, F. Solving bon- gard problems with visual language and pragmatic con- straints. Cognitive Science, e8(5))e1344) 2024.
Du, Y. and Mordatch, I. Implicit generation and modeling with energy based models. Advances in Neural Informa- tion Processing Systems, 32, 2019.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Dutordoir, V., Saul, A., Ghahramani, Z., and Simpson, F. Neural diffuusion processes. In International Conference on Machine Learming, pp. 8990-9012. PMLR, 2023.
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Bai, Y., Geng, X., Mangalam, K., Bar, A., Yuille, A., Darrell, T., Malik, J., and Efros, A. A. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.00785, 2023.
Garnelo, M.. Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural pro- cesses. arXiv preprint
2018.
Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., and Efros, A. Visual prompting via image inpainting. Ad- vances in Neural Information Processing Systems, 35: 25005-25017, 2022.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Barrett, D., Hill, F., Santoro, A., Morcos, A., and Lillicrap, T. Measuring abstract reasoning in neural networks. International conference on machine learning, pp. 511- 520. PMLR, 2018
Gray, J. R. and Thompson, P. M. Neurobiology of intelli- gence: science and ethics. Nature Reviews Neuroscience, 5(6):771-482, 2004.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Benny, Y., Pekar, N., and Wolf, L. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12557- 12565, 2021.
Hersche, M., Zeqiri, M., Benini, L., Sebastian, A., and Rahimi, A. A neuro-vector-symbolic architecture for solving raven's 's progressive matrices. Nature Machine Intelligence, 5(4):363-375. 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Hill. F., Santoro, A., Barrett, D. G., Morcos, A. S.. and Lillicrap, T. Learning to make analogies by con- trasting abstract relational structure. arXiv preprint arXiv: 0eee:900000010 20, 2019.
Cao, X., Lai, B., Ye, W. Ma, Y., Heintz, J., Chen, J., Cao, J., and Rehg, J. M. What is the visual cognition gap between humans and multimodal llms? arXiv preprint arXiv:2406.10444 2024.
Hu, S., Ma, Y., Liu, X., Wei, Y., and Bai, S. Stratified rule- aware network for abstract visual reasoning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1567-1574, 2021.
Cattell, R. B. Theory of fluid and crystallized intelligence: critical experiment. Journal of educational psychology, 54(1):1 1963.
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W. Attentive neural processes. arXiv preprint arXiv:1901.05: 2019.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I,, and Abbeel, P. Infogan: Interpretable representation
10

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint /3 20133 2013.
diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Little, D. R.,. Lewandowsky, S., and Griffiths, L. A bayesian model of rule induction in raven's progressive matrices. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 34, 2012.
Shi, F., Li, B., and Xue, X. Raven's progressive matri- ces completion with latent gaussian process priors. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 9612-9620, 2021.
Locatello, F., Weissenborn, D.., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525- 11538, 2020.
Shi, F., Li, B., and Xue, X. Compositional law parsing with latent random functions. In The Eleventh International Conference on Learning Representations, 2023.
Shi, F., Li, B., and Xue, X. Towards generative abstract rea- soning: Completing raven's wwen's progressive matrix via rule abstraction and selection. In The Twelfth International Conference on Learning Representations, 2024.
Lovett, A., Forbus, K., and Usher, A structure-mapping model of raven's progressive matrices. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2010.
Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 28, 2015.
Małkiński, M. and Mańdziuk, J. Deep learning methods for abstract visual reasoning: survey on raven's 's progressive matrices. arXiv preprint arXiv:2201.12388 2022a.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Malkiñski, M. and Mañdziuk, J. A review of emerging research directions in abstract visual reasoning. arXiv preprint arXiv:2202. 2022b.
Van Steenkiste, S. Locatello, F., Schmidhuber, J., and Bachem, O. Are disentangled representations helpful for abstract visual reasoning? Advances in Neural Infor- mation Processing Systems, 32, 2019.
McCloskey, M. Intuitive physics. Scientific american, 248 (4):122-131, 1983.
McMillen, J. The colossal book of short puzzles and prob- lems, 2007.
Wang. D.,, Jamnik, M., and Lio, P. Abstract diagrammatic reasoning with multiplex graph networks. arXiv preprint arXiv:2006.1 1977, 2020.
Mirza, M. Conditional generative adversarial nets. arXiv preprint IrXiv:I411.1780 2014.
Webb, T., Mondal, S. S., and Cohen, J. D. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36, 2024.
Mondal, S. S., Webb, T., and Cohen, J. D. Learning to reason visual objects. arXiv preprint 2xXiv:2303.02260 2023.
Wu, Y., Dong, H., Grosse, R., and Ba, The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint 1iXiv:2007.042122 2020.
Nie, W., Yu, Z., Mao, L., Patel, A. B., Zhu, Y., and Anand- kumar, A. Bongard-logo: A new benchmark for human- level concept learning and reasoning. Advances Neural Information Processing Systems, 33:16468-16480, 2020.
Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10155 2021.
Pekar, N., Benny, Y., and Wolf, L. Generating correct answers for progressive matrices intelligence tests. arXiv preprint arXiv:2011.004
2020.
Yuan, J., Chen, T., Li, B., and Xue, X. Compositional scene representation learning via reconstruction: A sur- vey. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(10): 1540-111140,, 2023.
Raven, C. and Court, J. Raven's progressive matrices. Western Psychological Services Los Angeles, CA, 1938.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Yun, X., Bohn, T., and Ling, C. A deeper look at bongard problems. In Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Cana- dian Al 2020, Onawa, ON, Canada, May 13-15, 2020. Proceedings 33, pp. 528-539. Springer, 2020.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Zhang, C.,, Jia, B., Zhu, S.-C., and Zhu, Y. Abstract spatial- temporal reasoning via probabilistic abduction and ecution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9736- 9746, 2021a.
Zhang, C., Xie, S., Jia, B., Wu, Y. N., Zhu, S.-C., and Zhu, Y. Learning algebraic representation for system- atic generalization in abstract reasoning. arXiv preprint arXiv:21 2021b.
Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W., and Chang, B. Mmicl: Empower- ing vision-language model with multi-modal in-context learning, arXiv preprint MXiv:2309.079155 2023.
Zheng, K., Zha, Z.-.,, and Wei, W. Abstract reasoning with distracting features. Advances in Neural Information Processing Systems, 32, 2019.
Zhuo, T. and Kankanhalli, M. Effective abstract reasoning with dual- network. In International Conference Learning Representations, 2021.
12

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
A. Datasets
In this section, we will introduce the datasets used for model training and testing. Tables 3 and 4 introduce the training, validation, and test splits of the datasets.
Table 3: The ID tasks used to train and evaluate the baselines, task-specific solvers and UCGS-T. #Samples = number of samples per dataset, #Images = number of images per problem, #Candidates = number of candidate per problem.
Dataset
RAVEN
PGM
Split
Train
Valid
Test
Train
Valid
Test
#Samples
42K
14K
14K
1.4M
5K
200K
#lmages
#Candidates
8
8
Image Size
128x 128
128x
Table 4: The ID-ZS and OOD-ZS tasks used to evaluate the baselines and UCGS-T. #Samples number of samples per dataset, #lmages number of images per problem, #Candidates number of candidate per problem.
Dataset
Gl-set
VAP
SVRT
O3-ID
VAP-ID
SVRT-ID
Split
Test
Test
Test
Test
Test
Test
#Samples
1K
200K
253
14K
14K
14K
#Images
4~5
6
8
5
6
8
#Candidates
NA
4
2
NA
8
2
Image Size
128x 128
128x 128
128x 128
128x 128
128x 128
128 128
Table 5: The subsets used to evaluate GPT-4o and UCGS-T. The datasets noted with are subsets sampled from corresponding complete datasets. #Samples number of samples per dataset, #Images number of images per problem, #Candidates number of candidate per problem.
Dataset
RAVEN*
PGM*
G1-set
VAP*
SVRT
O3-ID
VAP-ID
SVRT-ID
Split
Test
Test
Test
Test
Test
Test
Test
Test
#Samples
700
IK
1K
IK
253
700
700
700
#Images
9
9
4-55
6
8
5
6
8
#Candidates
8
8
NA
4
2
NA
8
2
Image Size
128 x 128
128x 128
128x
128
128x 128
128x 128
128 128
128 128
128 x 128
A.1. ID Tasks
RAVEN (Zhang et al., 2019). The RAVEN dataset is inspired by the widely used RPM tests. RAVEN provides structured and systematically generated
problems with clearly defined rules and visual concepts. Each RAVEN problem consists of nine-image matrix (3 x grid) where the bottom-right image is removed. The task is infer the missing image from set of candidate choices based on the underlying abstract rules. RAVEN contains seven configurations, each containing 10K problems with specific scene structure. The configurations incorporate five concepts that define problem variations: Shape (e.g., triangle, square, pentagon, etc.), Size (relative scale of objects), Color (e.g., white, gray, black, etc.) and Position/Number (spatial arrangement and number of objects within each image). Each problem follows one or more concept-specific rules that govern changes in these visual concepts: Constant (the concept remains unchanged). Progression
13

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
(the concept changes sequentially), Arithmetic (the concept follows mathematical relationship) and Distribution-of-TTie (the concept has three permutable values). We use problems from all seven configurations of RAVEN to train and test the models.
PGM (Barrett et al.. 2018). The PGM dataset is a large-scale RPM-like dataset. The problem structure of PGM problems similar to RAVEN. PGM problem also consists of a nine-image matrix, with the bottom-right image removed. The models infer the missing image based on the abstract rules of the given context and choose the correct answer from eight candidate images. The dataset contains 1.4M unique problems. The visual concepts are shape and line the color, number, position, size and type. The visual concepts follow systematically defined rules, which have five categories; progression. XOR, OR, AND, and consistent union. The PGM dataset provides eight generalization regimes, and the neutral regime for model training and evaluation.
A.2. ID-ZS Tasks
(a) Construct O3-ID
RAVEN Problem Panel
Problem Panel
RAVEN Candidate
Replace
(b) Construct VAP-ID
RAVEN Problem Panel
RAVEN Candidate Panel
Candidate Panel
Remove
the first
Construct SVRT-ID
Left Panel
Left Problem Panel
RAVEN Problem Panel
Split
right images
Query Panel
Right Panel
Right Problem Panel
Figure 4: Construction of the ID tasks.
O3-ID. As shown in Figure 4a, replace the bottom-right image of each 3 x 3 image panel with randomly selected distractor from the candidate panel to construct odd-one-out tests from RAVEN. This process transforms the bottom-right image into a rule-breas breaking image by modifying a specific visual concept of the correct image. The constructed O3-ID dataset shares the same visual concepts and abstract rules as RAVEN but follows the form of odd-one-out, where the models must identify the image breaking the rule of the problem panel.
14

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
VAP-ID. The construction of the VAP-ID task is relatively straightforward in Figure 4b. Unlike RAVEN, VAP presents 2 x 3 image matrix its problem panel, and the abstract rules can change different visual concepts in different rows. Here, we construct VAP problems using RAVEN's design of analogy. Specifically, we remove the final row and retain the first two rows (i.e., the first six images) of each RAVEN problem panel to generate problem of VAP-ID. In this way, VAP-ID problems preserve the same in-distribution visual concepts and abstract rules as RAVEN, while adopting a different structure of problem panels. These problems are designed to assess whether models can still parse the learned abstract rules and complete the task when the structure of problem panels changed.
SVRT-ID. In Figure 4c, we split each RAVEN problem panel into two parts to construct SVRT-style problems. The first two rows form the left panel, and the last two rows form the right panel. We then extract the bottom-right image from both panels to form the query panel, while the remaining images are kept as the corresponding problem panels. In this way of construction, the left and right problem panels exhibit different abstract rules of the same type. Therefore, the association between the query images and the problem panels is typically unambiguous.
A.3. OOD-ZS Tasks
G1-set (Mardziuk & Zychowski, 2019). The G1-set dataset evaluates the AVR ability of models through oud-one-out problems. Unlike RPMs, where the task is to infer missing images, an odd-one-out problem requires the models to recognize the oddest figure in a panel based on relational differences in visual concepts. In Gl-set, the odd figure can differ on the visual concepts size, shape, shading, rotation, etc. To ensure the problems are well-defined, the G1-set problems are constructed using a strict rule. In each problem with N figures, there is one subset of N figures that share common set of ~ 3 features. The remaining figure is the odd image that lacks this commonality. The difficulty of problems is controlled by the number of figures and the number of shared features.
VAP (Hill et al., 2019). The VAP dataset evaluates the mmalogy-making ability of models. Inspired by previous human and machine reasoning tasks like RPMs, VAP requires the models to identify abstract rules and apply them across different visual concepts. Each problem panel consists of source sequence and a target sequence. The source sequence has three images where an abstract rule is instantiated. The target sequence has two images where the removed position must be completed according the rule of the source sequence. The candidate panel provides four candidate answer images, including one correct answer and three distractors. The goal is to select the correct answer that best completes the target sequence by analogy with the source sequence. The core challenge in VAP is to apply learned rules across different visual domains. VAP defines four logical or mathematical abstract rules: XOR, OR, AND and progression. The abstract rules can be instantiated in seven different visual concepts: line type, line color, shape type, shape color, shape size, shape quantity and shape position. Each visual concept contains 10 possible values to increase the complexity of problems.
SVRT (Fleuret et al., 2011). The SVRT dataset evaluates the relational visual reasoning ability of models. Unlike traditional image classification tasks that rely on texture, SVRT problems require holistic pattern recognition based on spatial relationships among randomly generated abstract shapes. Each of the 23 problems in the dataset requires assigning query image to one of two panels, where the distinction is defined by an abstract rule governing the global arrangement of parts. Importantly, these rules are not based on the appearance, location, or topology of individual parts but on how multiple parts interact as a whole. The problems in SVRT involve relational concepts rather than low-level features. Some examples of reasoning principles used in the dataset include: proximity (are two or more shapes close to each other), similarity (do multiple shapes share the same size/form), symmetry (are the parts arranged symmetrically). topology (are certain shapes enclosed within others), counting (does the image contain even or odd number of objects) and identity relations (do two objects have the same shape or orientation). For example, one problem might require distinguishing two identical shapes vs. two different images.
B. Details of Models
B.1. UCGS-T
This section describes the architectures of learnable networks in UCGS-T and the choice of hyperparameters. We introduce the networks in the order of image encoder, image decoder. patch encoder, panel encoder and patch decoder.
Image Encoder:
4 x 4 Cony, stride 2, padding 1, 64, ReLU
15

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
4 x 4 Conv, stride 2. padding 64, ReLU
4 x 4 Conv, stride 2, padding 1, 128, ReLU
3 X 3 Conv, stride 1, padding 128
ResBlock, hidden 32, 128
ResBlock, hidden 32, 128, ReLU
4 x Conv, stride 2. padding 1, 64, ReLU
4 X 4 Conv, stride 2, padding 1, 128, ReLU
3 Cony, stride I, padding 128
ResBlock, hidden 32, 128
ResBlock, hidden 32, 128, ReLU
-
Conv, stride 1, 64
Image Decoder:
x 3 Conv, stride I, padding 128, ReLU
ResBlock, hidden 32, 128
ResBlock, hidden 32, 128, ReLU
4 X 4 Deconv, stride 2, padding 1, 64, ReLU
4 X 4 Deconv, stride 2, padding 128
ResBlock, hidden 32, 128
ResBlock, hidden 32, 128, ReLU
4 x 4 Deconv, stride 2, padding 1, 64, ReLU
4 X 4 Deconv, stride 2, padding 1, 64, ReLU
4 x 4 Deconv, stride 2, padding 1,
Patch Encoder: The linear network to encode position information is
LayerNorm, 128
Fully Connected, 128, ReLU
Fully Connected, 128
The patch encoder is 12-layer Transformer decoder where the hidden size is 128 and the number of attention heads is 8.
Panel Encoder: The panel encoder 12-layer Transformer decoder with the hidden size 128 and the number of attention heads 8. We use linear layer to convert the input keys and values hidden vectors. Another linear layer is introduced to convert the input queries to hidden vectors.
Patch Decoder: The panel encoder 12-layer Transformer decoder with the hidden size 128 and the number of attention heads 8. We use linear layer to convert the input keys and values to hidden vectors. Another linear layer is introduced to convert the input queries to hidden vectors. The output of the Transformer decoder is projected to the index of the discrete vector in the codebook e by linear layer without the bias parameter.
The image encoder and image decoder are first trained by setting the learning rate as 4 10 and batch size 64, referring to the original configuration in VQVAE (Van Den Oord et al., 2017). Then we freeze the parameters of the image encoder and image decoder, training the remaining part by setting the learning rate as 3 x 10 batch size as 128. We monitor the performance of UCGS-T on the validation set after each training epoch and save the checkpoint with the best validation accuracy. The parameters are updated by Adam optimizer (Kingma & Ba, 2014).
B.2. Ablation Baselines
Besides the patch-bacc basch backbone of UCGS-T, T, we introduce another two backbones. The object-centric backbone can distinguish objects in scenes and extract representations for individual objects. We adopt the encoder of STSN (Mondal et al., 2023) as the object-centric backbone, and train the backbone using the official code!. The monolithic backbone encodes
hhtpss//gituuh..om/Shaat Sireemnnreemmann
16

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
scene into one representation. The encoder and decoder of the monolithic backbone consist of continuous convolutional layers that map the image into single representation of size 512. We call the slots and representations extracted from the object-centric and monolithic backbones patches for convenience. We introduce two types of baseline conditional generative solvers: the Transformer-basee solver (Vaswani et al., 2017) and the ANP-based solver (Kim et al., 2019). All the parameters are updated by Adam optimizer (Kingma & Ba, 2014).
Transformer-bent Solver (Vaswani et al., 2017). The Transformer- conditional generative network inputs all context patches Z" i .......,. Nee Transformer encoder. The output of the Transformer decoder is the key and query of Transformer decoder, and the embeddings of the target positions {PE,li I..., ,....,Ny. Ne are regarded as the query of the Transformer decoder. The generative process is
h fKv (Zi),
1,.., No:
h kv TFEncoder (14wv)
h = So (PE.),
i= 1, 1,..., No:
(10)
h TFDecoder 129 h kv
z fo (n:))
No.
In the generative process, fKV. Jo. and fo are single-layer feedforward networks. We set the hidden size of the Transformer encoder and decoder as 512, the number of Transformer layers as 12, and the number of attention heads as 8.
ANP-based Solver (Kim et al., 2019). The ANP conditional generative baseline regards the mapping from context patches to target patches as stochastic functions. The context slots are used to construct the context of stochastic functions, and the model will sample a function from the posterior to map the embeddings of target positions into target patches. We set the size of the global latent 512, the number of attention layers as 12, and the number of attention heads as and other hyperparameters and the model architecture follow the 2D regression configuration in (Kim et al., 2019).
B.3. Task-specific Solvers
We use the official codes of PrAB², NVSA GCA4, ALANSS and RAISE" in the experiments. All task-specific solvers trained without additional annotation information. To stabilize the training process, ALANS is trained based on the pretrained checkpoint provided by the authors.
B.4. GPT-40
We design task-specific prompts for GPT-40, which are illustrated in Figure 5. In the prompts, we will provide the information about the problem structures, e.g. the problem panel of PGM is a x 3 matrix and the problem panel of VAP is 2 x 3 matrix. Each panel is regarded as an input image of GPT-4o.
B.5. Computational Resource
All the models are trained on single 24GB NVIDIA GefFree RTX 4090 GPU and implemented using PyTorch (Paszke et al., 2019).
C. Additional Experimental Results
C.1. Answer Generation
We provide additional examples of answer generation. Figures 6 and show the generated results for various AVR problems on the RAVEN and PGM datasets, respectively. Overall, the generated results RAVEN are more accurate, corresponding
hntpss/ggithubbco hnaps://gttub..com//lll/y hot.://bbooooeellyyyan/PP.P.E
"hups://githhh.com// http:/ggghhh..ooooeeeyyy/nngggg hwwpy//gghhhoooiaaaaggrrtteeeeee
17

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
the selection accuracy. primarily because the problems of RAVEN contain less noise, which has been discussed in previous works. UCGS-T achieves the best generation performance both RAVEN and PGM. UCGS-T can handle the uncertainty caused by noise effectively. For example, in the 4th sample of PGM, the shape and position of the generated object differ from the ground truth. TF-Patch generates relatively accurate results on RAVEN, but it tends to produce answers on PGM. Overall, TF-Patch generates fewer artifacts the images. Baselines using the object-centric backbone (TF-OCL and ANP-OCL) struggle to generate clear images, often producing large black regions. Baselines based on the monolithic backbone tend to produce samples that deviate significantly from the ground truths.
violating
References
Barrett, D., Hill, F., Santoro, A., Morcos, A., and Lillicrap, T. Measuring abstract reasoning in neural networks. In International conference on machine learning, pp. 511-520. PMLR, 2018.
Fleuret, F,, Li, T., Dubout, C., Wampler, E. K., Yantis, S., and Geman, D. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, (78(43):76211-7666 2011.
Hill, F., Santoro, A., Barrett, D., Morcos, A., and Lillicrap, T. Learning make analogies by contrasting abstract relational structure. In International Conference on Learning Representations, 2019.
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W. Attentive neural processes. arXiv preprint 1rXiv:190100576. 2019.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint
2014.
Maúdziuk, J. and Zychowski, A. Deepiq: A human-inspired ai system for solving iq test problems. In 2019 International Joint Conference Neural Networks (JCNN), pp. 1-8. IEEE, 2019.
Mondal, S., Webb, T., and Cohen, J. D. Learning to reason over visual objects. arXiv preprint urXiv:2303.022600 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
18

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
(a)
Each tisk conssts of quesiion image with a x mattix Eight the nine resilting odls riglt-land side the
cell
complete manix by flading the misdnn shipe amorg mukipl: posbll alienutisss.
("Resuli": "1")
option images corcest answer The fins inage the quesiie image The sasrd image is the option maye. where the uptinn given frum lell to right. and index the ougur the index cornect Plesse fallowing
GPT-4o
the
(b) VAP
usk
shipe, resilting empty. Your tadk is to complete the matii by findin the mising shipe among mukipl: posell allemaiivss One of the option images the conrect answer. The first image the questicn The acond image the option oupprn
riglt-han
("Resuitt: "1")
GPT-4o
image fullowing ("Roull". "Indoc")
Odd-one-out
The inpat a question ponel containing N imago, where each image onttains in abworst stope. Your rak is stentify the single image that doos siane the the N-1 relationsip exhibited image outpu mover the following ISON format: ("Roul"" "Index")
("Resull": "2")
GPT-4o
SVRT
Each tnk coniake of queeion puck and query pand. Each quotim punel contains images Each dapes The sets of geomeriic stapes orgaized aoccrding dificrent rules. query panel
m
D
query images Your to detemmiee which question panel's sule marches the rale diglayed in the query image. The ixput imagei qucation panel the
["Query 1": "I"
"Query"": "2")
GPT-40
c
outpur panel query images helong Using the following ISON format ['Queryl": "indes", "Quary"": "Index")
D
Figure 5: Task-specific prompts of GPT-4o.
19

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
Sample
UCGS-T
TF-Pateh
TF-OCL
0
TF-Mono
ANP-Patch
0
ANP-OCL
ANP-MonO
Figure 6: Comparison of generation results on RAVEN.
20

---

Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning
DOS
Somple
UCGS-T
TF-Pateh
TF-OCL
7F-Mono
ANP-Patoh
000
ANP-OCL
ANP-MonO
Figure 7: Comparison of generation results on PGM.
21


Reviews:
[{"review_id": "pEqVu1hG6L", "reviewer": "Reviewer_v5Nm", "summary": "The authors transform a series of different classical abstract visual reasoning tasks by making them all into a task of generating one missing data panel given the remaining set of example panels instantiating a visual concept, which is captured with conditional generative models. Using a new architecture based on transformers, the system is trained on several abstract visual reasoning tasks, performs well across tasks, and transfers even to unseen tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.\n\n**Q1: Concerns about the formulation and experiments of Bongard problems**\n\nSimilar to the task-specific solvers, UCGS is a framework designed to solve abstract visual reasoning (AVR) tasks with only visual input/output. Therefore, we adopt a simplified version of Bongard Problems (BPs) mentioned in [1], where the task of describing rules via natural language is transformed into a classification problem on query images. Since the modified BPs have a similar definition to SVRT, we validate the ability of models to solve the modified BPs based on SVRT in our experiments. We will provide a more detailed explanation in the revised manuscript to clarify the setting of BPs.\n\n[1] A review of emerging research directions in abstract visual reasoning.\n\n**Q2: Comparison and discussion to multimodal models and neuro-symbolic models**\n\nWe conducted additional experiments on multimodal LLMs GPT-4V and GPT-4o and task-specific solvers. Since UCGS-T can handle both selective and generative problems, we compare it to generative task-specific solvers PrAE [1], NVSA [2], GCA [3], ALANS [4] and RAISE [5] that support answer selection and generation, where PrAE and NVSA are neuro-symbolic models. The task-specific solvers are trained separately on each dataset without supervision of rule and attribute annotations to ensure consistency with the experimental setup of UCGS-T.\n\n| Models | RAVEN | PGM |\n| --- | --- | --- |\n| PrAE [1] | 13.6 | - |\n| NVSA [2] | 11.5 | - |\n| GCA [3] | 37.3 | 31.7 |\n| ALANS [4] | 50.1 | - |\n| RAISE [5] | 54.5 | 14.0 |\n| GPT-4V | 13.8 | - |\n| GPT-4o | 19.2 | - |\n| GPT-4o + Language Descriptions | 38.8 | - |\n| UCGS-T (Ours) | 64.6 | 38.1 |\n\nThe experimental results indicate that under the same experimental setup, UCGS-T outperforms the generative task-specific solvers.\n\n**Comparison with neuro-symbolic models.** Without additional annotations, neuro-symbolic models show a significant performance drop on RPMs. [1] reports PrAE achieves 65.0% accuracy with rule supervision, but it drops to 13.6% in our experiments. PrAE and NVSA rely on predefined rules and explicitly defined representations for specific datasets, making it hard for reasoning with undefined concepts and rules. UCGS-T does not rely on manually designed concepts and rules. It performs independent reasoning process on each visual concept, inferring concept and rule without additional annotations.\n\n**Comparison with GPT-4V and GPT-4o.** We exhibit GPT-4V and GPT-4o's performance on RAVEN from [6]. Both models have lower accuracy than UCGS-T, ALANS, RAISE and GCA. GPT-4o improves with candidate image descriptions but remains behind UCGS-T, ALANS, and RAISE. Since not designed for abstract visual reasoning, GPT-4o may require further prompt engineering and chain-of-thought design to improve the performance on RPM.\n\n[1] Abstract spatial-temporal reasoning via probabilistic abduction and execution.\n\n[2] A neuro-vector-symbolic architecture for solving Raven’s progressive matrices.\n\n[3] Generating correct answers for progressive matrices intelligence tests.\n\n[4] Learning algebraic representation for systematic generalization in abstract reasoning.\n\n[5] Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection.\n\n[6] What is the visual cognition gap between humans and multimodal llms?\n\n**Q3: Missing references to some papers of multimodal models and Bongard problems**\n\nThanks for the suggestions. We will cite them in the revised version.\n\n**Q4: Can the authors explain the large improvement in accuracy in the ID tasks with comparatively little improvement compared to baseline in the out of distribution tasks?**\n\nThe performance of models in ID tasks is influenced by the architecture and composed modules. In OOD tasks, the difference between the training data and unseen visual concepts or abstract rules dominate the performance of models, which may reduce the performance difference between baselines and UCGS-T.\n\n**Q5: The proposed architecture is rather intricate and complex. Are there any insights the authors would like to share beyond the performance on the benchmarks?**\n\nUCGS-T extracts global visual concepts from patch-based representations for reasoning. RPM tests often involve global rules, e.g., consistent object counts across panels, which patch-based and object-centric representations struggle to capture. UCGS-T addresses this by adding a module to extract and reason independently on visual concepts. Exploring better image tokenizers may help build models with stronger OOD reasoning capabilities.\n\n**Q6: In fig. 2 (a) it looks like there is a redundant positional embedding 2 /3.**\n\nThe positional embedding 2 indicates the target position and is used as the query vector in the panel encoder.", "claims_and_evidence": "While I could not run the code to replicate the experiments, the reported experiments and evaluations support the claims.", "methods_and_evaluation_criteria": "The authors use selection accuracy as a single evaluation criterion.", "theoretical_claims": "The paper does not contain any proof. Propositions 3.4 to 3.6 are rather basic intuitions that are translated from language to straightforward formulas.", "experimental_designs_or_analyses": "The abstract visual reasoning tasks are standard benchmarks.", "relation_to_broader_scientific_literature": "Abstract visual reasoning is of broad interest to the community. The present paper profits most from reformulating classic problems into one single problem, i.e., generating a new data point (panel) based on a number of positive examples (panels) from the same concept. While this is an interesting manipulation that allows the transfer across different AVR tasks, it is also a strong modification of the original problems, e.g. in the case of the Bongard problems, where the original goal is to produce a sentence describing the concept of one set versus the concept of a second set. \n\nConceptually, it is an important goal to learn more abstract visual concepts, but it is not clear how the current system that learns such implicit abstract representations would compare to neuro-symbolic systems, particularly in terms of explainability. \n\nIntuitively, it is unclear why the performance on the RPM task is comparatively low to other systems trained exclusively on this task and why out of distribution generalization is comparatively to the other systems.", "essential_references_not_discussed": "There are multimodal models that have addressed similar problem settings, e.g.\n- Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W. and Chang, B., MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning. In The Twelfth International Conference on Learning Representations.\nand neuro-symbolic systems, e.g.:\n- Hersche, M., Zeqiri, M., Benini, L., Sebastian, A. and Rahimi, A., 2023. A neuro-vector-symbolic architecture for solving Raven’s progressive matrices. Nature Machine Intelligence, 5(4), pp.363-375.\nIf the Bongard problems are part of this paper, then it seems necessary to cite\n- Yun, X., Bohn, T. and Ling, C., 2020. A deeper look at Bongard problems. In Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian AI 2020, Ottawa, ON, Canada, May 13–15, 2020, Proceedings 33 (pp. 528-539). Springer International Publishing.\n- Depeweg, S., Rothkopf, C.A. and Jäkel, F., 2024. Solving bongard problems with a visual language and pragmatic constraints. Cognitive Science, 48(5), p.e13432.", "other_strengths_and_weaknesses": "The goal of the study is very relevant, timely, and interesting.\n\nIt would be very helpful to contextualize the performance results: how well do multimodal models fair by comparison, and why? How do neurosymbolic models perform, and why?\n\nWhile I am convinced of the helpfulness of the old approach, “if you cannot solve a problem, solve a different problem”, it should be pointed out that e.g. the Bongard problems are much harder in their original formulation in which a sentence describing the concept has to be formulated in natural language.", "other_comments_or_suggestions": "Fig. 1 (d) does not show BPs but SVRT by Fleuret et al. (2011).\n\nIn fig. 2 (a) it looks like there is a redundant positional embedding 2 /3.\n\nBP is used in proposition 3.6, although the Bongard Problems are introduced later in the text and then disappear altogether.", "questions_for_authors": "The proposed architecture is rather intricate and complex. Are there any insights the authors would like to share beyond the performance on the benchmarks?\n\nCan the authors share any reasoning for the specific form of the \n\nCan the authors explain the large improvement in accuracy in the ID tasks with comparatively little improvement compared to baseline in the out of distribution tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8xyCozTDYx", "reviewer": "Reviewer_zHeV", "summary": "This paper presents a method for solving abstract visual reasoning tasks that aims to unify previous methods for different types of tasks (e.g. matrix reasoning vs. odd-one-out) and different modes for solving problems (e.g. classification vs. generation). The method is evaluated on various abstract visual reasoning tasks, with an emphasis on the ability to generalize between tasks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.\n\n**Q1: Comparison and discussion to RPM solvers and general-purpose systems**\n\nWe conducted additional experiments on the general-purpose systems GPT-4V and GPT-4o and classic task-specific RPM solvers. Since UCGS-T can handle both selective and generative problems, we compare it to generative task-specific solvers PrAE [1], NVSA [2], GCA [3], ALANS [4] and RAISE [5] that support answer selection and generation. The task-specific solvers are trained separately on each dataset without the supervision of rule and attribute annotations to ensure consistency with the experimental setup of UCGS-T.\n\n| Models | RAVEN | PGM |\n| --- | --- | --- |\n| PrAE [1] | 13.6 | - |\n| NVSA [2] | 11.5 | - |\n| GCA [3] | 37.3 | 31.7 |\n| ALANS [4] | 50.1 | - |\n| RAISE [5] | 54.5 | 14.0 |\n| GPT-4V | 13.8 | - |\n| GPT-4o | 19.2 | - |\n| GPT-4o + Language Descriptions | 38.8 | - |\n| UCGS-T (Ours) | 64.6 | 38.1 |\n\n**Comparison with task-specific solvers.** The experimental results indicate that under the same experimental setup, UCGS-T outperforms the generative task-specific RPM solvers. Since PrAE and ALANS only define visual concepts and abstract rules for RAVEN, they are incapable of solving PGM problems. Our experiments reveal that, without additional annotation information, the performance of the generative task-specific RPM solvers significantly declines. For example, [1] reports that PrAE achieves accuarcy of 65.0% after rule-supervised training, which declines to 13.6% in our experiments.\n\n**Comparison with general-purpose systems.** We also exhibit the results of the general-purpose systems GPT-4V and GPT-4o on RAVEN. The accuracy scores are reported by [6] where GPT-4V and GPT-4o have lower accuracy than UCGS-T, ALANS and RAISE. GPT-4o shows a notable performance improvement when provided with language descriptions of candidate images (from 19.2% to 38.8%), but still falls short of UCGS-T, ALANS, and RAISE. While the general-purpose systems GPT-4V and GPT-4o are not specifically designed for abstract visual reasoning, solving such tasks may require further prompt engineering and chain-of-thought design.\n\n[1] Abstract spatial-temporal reasoning via probabilistic abduction and execution.\n\n[2] A neuro-vector-symbolic architecture for solving Raven’s progressive matrices.\n\n[3] Generating correct answers for progressive matrices intelligence tests.\n\n[4] Learning algebraic representation for systematic generalization in abstract reasoning.\n\n[5] Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection.\n\n[6] What is the visual cognition gap between humans and multimodal llms?\n\n**Q2: Despite claiming to present a highly general model, it is overly focused on a very specific subset of idealized abstract visual reasoning problems.**\n\nAs stated in Lines 26–29 of the abstract, UCGS is \"general\" since it aims to solve multiple abstract visual reasoning (AVR) tasks in a unified manner, which are often treated as independent in task-specific solvers. We agree that AVR tasks are widely used not because researchers are interested in achieving high scores in these tasks, but because they can reveal the core reasoning abilities of AI. Our goal is precisely to develop a more general framework for AVR—one that can exhibit multi-task AVR ability as humans, rather than simply “solving these tasks.”\n\nTo this end, we use representative AVR tasks as benchmarks. While the matrix reasoning tasks may seem simple in form, building multi-task solvers on them remains challenging. As shown in the table, GPT-4V and GPT-4o can hardly achieve performance comparable to task-specific solvers. The performance of UCGS-T illustrate the effectiveness of the proposed framework in the typical AVR tasks. Importantly, selecting these tasks does not mean our framework is limited to them. UCGS can extend to other tasks, such as visual analogy extrapolation problems [1], which shares a similar structure with matrix reasoning.\n\n[1] A review of emerging research directions in abstract visual reasoning.\n\n**Q3: The propositions and proofs in section 3.1 seem somewhat unnecessary …**\n\nThe significance of the propositions consists of two aspects. On one hand, it demonstrates how different forms of classical reasoning tasks can be described as a conditional generation processes. On the other hand, it explains how generative and selective tasks can be unified into one framework. Specifically, some selective tasks can be reformulated as an implicit answer generation problem, where the goal is to search for an option among predefined candidates that matches the generated result. This design allows a single conditional generative model to solve both types of tasks simultaneously, eliminating the requirement of an additional scoring network for answer selection.", "claims_and_evidence": "The primary concern with this work is that, despite claiming to present a highly general model, it is overly focused on a very specific subset of idealized abstract visual reasoning problems. In general, abstract visual reasoning tasks have been created because we think that they tell us something important about the more general reasoning capabilities of our models, not because we are interested in solving these tasks per se. Thus, it is unclear why a 'general purpose solver' is needed for this specific subset of tasks. Despite being more general than some previous models in this literature (e.g., those that are specific to classification-based matrix reasoning tasks), the proposed approach is still very specific to a particular type of problem. It assumes that inputs will be presented in a set of discrete panels, that these panels will consist of relatively simple geometric forms, and that the objects will be entirely explainable via a relatively simple set of abstract rules. Thus, the proposed approach is very far from the 'general purpose abstract reasoner' that is promoted in the abstract and the introduction. \n\nThe performance of the model is also very poor in some cases. For instance, the iid performance on the RAVEN and PGM benchmarks is well below the performance of several models which are not included as baselines. The introduction touts the zero-shot reasoning abilities of the model, but the zero-shot generalization to new tasks is generally very poor (though somewhat better than the baselines). There is also no discussion or evaluation of the truly general-purpose systems (LLMs, VLMs, reasoning models like o1) that increasingly display strong abilities to solve these sorts of problems, while also not being limited to a very specific problem format.", "methods_and_evaluation_criteria": "There are many baselines from the literature on abstract visual reasoning in deep learning systems (e.g. architectures designed for problems like RAVEN and PGM) that are not discussed or compared with the proposed approach. The tasks that are considered are also very similar, despite the emphasis of the paper on generality. There are many other abstract visual reasoning tasks (Bongard problems, ARC, visual question-answering tasks, tasks involving reasoning over real-world images) that are important to consider when evaluating a putatively general-purpose abstract visual reasoning system, and that are increasingly solvable by LLMs or VLMs.", "theoretical_claims": "The propositions and proofs introduced in section 3.1, though correct, seem somewhat unnecessary for what ultimately turns out to be a relatively straightforward generative architecture for solving these types of abstract visual reasoning problems. The definitions introduced in this section also underscore the very specific domain to which the model can be applied (i.e. it is only applicable to problems involving panels governed by specific rules).", "experimental_designs_or_analyses": "The experiments are reasonable given the goals of the paper, but they are very limited to a specific type of abstract visual reasoning task.", "relation_to_broader_scientific_literature": "The paper primarily considers only other work that evaluates deep learning systems on abstract visual reasoning tasks. There is very little connection to other types of reasoning tasks, or the broader space of models that are increasingly able to solve a wide range of tasks.", "essential_references_not_discussed": "Within the domain of deep learning models designed to solve abstract visual reasoning tasks, the references considered are reasonable. But many of these models are not directly compared with the proposed approach, despite achieving better performance in some settings. There is also very little consideration of reasoning beyond these types of tasks, and very little discussion of other approaches to solving abstract visual reasoning tasks.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wUuWnB0Sz6", "reviewer": "Reviewer_r4C7", "summary": "- This paper proposes a unified framework (UCGS) for solving 4 different abstract visual reasoning tasks with a single deep network architecture. \n- The already existing abstract visual reasoning tasks are based on problem panels consisting of several images showing simple visual concepts following different abstract rules with the model's task being the selection of the correct missing image (Raven's Progressive Matrices, Visual Analogy Problems), the detection of an outlier w.r.t. the rule (Odd-One-Out), or the assignment of images to the correct problem panel (Bongard problems).\n- The authors introduce a formal generative framework that cuts these different tasks down to the estimation of conditional probabilites of images given the rest of the problem panel.\n- The paper provides a concrete implementation of the framework in form of a transformer architecture applied on vector quantized image patches and consisting of a hierarchy of modules to first encode patches per image, concepts on panel level, and finally generate a target image autoregressively patch by patch.\n- An experimental evaluation validates the effectiveness of the proposed framework and architecture in solving various AVR problems without retraining as well as generalization to unseen problems and tasks by showing results for three different settings with:\n  - known tasks with unseen combinations of known abstract rules and visual concepts (ID Tasks), \n  - unseen tasks with known abstract rules and visual concepts (ID-ZS), and \n  - unseen tasks with unseen abstract rules and visual concepts (OOD-ZS).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive suggestions. The detailed responses to the reviewer's comments are as follows.\n\n**Q1: Some statements could be misunderstood and the paper would benefit from a more precise formulation.**\n\nThank you for pointing out the inaccurate statements. We will carefully check the manuscript to modify or remove the statements in the revised version.\n\n**Q2: Comparison and discussion to task-specific solvers**\n\nWe conducted additional experiments on the task-specific solvers and multimodal LLMs GPT-4V and GPT-4o. Since UCGS-T can handle both selective and generative problems, we compare it to generative task-specific solvers PrAE [1], NVSA [2], GCA [3], ALANS [4] and RAISE [5] that support answer selection and generation. The task-specific solvers are trained separately on each dataset without the supervision of rule and attribute annotations to ensure consistency with the experimental setup of UCGS-T.\n\n| Models | RAVEN | PGM |\n| --- | --- | --- |\n| PrAE [1] | 13.6 | - |\n| NVSA [2] | 11.5 | - |\n| GCA [3] | 37.3 | 31.7 |\n| ALANS [4] | 50.1 | - |\n| RAISE [5] | 54.5 | 14.0 |\n| GPT-4V | 13.8 | - |\n| GPT-4o | 19.2 | - |\n| GPT-4o + Language Descriptions | 38.8 | - |\n| UCGS-T (Ours) | 64.6 | 38.1 |\n\nExperimental results show that UCGS-T outperforms generative task-specific solvers under the same setup. PrAE and ALANS cannot solve PGM problems since they only define visual concepts and abstract rules of RAVEN. Without additional annotations, task-specific solvers suffer significant performance drops, e.g., PrAE’s accuracy drops from 65.0% (reported by [1]) to 13.6% in our experiments. We exhibit the results of GPT-4V and GPT-4o on RAVEN from [6], where both models show lower accuracy than UCGS-T, ALANS and RAISE. The multimodal LLMs are not designed for abstract visual reasoning, therefore solving such tasks may require further prompt engineering and chain-of-thought design.\n\n[1] Abstract spatial-temporal reasoning via probabilistic abduction and execution.\n\n[2] A neuro-vector-symbolic architecture for solving Raven’s progressive matrices.\n\n[3] Generating correct answers for progressive matrices intelligence tests.\n\n[4] Learning algebraic representation for systematic generalization in abstract reasoning.\n\n[5] Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection.\n\n[6] What is the visual cognition gap between humans and multimodal llms?\n\n**Q3: Doubts regarding the replacement of the last image in Proposition 3.6**\n\nSVRT provides an image set for each rule containing different images generated by computer programs that follow the rule. Each image is assigned to only one panel of Bongard Problems (BPs). If the remaining images of a set are insufficient to form a complete panel, they will be dropped. Therefore, there are no duplicate images between BP panels even if one image is removed from the left and right problem panels.\n\n**Q4: The main paper is missing descriptions and references for the G1-set and SVRT datasets, while the supplementary material provides these.**\n\nThank you for the suggestion. We will add the descriptions and references in the first paragraph of Experiments.\n\n**Q5: Is the model training of VQVAE and transformer done end-to-end or as usual one after another?**\n\nVQVAE is pretrained before training the remaining modules. We leave the image reconstruction loss in Eq. 9 to make it possible to finetune VQVAE end-to-end in the training stage. But we find that setting \\lambda to 0 (i.e., freezing the parameters of VQVAE) is the best choice. Please refer to Appendix B.1 for the detailed descriptions.\n\n**Q6: Regarding the lack of clarity w.r.t. the used terms**\n\nIn this paper, tasks refer to different abstract visual reasoning tasks such as RPM and BP. Problems refer to individual problem instances. Definitions describe the form of different tasks. Visual concepts refer to image attributes like object size and color. Rules are changing patterns on attributes (e.g., progressive changes). Thanks for the helpful suggestion to make the terms clear. We will introduce the terms at the beginning of the method section.\n\n**Q7: The motivation of including an object-centric baseline**\n\nSome task-specific solvers [1, 2] have realized abstract visual reasoning with object-centric representations. Therefore, we adopt it as one of the typical approaches to explore the performance of different visual representations in UCGS.\n\n[1] Learning to reason over visual objects.\n\n[2] Systematic visual reasoning through object-centric relational abstraction.\n\n**Q8: Other comments or suggestions about writing**\n\nThanks for the constructive comments. In the revised version, we will clarify that the baselines are ablations on model architecture when introducing the baselines. And we will also remove the repeated parts (Lines 64-70 and the first paragraph in Section 3.1) in the manuscript.", "claims_and_evidence": "- The authors claim that \"UCGS can successfully solve various AVR tasks\" (lines 038 ff.) and the \"framework can solve tasks like RPM, Visual Analogy Problem, and Odd-one-out\" (lines 094 ff.).\n  - While the paper shows that the formal framework is flexible and can be used to tackle these tasks, the experimental results are still far away from completely solving these tasks. Therefore, this statement could be misunderstood and the paper would benefit from a more precise formulation.\n\n- The paper claims \"strong reasoning ability in ID tasks\" (lines 365 ff.).\n  - If I am not mistaken, the paper is missing a comparison with task-specific solvers that would put the documented experimental results in relation to what is possible with inductive biases specific to the tasks and answer the question of the cost of the unified framework and its advantage w.r.t. task flexibility.\n  - The provided baselines are rather ablations of the architecture implementing the general framework.\n\nThe remaining claims are supported by clear and convincing evidence, e.g., the zero-shot generalization of the reasoning ability to unseen tasks, novel rules and visual concepts by outperforming the random guess.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense.", "theoretical_claims": "I have checked all theoretical claims and their proofs (propositions 3.4 to 3.6).\n- Regarding proposition 3.6., I have some doubts regarding the replacement of the last image in the panel:\n  - The Bongard problem only describes the task of assigning a query image to the correct panel (left or right), not replacing an image from the panel with the query image.\n  - Can the removal of the last image (or any other image) result in the abstract rule of that panel not being unique anymore and in the worst case even making both query images fitting with both partial problem panels such that there is no unique solution anymore?\n  - In lines 183 ff., the authors say that if both problem panels (left and right) are sampled uniformly from the dataset, then the probabilities of the partial panels is 1 over the size of the dataset. If I am not mistaken, that should only be correct if you can make some additional assumptions about the dataset, e.g., that there are no problem panels with equal partial panels, if you remove the last image.", "experimental_designs_or_analyses": "From the paper and supplementary material, the experimental designs seem valid.", "relation_to_broader_scientific_literature": "I am not very familiar with the related works in the area of abstract visual reasoning. However,\n- the key contribution being the unified framework to tackle different tasks with a single model without retraining and enabling some generalization to unseen tasks and problems (combinations of abstract rules and visual concepts) seems novel,\n- therefore, the generalization ability to unseen tasks and problems is a finding unique to this paper,\n- as mentioned in the review section \"Claims and Evidence\", the paper misses the comparison with the performance of task-specific solvers, which do exist as mentioned in the introduction (lines 37ff. right column) and seem to perform better, as stated in the limitations \"further exploration is required to reach the performance achieved by the task-specific solvers\" (lines 431ff. right column).", "essential_references_not_discussed": "- The main paper is missing descriptions and references for the G1-set and SVRT datasets, while the supplementary material provides these.\n- As mentioned above, the paper mentions task-specific solvers but does not discuss the provided experimental results in relation to the performance of these task-specific solvers.\n\nBesides the above, I am not aware of any missing essential references.", "other_strengths_and_weaknesses": "Strengths:\n- The paper addresses a very challenging and interesting problem in learning abstract rules from visual data with a unified framework addressing multiple tasks without retraining.\n- It is mostly very clearly explained and easy to understand.\n  - The introduction, related work, and figure 1 introduce the topic of abstract visual reasoning very well.\n  - The unified framework (section 3.1) and transformer implementation (section 3.2) are explained clearly without being too detailed (hyperparameters etc. in the appendix).\n- The generalization capabilities to unseen tasks are exciting and unique to the paper with its unified framework.\n\nWeaknesses:\n- Some lack of clarity\n  - Is the model training of VQVAE and transformer done end-to-end or as usual one after another? The total loss in eq. 9 seems to suggest the former, which I would find surprising.\n  - The paper uses the terms \"visual concepts, rules, definitions, problems, and tasks\" without a clear definition. The meanings and differences became clear to me rather late in the paper (in sections 4.1.1 to 4.1.3).\n  - If I am not mistaken, the provided baselines are rather architecture ablations than existing approaches. If that is the case, the paper would benefit from making this clear.\n  - What is the motivation of including an object-centric baseline?", "other_comments_or_suggestions": "- Regarding the lack of clarity w.r.t. the used terms (abstract rules, visual concepts, etc.), the paper would benefit from defining them once early on and then using them consistently.\n- Lines 64 to 70 (the first two sentences of that paragraph in the left column) sound very repetitive, which seems like an accident.\n- The first paragraph in section 3.1. (before definition 3.1) is a bit repetitive w.r.t. the related work.", "questions_for_authors": "The most important questions are:\n1. Why do you not compare with task-specific solvers to put the provided experimental results in relation to them, if you mention that \"further exploration is required to reach the performance\" of them (see limitations lines 431 ff.)? \n  - This comparison is very important in my opinion for the evaluation to be complete. Worse performance of the proposed approach is to be expected and would therefore not change my positive view of the experimental results (if not too large).\n2. Please see my doubts regarding proposition 3.6. in review section \"Theoretical Claims\": Does the replacement and therefore removal of an image in the Bongard problem planels not affect the uniqueness of solutions and the probabilities in the proof?\n  - A convincing clarification would remove my doubts regarding this theoretical claim.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present a novel framework for solving visual reasoning tasks based on conditional generative models. The work was received with mixed scores by the reviewers. As positive aspects, they highlighted the goal and novel approach of the paper, aiming to generalize across tasks. Negative aspects include skepticism regarding the general direction of the paper, sitting between specialized classification approaches and very general reasoning methods based on LLMs/VLMs. Also, additional comparisons against neuro-symbolic methods and LLMs were requested.

In the rebuttal, the authors followed up with the requested comparisons, satisfying two out of three reviewers (Weak Accept). One reviewer remained unconvinced (Reject), raising general concerns about the direction of the paper in discussions, specifically, that it is neither fully general (in compared to LLM/VLM reasoning) nor does it beat fully specialized classification methods.

In light of the diverging situation, I also took a look at the paper. I believe that statements made by all three reviewers are correct and that the decision comes down to how much value we put on novel exploration if the results do not outperform all existing paradigms. As I strongly believe that such exploration is of value to the community, that this is a well written and executed paper, and I believe the general direction to be promising, I decided to follow the two positive reviewers and recommend to accept this work.
Decision: Accept (poster)