Hierarchical Perceptual and Predictive Analogy-Inference Network for Abstract Visual Reasoning

Authors: palc  block
batchnorm  conv
  relu
conv  batchnorm
relu  spn
seca  mlp
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-13 07:28:33

Abstract: Advances in computer vision research enable human-like high-dimensional perceptual induction over analogical visual reasoning problems, such as Raven's Progressive Matrices (RPMs). In this paper, we propose a Hierarchical Perception and Predictive Analogy-Inference network (HP$^2$AI), consisting of three major components that tackle key challenges of RPM problems. Firstly, in view of the limited receptive fields of shallow networks in most existing RPM solvers, a perceptual encoder is proposed, consisting of a series of hierarchically coupled Patch Attention and Local Context (PALC) blocks, which could capture local attributes at early stages and capture the global panel layout at deep stages. Secondly, most methods seek for object-level similarities to map the context images directly to the answer image, while failing to extract the underlying analogies. The proposed reasoning module, Predictive Analogy-Inference (PredAI), consists of a set of Analogy-Inference Blocks (AIBs) to model and exploit the inherent analogical reasoning rules instead of object similarity. Lastly, the Squeeze-and-Excitation Channel-wise Attention (SECA) in the proposed PredAI discriminates essential attributes and analogies from irrelevant ones. Extensive experiments over four benchmark RPM datasets show that the proposed HP$^2$AI achieves significant performance gains over all the state-of-the-art methods consistently on all four datasets.

Main paper: 
 introduction: 
 Analogical reasoning over abstract concepts has been researched for decades [23] , because of its significance in human cognition and potential applications in children assessment [4] , zero-shot learning [33, 34] , multimedia content understanding [28, 32, 42] , etc. The recent emergence of multimodal large language models has revolutionized all aspects of daily human tasks, but due to the lack of human-like fluid reasoning capabilities, their performance in abstract thinking is far from humans [15, 34] . Analogical reasoning ? Figure 1 : Overview of the proposed HP 2 AI framework. The proposed PredAI reasoner solves RPMs by predicting the 3rd entity in each row and verifying it against the truth, and enhancing important reasoning attributes in channel wise from multi-level perceptual features. This is done for the output of each of the 𝐽 PALC modules, which form HPALC.extracts high-level abstractions rationally from existing complex high-dimensional information, which enables abstract thinking beyond appearance details and comprehending unfamiliar scenarios in terms of objects, relationships and higher-order patterns [25] . It provides an effective means to assess a machine's capacity for abstraction, i.e., identify common higher-order rationales by abstracting from superficial features, similar to how human reasons. Analogical reasoning can be broadly categorized based on the nature of the relations being compared, e.g., categorical analogy [23] , visual analogy [9] and linguistic analogy [27] .Raven's Progressive Matrices (RPM) are typical visual analogy problems whose solutions rely on an analogy over visually perceived contents [40] . RPM problems have historically been studied in cognitive science to assess human intelligence, while recently studied in computer science to improve the reasoning ability of computers [8, 36] . To minimize the impact of language barrier and culture bias, the pictorial matrices are designed with images of regular-shaped polygons, e.g., triangles, circles, etc. Given a 3 × 3 pictorial matrix with the last one left blank, the objective is to identify the missing entry from eight candidate answers based on the visual context and inductive rules.Solving RPM problems usually involves two main steps: visual perception and analogical reasoning. For visual perception, most RPM solvers [14, 30, 41, 43] utilize shallow neural networks to extract local visual attributes such as polygon shape, size and color, as RPM images are often constructed using simple 2D polygons or lines [1, 40] . But these shallow networks may not capture the global image layout well. Solving RPM problem requires the ability to reason about a set of relational rules inherent to global and local attributes. Deficiencies in extracting global attributes will lead to the loss of key reasoning clues, thus limit the performance of the model. To bridge the gap, a set of hierarchically coupled Patch Attention and Local Context (PALC) blocks are proposed to extract features from multiple receptive fields. Each block in turn combines a couple of Local Context blocks built using residual blocks to capture relatively lower-level visual attributes, and a set of Patch Attention blocks to capture relatively higher-level spatial dependencies. Additionally, different from the methods that reason over object detection results [35] , the proposed perceptual encoder is integrated into an end-to-end visual reasoning model without extra supervision effort in object detection.Most existing methods often utilize object-level feature similarities to solve RPMs [2, 8, 43] , e.g., MRNet [2] and HCV-ARR [8] examine the pair-wise feature similarity over three rows 1 to detect recurrent object attributes. While the shared concrete object properties can be measured by similarity, the shared abstract relational structures can only be handled by analogy [6, 9] . Those models [2, 8, 43] may neglect the core nature of analogical reasoning, i.e., to align relational analogies from the base and apply them to predict the target [6] . Inspired by the prediction and matching paradigm in cognitive science [6, 36] , a Predictive Analogy-Inference (PredAI) framework is developed in this paper to conduct more robust analogical reasoning, in which the Analogy-Inference Blocks (AIBs) first model the abstract analogies from the base and then apply them to the target, rather than object-level similarities across different RPM images in most approaches [2, 8, 43] . As depicted in Fig. 1 , the proposed AIB explicitly models the analogy by first predicting the third feature set using the first two feature sets of each row through a Shared-weight Prediction Network (SPN), and then verifying the prediction against the ground-truth. Such a formulation well emulates the human's analogical reasoning process [6] , and overcomes the dilemma about "analogy" and "similarity" in RPMs.In the proposed framework, multi-level perceptual features from different hierarchies of the perceptual encoder are jointly utilized in the reasoning module PredAI, resulting in countless potential analogical inferences embedded in these features. To discriminate essential attributes and relations from irrelevant ones, a Squeezeand-Excitation Channel-wise Attention (SECA) is designed in the proposed PredAI, where the less relevant features and relations are suppressed at the squeeze stage and the more dominant ones are amplified with large weights at the excitation stage.Our contributions can be summarized as follows. 1) To tackle the limited receptive fields of shallow networks in existing RPM solvers, a hierarchically coupled perceptual encoder is proposed to perceive both the local visual attributes and the global panel layout at multiple receptive fields. 2) Instead of seeking for feature similarity in existing RPM solvers, a predicting-and-verifying paradigm is designed in the proposed AIB to directly model the reasoning rules embedded in RPM questions. 3) The proposed SECA highlights the most relevant attributes and rules for solving the problem and suppresses the irrelevant ones. 4) The proposed HP 2 AI significantly outperforms the state-of-the-art methods over 4 datasets, improving the previous best results from 98.2% to 99.3% on PGM [1] , from 95.8% to 98.8% on RAVEN [40] , from 96.5% to 99.4% on I-RAVEN [14] and from 97.1% to 98.6% on RAVEN-FAIR [2] , respectively. 
related work 2.1 visual reasoning: 
 Solutions of visual reasoning can be categorized into various groups [24] , including analogical (abstract) visual reasoning [1, 2, 8, 14, 24, 40] , visual question answering [16, 21, 29, 37] , visual commonsense reasoning [17, 19, 38, 39] , mathematical visual reasoning [5, 10, 18] and many others [11, 12] . They all address the problem of reasoning for the abstract concepts present in images or the hidden patterns that govern visual entities. Visual question answering (VQA) is one of the most adapted assessments for the general reasoning ability of linking vision and language modalities, where the goal is to answer a question by referring to an associated image [21] . Analogical visual reasoning spans a variety of tasks, e.g., RPMs [1, 40] , visual abductive reasoning (VAR) [20] , CLEVR-Matrices [25] and Bongard-LOGO problems [26] . RPMs are a typical category of analogical visual reasoning tasks, where the questions are designed in an abstract formulation using only 2D lines or shapes to minimize the impact of language barrier or culture bias. VAR [20] examines abductive reasoning ability of machine intelligence in everyday real-world situation by completing sets of visual events and inferring the hypothesis that can best explain the visual premise. Mondal et al. [25] developed an RPM-like problem dataset, CLEVR-Matrices, using realistically rendered 3D shapes based on the VQA dataset CLEVR [16] . The Bongard-LOGO problem [26] was developed to evaluate human-like analogy-making by interpreting a LOGO object in terms of another, with the target of discovering the concept that the positive images obey while negative images violate.Visual reasoning models for solving RPMs often consist of two modules: visual perception and relational reasoning. For visual perception, most solution models utilize consecutive convolution layers to extract visual features, e.g., CoPINet [41] and Rel-AIR [30] , and the latest PredRNet [36] all utilize residual networks [7] . Recently, MRNet [2] applies multi-scale convolutional encoders and HCV-ARR [8] adapts a mixed model of convolution blocks and transformer blocks to extract visual features from RPM images. For analogical reasoning over RPMs, early models often compare and contrast answer sets directly, e.g., CoPINet [41] explicitly contrasts candidate answers and highlights the difference between options while DCNet [43] contrasts both row features and candidate options. Research shows that different levels of inductive bias are incorporated into these network structures, and thereby improve the classification performance [24, 25] . Recently, MRNet [2] deducts the correct answer by minimizing the squared Euclidean distance between row features, to identify recurring patterns. In [8] , a reasoning module is designed based on the attention mechanism to dynamically learn the feature weights and highlight the important features to encode the rule representations. Most recently, PredR-Net [36] applies the neural prediction errors between the selected option and the predicted answer for reasoning. 
proposed method 3.1 overview of proposed method: 
 As shown in Fig. 2 , the proposed HP 2 AI consists of two modules. 2) Squeeze-and-Excitation Channel-wise Attention (SECA) that suppresses the irrelevant attributes and relations and highlights the essential ones, to concentrate on the most relevant features and analogies to solve the RPM problems. 𝐽 hierarchically organized PALC blocks, where features extracted from the early blocks generally focus on image fine details, whilst the latter stages are specialized to higher level features and the features become gradually sparse across channels. In traditional image encoders, features from early stages are considered too noisy for visual recognition and are usually passed into deeper stages for high-level abstraction [36] . But for RPM problems, small entity details are as important as high-level image semantics. We hence hierarchically extract features from multiple receptive fields and build PredAI block for each set of features at their respective receptive field. Furthermore, each PALC block forms a dual-branch structure to extract both local and global features, where the Local Context block perceives low-level image details of objects, and the Patch Attention block extracts global spatial semantics on images by modelling long-range dependencies through cross-patch multi-head self-attention [3, 22] .Formally, an RPM sample ⟨Q, A⟩ is composed of a 3 × 3 pictorial image matrix of size 𝐻 × 𝑊 with the last one missing, Q = {𝑸 1 , 𝑸 2 , . . . , 𝑸 8 } as the question set, and A = {𝑨 1 , 𝑨 2 , . . . , 𝑨 8 } as the answer set. Each answer image is appended to eight question images to form one complete 3 × 3 RPM panel. The goal is to determine the missing image from the eight candidate answers through 𝑘 × 𝑊 𝑘 is the number of rows. Each token vector is embedded into 𝑑 𝑇 dimensions through the same linear layers to obtain 𝑬 𝐹 ∈ R 𝑛 𝑇 ×𝑑 𝑇 , and a positional embedding 𝑬 𝑃 ∈ R 𝑛 𝑇 ×𝑑 𝑇 is added to 𝑬 𝐹 to form the input into cross-patch multi-head self-attention as 𝑭 0 = 𝑬 𝐹 + 𝑬 𝑃 . Next, multiple Patch Attention blocks are sequentially applied to extract the attentional features, with 𝑭 𝑗 -1 and 𝑭 𝑗 as the input and the output of the 𝑗-th block respectively. The local self-attention (LSA) computes the self-attention relations of local patches within non-overlapped regions, whilst regional self-attention (RSA) computes the global self-attention relations between different regions of patches. As shown in Fig. 2 , the extracted features for all 9 panel images are collectively denoted as F 𝑗 . Partitioning images into patch tokens and investigating patches with shifting windows is advantageous for understanding rich combinatorial relationships such as global spatial semantics, for solving challenging problem settings such as 2x2G, 3x3G and O-IG in RAVEN datasets [8, 25, 36] .Meanwhile, the Local Context block is designed for perceiving local details of entities in RPMs. Specifically, given an image 𝑰 ∈ R 𝐻 ×𝑊 , each stage contains three successive 3×3 convolution layers with a stride of 𝑆 to obtain multi-scale features. The input 𝒁 𝑗 -1 of each block is down-sampled and added to the output of the block through a shortcut to form a residual structure [7] , and produce 𝒁 𝑗 as the output of the 𝑗-th block. Collectively, the features for the 3 × 3 RPM panel for the 𝑗-th Local Context block can be denoted as Z 𝑗 . The output of the 𝑗-th PALC block is obtained by fusing both the output of 𝑗-th Local Context block and Patch Attention block,EQUATIONwhere 𝐷 𝑗 is the number of feature channels. Hierarchical features {X 1 , . . . , X 𝐽 } serve as multi-view cues for analogical reasoning.For a 3×3 RPM panel, the analogies are designed row-wisely [1, 40] , i.e., the underlying rules can be derived from each of the first two base rows and then the same rules are applied to the third target row to infer the missing image from the options. However, it is difficult to explicitly model the rules embedded in three images of each row.As shown in Fig. 3 , the proposed Analogical-Inference Block (AIB) transforms this complex and challenging problem into a predictingand-verifying paradigm, which takes the reasoning features of the first two columns as the input, predicts those of the third column, and verifies the prediction against the features of the third column. The Shared-weight Prediction Network (SPN) is expected to capture the underlying reasoning rules across three rows. Furthermore, to suppress the irrelevant attributes and rules, and highlight the important ones, a Squeeze-and-Excitation Channel-wise Attention (SECA) mechanism is proposed to adaptively weigh the features. Finally, multiple PredAI blocks are stacked to refine rules, especially the complicated ones. Analogy Inference. Most methods [2, 8, 43] attempt to refine inherent rules by maximizing the similarity over visually perceived row features. However, feature-level similarities do not necessarily assure similar relation-level analogies, e.g., two rows of images satisfying an Arithmetic rule for Number may have different visual layouts of objects in images (Fig. 2 ). In contrast, the analogy inference aims at extracting analogies that fulfill relations for all three row attributes, which emulates the process of human-like highlevel perceptual induction rather than purely focuses on similar visual features. In this paper, we attempt to take row-wise features perceived from the HPALC as the input, and outputs the high-level relation abstractions.Predicting-and-verifying Paradigm. It is difficult to explicitly derive the underlying rules, as the perceived visual features contain multiple object attributes and hence the reasoning rules over these visual features vary significantly. Taking the feature tensors X 𝑗 of the 𝑗-th PALC block as an example, we map X 𝑗 to 3 × 3 layout as Y 𝑗 ∈ R 3×3×𝐷 𝑗 and use Y 𝑗 as the reasoning features. For the 𝑖-th row, a naive implementation to derive the reasoning rule R 𝑖 is,EQUATIONwhere F ˆis the reasoning network and Y 𝑗 𝑖,:,: represent the reasoning features for the three images of the 𝑖-th row. To ensure that the rules are consistent across three rows, it is preferred to max F 𝑆 (R 1 , R 2 , R 3 ), where F 𝑆 measures the similarity of three rules. Note that it is difficult to directly deduce the consistent rules across three rows as in Eqn. where F 𝚽 is a Shared-weight Prediction Network (SPN). For each row, the third entity is first predicted using the first two entities by , so that F 𝚽 well models the underlying rules. In such a way, the Analogy Inference Block (AIB) extracts the analogy and applies it to predict the third entity in each row. Unlike the way in Eqn. (2) that the analogies are difficult to extract, the network parameters 𝚽 of AIB can be explicitly refined to capture the analogies. Shared-weighted Prediction Network. Many techniques have been attempted in literature to align the abstract analogies derived from visual features across all three rows, e.g., Euclidean similarity [2] or attention mechanism [8] . However, these methods focus on feature-level consistency, which does not necessarily align the underlying relations. In this paper, following the predicting-andverifying paradigm, we propose a simple but genius mechanism to restrict that the AIBs for different rows share the same weights 𝚽. Consequently, the predicted features are derived using same weights embedded in AIBs for different rows, to ensure that the same reasoning analogy is applied across three rows. Squeeze-and-Excitation Channel-wise Attention. Analogical inference through the proposed predicting-and-verifying paradigm offers an effective way to refine the network model and capture the underlying analogy, but it also poses a practical challenge, i.e., between any two nonidentical features there may be countless potential inferences [6] . The high-dimensional multi-scale features implicitly represent different visual attributes perceived from the RPM images. Given one attribute dominated by one potential relation, there might be countless potential relations between three entities if derived by neural layer computations.FTo tackle the challenge, a Squeeze-and-Excitation Channel-wise Attention (SECA) mechanism is integrated into the PredAI block to dynamically weigh the attributes across channels with different importance. Squeeze-and-Excitation [13] was originally designed to learn channel-wise feature dependencies to improve the network's ability to discriminate important features from less relevant ones. In this paper, the proposed SECA mechanism is designed to suppress the irrelevant features and rules that may be harmful for deriving the underlying analogies by squeezing the input feature map to a vector through global average pooling, and then the squeezed vector is excited with a set of channel-wise scaling factors that signify the importance of each channel.Specifically, the SECA is represented by the down-sampling operation with network parameters W In such a way, the irrelevant features are suppressed and the important ones are highlighted. The enhanced features from Eqn. ( 6) and predictive differences from Eqn. (4) are combined to form Ỹ𝑗 and added to the original input features, and passed through multilayer perceptrons F MLP for high-level abstraction,EQUATIONIntuitively, if the prediction error is small, Y ′𝑗 contains the enhanced features of the first two entities and the original features of the third entity for next stage optimization. In practice, in order to capture the complex rules, PredAI blocks F 𝑘 P are stacked asY 𝑗 𝑘 = F 𝑘 P (Y 𝑗 𝑘 -1 ), where the output from the (𝑘 -1)-th PredAI block is the input to the 𝑘-th PredAI block. Finally, the one-hot vector p ∈ {0, 1} 8 is derived through MLPs over all the reasoning features to determine which option is the correct answer,EQUATIONDiscussion. The proposed HP 2 AI is significantly different from previous reasoners [2, 8, 43] . Its superior performance as shown in the next section is achieved mainly because: 1) We make higherlevel analogy abstraction based on the perceived visual features instead of seeking object-level similarity in most previous methods [2, 8, 43] . 2) To explicitly model the underlying rules embedded in various visual attributes, we transform the problem of analogyinference into a predicting-and-verifying problem. 3) To ensure the embedded rules are identical across rows, a Shared-weight Prediction Network is designed in the analogy-inference blocks. 4) To effectively discriminate essentials from countless possible rules associated with various attributes, the proposed SECA mechanism assigns different weights to different feature channels to signify important attributes and weaken irrelevant ones. 
experiments 4.1 experimental settings: 
 The proposed HP 2 AI is systematically compared with nine stateof-the-art methods, WReN [1] , CoPINet [41] , DCNet [43] , SRAN [14] , MRNet [2] , AlgebraicMR [35] , HCV-ARR [8] , STSN [25] and PredRNet [36] on four publicly available RPM benchmark datasets, PGM (PGM) [1] , original RAVEN (O-RVN) [40] , Impartial-RAVEN (I-RVN) [14] and RAVEN-FAIR (RVN-F) [2] . The key dataset statistics are summarized in Table 1 . Further details of the compared methods and datasets are provided in the supplementary material. We strictly follow the standard evaluation protocol in [1, 40] . The input image size is set to 80 × 80 for RAVENs and PGM. The datasets are split into training, validation, and test sets, where the validation set is utilized to determine the hyper-parameters of the model. No other forms of auxiliary supervision (e.g., metadata [1] ) are incorporated during training. We report the results on Neutral, Interpolation and Extrapolation regimes for the PGM dataset, Table 2 : Comparison with state-of-the-art on the original RAVEN dataset [40] . † indicates that the original method is based on contrasting over candidate answers. The proposed method significantly outperforms all the compared methods over 7 configurations consistently.Accuracy (%) on Different Configurations Avg. '19) [41] 91. 99.4/98.6 100.0/100.0 99.9/99.4 97.4/96.9 99.9/99.9 100.0/99.9 100.0/99.7 98.8/94.2 similar to many previous models [2, 25, 36, 43] . For RAVENs, we adhere to the settings described in their original papers for a fair comparison. Adam optimizer is used with a learning rate of 1e-3 and a batch size of 128. For the RAVENs, the weight decay is set at 1e-5, while for the PGM, it is 1e-7 due to their larger quantity.Center 2x2G 3x3G L-R U-D O-IC O-IG † CoPINet (NIPSOriginal RAVEN Dataset [40] contains a loophole that the correct answer can be derived by simply aggregating the most common properties from the answer options, without examining the question at all, as identified in [2, 14] . We hence conduct evaluations without contrasting over options, following most previous models [2, 8, 36] , and the results are summarized in Table 2 . From Table 2 , we can see that some methods utilizing the contrast information achieve high accuracy, while their performance drops if the contrast information is not utilized, e.g., MRNet [2] and HCV-ARR [8] . In contrast, the proposed method reaches the mean accuracy of 98.8% without any auxiliary contrast information over options, which outperforms the second-best method, PredRNet, by 3.0%, and it is even superior to the methods using the contrast information. The proposed method significantly outperforms all the compared methods for all configurations consistently. Specifically, it achieves significant improvements on complicated configurations where high-level spatial positions are considered, e.g., 2x2G, 3x3G and O-IG, which demonstrates the effectiveness of the Patch Attention blocks in visual perception for global feature extraction. The proposed PredAI module effectively models the underlying relations following the designed analogy inference mechanism, which yields one shared rationale for each attribute across rows, and discovers a wide range of reasoning rationales for better reasoning. I-RAVEN Dataset [14] has been developed to eliminate the loophole in the original RAVEN dataset [40] . Table 3 summarizes the reasoning accuracy on the I-RAVEN dataset [14] . The proposed HP 2 AI significantly outperforms the state-of-the-art methods on the I-RAVEN dataset, achieving an average reasoning accuracy of 99.4%, with 100.0% perfect or 99.9% near-perfect over five out of the seven problem configurations. Compared with the second best method, PredRNet [36] , the superiority is 2.9% on average, with massive improvements on those complicated configurations, e.g., 6.2% on 3x3G and 11.1% on O-IG. All these results are achieved by hierarchically perceiving RPM images in multiple receptive fields and the effectiveness of the proposed PredAI blocks in discovering high-level abstractions of embedded analogies.RAVEN-FAIR Dataset [2] has been developed to avoid the loophole in the original RAVEN dataset by generating option images with more randomness. The results of comparison to state-of-theart methods are summarized in Table 3 . The proposed HP 2 AI achieves an average reasoning accuracy of 98.6% on the RAVEN-FAIR dataset [2] , which consistently outperforms all the compared methods, except that for O-IC configuration it achieves the same accuracy of 99.7% as the second best method. Compared with the second best method PredRNet [36] , the performance gains on challenging configurations are most significant, e.g., 2.1% on 2x2G, 4.3% on 3x3G and 3.0% on O-IG.In summary, the proposed method shows significant performance gains on all three RAVEN datasets consistently, particularly on complex 2x2G, 3x3G and O-IG configurations involving spatial relations, which confirm the positive contribution of the proposed HPALC and PredAI modules. PGM Dataset [1] has been widely used for evaluating the capability of abstract reasoning [2, 14, 25, 36, 43] . The proposed method is compared with state-of-the-art models on the PGM dataset in Table 4 . Some methods such as CoPINet [41] and DCNet [43] that perform satisfactorily on the RAVENs may not perform well on the PGM dataset. This is mainly because these models adopt RAVEN-specific inductive biases rather than investigate the core of analogical reasoning, and hence have poor generalization abilities. The proposed method achieves 99.3% reasoning accuracy, which significantly outperforms the 98.2% accuracy of the previous best method STSN [25] and the 97.4% accuracy of the second best method PredRNet [36] on the Neutral regime. On the other two OOD regimes, the proposed method also demonstrates good generalization ability by improving the previous best method STSN [25] by 1.5% and 2.2%, respectively. The PGM dataset is 20 times larger than RAVENs, and the rules are applied either in rows or in columns. Despite these additional constraints, our method is able to obtain a significant gain over a rather high accuracy of 98.2%, thanks to the hierarchical design in HPALC for visual perception from multiple receptive fields and the predicting-and-verifying design in the PredAI reasoning blocks.Ablation Studies of Major Components. We ablate the two main modules of the proposed HP 2 AI on three RAVEN-style datasets [2, 14, 40] . We select three representative models such as MRNet [2] , HCV-ARR [8] and PredRNet [36] as the baselines. We disentangle the perception module and reasoning module for the baselines, and substitute the respective module by the proposed one. Specifically, we ablate the proposed HP 2 AI using perception modules Multi-Scale Encoder (MSE) from MRNet [2] , Hierarchical ConViT (HCV) from HCV-ARR [8] and 4-Block ResNet (RN-4B) from Pre-dRNet [36] . We also ablate the proposed HP 2 AI using reasoning modules Relation Module with Pattern Module (RM+PM) from MRNet [2] , Attention-based Relation Reasoner (ARR) from HCV-ARR [8] and Predictive Reasoning Block (PRB) from PredRNet [36] .The results are summarized in Table 5 . We can observe the consistent performance gain brought by the two proposed modules over the baselines. The average gains over three datasets brought by the proposed HPALC and the proposed PredAI are 1.3% and  
conclusion: 
 In this work, a novel HP 2 AI network is proposed to solve Raven's Progressive Matrices, one of the most frequently-used assessments for human's reasoning capabilities. The proposed HPALC module simultaneously extracts the fine image details and global image semantics through the hierarchical Patch Attention and Local Context blocks across multi-level receptive fields. Instead of detecting object similarity, the proposed PredAI module models and exploits the inherent analogical reasoning rules embedded in RPM questions. We also propose the Squeeze-and-Excitation Channel-wise Attention to discriminate the importance of feature channels and hence to control their contribution to the final decision. Extensive comparisons with nine state-of-the-art models on four benchmark datasets demonstrate that the proposed HP 2 AI significantly outperforms all state-of-the-art models consistently in all configurations and on all four datasets.  
 
 Appendix: Without losing generality, only row-wise relations are illustrated but column-wise relations are used in experiments whenever necessary. 


Decision: This paper obtained two borderline accepts and one borderline reject in the final evaluation. However, the borderline reject reviewer is not against the acceptance of this paper given the improvements in the presentation of hypotheses and solutions can be made in the camera-ready version. 

The AC would like to accept this paper for its contribution to a novel network of abstract visual reasoning and its strong performance shown in the experiments. The AC urges the authors to make more efforts on editing the paper for a clearer presentation of hypotheses and solutions as suggested by the third reviewer.

Reviews: review: paper_summary: paper_summary: The author proposed $HP^2AI$, a method that targets analogical visual reasoning problems. The proposed method contains two different modules: the visual perception module and the analogical reasoning module. Specifically, the visual perception module can not only capture low-level features like type, size, and color but also capture high-level features like position, and number. The analogical reasoning module uses the first two images to predict the third image and optimize the model. Experiment results show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well-organized and easy to follow.
2. The paper provides comprehensive experiment results to prove the effectiveness of the proposed method. 
3. The idea of the paper is straightforward and follows the intuition of humans.
limitations: limitations: 1. According to the author's claim, at shallower stages, the visual perception module should focus on low-level features like shape, and color. However, as shown in Figure 4, the t-SNE results show that the features of HPALC are already unique for different configurations.  In my understanding, the low-level feature should be similar in these configurations, why the t-SNE results are unique in these configurations?
2. In the HPALC, in my understanding, the author used the addition operation to fuse the $F^j$ and $Z^j$, could you use the concatenate operation to fuse these two features and provide the quantitative results?
3. As shown in Figure 5, looks like the verification error keeps decreasing for 100 epochs, whether the model can get higher accuracy if keep training?
paper_summary: paper_summary: The paper introduces HP2AI, a Hierarchical Perception and Predictive Analogy-Inference network designed for abstract visual reasoning tasks like Raven’s Progressive Matrices (RPMs). HP2AI uses a perceptual encoder with Patch Attention and Local Context blocks to capture local and global image features, and a Predictive Analogy-Inference module to model analogical reasoning rules. Experiments on four RPM benchmark datasets show that HP2AI outperforms current state-of-the-art methods in accuracy.
strength_weakness: strength_weakness: 1. HP2AI consistently outperforms existing models across multiple RPM benchmark datasets, indicating its strong reasoning capabilities.
2. The Predictive Analogy-Inference module enhances the model’s ability to understand and apply analogical reasoning, moving beyond simple feature similarity.
3. The model demonstrates strong performance in different test regimes, including neutral, interpolation, and extrapolation settings.
limitations: limitations: 1. While the model shows strong performance on the tested RPM datasets, its scalability to larger and more diverse datasets is not thoroughly explored. Real-world applications often involve more varied and complex data, and it is unclear if the HP2AI model can maintain its performance in these scenarios without significant modifications.
2. The paper provides limited details on the implementation aspects, such as hyperparameter settings, training durations, and computational requirements.
3. The strong performance of HP2AI on the benchmark datasets raises concerns about potential overfitting. The model might be overfitting to the specific characteristics of the RPM datasets rather than learning generalizable reasoning skills. This is particularly concerning given the model's high accuracy scores, which might not translate to unseen data or different types of visual reasoning tasks.
4. The effectiveness of the HP2AI model is heavily dependent on the quality and structure of the training data. If the training data is not well-curated or if it lacks diversity, the model's performance could degrade significantly.
paper_summary: paper_summary: The paper describes a model to solve Raven’s Progressive Matrices (RPMs) problems, in which the model must pick the correct image out of a pool of answers that completes the third row of a 3x3 image matrix, following the rules that can be deducted from the first two rows. 

The main contributions seem architectural and specifically tailored to improve performance on RPM-related datasets and comprise custom modules for feature extraction (HPALC) and reasoning (PredAI).
strength_weakness: strength_weakness: Results of experiments show an improved performance on multiple datasets.
limitations: limitations: The main limitation, in my opinion, is that the work is out of scope for the Multimedia conference. Although the MM community has thrived on progress in neural processing pipelines, the work is not multimodal and does not tackle multimedia. Indeed, the state-of-the-art methods the authors compared with have been presented to neural processing/computer vision/general AI/ML venues.  

Moreover, the presentation of the proposed architecture should be made clearer, as there are many modules and acronyms that the reader should keep in mind, each with its specific function and peculiarities and most containing custom modifications. This makes it very difficult to discern which component is a reused one or an incremental change of an existing one (e.g., HPALC seems to me like a two-branch parallel ViT-ResNet architecture) and which is an original formulation. Some proposed modules in the architecture are justified in the text but not rigorously tested; e.g., the authors introduced SECA to tackle a particular challenge discussed in L484-493. How did the authors quantitatively measure that the introduced mechanism solved or mitigated that hypothesized issue? 

For this reason, the overall architecture feels like a bag of tricks guided by performance and inductive biases rather than hypothesis testing. Thus, I did not see clear takeaway messages in this study. I perceived most of the design choices justified as “because it works" without taking the opportunity to analyze “why it works”.