Effective Abstract Reasoning with Dual-Contrast Network

EFFECTIVE ABSTRACT REASONING WITH DUAL-CONTRAST NETWORK Tao Zhuo , Mohan Kankanhalli School of Computing , National University of Singapore zhuotao @ nus.edu.sg , mohan @ comp.nus.edu.sg ABSTRACT As a step towards improving the abstract reasoning capability of machines , we aim to solve Raven ’ s Progressive Matrices ( RPM ) with neural networks , since solving RPM puzzles is highly correlated with human intelligence . Unlike previous methods that use auxiliary annotations or assume hidden rules to produce appropriate feature representation , we only use the ground truth answer of each question for model learning , aiming for an intelligent agent to have a strong learning capability with a small amount of supervision . Based on the RPM problem formulation , the correct answer filled into the missing entry of the third row/column has to best satisfy the same rules shared between the first two rows/columns . Thus we design a simple yet effective Dual-Contrast Network ( DCNet ) to exploit the inherent structure of RPM puzzles . Specifically , a rule contrast module is designed to compare the latent rules between the filled row/column and the first two rows/columns ; a choice contrast module is designed to increase the relative differences between candidate choices . Experimental results on the RAVEN and PGM datasets show that DCNet outperforms the state-of-the-art methods by a large margin of 5.77 % . Further experiments on few training samples and model generalization also show the effectiveness of DCNet . Code is available at https : //github.com/visiontao/dcnet . 1 INTRODUCTION . Abstract reasoning capability is a critical component of human intelligence , which relates to the ability of understanding and interpreting patterns , and further solving problems . Recently , as a step towards improving the abstract reasoning ability of machines , many methods ( Santoro et al. , 2018 ; Zhang et al. , 2019a ; b ; Zheng et al. , 2019 ; Zhuo & Kankanhalli , 2020 ) are developed to solve Raven ’ s Progress Matrices ( RPM ) ( Domino & Domino , 2006 ; Raven & Court , 1938 ) , since it is widely believed that RPM lies at the heart of human intelligence . As the example shown in Figure 1 , given a 3× 3 problem matrix with a final missing piece , the test taker has to find the logical rules shared between the first two rows or columns , and then pick the correct answer from 8 candidate choices to best complete the matrix . Since the logical rules hidden in RPM questions are complex and unknown , solving RPM with machines remains a challenging task . As described in ( Carpenter et al. , 1990 ) , the logical rules applied in a RPM question are manifested as visual structures . For a single image in the question , the logical rules could consist of several basic attributes , e.g. , shape , color , size , number , and position . For the images in a row or column , the logical rules could be applied row-wise or column-wise and formulated with an unknown relationship , e.g. , AND , OR , XOR , and so on ( Santoro et al. , 2018 ; Zhang et al. , 2019a ) . If we can extract the explicit rules of each question , the problem can be easily solved by using a heuristicsbased search method ( Zhang et al. , 2019a ) . However , given an arbitrary RPM question , the logical rules are unknown . What ’ s worse - even the number of rules is unknown . As a result , an intelligent machine needs to simultaneously learn the representation of these hidden rules and find the correct answer to satisfy all of the applied rules . With the success of deep learning in computer vision , solving RPM puzzles with neural networks has become popular . Because the learned features might be inconsistent with the logical rules , many supervised learning methods , e.g. , DRT ( Zhang et al. , 2019a ) , WReN ( Santoro et al. , 2018 ) and LEN ( Zheng et al. , 2019 ) , MXGNet ( Wang et al. , 2020 ) and ACL ( Kim et al. , 2020 ) , not only use the ground truth answer of each RPM question but also the auxiliary annotations ( such as logical rules with shape , size , color , number , AND , OR , XOR ) to learn the appropriate feature representation . Although auxiliary annotations provide lots of priors about how the logical rules are applied , noticeable performance improvement is not always obtained on different RPM problems , such as in the results reported in Table 1 . Moreover , such a learning strategy requires additional supervision . When auxiliary annotations are not available , it will fail to boost the performance . For example , DRT ( Zhang et al. , 2019a ) can not be applied to PGM dataset ( Santoro et al. , 2018 ) for the lack of structure annotations . To overcome the constraint of using auxiliary annotations , a recent method CoPINet ( Zhang et al. , 2019b ) only uses the ground truth answer of each question . Meanwhile , to produce the feature representation of hidden rules , CoPINet assumes there are at most N attributes in each problem , and each of which is subject to the governance of M rules . However , due to N and M being unknown for arbitrary RPM problems , such an assumption is still too strong . In this work , we aim to learn the abstract reasoning model by using only the ground truth answer of each question , and there is not any assumption about the latent rules . According to the RPM problem formulation ( Carpenter et al. , 1990 ) , it can be concluded that finding the correct answer of a RPM puzzle mainly depends on two contrasts of rules : ( 1 ) compare the hidden rules between the filled row/column and the first two rows/columns to check whether they are the same rules ; ( 2 ) compare all candidate choices to check which one best satisfies the hidden rules . Unlike making specific assumptions about latent rules that are only valid for particular cases , the above two contrasts are general properties of all RPM problems . Considering above two contrasts , we propose a simple yet effective Dual-Contrast Network ( DCNet ) to solve RPM problems . Specifically , a rule contrast module is used to compute the difference between the filled row/column and the first two rows/columns , which checks the difference between latent rules . Additionally , the second choice contrast module is used to increase the relative differences of all candidate choices , which helps find the correct answer when confusingly similar choices exist . Experiments on two major benchmark datasets demonstrate the effectiveness of our method . In summary , our main contributions are as follows : • We propose a new abstract reasoning model on RPM with only ground truth answers , i.e . there are not any assumptions or auxiliary annotations about the latent rules . Compared to previous methods , the problem setting of our method is more challenging , as we aim for an intelligent agent to learn a strong model with a small amount of supervision . • We propose a simple yet effective Dual-Contrast Network ( DCNet ) that consists of a rule contrast module and a choice contrast module . By exploiting the inherent structures of each RPM with basic problem formulation , robust feature representation can be learned . • Experimental results on RAVEN ( Zhang et al. , 2019a ) and PGM ( Santoro et al. , 2018 ) datasets show that our DCNet significantly improves the average accuracy by a large margin of 5.77 % . Moreover , from the perspective of few-shot learning , DCNet outperforms the state-of-the-art method CoPINet ( Zhang et al. , 2019b ) by a noticeable margin when few training samples are provided , see Table 3 and 4 . Further experiments on model generalization also show the effectiveness of our method , see Table 5 and 6 . 2 RELATED WORK . 2.1 VISUAL REASONING . One of the most popular visual reasoning tasks is Visual Question Answering ( VQA ) ( Antol et al. , 2015 ; Zellers et al. , 2019 ; Fan et al. , 2020 ) . To aid in the diagnostic evaluation of VQA systems , Johnson et al . ( 2017 ) designed a CLEVR dataset by minimizing bias and providing rich ground-truth representations for both images and questions . It is expected that rich diagnostics could help better understand the visual reasoning capabilities of VQA systems . Recently , to understand the human actions in videos , Zhou et al . ( 2018 ) proposed a temporal relational reasoning network to learn and reason about temporal dependencies between video frames at multiple time scales . Besides , for explainable video action reasoning , Zhuo et al . ( 2019 ) proposed to explain performed actions by recognizing the semantic-level state changes from a spatio-temporal video graph with pre-defined rules . Different from these visual tasks , solving RPM puzzles depends on sophisticated logical rules acquiring human intelligence ( Raven & Court , 1938 ; Santoro et al. , 2018 ; Zhang et al. , 2019a ) . Therefore , it is expected that solving RPM problems with machines could help better understand and perhaps improve the abstract reasoning ability of contemporary computer vision systems . 2.2 COMPUTATIONAL MODELS ON RPM . The early methods on RPM ( McGreggor & Goel , 2014 ; McGreggor et al. , 2014 ; Mekik et al. , 2018 ) often compute the feature similarity of images with a hand-crafted representation . Besides , structural affinity ( Shegheva & Goel , 2018 ) with graphical models are also used in RPM problems . Due to the lack of large-scale dataset , the early methods are only evaluated on a small dataset . Recently , with the introduction of deep learning in pattern recognition ( Deng et al. , 2009 ) , solving RPM with deep neural networks has becomes the main approach . For automatic RPM generation , Wang & Su ( 2015 ) introduced an abstract representation method for RPM by using the first-order logic formulae , and they applied three categories of relations ( i.e . unary , binary , and ternary ) . Hoshen & Werman ( 2017 ) first trained a CNN model to measure the IQ of neural networks in a simplistic evaluation environment . Santoro et al . ( 2018 ) measured the abstract reasoning capability of machines with several popular neural networks , such as LSTM , ResNet and Wild-ResNet . Besides , Zhuo & Kankanhalli ( 2020 ) studied the effect of ImageNet pre-training ( Deng et al. , 2009 ) for RPM problems and proposed a novel unsupervised abstract reasoning method by designing a pseudo target . To further improve the feature representation with auxiliary annotations , Santoro et al . ( 2018 ) proposed a WReN network that formulates pair-wise relations between the problem matrix and each individual choice in embedding space , independent of the other choices . Zhang et al . ( 2019a ) generated a RAVEN dataset with the structure annotations for each RPM problem and proposed a Dynamic Residual Tree ( DRT ) method . In addition , Zheng et al . ( 2019 ) proposed a Logic Embedding Network ( LEN ) with distracting features , which also uses the auxiliary annotations to boost the performance and designed a teacher model to control the learning trajectories . MXGNet ( Wang et al. , 2020 ) is a multi-layer graph neural network for multi-panel diagrammatic reasoning tasks . For better performance , MXGNet also uses auxiliary annotations for model training . ACL ( Kim et al. , 2020 ) uses symbolic labels to generate new RPMs with the same problem semantics but with different attributes ( e.g. , shapes and colors ) . Discarding the use of auxiliary annotations , CoPINet ( Zhang et al. , 2019b ) only utilizes the ground truth answer of each RPM question . By combining contrasting , perceptual inference , and permutation invariance , CoPINet achieves the state-of-the-art performance . However , it assumes the maximum number of attributes and rules is known . Different from the “ objective-level contrast ” in ( Zhang et al. , 2019b ) that treats the complete problem matrix ( 9 images , including a filling choice ) as an object for comparison , our model consists of a rule contrast module and a choice contrast module and we only compare the features of different rows/columns ( 3 images ) . Thus our method is more effective to exploit the inherent structure of RPM on rows/columns . Moreover , no assumptions about attributes and number of rules are made .

Summary: The paper proposes a neural network based approach called Dual-Contrast Network (DCNet) to solve Raven’s Progressive Matrices (RPM). The approach consists of a rule contrast module that compares the latent rules between the unfilled (third) row/column and the filled (first and second) rows/columns, a choice contrast module that helps in picking the correct choice among the given eight choices, and finally uses a 2-layer MLP to predict scores for the choices. Different from previous approaches, the only supervision used in the proposed approach is the ground-truth choice. The approach achieves state-of-the-art performance on RAVEN and PGM datasets.