Learning Algebraic Representation for Abstract Spatial-Temporal Reasoning

1 INTRODUCTION . “ Thought is in fact a kind of Algebra. ” —William James ( James , 1891 ) Imagine you are given two alphabetical sequences of “ c , b , a ” and “ d , c , b ” , and asked to fill in the missing element in “ e , d , ? ” . In nearly no time will one realize the answer to be c. However , more surprising for human learning is that , effortlessly and instantaneously , we can “ freely generalize ” ( Marcus , 2001 ) the solution to any partial consecutive ordered sequences . While believed to be innate in early development for human infants ( Marcus et al. , 1999 ) , such systematic generalizability has constantly been missing and proven to be particularly challenging in existing connectionist models ( Lake & Baroni , 2018 ; Bahdanau et al. , 2019 ) . In fact , such an ability to entertain a given thought and semantically related contents strongly implies an abstract algebra-like treatment ( Fodor et al. , 1988 ) ; in literature , it is referred to as the “ language of thought ” ( Fodor , 1975 ) , “ physical symbol system ” ( Newell , 1980 ) , and “ algebraic mind ” ( Marcus , 2001 ) . However , in stark contrast , existing connectionist models tend only to capture statistical correlation ( Lake & Baroni , 2018 ; Kansky et al. , 2017 ; Chollet , 2019 ) , rather than providing any account for a structural inductive bias where systematic algebra can be carried out to facilitate generalization . This contrast instinctively raises a question—what constitutes such an algebraic inductive bias ? We argue that the foundation of the modeling counterpart to the algebraic treatment in early human development ( Marcus , 2001 ; Marcus et al. , 1999 ) lies in algebraic computations set up on mathematical axioms , a form of formalized human intuition and the starting point of modern mathematical reasoning ( Heath et al. , 1956 ; Maddy , 1988 ) . Of particular importance to the basic building blocks of algebra is the Peano Axiom ( Peano , 1889 ) . In the Peano Axiom , the essential components of algebra , the algebraic set and corresponding operators over it , are governed by three statements : ( 1 ) the existence of at least one element in the field to study ( “ zero ” element ) , ( 2 ) a successor function that is recursively applied to all elements and can , therefore , span the entire field , and ( 3 ) the principle of mathematical induction . Building on such a fundamental axiom , we begin to form the notion of an algebraic set and induce the operator along with it to construct an algebraic structure . We hypothesize that such a treatment of algebraic computations set up on fundamental axioms is essential for a model ’ s systematic generalizability , the lack of which will only make it sub-optimal . To demonstrate the benefits of such an algebraic treatment in systematic generalization , we showcase a prototype for Raven ’ s Progressive Matrices ( RPM ) ( Raven , 1936 ; Raven & Court , 1998 ) , an exemplar task for abstract spatial-temporal reasoning ( Santoro et al. , 2018 ; Zhang et al. , 2019a ) . In this task , an agent is given an incomplete 3ˆ3 matrix consisting of eight context panels with the last one missing , and asked to pick one answer from a set of eight choices that best completes the matrix . Human ’ s reasoning capability of solving this abstract reasoning task has been commonly regarded as an indicator of “ general intelligence ” ( Carpenter et al. , 1990 ) and “ fluid intelligence ” ( Spearman , 1923 ; 1927 ; Hofstadter , 1995 ; Jaeggi et al. , 2008 ) . In spite of the task being one that ideally requires abstraction , algebraization , induction , and generalization ( Raven , 1936 ; Raven & Court , 1998 ; Carpenter et al. , 1990 ) , recent endeavors unanimously propose pure connectionist models that attempt to circumvent such intrinsic cognitive requirements ( Santoro et al. , 2018 ; Zhang et al. , 2019a ; b ; Wang et al. , 2020 ; Zheng et al. , 2019 ; Hu et al. , 2020 ; Wu et al. , 2020 ) . However , these methods ’ inefficiency is also evident in systematic generalization ; they struggle to extrapolate to domains beyond training , as pointed out in ( Santoro et al. , 2018 ; Zhang et al. , 2019b ) and shown later in this paper . To address the issue , we introduce the ALgebra-Aware Neuro-Semi-Symbolic ( ALANS2 ) learner . At a high-level , the ALANS2 learner is embedded in a general neuro-symbolic architecture ( Yi et al. , 2018 ; Mao et al. , 2019 ; Han et al. , 2019 ; Yi et al. , 2020 ) but has on-the-fly operator learnability and hence semi-symbolic . Specifically , it consists of a neural visual perception frontend and an algebraic abstract reasoning backend . For each RPM instance , the neural visual perception frontend first slides a window over each panel to obtain the object-based representations ( Kansky et al. , 2017 ; Wu et al. , 2017 ) for every object . A belief inference engine latter aggregates all object-based representations in each panel to produce the probabilistic belief state . The algebraic abstract reasoning backend then takes the belief states of the eight context panels , treats them as snapshots on an algebraic structure , lifts them into a matrix-based algebraic representation built on the Peano Axiom and the representation theory ( Humphreys , 2012 ) , and induces the hidden operator in the algebraic structure by solving an inner optimization ( Colson et al. , 2007 ; Bard , 2013 ) . The algebraic representation for the answer is predicted by executing the induced operator : its corresponding set element is decoded by isomorphism established in the representation theory , and the final answer is selected as the one most similar to the prediction . The ALANS2 learner enjoys several benefits in abstract reasoning with an algebraic treatment : 1 . Unlike previous monolithic models , the ALANS2 learner offers a more interpretable account of the entire abstract reasoning process : the neural visual perception frontend extracts object-based representations and produces belief states of panels by explicit probability inference , whereas the algebraic abstract reasoning backend induces the hidden operator in the algebraic structure . The corresponding representation for the final answer is obtained by executing the induced operator , and the choice panel with minimum distance is selected . This process much resembles the topdown bottom-up strategy in human reasoning : humans reason by inducing the hidden relation , executing it to generate a feasible solution in mind , and choosing the most similar answer available ( Carpenter et al. , 1990 ) . Such a strategy is missing in recent literature ( Santoro et al. , 2018 ; Zhang et al. , 2019a ; b ; Wang et al. , 2020 ; Zheng et al. , 2019 ; Hu et al. , 2020 ; Wu et al. , 2020 ) . 2 . While keeping the semantic interpretability and end-to-end trainability in existing neurosymbolic frameworks ( Yi et al. , 2018 ; Mao et al. , 2019 ; Han et al. , 2019 ; Yi et al. , 2020 ) , ALANS2 is what we call semi-symbolic in the sense that the symbolic operator can be learned and concluded on-the-fly without manual definition for every one of them . Such an inductive ability also enables a greater extent of the desired generalizability . 3 . By decoding the predicted representation in the algebraic structure , we can also generate an answer that satisfies the hidden relation in the context . This work makes three major contributions : ( 1 ) We propose the ALANS2 learner . Compared to existing monolithic models , the ALANS2 learner adopts a neuro-semi-symbolic design , where the problem-solving process is decomposed into neural visual perception and algebraic abstract reasoning . ( 2 ) To demonstrate the efficacy of incorporating an algebraic treatment in abstract spatialtemporal reasoning , we show the superior systematic generalization ability of the proposed ALANS2 learner in various extrapolatory RPM domains . ( 3 ) We present analyses into both neural visual perception and algebraic abstract reasoning . We also show the generative potential of ALANS2 . 2 RELATED WORK . Quest for Symbolized Manipulation The idea to treat thinking as a mental language can be dated back to Augustine ( Augustine , 1876 ; Wittgenstein , 1953 ) . Since the 1970s , this school of thought has undergone a dramatic revival as the quest for a symbolized manipulation in cognitive modeling , such as “ language of thought ” ( Fodor , 1975 ) , “ physical symbol system ” ( Newell , 1980 ) , and “ algebraic mind ” ( Marcus , 2001 ) . In their study , connectionist ’ s task-specific superiority and inability to generalize beyond training ( Kansky et al. , 2017 ; Chollet , 2019 ; Santoro et al. , 2018 ; Zhang et al. , 2019a ) have been hypothetically linked to a lack of such symbolized algebraic manipulation ( Lake & Baroni , 2018 ; Chollet , 2019 ; Marcus , 2020 ) . With evidence that an algebraic treatment adopted in early human development ( Marcus et al. , 1999 ) can potentially address the issue ( Bahdanau et al. , 2019 ; Mao et al. , 2019 ; Marcus , 2020 ) , classicist ( Fodor et al. , 1988 ) approaches for generalizable reasoning used in programs ( McCarthy , 1960 ) and blocks world ( Winograd , 1971 ) have resurrected . As a hybrid approach to bridge connectionist and classicist , recent developments lead to neuro-symbolic architectures . In particular , Yi et al . ( 2018 ) demonstrate a neuro-symbolic prototype for visual question answering , where a perception module and a language parsing module are separately trained , and the predefined logic operators associated with language tokens are chained to process the visual information . Mao et al . ( 2019 ) soften the predefined operators to afford end-to-end training with only question answers . Han et al . ( 2019 ) and Yi et al . ( 2020 ) use the hybrid architecture for metaconcept learning and temporal causal learning , respectively . ALANS2 follows the classicist ’ s call but adopts a neuro-semi-symbolic architecture : it is end-to-end trainable as opposed to Yi et al . ( 2018 ; 2020 ) and the operator can be learned and concluded on-the-fly without manual specification ( Yi et al. , 2018 ; Mao et al. , 2019 ; Han et al. , 2019 ; Yi et al. , 2020 ) . Abstract Visual Reasoning Recent works by Santoro et al . ( 2018 ) and Zhang et al . ( 2019a ) arouse the community ’ s interest in abstract visual reasoning , where the task of Raven ’ s Progressive Matrices ( RPM ) is introduced as such a measure for intelligent agents . Initially proposed as an intelligence quotient test for humans ( Raven , 1936 ; Raven & Court , 1998 ) , RPM is believed to be strongly correlated with human ’ s general intelligence ( Carpenter et al. , 1990 ) and fluid intelligence ( Spearman , 1923 ; 1927 ; Hofstadter , 1995 ; Jaeggi et al. , 2008 ) . Early RPM-solving systems employ symbolic representations based on hand-designed features and assume access to the underlying logics ( Carpenter et al. , 1990 ; Lovett et al. , 2009 ; 2010 ; Lovett & Forbus , 2017 ) . Another stream of research on RPM recruits similarity-based metrics to select the most similar answer from the choices ( Little et al. , 2012 ; McGreggor & Goel , 2014 ; McGreggor et al. , 2014 ; Mekik et al. , 2018 ; Shegheva & Goel , 2018 ) . However , their hand-defined visual features are unable to handle uncertainty from imperfect perception , and directly assuming access to the logic operations simplifies the problem . Recently proposed data-driven approaches arise from the availability of large datasets : Santoro et al . ( 2018 ) extend a pedagogical RPM generation method ( Wang & Su , 2015 ) , whereas Zhang et al . ( 2019a ) use a stochastic image grammar ( Zhu et al. , 2007 ) and introduce structural annotations in it , which Hu et al . ( 2020 ) further refine to avoid shortcut solutions by statistics in candidate panels . Despite the fact that RPM intrinsically requires one to perform abstraction , algebraization , induction , and generalization , existing methods bypass such cognitive requirements using a single feedforward pass in connectionist models : Santoro et al . ( 2018 ) use a relational module ( Santoro et al. , 2017 ) , Steenbrugge et al . ( 2018 ) augment it with a VAE ( Kingma & Welling , 2013 ) , Zhang et al . ( 2019a ) assemble a dynamic tree , Hill et al . ( 2019 ) arrange the data in a contrasting manner , Zhang et al . ( 2019b ) propose a contrast module , Zheng et al . ( 2019 ) formulate it in a student-teacher setting , Wang et al . ( 2020 ) build a multiplex graph network , Hu et al . ( 2020 ) aggregate features from a hierarchical decomposition , and Wu et al . ( 2020 ) apply a scattering transformation to learn objects , attributes , and relations . In contrast , ALANS2 attempts to fulfill the cognitive requirements in a neuro-semi-symbolic framework : the perception frontend abstracts out visual information , and the reasoning backend induces the hidden operator in an algebraic structure . 3 THE ALANS2 LEARNER In this section , we introduce the ALANS2 learner for the RPM problem . In each RPM instance , an agent is given an incomplete 3ˆ 3 panel matrix with the last entry missing and asked to induce the operator hidden in the matrix and choose from eight choice panels one that follows it . Formally , let the answer variable be denoted as y , the context panels as tIo , iu8i “ 1 , and choice panels as tIc , iu8i “ 1 . Then the problem can be formulated as estimating P py | tIo , iu8i “ 1 , tIc , iu8i “ 1q . According to the common design ( Santoro et al. , 2018 ; Zhang et al. , 2019a ; Carpenter et al. , 1990 ) , there is one operator that governs each panel attribute . Hence , by assuming independence among attributes , we propose to factorize the probability as P py “ n | tIo , iu8i “ 1 , tIc , iu8i “ 1q9 ź a ÿ T a P pya “ n | T a , tIo , iu8i “ 1 , tIc , iu8i “ 1qP pT a | tIo , iu8i “ 1q , ( 1 ) where ya denotes the answer selection based only on attribute a and T a the operator on a. Overview As shown in Fig . 1 , the ALANS2 learner decomposes the process into perception and reasoning : the neural visual perception frontend extracts the belief states from each of the sixteen panels , whereas the algebraic abstract reasoning backend views an instance as an example in an abstract algebra structure , transforms belief states into algebraic representations by representation theory , induces the hidden operators , and executes the operators to predict the representation of the answer . Therefore , in Eq . ( 1 ) , the operator distribution is modeled by the fitness of an operator and the answer distribution by the distance between the predicted representation and that of a candidate .

Summary: This work proposes a new learner bridging the gap between connectionists and classicists in the task of Raven’s Progressive Matrices (RPM). It relies on a CNN to extract visual features and then uses an algebraic abstract reasoning module to infer the operators of an RPM instance, which allows applying the inferred operator on the RPM instance to predict potential solutions according to various attributes. The most likely solution according to the ensemble of the attributes is then selected as an answer.