Neural Structure Mapping For Learning Abstract Visual Analogies

1 INTRODUCTION . The ability to form abstractions of ‘ concepts ’ from information and then reason about them is central to human intelligence ( Lake et al. , 2015 ) . Abstractions enable humans to quickly learn concepts from few examples , and then reason systematically about new concepts by composing previously understood concepts ( Nam & McClelland , 2021 ) . Over the last decade , artificial neural networks have demonstrated human-level performance in building useful abstractions from data when tested on well-defined and constrained tasks . In computer vision , deep learning ( DL ) models that learn visual abstractions from raw images show strong validation performance on curated test datasets for image recognition ( Krizhevsky et al. , 2012 ) , object detection ( Ren et al. , 2015 ) , and scene classification ( Zhou et al. , 2017 ) . However , unlike humans , DL models struggle with isolating these abstractions and systematically applying them to out of distribution test scenarios ( Greff et al. , 2020 ) . One important way in which humans both build and reason about abstractions is through analogies ( Mitchell , 2021 ) . There are several theories in human cognitive science about how humans perform analogical reasoning . Structure Mapping Theory ( SMT ) posits that perceptual information can be broken down into a domain consisting of objects and attributes , and structural relations between the attributes in the domain ( Gentner , 1983 ) . Consequently , SMT defines an analogy as a mapping between the structural relations across two domains , with no mapping of the attributes . For example , to draw an analogy between the solar system and an atom , we can map the relational struc- ture ( Planet revolves around Sunùñ Electron revolves around Nucleus , Sunmore massive than Planets ùñ Nucleus more massive than Electrons ) . However , the domain attributes , such as the Sun being yellow or hot are not mapped to the Nucleus in drawing an analogy . This idea behind structure mapping was utilized by Hill et al . ( 2019 ) as a prior in training ML models for learning analogies . They first constructed a dataset on learning abstract visual analogies from Raven ’ s progressive matrices ( RPMs ) ( Raven , 2000 ) . The abstract visual analogy task is made up of two rows of five context panels , and four possible candidate panels to complete the visual analogy . The learner must understand the higher-order relation constituted by the first row of three context panels , and then choose the candidate that satisfies the same relation with the two context panels in the bottom row as shown in Figure 1 . Hill et al . ( 2019 ) showed that if the candidates presented during training were curated to maximize differences in structural relations , while being perceptually similar , the models performed more accurately in learning visual analogies . Thus , the authors verifed that incorporating structural differences to maximize the importance of structure mapping in the learning procedure also works for ML models . However , it is not feasible to build or curate datasets to always exploit a prior in the dataset to maximize structure learning . This would involve knowing a combinatorially large number of possible relational structures beforehand and manually selecting data to maximize learning these structures , a requirement that scales poorly with an increase in possible relational structures . To address this issue , we propose a two-stage Neural Structure Mapping ( NSM ) framework to learn abstract visual analogies ( Figure 2 ) . Our first-stage involves a visual relationship encoder that takes as input the visual context from the source domain and predicts the component visual concepts : objects , attributes and relationship in a multi-task manner . We then use the derived relation to dynamically build an analogy inference engine . Similar to structure mapping in humans , our engine utilizes only the relation from the source domain to pick the candidate panel that best fits the analogy . We incorporate compositional reasoning into our approach by using a neural module network ( Andreas et al. , 2016 ; Johnson et al. , 2017b ; Csordás et al. , 2021 ) as our engine . Our modular neural engine dynamically adapt its structure to align ( Xu et al. , 2020 ) with the structural relationship , thus explicitly incorporating the structure mapping process in our NSM model . We train and evaluate our approach on five different generalization splits of the visual analogy dataset . These splits were proposed by Hill et al . ( 2019 ) to test systematic generalization ( Bahdanau et al. , 2018 ) across object and attribute values in learning visual analogies . We show that our relationship encoder achieves a high degree of accuracy in isolating the relationship from the source domain , and , as expected , there is no performance drop across generalization splits or training regimes . For our analogy inference engine , we empirically show that it explicitly captures the structure mapping process , while generalizing systematically in learning the structural relation across visual domains . 2 RELATED WORK . Learning Analogies Gentner ( 1983 ) ’ s SMT defines an analogy as a comparison in which relational predicates , but few or no object attributes , can be mapped from base to target domains . Symbolic models based on SMT ( Falkenhainer et al. , 1986 ) , including those designed for RPMs ( Lovett & Forbus , 2017 ) , rely on first extracting a rule based representation of the domains from the perceptual input . Other cognitive analogy models based on Active Symbol theory ( Hofstadter & Mitchell , 1994 ; Mitchell , 1993 ) or High-Level Perception theory ( Chalmers et al. , 1992 ) do not separate the structure extraction and mapping process , nor rely on mapping the structure syntax across domains . Our approach is not directly related to these models . Hill et al . ( 2019 ) introduced the dataset for learning analogies from RPMs and utilized several neural network architectures to directly learn from visual data . Their key contribution was to introduce the SMT prior into the dataset during candidate selection . Webb et al . ( 2020 ) introduced Temporal Context Normalization to explicitly learn visual features that can support extrapolation , and tested it on the Visual Analogy Extrapolation Challenge ( VAEC ) dataset . Both of these approaches are complementary to ours . Other DL approaches most similar to ours are the Part-Composition Model ( Ichien et al. , 2021 ) and Chen et al . ( 2019 ) ’ s approach . However , unlike our model , these models rely on availability of very structured intermediate representations like semantic segmentation maps , and do not use the extracted structural representations to explicitly configure the reasoning model . Abstract Reasoning Hill et al . ( 2019 ) ’ s dataset is directly derived from the Procedurally Generated Matrices ( PGM ) dataset introduced by Barrett et al . ( 2018 ) to test the ability of neural networks to perform abstract reasoning on RPM problems . RPM tests are an important measure of fluid intelligence in humans ( Raven , 2000 ) , and the PGM dataset provided a sufficiently large sample size for training neural networks on this problem . Following this work , two new RPM datasets were also released : RAVEN ( Zhang et al. , 2019 ) , which utilized additional rules and structured rule annotations , and V-PROM ( Teney et al. , 2020 ) , which utilized real images . DL approaches to RPM tasks can be roughly categorized into three types . The first is relation learning approaches , such as WReN ( Barrett et al. , 2018 ) , which models all pairwise relationships between the matrix panels using relation networks ( Santoro et al. , 2017 ) , and MXGNet ( Wang et al. , 2020 ) , which learns row-based node embeddings and then performs a graph classification on the resulting candidate graphs . Second is rule learning approaches , such as DRT ( Zhang et al. , 2019 ) , that learns a structured Stochastic Image Grammar of the abstract rules , and LEN ( Zheng et al. , 2019 ) that utilizes a logic embedding network along with a curriculum to learn with increasingly distracting features . Third is object-centric learning , such as Rel-AIR ( Spratley et al. , 2020 ) , that obtains object embeddings generated via Attend-Infer-Repeat for each panel , and Pekar et al . ( 2020 ) ’ s method that utilizes both Variational AutoEncoders and an adversarial loss for candidate generation . Our model can be broadly fit into rule learning by way of the encoder , followed by relation learning via the analogy inference engine . However , our general approach towards explicitly extracting structure and mapping it is complimentary to these ideas and can be used in conjunction with several of them . Systematic Generalization Systematic generalization refers to the ability to generalize to novel concepts ( out of distribution data ) by understanding them as compositions of known concepts ( Bahdanau et al. , 2018 ) . It is underpinned by three important characteristics : ( 1 ) systematicity , the ability to generalize to semantically related concepts ; ( 2 ) productivity , the ability to iteratively apply compounding to generalize from constituent concepts to their recurrence ; and ( 3 ) localism , the ability to iteratively apply reductionism to generalize from repeated constituent concepts to the singular . The ability to generalize systematically has been previously explored in human cognition ( Fodor & Pylyshyn , 1988 ) , natural language processing models ( Lake & Baroni , 2018 ) , and languagegrounded computer vision models ( Agrawal et al. , 2017 ) . It has its origins in human cognition and was formalized by Fodor ( 1975 ) under the ‘ Language of Thought ’ hypothesis . Cognitive scientists have expressed reservations about the ability of connectionist models to generalize systematicitally ( Fodor & Pylyshyn , 1988 ; Marcus , 2019 ) . Investigation of neural language models showed that systematic generalization still poses a challenge for DL ( Lake & Baroni , 2018 ; Loula et al. , 2018 ) . A key reason deep neural networks lack systematicity is due to their propensity towards ‘ shortcut learning ’ ( Geirhos et al. , 2018 ; 2020 ) . This suggests that DL models rely on exploiting a few number of predictive features instead of considering all possible information about the data while drawing conclusions . Consequently , this leads to poor generalization when the key predictive features are changed , even though the central evidence remains the same . For example , DL approaches to solving RPM problems were discovered to rely on evaluating the mode across all candidates in the RAVEN dataset and a significant drop in generalization performance was observed when the models were tested without the shortcut on the RAVEN-Fair dataset ( Spratley et al. , 2020 ) . On the other hand , neural module networks ( Andreas et al. , 2016 ; Johnson et al. , 2017b ) generalized well systematically when their layout aligned well with specific language-grounded reasoning problems ( Bahdanau et al. , 2018 ) . Hence , we utilize a modular approach in building our analogy inference engine , which in turn utilizes the relationship structure inferred by our encoder to automatically align itself to the problem . This is a form of conditional computation . 3 PROBLEM SETUP . The abstract visual analogy task ( Hill et al. , 2019 ) falls under the umbrella of Raven ’ s Progressive Matrices ( Raven , 2000 ) designed to test abstract reasoning and fluid intelligence in human and artificial agents . Each question consists of five context panels P 1´5con , and four candidate panels P 1´4can . The context panels are arranged in a matrix with two rows of three columns each , with the final panel in the second row missing . The first row of panels P 1´3con express a semantically related triplet R “ to , a , ru , that is composed of two lower-order perceptual visual concepts : object o and attribute a , and one higher-order semantic visual concept : relationship r. The objective is to choose a candidate c P Ť4 i “ 1tP icanu that , upon substitution in place of the missing panel , represents the same relationship as the first row of panels . The object ( o P { line , shape } ) and attribute ( a P { quantity , colour , type , size , position } ) together constitute the visual domain d of the triplet R. Each possible attribute can take ten values vpaq P t1 , 2 , ... , 9 , 10u ( normalized ) . The domain of the first row of panels P 1´3con defines the source domain dsource of the analogy , and the domain of the second row of panels tP 4´5con , cu defines the target domain dtarget . The dataset consists of seven unique possible domains such that d P { shape quantity , shape colour , shape type , shape size , shape position , line type , line colour } . To test the systematic generalization in learning visual analogies , the dataset has five different generalization splits that require the ability to recognize the relationship across both novel domain d and attribute a values : • Novel Domain Transfer : The training set consists of 42 ordered pairs of dsource and dtarget while the remaining 7 ( 7 * 7 - 42 ) pairs are present only in the test set i.e . dtrainsource Ś dtraintarget X dtestsource Ś dtesttarget “ H. • Novel Domain Type ( line type and shape color ) : The training set does not contain any problems with the held-out domain i.e . dtrain R { line type } or dtrain R { shape colour } , while each problem in the test set involves the held-out domain . • Novel Domain Values ( Interpolation and Extrapolation ) : The training and test sets are made of two mutually exclusive sets of attribute values . For the extrapolation split , vpaqtrain P t1 , 2 , 3 , 4 , 5u while vpaqtest P t6 , 7 , 8 , 9 , 10u . For interpolation , vpaqtrain P t2 , 4 , 6 , 8 , 10u while vpaqtest P t1 , 3 , 5 , 7 , 9u . Each of the splits presents a unique challenge in terms of generalization , yet shares the idea that understanding an analogy involves identifying the higher-order visual relation from the source domain and applying it to a target domain . The relationship ( r P { progression , AND , OR , XOR } ) in the visual analogy dataset can be of classified into two types : Unary and Binary . The Unary relation , progression , is composed of a function applied to a single panel to produce the next panel in a row . On the other hand , the Binary relations , ( AND , OR , XOR ) , are composed of a function applied to the first two panels in a row to generate the final panel . Thus , the relationship defines the semantics across a row of panels and is the core abstraction of analogical reasoning . Hill et al . ( 2019 ) hypothesized that introducing a prior on the learning process that requires the learner to correctly identify the relationship would align with how humans learn structure for analogical mapping . They designed this prior by carefully controlling the candidate panels P 1´4can presented during model training ( Figure 3 ) . This regime , called Learning Analogies By Contrasting ( LABC ) , consists of candidates that are all semantically possible to complete the visual analogy . Each candidate satisfies one of the four possible relationships with the target domain panels . However , only the correct panel satisfies the relationship in the source domain panels . The authors found that LABC training significantly improved the ability to learn analogies in DL models compared to training with merely perceptually possible candidates ( Normal training ) . In fact , even a mixed training regime consisting of both LABC and Normal problems showed significant improvement on the Normal training regime . To simplify presentation , in what follows we will use the term “ Contrasting ” for the use of such examples during training ( LABC ) and in evaluation .

Summary: This paper proposes a new architecture for learning visual analogies, based on Gentner’s Structure Mapping Theory for how humans might draw analogies. Gentner’s theory proposes representing the relationships between objects explicitly, so that this relational structure can be reused in new domains (and suggests that this commonality in structure is what permits analogies to be made between perceptually dissimilar objects). The authors propose a neural network model architecture and test it on the Raven’s Progressive Matrices dataset. The proposed architecture first splits a series of ‘source’ visual scenes into objects, attributes and the relationships between those scenes, before feeding just the relationship head into a second network. The second network then switches between two different architectures (depending on the relation fed in). The architecture in the second network (whichever is chosen) receives the ‘source’ relationship and two ‘target’ scenes before trying to predict which of a set of 4 candidate  ‘target’ scenes completes the visual analogy between source and target. The authors test their architecture on the generalisation splits in the RPM dataset and compare test accuracy results to the baseline models used by Hill et al, 2019. The authors show that their model (which builds in additional architectural structure) performs better at a subset of tests than more general architectures.