Neural Structure Mapping For Learning Abstract Visual Analogies
Authors: Shashank Shekhar
Graham W. Taylor
Venue: ICLR 2022 Submitted (2022)
Keywords: cognitive science
analogy
psychology
cognitive theory
cognition
abstraction
abstract reasoning
generalization
systematic generalization

Abstract: Building conceptual abstractions from sensory information and then reasoning about them is central to human intelligence. Abstract reasoning both relies on, and is facilitated by, our ability to make analogies about concepts from known domains to novel domains. Structure Mapping Theory of human analogical reasoning posits that analogical mappings rely on (higher-order) relations and not on the sensory content of the domain. This enables humans to reason systematically about novel domains, a problem with which machine learning (ML) models tend to struggle. We introduce a two-stage neural framework, which we label Neural Structure Mapping (NSM),  to learn visual analogies from Raven's Progressive Matrices, an abstract visual reasoning test of fluid intelligence. Our framework uses (1) a multi-task visual relationship encoder to extract constituent concepts from raw visual input in the source domain, and (2) a neural module net analogy inference engine to reason compositionally about the inferred relation in the target domain. Our NSM approach (a) isolates the relational structure from the source domain with high accuracy, and (b) successfully utilizes this structure for analogical reasoning in the target domain.

Under review as a conference paper at ICLR 2022
NEURAL STRUCTURE MAPPING FOR LEARNING ABSTRACT VISUAL ANALOGIES
Anonymous authors Paper under double-blind review
ABSTRACT
Building conceptual abstractions from sensory information and then reasoning about them is central to human intelligence. Abstract reasoning both relies on, and is facilitated by, our ability to make analogies about concepts from known domains to novel domains. Structure Mapping Theory of human analogical rea- soning posits that analogical mappings rely on (higher-order) relations and not on the sensory content of the domain. This enables humans to reason systematically about novel domains, a problem with which machine learning (ML) models tend struggle. We introduce a two-stage neural framework, which call Neural Structure Mapping (NSM), to learn visual analogies from Raven's 's Progressive Matrices, abstract visual reasoning test of fluid intelligence. Our framework uses (1) a multi-task visual relationship encoder to extract constituent concepts from raw visual input in the source domain, and (2) neural module net-based analogy inference engine to reason compositionally about the inferred relation in the target domain. Our NSM approach (a) isolates the relational structure from the source domain with high accuracy, and (b) successfully utilizes this structure for analogical reasoning in the target domain.
INTRODUCTION
The ability to form abstractions of 'concepts' from information and then reason about them is cen- tral to human intelligence (Lake al. 2015). Abstractions enable humans quickly learn concepts from few examples, and then reason systematically about new concepts by composing previously understood concepts (Nam & McClelland 2021 Over the last decade, artificial neural networks have demonstrated human-level performance in building useful abstractions from data when tested on well-defined and constrained tasks. In computer vision, deep learning (DL) models that learn visual abstractions from raw images show strong validation performance on curated test datasets for image recognition (Krizhevsky et al.] 2012). object detection (Ren al., 2015), and scene classifi- cation (Zhou al. 2017). However, unlike humans, DL models struggle with isolating these ab- stractions and systematically applying them to out of distribution test scenarios (Greff et al. 2020).
One important way in which humans both build and reason about abstractions through analo- gies (Mitchell] 2021). There several theories in hu- man cognitive science about how humans perform ana- logical reasoning. Structure Mapping Theory (SMT) posits that perceptual information can be broken down into a domain consisting of objects and attributes, and structural relations between the attributes in the do- main (Gentner 1983). Consequently, SMT defines an analogy as a mapping between the structural relations across two domains, with no mapping of the attributes. For example, to draw analogy between the solar system and an atom, we can map the relational struc- ture (P (Planes revolves around Sun Electron revolves around Nucleus, Sun more massive than Planets = Nucleus more massive than Elect rons). However, the domain attributes, such the Sun being yellow hot mapped the Nuc luus in drawing analogy.
Source Domain: Line Type
Relation: OR
?
Target Domain: Shape Posttion
Candidates
Figure 1: Abstract visual analogy problem

---

Under review as a conference paper at ICLR 2022
Classity
Answer:
Relationship Encoder
Analogy Inference Engine
LSTM
LSTV
Typa
Filter objects
Filter objects lusic
Filter objects
ON
LOTM
CNN
CNN
?
Figure 2: Neural Structure Mapping overview. Inset: RPM Analogy Problem from Fig Left (yellow box): Visual Relationship Encoder to extract the object, attribute and relationship from the first row of panels. Right (green box): Analogy Inference Engine that uses the relationship label to configure a neural module net for matching the correct candidate to the second row of panels.
prior in training ML models for learning analogies. They first constructed a dataset on learning abstract visual analogies from Raven's progressive matrices (RPMs) (Raven 2000). The abstract visual analogy task is made up of two rows of five context panels, and four possible candidate panels to complete the visual analogy. The learner must understand the higher-order relation constituted by the first row of three context panels, and then choose the candidate that satisfies the same relation with the two context panels in the bottom row shown Figure T Hill al. (2019) showed that if the candidates presented during training were curated to maximize differences in structural relations, while being perceptually similar, the models performed more accurately in learning visual analogies. Thus, the authors verifed that incorporating structural differences to maximize the importance of structure mapping the learning procedure also works for MI. models.
This idea behind structure mapping utilized by Hill et al. 2019
However, it is not feasible to build or curate datasets to always exploit a prior in the dataset to max- imize structure learning. This would involve knowing a combinatorially large number of possible relational structures beforehand and manually selecting data maximize learning these structures, a requirement that scales poorly with an increase in possible relational structures. address this issue, propose two-stage Neural Structure Mapping (NSM) framework learn abstract visual analogies (Figure 2). Our first-stage involves a visual relationship encoder that takes as input the visual context from the source domain and predicts the component visual concepts: objects,, attributee and rel a atioonshi i in a multi-task manner. We then use the derived relation to dynamically build an analogy inference engine. Similar structure mapping humans, our en- gine utilizes only the relation from the source domain to pick the candidate panel that best fits the analogy. We incorporate compositional reasoning into our approach by using neural mod- ule network (Andreas et al.) 2016 Johnson et al.) 2017b] Csordás et al., 2021 as our engine. Our modular neural engine dynamically adapt its structure align (Xu al. (020)) with the structural rel at sonshipp p, thus explicitly incorporating the structure mapping process in our NSM model.
We train and evaluate our approach on five different generalization splits of the visual analogy dataset. These splits were proposed by Hill et al.) (2019) to test systematic generalization (Bah- danau et al., 2018) across object and attribute values learning visual analogies. We show that our relationship encoder achieves a high degree of accuracy in isolating the relationship from the source domain, and, as expected, there is no performance drop across generalization splits or training regimes. For our analogy inference engine, empirically show that it explicitly captures the struc- ture mapping process, while generalizing systematically in learning the structural relation across visual domains.
2

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
Learning Analogies Gentner (1983) 's SMT defines an analogy as a comparison in which rela- tional predicates, but jew or no object attributes, can be mapped from base to target domains. Symbolic models based on SMT (Falkenbainer et al. 1986). including those designed for RPMs (Lovett & Forbus 2017), rely on first extracting a rule based representation of the domains from the perceptual input. Other cognitive analogy models based on Active Symbol theory (Hofstadter & Mitchell 1994] Mitchell, 1993) or High-Level Perception theory (Chalmers et al. [992] do not sep- arate the structure extraction and mapping process, nor rely on mapping the structure syntax across domains. Our approach is not directly related to these models.
Hill et al. (2019) introduced the dataset for learning analogies from RPMs and utilized several neural network architectures to directly learn from visual data. Their key contribution was to introduce the SMT prior into the dataset during candidate selection. [Webb et al. (2020) introduced Temporal Context Normalization to explicitly learn visual features that can support extrapolation, and tested it on the Visual Analogy Extrapolation Challenge (VAEC) dataset. Both of these approaches complementary to ours. Other DI. approaches most similar to ours are the Part-Composition Model [Ichien et al.) 202I) and Chen et al. (2019)'s 's approach. However. unlike our model, these models rely on availability of very structured intermediate representations like semantic segmentation maps, and do not use the extracted structural representations to explicitly configure the reasoning model.
Abstract Reasoning Hill et al. 2019 's dataset is directly derived from the Procedurally Gener- ated Matrices (PGM) dataset introduced by Barrett et al. (2018) to test the ability of neural networks to perform abstract reasoning on RPM problems. RPM tests are an important measure of fluid intel- ligence in humans (Raven 2000), and the PGM dataset provided a sufficiently large sample size for training neural networks on this problem. Following this work, two new RPM datasets were also re- leased: RAVEN (Zhang et al. 2019)) which utilized additional rules and structured rule annotations, and V-PROM (Teney et al. (020). which utilized real images.
DL approaches to RPM tasks can be roughly categorized into three types. The first is relation learning approaches, such as WReN (Barrett et al. 2018), which models all pairwise relationships between the matrix panels using relation networks (Santoro et al., (017). and MXGNet (Wang et al.)
2020), which learns row-based node embeddings and then performs a graph classification on the resulting candidate graphs. Second is rule learning approaches, such as DRT (Zhang et al. 2019) that learns a structured Stochastic Image Grammar of the abstract rules, and LEN (Zheng et al. 2019)
that utilizes logic embedding network along with curriculum to learn with increasingly distracting features. Third is object-centric learning. such as Rel-AIR (Spratley et al., 2020), that obtains object embeddings generated via Attend-Infer-Repe
for each panel, and Pekar et al. (2020)'s 's method that utilizes both Variational AutoEncoders and an adversarial loss for candidate generation. Our model can be broadly fit into rule learning by way of the encoder, followed by relation learning via the analogy inference engine. However, our general approach towards explicitly extracting structure and mapping it complimentary to these ideas and can be used in conjunction with several of them.
Systematic Generalization Systematic generalization refers to the ability generalize to novel concepts (out of distribution data) by understanding them as compositions of known concepts (Bah- danau et al. 2018). It underpinned by three important characteristics: (1) systematicity, the ability to generalize to semantically related concepts; (2) productivity, the ability to iteratively apply com- pounding to generalize from constituent concepts to their recurrence; and (3) localism, the ability to iteratively apply reductionism to generalize from repeated constituent concepts to the singular. The ability to generalize systematically has been previously explored in human cognition (Fodor & Pylyshyn| 1988), natural language processing models (Lake & Baroni 2018). and language- grounded computer vision models (Agrawal et al. 2017). has its origins in human cognition and was formalized by Fodor (975) under the 'Language of Thought" hypothesis.
Cognitive scientists have expressed reservations about the ability of connectionist models to gener- alize systematicitally (Fodor & Pylyshyn 1988, Marcus 2019). Investigation of neural language models showed that systematic generalization still poses a challenge for DL (Lake & Baroni 2018 Loula et al.) 2018). A key reason deep neural networks lack systematicity is due to their propensity towards "shortcut learning' (Geirhos et al.) 2018 2020). This suggests that DL models rely on exploiting a few number of predictive features instead of considering all possible information about
3

---

Under review as a conference paper at ICLR 2022
the data while drawing conclusions. Consequently, this leads to poor generalization when the key predictive features are changed, even though the central evidence remains the same. For example, DL approaches to solving RPM problems were discovered to rely on evaluating the mode across all candidates in the RAVEN dataset and a significant drop in generalization performance was observed when the models were tested without the shorteut on the RAVEN-Fail dataset (Spratley al. 2020).
On the other hand, neural module networks (Andreas et al. 2016, Johnson et al. (017b) general- ized well systematically when their layout aligned well with specific engguage-rrunndee reasoning problems (Bahdanau et al. 2018). Hence, we utilize modular approach in building our analogy inference engine, which in turn utilizes the relations cnssip structure inferred by our encoder to automatically align itself to the problem. This is a form of conditional computation.
3 PROBLEM SETUP
The abstract visual analogy task (Hill et al., 2019) falls under the umbrella of Raven's Progressive Matrices (Raven 2000) designed to test abstract reasoning and fluid intelligence in human and artificial agents. Each question consists of five context panels P and four candidate panels Pl The context panels are arranged in a matrix with two rows of three columns each, with the final panel in the second row missing. The first row of panels P1-3 express a semantically related triplet R = fo, a, r). that is composed of two lower-order perceptual visual concepts: ob ject and attribute a, and one higher-order semantic visual concept: relationshil r. The objective is choose candidate i panel, represents the same rwwatieenssiiee as the first row of panels.
) that, upon substitution in place of the missing
E {line, shape)) and attribute (a (quantity, colour, type, size, position) together constitute the visual domain d of the triplet R. Each possible attribute can take ten values v(a) (1.2...., 9, 10) (nor- malized). The domain of the first row of panels Pl-3 defines the source domain dsoucce of the analogy, and the domain of the second row of panels (Pim3, ,e) defines the target domain durget-- The dataset consists of seven unique possible domains such that d E {shape quantity, shape colour, shape type, shape 3 size, shape position, line type, line colcur). To test the systematic generalization in learning visual analogies, the dataset has five different generalization splits that require the ability to recognize the relationshij across both novel domain d and attribute values:
The object (o
Novel Domain Transfer: The training set consists of 42 ordered pairs of dyource and durget while the remaining (7*7- 42) pairs are present only the test set i.e.
source X darget
dest source X deest
Ø
Novel Domain Type (line type and shape color): The training set does not contain any problems with the held-out domain i.e. d'min & {line type} or drainn & (shape colour), while each problem the test set involves the held-out domain.
Novel Domain Values (Interpolation and Extrapolation): The training and test sets are made of two mutually exclusive sets of attribute values. For the extrapolation split, v(a)train E {1, 2, 3, 4, 5] while o( ((() (1,3,5,7.9).
jest {6, 7, 8, 9, 10). For interpolation, o(a) train E {2,4, 6, 8, 10) while v(a)
Each of the splits presents a unique challenge in terms of generalization, yet shares the idea that understanding an analogy involves identifying the higher-order visual relation from the source do- main and applying it to target domain. The relationshp (r E (progression, AND, OR, XOR)) in the visual analogy dataset can be of classified into two types: Unary and Binary. The Unary relation, progressicn inn, is composed of function applied to a single panel to pro- duce the next panel in row. On the other hand, the Binary relations, (AND, OR, XOR), are composed of a function applied to the first two panels in row to generate the final panel. Thus, the relatioonshp defines the semantics across a row of panels and is the core abstraction of analogical reasoning.
Hill et al. (019) hypothesized that introducing a prior on the learning process that requires the learner to correctly identify the relationshi would align with how humans leam structure for analogical mapping. They designed this prior by carefully controlling the candidate panels pl-4 presented during model training (Figure 3). This regime, called Learning Analogies By Contrasting
4

---

Under review as a conference paper at ICLR 2022
Source Domain: Number Relation: Progression
Domain Shape Number Relation:
shopes
shapes
15 Srayes
5 shaney
shapep
7 shappy
*
Dark
Lighter
Lightest
Dark
Lighoest
Target Domann: Shape Color
Targe: Domaan: Shape Coler
AND
OR
Progression
XOR
Progression
(a) Contrasting
(b) Normal
Figure 3: Two different types of candidates for the same target domain. (a) Contrasting candidates, each of which satisfies a relationship structure and requires identifying semantic structure during candidate selection. (b) Normal candidates, which are merely perceptually similar to the context.
(LABC), consists of candidates that all semantically possible complete the visual analogy- Each candidate satisfies one of the four possible re relatiinns ps with the target domain panels. However, only the correct panel satisfies the relationshis in the source domain panels. The authors found that LABC training significantly improved the ability to learn analogies in DL models compared to training with merely perceptually possible candidates (Normal training). In fact, even mixed training regime consisting of both LABC and Normal problems showed significant improve- ment on the Normal training regime. To simplify presentation, in what follows we will use the term "Contrasting" for the use of such examples during training (LABC) and in evaluation.
4 METHOD
In work, take complimentary approach to Hill et al. (2019) for learning abstract visual analogies in RPMs. Instead of depending on explicitly labeled candidates for mapping relational structure, build model that dynamically configures its structure based the relational structure extracted from the source domain. Internally, our approach is made up of two different neural networks that correspond to the two separate tasks in our pipeline. The first step is the Visual Relationship Encoder, Rpred = o(PL=)), that predicts the visual relationship triplet R {o, r). encoded the source domain panels P1-3 Next is the Analogy Inference Engine, < (P1-5 Pl-1,r), that takes the predicted source relation r, and selects the candidate ce U that maximizes the probability this relation in the target domain p4-5, We next provide detailed discussion each of these networks in our pipeline. For implementation details of the networks please refer to Appendix
4.1 VISUAL RELATIONSHIP ENCODER
The Visual Relationship Encoder, Rpeed o( (P..m)) segregates the abstract relationship triplet in the source domain panels into its constituent ob object, attribute, and re velationship This step of our NSM model corresponds the structure extraction phase in SMT in which humans segregate the higher-order relation concepts in perceptual information from the underlying domain concepts.
The encoder is made up of multi-task neural network. We use convolutional neural network (CNN) Co to first extract a visual feature vector vis), = C&( for each context panel. The extracted visual features are then sequentially fed into long short term memory network (LSTM) Rs to generate visual sequence features seqe of the source domain. The hidden layer vector after the third sequence panel is taken as the encoded features of the source domain seq, = Rs (v)
The source domain feature vector seqş is then passed to fully connected layer Ishared FCyharcd that shared between all three visual components. The shared linear fea-

---

Under review as a conference paper at ICLR 2022
Glassifer
Cleasifier
Binary
Binary
Binary
Unary
Unary
Unary
Stom
Stem
Stem
Domain: Shape Size
Relation: Progression
Domain: Shape Position
Relation: AND
(a) Unary Relation
(b) Binary Relation
Figure 4: Types of relati onship and corresponding Analogy Inference Engine layout. Only correct candidate panel shown. See Section 4.2 for details
ture vector Ishared is then passed through three different fully connected layers Pask FCrask (rrruss([ssanrg (Ishared) for each task E fobject, attribute, relation] classification. Finally, each pask is passed through fully connected layer out FC (Jaak) of sizes 2, 5, and 4 for the object, attribute, and relation prediction tasks respectively. Softmax over the outputs out yields a probability distribution over the possible fo, a, r values that constitute the triplet R.
4.2 ANALOGY INFERENCE ENGINE
The Analogy Inference Engine, T( (P4-5 11-4,,, , maps the source relationship r extracted by the Visual Relationship Encoder to the target domain P4-6 to identify the correct candidate e that completes the visual analogy. This step of our NSM model corresponds to the structure mapping phase in SMT, in which humans apply relations extracted from the source domain to a new domain to reason about concepts in the target domain. To generalize systematically to new domains and attribute values, utilize modular approach (Andreas et al. (2016) to build analogy inference engine. Similar to Johnson et al. (2017)) and Bahdanau et al. (2018), we use modules with generic architecture. Our engine is made up of four different types of neural modules:
The Stem module performs a series of convolution operations with stride=2. It takes the original grayscale images size x 160 x 160 as input and returns visual feature map of size x 9 (C x H. x W). Each of con and pI-4 panels is first passed through the Stem module to extract visual features before being processed by the rest of the engine.
The Unary module is residual block with two
convolution layers. It receives one feature map (C x H x W) as input, and returns one feature map of (C x H x W) as output.
The Binary module receives two feature maps, concatenates them along the channel dimension, and uses 1 x I convolution to project them to C dimensions. It then passes the combined feature maps through a residual block, and returns one feature map of (C H x W) as output.
The classifies module also receives two feature maps, and projects them C dimensions. It then flattens the feature maps and passes them through two fully connected layers to generate probability distribution over the relations.
Each candidate panel is fed parallely to the engine to generate its predictions for all four relations. In order to generate the candidate selection, we subset the probability re relat atnnship r (Tped dur- ing inference) for all candidates, and select the candidate with the highest corresponding probability.

---

Under review as a conference paper at ICLR 2022
The layout of the module network is chosen adaptively from two possible layouts based the source domain relation at inference time. Each layout first processes the individual panels through the em module extract visual features, followed by the Unary module. In the first layout, the Unary ry module outputs for the target domain panels P1-5 con are combined using a Bi na nary module, and the outputs for the second panel P and each candidate panel P are combined using another Binary module (Figure 44). The outputs of both the Binary modules are then fed into the Classifier module. In the second layout, the Unary module outputs for the target domain panels con are similarly combined using a Binary module. However, the output of this B Binary module and the Unary candidate output directly fed into the Classi fier module (Figure 44b) We provide further discussion on these layouts in Appendix A
5
EXPERIMENTS AND RESULTS
5.1 VISUAL RELATIONSHIP ENCODER
Training The Visual Relationship Encoder & is trained using the source domain context panels Pi and the visual relationship labels R = {o, r). The cross-entropy loss for each task is obtained using the multi-task network's predictions and triplet labels. The total encoder loss calculated as weighted sum of the individual losses for object, attribute, and relation prediction:
Lancoder Lobject + 3 * Cattibute + 7 Crelationship
Empirically, we found that the values of a 0.5, 3 0.5.y 2.0 led higher validation accuracy. The model was trained for 100 epochs using the Adam optimizer (Kingma & Ba 2015) with a learning rate le - For each generalization split, the model was trained with contrasting, normal, as well as mixed candidates. The model with the best validation accuracy was chosen for testing (the validation set consisted of a similar set of analogy candidates as the training set).
5.1.1 RESULTS
We test the relationship prediction accuracy of our encoder across all five generalization splits, with each possible set of analogy candidates (Contrasting, Normal, Mixed). Our relationship encoder was able to isolate the component relationship from the source domain with high degree of accuracy (min: 82.4%, max: 86.04%) across all generalization splits and all three types of candidates (Figure 5). Furthermore, since the encoder utilizes only the source domain information in determining the relation, we observe no drop between training and test accuracy across all generalization splits.
By identifying the visual relationship in the source domain, the output of the visual relationship encoder is able to determine the layout for the subsequent analogy inference engine. Since the inference engine chooses between only two possible layouts, a baseline that always chooses the majority class layout would lead to correct choice in 75% of problems. Hence, our encoder must
Mixel
Eoorrating
Nermnl)
Figure 5: Relationship prediction test accuracy of our Visual Relationship Encoder. Results are first grouped (L-R) by test candidate regimes (contrasting, normal, and mixed candidates) and then cub-grouped (L-R) by training candidate regimes. Since the encoder is trained on the source domain panels, it does not have generalize systematically and performs consistently across all splits.
7

---

Under review as a conference paper at ICLR 2022
achieve high degree of accuracy in identifying the correct layout to beat this baseline. Our encoder is indeed able to identify the layout matching the reasoning problem with near perfect accuracy (min: 98.42% max: 99.9%). The full results for layout prediction across all generalization splits and training regimes are in Table 15 Appendix D
5.2 ANALOGY INFERENCE ENGINE
Training The Analogy Inference Engine ı is trained using the target domain context panels Pilm the candidate panels & the ground truth candidate label, and the ground truth visual relationship label r. The engine was trained for 100 epochs using the Adam optimizer with a learning rate of le-t. Similar to the first stage, the engine was also trained with contrasting, normal, as well as mixed candidates. The model with the highest validation accuracy was chosen for testing (validation set consisted of a similar set of analogy candidates the training set).
Testing For testing, we rely on the relationship label rpedt predicted by our encoder instead of ground truth r. While all baselines (see below) are trained with the full set of context panels p1-5 our Analogy Inference Engine only utilizes the target domain panels P4-5 for context. Hence, we also provide the results
full context model which utilizes all context panels p1-5 For this purpose, we leverage our encoder which is trained with the remaining context panels P - p3 We use to first extract the relat ialationnae ionship r from the source domain as discussed in Section In addition, also generate candidate probabilities from the encoder during inference, by considering each candidate in parallel followed by a Softmax over the probability of r across all candidates as
deseribed in Section 42 The candidate probabilities of the full context NSM model calculated as 1:1 ensemble between the probabilities from both the inference engine and the encoder.
Baselines We compare the candidate selection accuracy of our analogy inference engine with each model utilized by Hill et al. (2019). These baselines include a CNN-LSTM model, a standard ResNet50 (with input channels for 9 panels), parallel ResNet50 (with 6 channels and each can- didate fed in parallel), and a Wide Relation Network. Each baseline is trained for 100 epochs using the Adam optimizer with learning rate of le-4, with contrasting, normal, and mixed candidates.
5.2.1 RESULTS
Does the modular analogy inference engine generalize systematically?
We hypothesize that choosing a modular engine for analogy inference enables compositional rea- soning about novel domain values (objects and attributes). In Table we compare the generalization performance of our engine on the Novel Attribute Value: Extrapolation and Novel Domain: Line Type test splits of the analogy dataset. Our approach achieves the highest test accuracy across all three training candidate regimes in the Novel Domain setting when tested with contrasting can- didates. Furthermore, our approach yields high test accuracy (highest within < 1%) for the Contrasting and Normal training candidate regimes in the Novel Attribute Value setting. Thus, our NSM approach is better at systematic generalization novel visual domains than monolithic neural
Test Accuracy % (Contrasting/Norer Normal)
Model
Novel Domain
Novel Attribute Value
(Line Type)
(Extrapolation)
Contrasting
Normal
Mixed
Contrasting
Normal
Mixed
CNN-LSTM (Hill et al 2019
76/50
45757
75754
62745
43/44
56/39
ResNet
25.01/24.95
41.8/46.9
25.01/24.95
78.97/55.2
48,5/49.55
62.59/52.73
Res Nes-aalParll
79.35/66.51
52.43/6.77
79.67775.9
61.97/57.23
54.86/56.69
63.71/57.17
WReN
73.71/57.49
53.42/62.64
61.1749.33
74,25/61.36
61,23/61.4
70.57/63.1
NSM (ours)
78. 14/59.55
70.64/64.31
78.53/64.1
65.24/57.14
62.33/58.00
58.4/50.98
NSM (full context)
79.75/59.55
76.18/62.43
80.57/65.74
73.36/59.94
66.75/59.8. 84
63.47/52.92
Table 1: Candidate prediction test accuracy comparison for Novel Domain and Novel Attribute val- ues. Our approach achieves the best generalization accuracy across all models in the higher number of possible candidate scenarios (4/12), and is within < 1% of the best generalization performance in half of all scenarios (6/12). Higher accuracy = Better systematic generalization.
8

---

Under review as a conference paper at ICLR 2022
Thet Contmoding
Thet Nornal
CABANTU
Raale-Padd FareeFaaddl
(a) Novel Attribute Value: Extrapolation
(b) Novel Domain: Line Type
Figure 6: Systematic generalization test performance with Normal training candidates which lack prior on the dataset that maximizes structure learning during training.
networks. For completeness. we provide the results for both our evaluation setups across all training and test regimes in Tables [ and 17 in Appendix D.2
Does our model's prior enable it learn structure mapping?
We specifically evaluate the Normal training scenario where the candidates presented the learner are drawn randomly and lack the contrasting prior which explicitly promotes structure learning. Since our NSM model explicitly captures and maps the structure of the visual analogy, thus making up for the lack of prior in the candidates, trained with normal candidates would outperform other baselines. We visualize the generalization performance of this training scenario for the Novel Domain Value: Extrapolation and Novel Domain: Line Type generalization splits in Figure 6 We observe that our model outperforms other networks when averaged across both types of candidates during testing, demonstrating that our model's prior enables it to learns structure mapping better than other models.
anticipate that the test performance of our model
How important structure mapping in drawing the correct analogy?
We verify the importance of explicitly mapping structure on our NSM approach by comparing our results when the relationship label inferred in the first step matches the ground truth relat ionship label. We present the confusion matrix in Table where it can be seen that the test candidate selection accuracy for the correct structure mapping (diagonal entries) significantly higher than incorrect structure mapping.
Prog.
XOR
OR
Prog
AND
Prog
XOR
OR
Prog.
90_36
16.72
22.53
10.88
Prog.
90.81
2.60
9.12
0.01
Prog.
89.93
30.44
35.57
21.44
XOR
20.83
86.49
37.67
12.32
XOR
6.61
89.36
30.04
194
XOR
33.33
83,58
63.39
22,83
OR
43.86
52.47
94.80
6.01
35.69
96.19
31.98
52.55
93.42
11.97
AND
5.86
6.18
2.76
94.72
AND
0.30
3.27
0.00
96.36
AND
11.51
9.15
5.57
93.06
(a) Test Accuracy (Mixed)
Test Accuracy (Contrasting)
Test Accunacy (Normal)
Table 2: Test accuracy with correct vs incorrect structure mapping. Correct Mapping == ground truth relation (rows) matches the relationship used to inform the analogy inference engine (columns).
6 CONCLUSIONS AND FUTURE WORK
In this work, we introduce a two-stage neural framework for learning abstract visual analogies based on the Structure Mapping Theory human analogy making. Our first stage is multi-task Visual Relationship Encoder that corresponds to structure extraction from perceptual information in hu- mans. Our second stage a modular Analogy Inference Engine that corresponds to mapping higher- order relations for structure mapping in humans. Our approach is able to generalize systematically to novel target domains, compensate for the lack of prior candidate selection, and successfully exploit the process of extracting and mapping the relationship structure. In future work, we plan to investigate search-bar baaec (e.g. Neural Architecture Search) as well as differentiable (e.g. attention- based) methods for generating the layouts of our Analogy Inference Engine. Furthermore, would like explore approaches that combine our structure mapping framework into existing methods for reasoning about Raven's Progressive Matrices beyond analogy making.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split of the visual question answering (vqa) v1. 0 dataset. arXiv preprint arXiv: 2017.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48. IEEE, 2016.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2018.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 511-520. PMLR, 2018.
David J Chalmers. Robert M French, and Douglas R Hofstadter. High-level perception, representa- tion, and analogy: critique of artificial intelligence methodology. Journal of Experimental & Theoretical Artificial Intelligence, 4(3):185-211 1992.
Kezhen Chen, Irina Rabkina, Matthew McLure, and Kenneth D Forbus. Human-like sketch object recognition via analogical learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 1336-1343, 2019.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? in- specting functional modularity through differentiable weight masks. In International Confer- ence Learning Representations, 2021. URL https: htttps://operree// /oppnreveeeeeeeeeeeeeeeeeeeeeeee net. /eetfforteeeeeeeeeeeeeeeeeeeeeee forum?id- 7uVcpu- gMD
Brian Falkenhainer, Kenneth Forbus, and Dedre Gentner. The stracture-mapping engine, volume 1275. Department of Computer Science, University of Illinois at Urbana-Champaign, 1986.
Jerry A Fodor. The language of thought, volume 5. Harvard university press, 1975.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analy- sis. Cognition, 28(1-2):3-71, 1988.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.1225: 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(11):665-673, 2020.
Dedre Gentner. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2): 155-170, 1983.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXi::2012.05208 2020.
Felix Hill, Adam Santoro, David Barrett, Ari Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. In International Conference on Learning Representations, 2019. URL http://s/////eeerriiieeeee net fow.mmiidsss. SyleiiiooyyLyyyyCyyy LYYCCcFm
Douglas R Hofstadter and Melanie Mitchell. The copycat project: A model of mental fluidity and amalogy-making 1994.
N Ichien, Q Liu, S Fu, KJ Holyoak, AL Yuille, and H Lu. Visual analogy: Deep learning versus compositional models. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society, 2021.
10

---

Under review as a conference paper at ICLR 2022
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 3668-3678, 2015.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition,
pp. 2901-2910, 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffmann, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reason- ing. Proceedings of the IEEE International Conference on Computer Vision, pp. 2989-2998, 2017b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Int. Conf. on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-ssequenc recurrent networks. In International Conference on Machine Learning, pp. 2882- PMLR, 2018.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human- concept learning through probabilistic program induction. Science, 35006266);1332-13333 2015.
João Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional gen- eralization in recurrent networks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLPP. Analyzing and Interpreting Neural Networks for NLP, pp. 108-114, 2018.
Andrew Lovett and Kenneth Forbus. Modeling visual problem solving as analogical reasoning. Psychological review, 124(1):60, 2017.
Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2019.
Melanie Mitchell. Analogy-making as perception: A computer model. 1993.
Melanie Mitchell. Abstraction and analogy-making in artificial intelligence. arXiv preprint arXiv:2102.10717, 2021.
Andrew Joohun Nam and James L McClelland. What underlies rapid learning and systematic gen- eralization in humans. arXiv preprint arXiv:2107.06994 2021.
Niv Pekar, Yaniv Benny, and Lior Wolf. Generating correct answers for progressive matrices intel- ligence tests. Advances in Neural Information Processing Systems, 33, 2020.
John Raven. The raven's progressive matrices: change and stability over culture and time. Cognitive psychology, 41(1):1-45 2000.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), pp. 91-99, 2015.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Ad- vances in Neural Information Processing Systems, 30, 2017.
Steven Spratley, Krista Ehinger, and Tim Miller. A closer look at generalisation in raven. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part XXVII 16, pp. 601-616. Springer, 2020.

---

Under review as a conference paper at ICLR 2022
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton den Hengel.
V-prom: A benchmark for visual reasoning using visual progressive matrices. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 12071-12078, 2020.
Duo Wang, Mateja Jamnik, and Pietro Lio. Abstract diagrammatic reasoning with multiplex graph networks. arXiv preprini arXiv:2006. 197, 2020.
Taylor Webb, Zachary Dulberg, Steven Frankland, Alexander Petrov, Randall O'Reilly, and
Jonathan Cohen. Learning representations that support extrapolation. In International Conference Machine Learning, pp. 10136-10144 PMLR, 2020.
Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. What can neural networks reason about? In International Conference on Learning Representa- fions, 2020. URL
net/ wet.ee/oooomm..rr....eeee...
Licheng Yu. Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chur Zhu. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Kecheng Zheng, Zheng-Jun Zha, and Wei Wei. Abstract reasoning with distracting features. Ad- vances in Neural Information Processing Systems, 32:5842-5853, 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
12

---

Under review as a conference paper at ICLR 2022
A HOW DO WE CHOSE THE MODULE NETWORK LAYOUTS?
The generic architecture of our modules allows for a combinatorially large number of possible ar- chitectural layouts of our inference engine. Previous work in anguage-grounded reasoning tasks like VQA (Johnson et al., 2017b) or referring expression comprehension (Yu et al. (011)) uses a se- mantic parse tree generated from natural language generate layouts for neural module networks. Since our inference engine (r)) relies only on the relationship label from the source domain, it lacks the rich compositional structure available in language grounded reasoning to arrange the modular network layout. However, generating the proper layout for neural module networks is fundamental to their ability to generalize systematically (Bahdanau et al. 2018).
-anguage-roouneed
we dynamically chose between two layouts of our neural module net- work based on the relationn iathhip type. Our rationale behind choosing these layouts comes from the knowledge of the possible relationnn iotthipss our engine has to reason about. Each lay- out aligns algorithmically (Xu et al. 2020) with reasoning algorithm for identifying the relation- ship across three panels. The first layout models Algorithm I to identify a Unary relationship like progression (not to be confused with a Unary module) between panels 1, 2 and 3. The second layout models Algorithm 2 to identify Binary relationship (OR, AND, XOR) across panels 1 and 2 in the last panel.
As discussed in Section 4.2
Algorithm To Identify Unary Relation
Algorithm To Identify Binary Relation
Require: Panels P1, Ps. Ps
Require: Panels P1, Ps, Ps
I: D1 extrectObjeerssnndAAtibinit
1: D1
2: D2 + extracrObijeccsssnAAAttttbbbeeee
2: D2 + extracrohiierrsshddArrbbbbss (2))
3: D3 extractOOhjectssAnAAArriii (Ps)
3: D3 extrac(OhieectssnnAAtiibbuue P3)
4: R12 - artihmetic D1, D22)
4: R12 x binaryOperation Dr,, D2)
5: R23 artihmettiOOperation(]] D3)
Result isConsistent((12, D3)
6: Result + is isConsistem( (112, R2a)
6: return Result
7: return Result
We do not explicitly train our modules to correspond any individual functional form. Instead, expect the module instantiations to learn the corresponding function in the reasoning step simply by learning from the classification loss. For example, in the reasoning algorithms above, a Unary module can learn and binaryOperation(). and Classi fier module can learn isCosiistent() functions. This enables the modules to share parameters across relation functions (such as two different instantiations of binuryOperation() for AND and OR), as well as across objects and attributes. This also enables us to utilize general purpose modules which could potentially be arranged in several possible combi- nations depending on the reasoning problem, and can be easily extended beyond the abstract visual analogy setting of only four relat ionships to large number of possible relat itnhships in
Binary module can learn artihmeticOperatton//
real-world datasets like Scene Graph (Johnson et al.) 2015) and CLEVR (Johnson et al. 2017a).
Do algorithmically aligned engine layouts learn corresponding relations better?
We chose two possible layouts for our modular engine based on their alignment with algorithms for identifying Unary or Binary relation across three panels in RPM problems. We expect Layout- A (Figure 4ab to better align with Unary (progression problems and Layout-B (Figure 4b) to better align with Binary (AND, OR, XOR) problems. In order to verify this empirically we train each layout individually with similar training hyperparameters as the full engine. We control the overall validation accuracy to select trained models which perform similarly across the full dataset. Next, we compare the test accuracy for Unary and Binary relations for the trained Layout-A and Layout-B models in Table
In practice, we found significant stochasticity in the performance between Unary and Binary relation types across both layouts. From the observed results, we cannot conclusively say that our layouts necessarily align better with one or the other relationship problem. One possible explanation for this could be that there can be alternate algorithms for evaluating the same relation, which align algorithmically with these layouts. This is similar to Xu et al. (2020)'s 's observation that neural network which would have not aligned with the Bellman Ford algorithm was found to align well with different algorithm for the same dynamic programming problem.
13

---

Under review as a conference paper at ICLR 2022
Moduar
Training Accurary
Teat Accurney (Cinireding)
Accurses (Nurmal)
Accurary (Mixed)
9181913
91.7299011
89.7499889
79.549632 74357536
90.26006
91.47/92033 92209136
90.6690
91,009311
85,438
30,99924
32
7769922
3001184
836543035
Table 3: Comparison of the analogy inference engine layouts on corresponding (unary/binary) re relate onsshi types
Does having an adaptive engine enable better analogy inference?
We use an adaptive modular engine for analogy inference which chooses between two possible module net layouts at inference time. This enables our engine to compositionally reason about the underlying relation. We expect our adaptive setup to outperform a static layout since the engine can then chose to use better fit layout according the mapped relation ainihhhip structure. To verify hypothesis, candidate layout described in Section 4.2 as a possible inference engine. We call these fixed models Engine-A A and Engine-B, B, and train them sumilar our full model. We then compare the test accuracy performance of our full model, which we label Engine-Full, against the test accuracy fixed models.
compare
input-adaptive engine to fixed engine. We chose each
Inference
Training Accuracy %
(Conirsting)
Accuracy (Normal)
Test Accuracy (Mixed)
Contrasting
Contrsting
Contrasting
Engine A
93.39
91.41
93.46
92.88
90.66
94.44
77.00
90.63
91.91
84.94
90.65
93.18
Engine B
93.24
92.07
93.40
92.47
87.70
91.16
75.01
91.02
9206
83.74
89.3
93.11
Engine
93.12
91.4
93.32
90.40
93.30
7437
88.92
96.02
89.06
91.65
Table Comparison of fixed analogy inference engines versus the full adaptive engine
Empirically, we found that the adaptive engine not able to outperform the fixed engines, contrary to our expectation. We believe there are two main reasons for this: (1) Our individual engines are not sufficiently differentiated in their performance Table 3). so there is not a significant performance gain in choosing one over the other, and (2) the number of possible reasoning scenarios quite limited in the abstract visual analogy problem, and hence the adaptive engine is not able to benefit from learning large number of mixture of experts.
their respective reasoning tasks (as shown in
14

---

Under review as a conference paper at ICLR 2022
B VISUAL RELATIONSHII ENCODER ABLATIONS
B.1 MULTI-TASK VERSUS MULTI-LABEL. VISUAL RELATIONSHIP ENCODER
In our two-step structure mapping approach, we treat the structure extraction problem a multi- task learning problem where the selationshi object, attribute prediction are treated as separate learning tasks. We could have alternatively treated the structure extraction problem as multi-label classification problem and instead predicted the full triplet R {o, a, r from our model. multi-label problem could in principle learn from added network interactions between the final classification layer weights for the domain components.
In order to draw a comparison between both these learning approaches, we trained a multi-label visual relationship encoder which predicts the triplet R {o, r) its output. We replaced the & FC**k (rshared) layers in out multi-task encoder with ysharedz FC (1 layer, and the final classifiers out FC (Mask) of sizes 2, 5, and 4, with one classifier outs FCqu (1shaaed)))) of size 11 (2+5+4). For training, we used exactly the same procedure as the multi- task encoder (described in Section
We compared the performance of these models by evaluating there test accuracy across three differ- ent generalization splits: Novel Domain Transfer, Novel Domain: Line Type, and Novel Attribute Value: Extrapolation. Since our encoder is not trained with the candidate panels, we restrict our- selves to one training regime (Contrasting) for this comparison. Our observations are reported below Table
Extegniation Acceray
Novd Dareis Turnds & Aucersc
Newd Derean (irctyel &. Accuecs
1811 DEREITRR
Tranzz 65.90
Son
86.11
$484
$4.98
$4.1
85.31
94.13
$4.38
$4.55
45.16
$5.15
85.66
Table 5: Comparison of relat loisshipp prediction accuracy of a multi-task versus multi-label visual relationship encoder
We found the performance of the two types of relationship encoders to be quite similar in terms of the test accuracy for predicting the relat iatiions tenhhhhp from the source domain panels. In fact, the multi- label encoder was slightly better than the multi-task encoder across all generalization splits. Despite this, we still went ahead with using the multi-task encoder in our two-step approach since the multi- task learning network scales much better with the number of objects, attributes, and relationships in the perceptual domain.
B.2 MULTI-TASK VERSUS RELATIONSHIP-ONE VISUAL RELATIONSHIP ENCODER
We also performed ablation study on predicting only the visual re relatic aationship from the coder and comparing it with the full-task of predicting re l at i onship i, ject, attri bute. We found that predicting only the relationship structure is competitive with a multi-task predictor in the Novel Domain Transfer regime, and since our encoder does not perform generalization across domains candidate types this should hold for other regimes as well. This does not take away from our central contribution of incorporating structure mapping into our model, rather it only highlights the importance of structure extraction as the first-step of our model.
Novel Domain Trunster & Accunacy (C (Contrastive Training)
Novel Domain Thinsfer & Accumey (Normal Training)
Encoder
Traning
(Contrating)
(NoemN))
(Mixdd)
Traiming
(Comtrasting)
1sst (Normal)
Test
Malt-task
86.56
82.32
$2.75
81.6
36.66
82.40
83,02
82.71
Relationshhp-sni
$4.00
83,61
83.6
8361
85.74
83,88
83.00
83.74
Table 6: Comparison of relationshte prediction accuracy of multi-task versus relationship- only visual relationship encoder
15

---

Under review as a conference paper at ICLR 2022
C IMPLEMENTATION DETAILS
C.1 VISUAL RELATIONSHIP ENCODER
C.2 ANALOGY INFERENCE ENGINE
Index
Layer
Output Size
Index
Layer
Output Size
Panel Image Inpul
Image Inpul
160 160
Conv(3 x stride
8
Conv(3 33, .8 stride
BatchNorn
8 79 79
BatchNorm
8 79 79
RcLU
8 79
RcLU
8 79 79
5
Conv(3 3,8 8- 8, stride 2)
8 39 39
Conv(3 3,8 8, strid 2)
8 39 39
6
BatchNorm
8 39 39
6
BatchNorm
8 39 39
ReI.U
8 39
Rel.U.
x 39 39
Conv(3 3,8- stride 2)
8 x
Convl3 3, 3.8- stride
19 19
10
BatchNorm
8 19
BatchNorm
19 19
12
ReLUU
T9 T9
ReL.
19
13
Conv(3 8-8-- stride
9 9
Conv(3 x 3, stride
8x
14
BatchNorm
8
14
BatchNorm
8x9x9
15
ReLU
8 xx
15
ReLU
8x9x9
16
FC(81929 128)
128
17
BatchNorm
128
Table 10: Stem module architecture
13
RcLU
Table 7: Encoder CNN architecture
Index
Layer
Output Size
Previous Module Output
8x9x 9
2
Conv(3 x 3, 8)
8x9x9
Index
ReLU
8x9x9
Eacuder-CINN Ousput
LSTM
ourput final hidden state 128
Conv(3 3,8
9x9x9
2
(hidden dim = 128.
Residual: Add (1) (4)
8
length: 3)
final well state = 128
ReLU
8x9x9
Table 8: Encoder LSTT architecture
Table 11: Unary module architecture
Layer
Output
Encoder LSTM linal hidden state
128
Index
Layer
Output Size
3
FC(128 - 128)
128
Previous Module Output
x 9
ReLU
2
Previous Module Output
8x9x9
FC2ae(128 64)
3
Concatenate (1) and (2)
16x
7
ReLU
64
4
Conv(1 1, 16
8x9x9
5
FCdnssifior.... (64 Task
Tast
5
ReLU
8x9x9
Conv(3 33,8-88)
8X9x9
Table 9: Encoder Multi-Task classifier ar- chitecture. [Task] = 2, 5, for object, attribute, and relationship classi- fiers respectively
ReL.U
8x9x9
Conv(3 3,8,3xx38-->
8x9x9
Residual: Add (5) and (8)
8x9x9
10
ReLU
8x9x9
Table 12: Binary module architecture
Index
Layer
Output Size
Previous Module Output
8x9x9
2
Previous Module Output
8x9x9
3
Concatenate (1) and (2)
16x9x9
4
Conv(1 x 1, 16 8)
8x9x9
3
ReLU
8x9x9
6
FC(8+979 - 256)
256
7
ReLU
8
FC(256
Ir]
Soltmax([c]
9
(over subset of (8) along index Forod all [c] candidates)
[c]
Table 13: Classifier module architec-
ture. |r| = |c| =4.
16

---

Under review as a conference paper at ICLR 2022
D SUPPLEMENTARY RESULTS
D.1 VISUAL RELATIONSHIP ENCODER
SPLR
Training Arcuracy Contnating
Accuracy
(Omntrading)
Test Accuracy
(Nommll)
Arcuracy
Normal
Mixed
Contrasing
Nomal
Mixed
Contracing
Nomal
Mised
Coitraiing
Nornal
Mined
Exmpelatio
86.25
36.44
36.75
$4.75
54.84
85.56
$4.75
$4.92
36.06
$4.76
$4.85
K330
Interpolatin
86 34
86-54
86.96
84.71
64.86
85.57
84.7
85.01
85.81
64.00
84.91
85.09
Novel Demain Transfer
35.96
36.20
86.30
8232
$2.00
83:18
8275
5300
50.83
52.51
K2.71
K3.51
Novel Damain (olage wdlort
36.88
36.48
86.27
85.66
14.00
8335
3500
14.8
13.0
14.95
K5.5
Novel Domain
86.16
86.30
8833
85.00
85
$4.58
$4.00
53,46
84.55
64.85
K3.40
Table 14: Relationship prediction accuracy of our visual relationship encoder
GENERAL
Training Accurncy S
Test Accurncy S (Contracting)
Test Aecuracy & (Normal)
Ics Accuracy
(Mised)
Commmating
Normal
Mixsed
Contracing
Nomal
Mixed
Contracing
Nomaa
Mixed
Contrasting
Nornal
Mined
Extrppolatio
99.86
99.82
99.99
99.50
99.46
99.90
99.49
99,46
99,88
99.50
99.45
99,89
Ineerplaaion
9992
9939
9939
99.47
99.38
99.38
99.41
99.39
99.83
99.44
99.59
99385
Nosel Daman Trmnler
9985
09.38
09.77
0831
98.48
98.36
08.42
18.99
98.33
58.51
08.53
98.87
Novel Demain (shae colori
99.75
99.73
99.45
99.43
99.62
99.45
99.44
98.45
99.43
99.63
19.33
Table 15: Engine layout prediction accuracy of our visual relationship encoder
D.2 ANALOGY INFERENCE ENGINE
GENIRALIEATIONSPIIII
Training
(Contristing)
(Normall
UMiiee
Nanna
Mixal
Extmpooation
Transer
Damain (slape wolrr) Novel Domain (line lxpa)
94.0
95.38
78.4
20.64
78:33
60.33
64.30
64.10
69.18
67.45
71.30
Table 16: Candidate prediction accuracy of our two-step model
PATIONL
Training Accuracy Coninsting
(Cuirasting)
Accuracy (Mised)
Contracing
Contrasing
Contrasting
Extrppolatoo
Interpelaion
81.71
Novel Domain Trasfer
9081
91.25
8837
$8.61
91.40
7302
$6.00
58.15
80.00
87.71
88.75
Novel Demain Glape edlor)
05.26
96.91
97.23
7W43
79.30
85115
38.0
66.18
69.66
68.25
72.85
76.00
Nove: Domain line lypa)
0113
94.8
94.4
79.3
76.15
80.77
39.33
00.0
60.74
07.3
19:30
72315
Table 17: Candidate prediction accuracy of our full-context ensemble
17


Reviews:
[{"review_id": "ycC9FSq6_A-", "reviewer": "Reviewer_QLDy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focused on learning abstract visual analogies, which is an important problem for machine learning models to be able to generalize. To illustrate the effectiveness of learned analogies, the authors chose a carefully designed benchmark that aims at solving generalization problems under the domain of RPM problems. This benchmark is deliberately designed to challenge models' performance on the task of novel domain (novel concept), novel domain type (novel concept composition) and novel domain values (unseen values for known concepts), which from my perspective, is sufficiently difficult as a starting-step task and a good direction to go. However, I strongly feel that the authors did not distinguish their work from existing ones. Given that the benchmark, generalization tests, and learning setup (learning by contrasting) are defined in prior works, the major contribution of this paper should have come from model design and learning method, but the novelty of the proposed model is somewhat weak. First of all, the idea of leveraging object-centric representations for solving abstract reasoning tasks has been widely discussed in RPM solving tasks, which makes the visual relationship encoder conceptually not a new thing. Further, the proposed model claimed to have borrowed ideas from modular approaches for the analogy inference engine, but the unary/binary operators mentioned are designed specifically for this RPM-based task and have fixed structures (though the authors discussed dynamic structures in the appendix). This makes the proposed two structural layouts a bit weak as (1) it lacks semantic information when compared to modular networks (does not induce anything we could use for other similar tasks), and (2) it might not meet the need for more complex scenarios (e.g. real-world analogies) as a structural mapping engine. This is somewhat supported by the fact that ResNet-Parallel could be achieving similar results under certain training settings. Given these facts, the proposed model seems to be constrained on the current task and does not have strong potential as a general solution to visual analogy learning. \n\nHere I list some points that the authors could clarify or correct me if understood them wrong during rebuttal. \n(1) To find the best model that can generalize well given the current test settings, I assumed we should be looking at normal testing accuracy on each training regime since compared with contrasting candidates, the randomly sampled candidates can be more confusing. If this is the case, should we be looking for the best model that could achieve the best normal testing accuracy? (in this case, perhaps ResNet-Parallel or WReN?)\n(2) It is a bit hard to see the difference between training setups and target tasks except for the obvious gap of novel domain transfer. As all visual encoders are trained given only the source domain context panels, one could imagine this task should not be too different for each generalization split. Then is there a potential reason why such a gap exists for different training setups or splits? Is it because of the joint training of later modules or distribution shift in the benchmark itself?\n(3) There are minor errors on paper writing including typos and figure errors (e.g., the highlighting candidate in Figure.1 is incorrect).", "summary_of_the_paper": "This paper targets the problem of abstract reasoning, with a special focus on the task of learning visual analogies. The authors propose a multi-stage neural network (Neural Structure Mapping, NSM) for decomposing the problem into vision relationship recognition and concept inference. They tested their model on an existing RPM (Raven's Progressive Matrices) based visual analogy benchmark that contains different systematic generalization tests and outperformed existing models. The authors made further discussion on these experimental results to support their proposals on model designs.", "summary_of_the_review": "Although this paper works on an interesting and potentially impactful problem, the current design of the model and illustration does not fully support the claim the authors made and is somewhat incremental on technical novelty. Therefore, I recommend a rejection at this time with the hope that the authors can make this submission stronger in the next version.\n\nPost rebuttal:\nAfter reading the authors' response, I still feel that the current submission is somewhat incremental and needs more solid results for justifying the authors' claims. Scores are increased from 3 to 5 given the clarifications of the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZVWkrHLxoGt", "reviewer": "Reviewer_zpNi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nInteresting proposal to implement an architecture that embodies a classic theory of analogy learning (Gentner 1983). \nVisual relationship encoder architecture was clear and logical, and the relationship to Gentner’s Structure Mapping Theory was clear in this component, as the Visual Relationship Encoder outputs an explicit representation of ‘structure’, which is here manifest in the relationship between separate visual scenes. \n\n\nWeaknesses:\n\nClarity of text, theory and justification for work:\nThe paper was difficult to read in parts, as some concepts were poorly explained. For example, the main paper that this work is based on seems to be somewhat misrepresented at several points in the text.  “It is not feasible to build or curate datasets to always exploit the structure mapping prior”, “their key contribution was to introduce the SMT prior into the dataset…”. “Hill et al (2019) hypothesized that introducing a prior on the learning process that requires the learning to correctly identify the relationship would align with how humans learning structure for analogical mapping”. The work by Hill et al demonstrates that when neural networks are trained with candidates that embody contrasting relational structures (akin to a curriculum), this enables the networks to learn to make analogies more effectively.  It is not clear what the authors mean by “Structure Mapping Theory prior” in these and other places in the text, but it does not give a good intuition for what this highly related piece of work contributes to the literature. When relating the current paper to other work, the authors also do not explain specifically how their architecture differs from those proposed in previous work, which is important as the architecture is the point of novelty in this paper.\n\nThe authors justify this work by making the point that in order to learn to make analogies, a researcher cannot rely solely on using class contrasts when presenting candidate options (i.e. the method proposed by Hill et al 2019), as these require knowledge of which relationship each candidate embodies, knowledge which may not always be available to a labeller. This is a fair point to make, however here the authors are proposing a new architecture that they also train and test using the contrast method illustrated by Hill et al, so it is not as though this paper proposes an alternative to class contrasts in training. In fact, the results show that the authors’ method works best in comparison to other methods precisely under the Hill et al contrast conditions. As far as I can see, there are no test splits in which the architecture proposed here performs best under Normal (not contrast) training and test conditions. The argument that clever contrast training is “not feasible” therefore seems a little misplaced.\n\nInterestingly, in the appendix the authors show that the adaptivity of their inference engine does not enable better analogy inference than either of the static engines. It makes the reader question why the authors left this extra complexity in their design when they show it has no impact on performance, and presumably requires additional resources to train (many more weights for this extra section in the Inference Engine). The authors do not provide a clear logical reason for its inclusion. \n\nResults:\nThe generalisation results appear to be somewhat cherry picked. The previously published dataset the authors use includes 5 generalisation splits, which each assess a different component of systematic generalisation. However the authors present results from just 2 of these 5 splits in the main text (Table 1 and Figure 6), and no explanation is given for why these 2 splits were the ones presented. In appendix D the authors do provide results for their method across all generalisation splits, but there is no comparison to other methods here so the values are hard to interpret. To properly and fairly communicate their results, the authors should present data (as in Table 1) for all generalisation splits and compare these fully to other work.\n\nFigure 5 presents performance results for just the Visual Relationship Encoder, but split across all 5 different generalisation splits and also across different contrast conditions, as though each of these combinations of conditions tests a different hypothesis. However, as the encoder takes as input only the first three (source) context squares, and generalisation splits are constructed so as to isolate different relationships between the source and target squares, these many (45) bars dont tell the reader much more than what a single bar would have communicated: that the Visual Relationship Encoder predicts with about ~85% accuracy. Correspondingly the bars are all virtually identical. Presenting them all in this way implies that the Visual Relationship Encoder might be somehow performing different types of systematic generalisation, when this is really not the case.\n\nFinally, it is nice that the authors try to determine how important the relationship output from the Visual Relationship Encoder was for performance of the Inference Engine. However the way this specific experiment was run confounds the usefulness for the Inference Engine of knowing the relation per se, with potential correlations in performance between the two network modules on each analogy example. The correct way to run this experiment would be to artificially provide the Inference Engine with all possible relationship labels (a different 'fake' relation label per run), and then evaluate whether the Inference Engine performed better when the relationship provided matched the true underlying relationship.\n", "summary_of_the_paper": "This paper proposes a new architecture for learning visual analogies, based on Gentner’s Structure Mapping Theory for how humans might draw analogies. Gentner’s theory proposes representing the relationships between objects explicitly, so that this relational structure can be reused in new domains (and suggests that this commonality in structure is what permits analogies to be made between perceptually dissimilar objects). The authors propose a neural network model architecture and test it on the Raven’s Progressive Matrices dataset. The proposed architecture first splits a series of ‘source’ visual scenes into objects, attributes and the relationships between those scenes, before feeding just the relationship head into a second network. The second network then switches between two different architectures (depending on the relation fed in). The architecture in the second network (whichever is chosen) receives the ‘source’ relationship and two ‘target’ scenes before trying to predict which of a set of 4 candidate  ‘target’ scenes completes the visual analogy between source and target. The authors test their architecture on the generalisation splits in the RPM dataset and compare test accuracy results to the baseline models used by Hill et al, 2019. The authors show that their model (which builds in additional architectural structure) performs better at a subset of tests than more general architectures.", "summary_of_the_review": "Overall,  this paper tested an interesting hypothesis based on a classic idea from cognitive science. However the results were not thoroughly analysed, and there was a  lack of clarity in the explanation of ideas that made it hard to situate this contribution of this paper in the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UciJqzcDztU", "reviewer": "Reviewer_Kq5G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths and weaknesses\n- strengths:\n  - Clear explanation of motivation\n  - The ability to reason with analogies seems like an important step towards intelligent systems that reason in the same way that humans do\n  - The proposed method performs well against existing work and the presented baselines\n- weaknesses (see detailed comments below): \n  - The information available at training time may have made the task too easy, so that meaningful comparison with existing work cannot be made\n  - The setup for the task seems very simple, and it doesn't seem appropriate to call it \"analogy learning\" \n  - Claims about the effectiveness of NSM's components could be better supported by ablation studies\n  - NSM achieves good performance when semantically-contrasting alternatives are not available at train time. But given the simplicity of the environment, it's unclear when collecting these alternatives would be an obstacle. \n\n# General comments:\n- Sec 4: The task as it is presented in this work seems strictly easier than the task attempted by Hill 2019 or any of the baselines.  Analogical reasoning is a hard problem, and part of that difficulty is learning to divide the world into relationship categories. In the NSM approach, this information is provided in a fully-supervised manner at train time. Building this into the method seems to make the problem much easier. Thus, a fair comparison with the baselines cannot really be made, since none of the other baselines have access to the ground truth relationship at train time.\n- Sec 4: Additionally, this approach does not scale well with increasing relationship types. The number of relationships is fixed, and if you want to expand the set of relationships that NSM can handle, you must manually categorize examples according to these new relationships and manually create new neural-module layouts.\n- In Table 2, it is noted that lower inference engine performance correlates with lower encoder performance. It is claimed that this shows that the encoder is an essential part of the system. However, this is not necessarily the case. It could be that there is some quality that makes the encoder fail and also makes the inference engine fail. Perhaps this corresponds with a type of relationship. For example, the encoder and inference engine could both have a hard time on relationship $X$ and an easy time on everything else. One way to check for this, would be to stratify results by relationship type. The other way to directly test this would be to run the inference engine with the correct relationship and the inference engine with the incorrect relationship and compare inference engine performance. \n- I'm not entirely convinced that the system as described can be said to have \"learning analogies\". The domain seems very simple and far away from the type of reasoning that humans do, in which much more sensory data must be abstracted away. However, I'm aware that it is common to talk about RPM in the context of analogy learning, so I willing to concede that others may disagree. \n\n# Clarifying questions/suggestions\n- Sec 3: Is there a reason why the \"extrapolation\" category is mentioned here, but doesn't appear in the results in section 5?\n- Sec 4: \"Instead of depending on explicitly labeled candidates for mapping relational structure,\"\n  - Can you explain what this means? Is it very difficult to collect semantically plausible candidates? In this domain, can't the examples be generated automatically? And doesn't the proposed NSM approach also requires access to the labeled relationship for each training example anyways? \n- Sec 4.1: Why is the encoder also predicting the object and the attribute? Only the predicted relationship matters for the Analogy Inference Engine, so why is multi-task learning necessary? If it is supposed to improve the prediction of the relationship, could you include an ablation study that shows this?\n- Sec 4.2: Can you explain why the architecture for the Unary module and Binary module need to be different? A priori, either architecture seems like a reasonable choice. For example, I could imagine that for a unary relation, selecting the candidate involves knowing the relationship and features of the first two matrices and nothing else.\n  - In Appendix A, it seems like both layouts perform equally well across all types of relationships. So why is there a need to dynamically choose between them?\n- Sec 5.1 and 5.1.1 \"contrasting, normal, as well as mixed candidates\"\n  - I think I'm missing something here. If I understand correctly, the candidate types have no bearing on the relationship encoder training? The candidates are never fed as input to the encoder, correct? So why are the results given according to candidate selection type?\n- 5.2 \"The candidate probabilities of the full context NSM model are calculated as a 1:1 ensemble between the probabilities from both the inference engine and the encoder\"\n  - This seems like an important piece of information that should probably appear earlier -- perhaps in section 4?\n  - I'm not understanding how the encoder is used to select the candidates. The Visual Relationship Encoder as shown in figure 2 has no way to take the candidate as input, correct?\n- Sec 5 Table 1: What numbers are being reported here? The title suggests that each cell contains the test perfomance on the contrasting data and the test performance on the normal data, separated by a slash. But the caption suggests that each cell contains the train/test performance.\n- Sec 5 Table 1: \"Our approach achieves the best generalization accuracy for the most number of possible train/test candidate scenarios across all models (4/12)\"\n  - Where does the \"12\" come from? Should it say 36, since there are 36 cells in the table?\n- Sec 5: Do Table 1 and Figure 6 show the same information? If so, it makes sense to only keep one of them.\n\n# small suggestions\n- Figure 5: The y-axis can be scaled down\n- The numbers on Figures 5 and 6 were a little too small\n- If Table 1 and Figure 6 show the same information, one of the them can probably be omitted.\n- pg 6. \"fed parallely\" --> \"fed in parallel\" perhaps?\n\n", "summary_of_the_paper": "This paper tackles the problem of analogical reasoning. In particular, it presents a framework for learning the Raven Progressive Matrices (RPM) task, an abstract analogy task.\n\nIn the RPM task, a sequence of three images from a source domain are given. There is some relationship that holds for the sequence, e.g. the third image is the union of the first two. Then, given an incomplete sequence of two images from the target domain, the third image must be chosen from a list of four possible candidates.\n\nThe proposed Neural Structure Mapping (NPM) system consists of two pieces. The first piece is the Visual Relationship Encoder. Given the source sequence of images, the encoder predicts the type of relationship exhibited in the sequence. This information is passed to the second piece, the Analogy Inference Engine. The architecture of the engine is assembled dynamically, according to the predicted relationship. The assembled network takes the target sequence and the candidate matrices as input and selects the completion of the sequence from among the candidates.\n\nThe encoder is trained with the ground truth relationship labels. The engine is trained using the ground truth candidate labels. \n\nThe paper presents an experiment to test systematic generalization, in which particular attributes are held out during train time. The NSM system is found to achieve better performance.\n\nIn contrast to Hill 2019, which presents the model with semantically-contrasting alternative candidates at train time, NSM achieves good performance even when the alternative candidates are not necessarily semantically related. \n", "summary_of_the_review": "The problem of learning to reason with analogies is important, and this paper makes an attempt towards doing this in a structured way, which seems like a principled move. However, as it is currently, I cannot recommend the paper for publication. The task being learned has been made too easy, so that claims about effectiveness cannot be supported. But if the authors work on a version of this study, in which the relationship types are latent/unobserved at train time, I think it would be a very promising step towards better analogical-reasoning systems. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O_5_im6SiMw", "reviewer": "Reviewer_opcn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The authors clearly know a lot about the literature about analogical reasoning and RAVEN. This paper provides an excellent review of related work in a concise manner.\n\n- Technical details are largely missing. There's even not a single equation in the entire paper. This makes it difficult to believe that the proposed method would work as the authors claimed. And if it does, it poses additional questions on how general it is to other analogical problems in general.\n\n- Results are weak. It's hard to see if the proposed method is indeed better than prior work in terms of performance. For instance, under the extrapolation setting with mixed data, the WReN is much better than the proposed one. This should be the best evidence to see if the proposed method is better.\n\n- Figures are very difficult to understand. For instance, there's no explanation of what \"filter\" means in Figure 1. Bar plots in Figure 6 should have standard diveration or some other metrics in addition to the bar.", "summary_of_the_paper": "Motivated by the structure mapping theory, this paper proposes an ad-hoc solution to address the classic RAVEN problem. The authors claim the proposed method is more general than previous methods while achieving good performance.", "summary_of_the_review": "Good literature review and motivation, but difficult to tease out the technical contributions with weak and ambiguous results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2ZDCl6WA6B", "reviewer": "Reviewer_z4sn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like how the authors motivate the work in a way that can be connected to the classic structure mapping engine, as this theory is an agreed-on approach in cognition for how analogy is made. And certainly this theory could be better leveraged for the learning community to solve the task of interest in this work. I also appreciate the engineering efforts the authors make in this work that notably improve over earlier methods in this challenging task.\n\nHowever, I'm particularly concerned on the novelty of this work. Using a neural modular architecture is anything but new in the reasoning community. Earlier works have extensively explored the potential of such an approach in VQA, and moreover, the NS-CL method further manages to jointly train both the visual component and the reasoning component. In this work, however, the visual component is neither jointly trained with the analogy part, nor is the analogy module any different from existing modular construction methods. What's worse, there are only two possible network layouts, a much simpler design than existing cases in VQA. Therefore, I only see this work as an application of the existing modular method in a new domain, with very little novelty. In addition, the DRT model in Zhang et. al. is exactly the modular approach, despite the fact that they call it DRT instead of neural modular network. And that model was among the very first trials in such tasks.\n\nBesides, the authors motivate this intuitive method with SMT, which becomes no more than a story wrapper in the paper. The theory is there, but apart from *claiming* the theory and the model are related, there is not any connection between the theory and the model: in what sense is analogy inferred? The method just simply fixes the structure and selects it based on a predicted label trained from ground truth.\n\nFor the experimental evaluation, I'm not sure why the authors only select the simple models to compare. Over the years, there have been quite some new works in this topic, some have already been mentioned in the literature review, *e.g.*, MXGNet, LEN, Rel-AIR, SCL, et. al.. What about comparing with them? I also notice that the dataset used is an incomplete version the traditional Raven task. What is the model's performance on the full matrix?\n\nBesides, if you only need the relation, why bother predicting object and attribute?", "summary_of_the_paper": "A model called Neural Structure Mapping (NSM) is introduced to solve the task of abstract visual analogy making. The NSM model consists of a visual relationship encoder and an analogy inference engine. The visual relationship encoder extracts the visual domain elements, including object, attribute, and relation, while the analogy inference engine is a neural modular architecture that constructs the model layout based on the relation and predicts the final answer. On the dataset proposed by Hill et. al., the NSM shows better performance than other baselines.", "summary_of_the_review": "I'm concerned on the novelty of the work and the evaluation performed.\n\nI have read the rebuttal and other reviews. As the authors agree on most of the comments, I decide not to raise the score. But I do acknowledge that\n1. The DRT may not fall into the modular network domain though the idea looks very similar.   \n2. I cannot quite connect the model and the structural mapping theory despite other reviewers' opinion.\n3. I would still encourage comparison with other models as the tasks considered are related.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript presents an approach to handling abstract visual analogy tasks where panels of drawings are shown with a missing entry. One of a number of candidate drawings must be chosen to complete the panel. Reviewers brought up several concerns:

1. The task performed was made considerably easier by providing additional annotations at training time. This was not the case in the original task in prior work that the manuscript builds on. No convincing explanation was provided as to why this change is critical to accommodate the manuscript's contributions.

2. A key feature of the approach, the adaptive modular design, does not seem to contribute much. The authors rightly point out this may be a limitation of current benchmarks. Reviewers were sympathetic to this view but that leaves the manuscript in a tough spot: a central contribution cannot be evaluated. What is even worrisome is that without evaluating the effectiveness of the adaptive design we cannot know if it is working at all. What if adaptivity is required for some future analogy tasks but it turns out that this approach, despite seeming to be adaptive, falls short?

3. Another contribution, the multi-task encoder does not seem to provide much value in ablation experiments. The manuscript would be improved if this feature was removed or its usefulness was demonstrated.

A number of smaller issues were also brought up by reviewers.

Throughout the responses to reviewers the authors highlight that their central contribution is incorporating a structure mapping prior "The central contribution of our method is introducing a structure mapping prior ...". I would like to draw the author's attention to the fact that they had to remind 3 of the 4 reviewers to focus on this rather than another aspect of the work. That clearly indicates that the manuscript and work needs a shift in focus. I would suggest that authors double down on their structural mapping prior, eliminate all other features which turned out to be controversial or impossible to evaluate, and demonstrate the utility of their idea in two domains, i.e., including another domain. This would really highlight the core contribution.

Unfortunately, what may turn out to be a good idea, the structural mapping prior, is lost among many other complexities. I hope the authors are not discouraged and that we see this line of work again in the future.
Decision: Reject