Analogical Reasoning for Visually Grounded Compositional Generalization
Authors: Bo Wu
Haoyu Qin
Alireza Zareian
Carl Vondrick
Shih-Fu Chang
Venue:  (2021)

Abstract: Children acquire language subconsciously by observing the surrounding world and listening to descriptions. They can discover the meaning of words even without explicit language knowledge, and generalize to novel compositions effortlessly. In this paper, we bring this ability to AI, by studying the task of multimodal compositional generalization within the context of visually grounded language acquisition. We propose a multimodal transformer model augmented with a novel mechanism for analogical reasoning, which approximates novel compositions by learning semantic mapping and reasoning operations from previously seen compositions.  Our proposed method, Analogical Reasoning Transformer Networks (ARTNet), is trained on raw multimedia data (video frames and transcripts), and after observing a set of compositions such as "washing apple" or "cutting carrot", it can generalize and recognize new compositions in new video frames, such as "washing carrot" or "cutting apple". To this end, ARTNet refers to relevant instances in the training data and uses their visual features and captions to establish analogies with the query image. Then it chooses a suitable verb and noun to create a new composition that describes the new image best. Extensive experiments on an instructional video dataset demonstrate that the proposed method achieves significantly better generalization capability and recognition accuracy compared to state-of-the-art transformer models.

Under review as a conference paper at ICLR 2021
ANALOGICAL REASONING FOR VISUALLY GROUNDED COMPOSITIONAL GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Children acquire language subconsciously by observing the surrounding world and listening to descriptions. They can discover the meaning of words even with- out explicit language knowledge, and generalize to novel compositions effort- lessly. In this paper, we bring this ability to AI, by studying the task of multimodal compositional generalization within the context of visually grounded language ac- quisition. We propose a multimodal transformer model augmented with a novel mechanism for analogical reasoning, which approximates novel compositions by learning semantic mapping and reasoning operations from previously seen com- positions. Our proposed method, Analogical Reasoning Transformer Networks (ARTNET), is trained on multimedia data (vidco frames and transcripts), and after observing a set of compositions such as "washing apple" or "cutting carrot", it can generalize and recognize new compositions in new video frames, such "washing carrot" or "cutting apple". To this end, ARTNET refers to relevant in- stances the training data and uses their visual features and captions establish analogies with the query image. Then it chooses suitable verb and noun to cre- ate new composition that describes the new image best. Extensive experiments on an instructional video dataset demonstrate that the proposed method achieves significantly better generalization capability and recognition accuracy compared to otate-of-the-art transformer models.
INTRODUCTION
Visually grounded Language Acquisition (VLA) is an innate ability of the human brain. It refers to the way children learn their native language from scratch, through exploration, observation, and listening (i.e., self-supervision). and without taking language training lessons (i.e., explicit supervi- sion). 2-year-old children are able to quickly learn semantics of phrases and their constituent words after repeatedly hearing phrases like "washing apple". or "cutting carrot" and observing such sit- uations. More interestingly, they will also understand new compositions such as "washing carrot" "cutting apple", even before experiencing them. This ability of human cognition is called com- positional generalization (Montague) (1970); Minsky| (1988); [ake et al. (2077)). It helps humans use limited of known components (vocabulary) to understand and produce unlimited new com- positions (e.g. verb-noun, adjective-noun, or adverb-verb compositions). This is also one of the long-term goals of Artificial Intelligence (AI), e.g. in robotics, where it enables the robot learn new instructions that they have never heard before.
Nevertheless, contemporary machine intelligence needs to overcome several major challenges of the task. On one hand, learning compositional generalization can be difficult without using data- hungry models. The power of existing language models mainly rely on large-scale language corpora (Lake & Baroni] (2017); [Pennington et al. (2014); Devlin et al. (2018)). They are still inadequate at compositional generalization (Marcus ( 1998); Lake & Baroni (2188); Surís et al. (2019). Their goal is to recognize training examples rather than focusing what is missing from training data. On the other hand, the designed model should close the paradigmatic gap (Nikolaus et al. (009)) between seen compositions and new compositions. For instance, given seen verb noun compositions "LA" and "2B" (the digit indicates verb, the letter indicates noun), the model should be able to link seen compositions to compositions (like "IB" or "2A") in completely new cases.

---

Under review as a conference paper at ICLR 2021
Image-Sentence Sentence Data
1.Discover analogical
2. Analogical reasoning retrieved reference samples
Predicting the correct composition
examples from reference set
post
TOLe
camoo
[wash)] (corroi] with water
(WN)) (0033)
Figure 1:
We propose multimodal language acquisition approach inspired by human language learning processes consisting of three steps: association, reasoning, and inference.
Different from previous work (Johnson al. (0177); Baradel et al.) (2018) : Santoro et al. (017)),
bring the power of compositional generalization to state-of-the-an language models by incorporat-
ing Analogical Reasoning (AR) (Gentner & Smith 2012); Littleorre (2008); Vosniadou & Ortony (1989 ). An analogy is a comparison between similar concepts or situations, and AR is analogical semantic reasoning that relies upon an analogy. The human brain spontaneously engages AR to make sense of unfamiliar situations in every day life [Vamvakoussi (019)). Inspired by the AR pro- cess in the human brain, we design the counterpart for machine language acquisition. To this end, we create a language model that generate appropriate novel compositions by relevant seen composi- tions, and forming analogies and appropriate arithmetic operations to express the new compositions (e.g. "washing carrot" "washing apple' "cutting carrot" "cutting apple"). We describe this process in three steps: association, reasoning. and inference, as shown in Figure
Given an image (a video frame in case) and narrative sentence describing it, mask the main verb-noun composition from the sentence, and ask the model to guess the correct composition that completes the sentence, considering the provided image. To this end, we propose novel self-supervised and reasoning-augmmnnted framework, Analogical Reasoning Transformer Networks (ARTNET). ARTNET adopts multimodal transformer (similar to ViLBERT (Lu et al. 2019)) as its backbone to represent visual-textual data in a common space. Then it builds three novel modules on top of the backbone that corresponds to the aforementioned AR steps: association, reasoning, and inference. First, we design Analogical Memory Module (AMM), which discovers analogical exemplars for given query scenario, from a reference pool of observed samples. Second, we propose Analogical Reasoning Networks (ARN), which takes the retrieved samples as input, selects candidate analogy pairs from the relevant reference samples, and learns proper reasoning operations over the selected analogy pairs, resulting an analogy context vector. Third, devise a Conditioned Composition Engine (CCE), which combines the analogy context vector with the representations of the query sample to predict the masked words and complete the target sentence with a novel composition.
We show how ARTNET generalizes to new compositions and excels in visually grounded language acquisition by designing experiments in various evaluations: novel composition prediction, assess- ment of affordance, and sensitivity to data scarcity. The results the ego-centric video dataset (EPIC-Kitchens) demonstrate the effectiveness of the proposed solution in various aspects: accu- racy, capability, robustness, etc. The project code publicly available at hhttps//giituu..com/XXX
The main contributions of this paper include the following:
We call attention to a challenging problem, compositional generalization, the context of machine language acquisition, which has seldom been studied.
We propose ideas supported by human analogical reasoning: approximating new verb-noun compositions by learned arithmetic operations over relevant compositions seen before.
We propose a novel paasoning-augmente architecture for visually grounded language ac- quisition, which addresses the compositional generalization problem through association and analogical reasoning.
We evaluate the proposed model in various aspects, such as composition prediction, validity test, and robustness against data scarcity. The results show that ARTNET achieves signif- icant performance improvements in terms of new composition accuracy, over a large-scale video dataset.
2

---

Under review as a conference paper at ICLR 2021
A. Analogical Memory Module (AMI))
Conditioned Compositor Engine (CCE)
wash carrot
Target Tolien Bequence
Muttmedal Ensader
MASK
UNITER
dirooioeed Transtormation
Prediction
Losa Function
Sep
Erc
Referencc Discaveringg
Enard Waigh
a Ouery
Contest
Pairel Makoy Gunsil
Peference
Start
Sequerios
furoumbd
[a1
:
Sep
UNITER
Laae
atby
Aound Antreneti:
Networks NAA
An Anology
Stari
Eserice
1200 CarreE
Sap
End
youss Trwee
Ject Tokan
Analogical Reasoning Networks (ARN)
Figure 2: Analogical Reasoning Transformer Networks (ARTNeT): The proposed reasoning-
augmented architecture for compositional generalization via analogical reasoning.
2 ARTNET: ANALOGICAL REASONING TRANSFORMER NETWORKS
Our goal is to develop a framework that can support multimodal compositional generalization through learning in visual-textual environment. The proposed framework learns acquire the meaning of phrases and words from image-sentence pairs and to create novel compositions via rea- soning. We call the framework Analogical Reasoning Transformer Networks (ARTNET), due to its ability to establish analogies with the previously seen, relevant scenarios, and perform reasoning operations generalize composition for the new scenario. Figure 2 illustrates an overview of ARTNET, which is composed of multimodal encoder backbone, followed by three main modules: Analogical Memory Module (AMM), Analogical Reasoning Networks (ARN), and Conditioned Composition Engine (CCE). We elaborate each component in the rest of this section.
2.1 MULTIMODAL ENCODER BACKBONE
The backbone network is responsible for encoding maage-sentence pairs into compositional semantic representations. To achieve this, we utilize the emerging multimodal transformers (e.g. UNITER (Chen al. (2019) or ViLBERT (Lu al. (2019))), which have recently achieved great success in various vision-language tasks. These models take a set of visual and textual tokens (e.g. objects and words), and extract multimodal embedding for each token, which is contextualized by all the other tokens through layers of multi-head attention. We follow the architecture of UNITER, as it performs slightly better than ViLBERT and other similar models. Note that since our goal is language acquisition, we intentionally do not use the pretrained weights of UNITER, which are trained on a large-scale corpus. Instead, we train the backbone from scratch on our limited data.
2.2 ANALOGICAL MEMORY MODULE
AMM plays the role analogical association. Like finding a useful puzzle piece, we propose AMM to discover the most useful reference samples for analogical reasoning in target scenario. Given a target image-sentence pair (query), where some tokens in the sentence are masked, we randomly select N (N 200 in our experiments) sample image-sentence pairs from the training data create a reference pool, and find the Top-K most relevant exemplars from that pool. To this end, we measure multimodal relevance score between the query and each reference. Here, we the initial embedding of each token on the query and reference samples as described in the Section 2. Given target and reference sample, we define the multimodal relevance score as combination of visual and text relevance between the corresponding sets of tokens. For visual tokens, we compute the mean cosine similarity of every pair of tokens from the query and reference token sets. For the language part, the contextual background words that are not masked can provide linguistic clues for semantic relevance. Thus, we compute the Jaccard Index (Hamers et al. (1999) between two sentences as textual relevance. Specifically, the multimodal relevance score is
Σ. Σ, cos( 1+
1
WT O WR
N.
(1)
WT W W
2
3

---

Under review as a conference paper at ICLR 2021
where Wr and Wa are the set of target words and reference words, NN, is the number of visual token pairs, and Pr, and FR, represent the visual embeddings of the iu. visual token of the query and the Jeh visual token of the reference. After computing the scores, AMM ranks reference samples with respect to their relevance scores and selects the Top-K most relevant samples for the given query.
2.3 ANALOGICAL REASONING NETWORKS
Given the retrieved analogical exemplars, we devise neural network with reasoning ability to enrich the original representation of the masked compositions by making analogies with the seen compositions. The idea is to exploit the semantic relation mapping between the candidate analogy composition and the target composition. To this end, we represent the target masked composition as a query vector 4, by concatenating the multimodal transformer embeddings of the masked words of that composition (typically a verb and noun from the target sentence) and learning the repre- sentations of ordered constituents in a composition based on a Long Short-Tern Memory (LSTM) (Zhou al. (2055). Next, apply the multimodal encoder backhone (as mentioned above) on the retrieved analogy samples, and parse each sample into candidate analogy compositions (pairs of tokens). Since the goal is language acquisition, we do not rely predefined grammar rules or pretrained models to generate the candidate compositions, such as applying part-of-speech tagging and taking each verb-noun pair. Instead, we enumerate all pairs of adjacent words from each re- trieved sentence, and all pairs of detected image regions from each retrieved image. The multimodal resulting set of pairs are called analogy pairs hereafter.
The core of ARN consists of three neural network modules for analogy attention, analogical rea- soning, and analogy transformation. Analogical attention learns the importance of each pair of can- didate analogy composition and query vector respectively and generates analogy aggregation from each modality independently. Analogical reasoning is designed to learn the appropriate arithmetic operations from analogy compositions for reasoning. It consists of modality-wise transformations and Neural Arithmetic Logic Units (Trask et al. 2018)) with multiple layers of Neural Accumulator (NAC) (Trask et al. (2018)). NAC is simple but effective operator that supports the ability to learn addition and subtraction. This module is applied on the analogy pairs, and computes a single vector that represents the output of some reasoning operations, optimized for our task through gradient descent. Through the analogy transformation, ARN generates the sequential representations of final analogy context vector. Specifically, ARN can be denoted as
exp expa([>"::1 [rT]: :44)
ca' Lngh Qij Σn expaall" (rr" r g) (P: (
Analogical Attention,
(2)
he = (Nnc(luo (c (c())) c c- (STM(h)))
Analogical Reasoning,
Analogy Transformation,
where e and represent the vision and language modalities, r."h and r (m modality indicator) the image regions or text words of candidate analogical compositions. Io and gI are modality transformations that contains two-layer fully connected networks with ReLU activation and dropout, and T represents matrix transpose. The output of ARN is the vector c, which is called analogical context vector, and will be used to augment the composition representations.
2.4 CONDITIONED COMPOSITION ENGINE
After analogical reasoning, we create a potentially novel composition based on both the initial com- prehension of the given scenario and the result of analogical reasoning. To this end, CCE is designed to enrich the representations of the masked tokens through a conditioned learning network. It takes the analogical context vector as contextual knowledge. Let {I1,..., mi, mi.+1..... ZNN be the input elements of a multimodal encoder, and m or m 1 are the l-th layer of the representations of the masked words. CCE uses the multimodal transformers transform the embedding features of both each masked word and the analogical context vector. Then it predicts the masked word by aggregating from linguistic clues of all the other unmasked elements. The embedding feature h; of
4

---

Under review as a conference paper at ICLR 2021
the i-th masked word computed by:
h,t w ReLU(W] oftt) ) b2t1
Context-conditioned
(5)
h GeLU(W!!+ 2 b5+2)
Feed-forward
(6)
h4 = LayerNorm(h
Feed-forward
(7)
where W W b/+1 and b,+1 learnable weights and biases for the context-conditioned
transformation. W/+? and 6/42 learnable weight and bias for feed-forward transformation, respectively. Given the contextual representation of the masked word h;, the model predicts the masked word by multiplying its contextual representation with word embedding matrix which trained with the rest of the network, vi; = 2wh4.
2.5 LEARNING OBJECTIVES
Masked Composition Acquisition Our model learns to perform language acquisition by filling in masked words. At both train and test time, we give the model a target example, with reference set sampled from the training dataset. Note that our masking policy different at training time and test time (details in Section During training and validation, randomly mask multiple words and visual tokens. However, during test, we only mask one verb-noun composition.
Objective Function We train the model acquire words by directly predicting them. We measure the Cross-Entropy loss between the predicted word ú; and true word w. over a vocabulary of size C, denoted L, This objective is the same in the original BERT (Devlin also learn visual reconstruction via a regression task. The visual loss Z. is a Triplet Metric loss (Weinberger al. (2006)) force linear projection of i/ to be closer ou (1:) than Φb. (Vkges) and
al. (2018)). We
(.(-) is the visual representation network (ResNet). Because the objectives are complementary, we train the parameters of the proposed model by minimizing the sum of losses:
min
Wi log(t::) Amax(0, + ||u (v:))l ||0: (n(ow)))ll)
(8)
where À E IR. is the parameter to balance the loss terms (modalities), is the triplet margin and θ represents the trainable parameters the entire network.
EXPERIMENTS
We compare our method (ARTNET) and baselines on new and seen composition acquisition tasks. To demonstrate the quantitative and qualitative results, we evaluate our method in a variety of aspects including performance comparison, validity test, incremental data setting, and case study.
3.1 EXPERIMENT SETTINGS
Dataset We use the instruction dataset EPIC-Kitchens (Damen al. (2018) in our experiments. The dataset consists of 55 hours of egocentric videos across 32 kitchens. Each video clip has a narrative sentence, For each video frame, we use the object bounding boxes officially provided with EPIC-Kitchens, which are produced by running faster R-CNN (Ren al. (2055) too. We discard the object labels due to strict constraints in the language acquisition scenario. Importantly, we train our model our video dataset only, without any language pre-training We partition the dataset to ensure new com- positions used in testing have never been seen in training. The test subsets of new/seen composition tasks are disjoint (without overlap) and equal-size (each set contains 29K samples related to 238/861 unique new/seen compositions); the train subsets of the tasks are identical (142K samples with 3675 unique compositions). The dataset was prepared by the following steps: (1) We took all annotated compositions from the Epic-Kine Kitchens and split them two groups of compositions seen consist- ing of 3675 unique compositions, 124 verbs and 254 nouns (vocabulary size), and new consisting of 238 new compositions, 90 verbs and 149 nouns. All the verbs nouns of the new compositions in the seen compositions too. (2) We randomly sampled 140K samples from the seen composition group shared train set and each seen composition has least sample in the train set. We
create image-sentence pairs by selecting one key frame of each video clip.
5

---

Under review as a conference paper at ICLR 2021
New Composition
Seen Composition
Models Metries (%)
Top-1 Acc.
Top-5 Acc.
Top-1 Acc.
Top-5 Acc.
BERT (from scratch)
2.25
6.85
6.47
26.42
BERT (pre-trained)
2.06
10.41
7.89
27.10
Multimodal BERT (language pretrained)
3.05
23.25
14.79
35.05
Multimodal BERT (from scratch)
5.97
36.63
23.22
57.81
Proposed ARTNET (from scratch)
8.00 (+2.03)
41.08 (+4.45)
22.25
57.74
Table 1: Performance Comparison: Accuracy comparison on new and seen composition tasks.
also sampled a set of seen composition samples (about 20% of the train set) as the test set of seen compositions. (3) The samples of the new compositions were used create the test set of the new compositions (size about 20% of the train set). To ensure the focus is on new compositions, rather than new words, we removed new compositions that contain new words not seen in the train set.
Evaluation Metries To compare the performance of our model against baselines, we evaluate the ability to acquire new or seen compositions (e.g. verb-noun word pairs). During testing, the model takes paired image- sentence data as input, and the target verb-noun composition replaced with a special "mask" token. The objective is to predict the masked words from the complete vocabulary of all words (unaware the part of speech). We adopt Top-1 and Top-5 accuracy to measure the performance, which calculates the average accuracy over the predicted nouns and verbs with the top N highest probability. The prediction is correct when both noun and verb are correctly predicted.
3.2 PERFORMANCE COMPARISON WITH BASELINES
We compare our model with state-of-the-art methods of language modeling or multimodal language modeling (implementation and more training details of our model and baselins are in the Section Appendix). We consider three variants of the BERT family (Devlin et al. (2018): Chen et al. (2019)
BERT (language only) is powerful language transformer model (Devlin et al. (2018) that achieved state-of-the-art art performance in multiple tasks. We adopt two learning settings: (1) train the BERT model from scratch on the sentences of our task data; (2) utilize the pre-trained BERT model (Zhu et al. (2015); Chelba et al. (2013)) and fine-tune BERT on the task data. Note that the pretrained BERT has the advantage of scale data, which is different with language acquisition.
Multimodal BERT (language vision) is a generalization of BERT that receives both visual and textual tokens as input, which usually come from an image-sentence pair. Here we use the same architecture as our backbone UNITER (Chen et al. (2019), ) and adopt a pre-trained ResNet-18 (He et al. (2016) to extract visual features. We train from scratch on our task dataset.
new compostion predictions [pul] [sousage] onto oan targtt [saussee onta pan rfft put pan
rett: put of
nefi: all por
reft: on
rett: o rett, Lif net): pan met, hilling neth: sousage, mi2 jillimg
raft: pan
0.11
ref2: barn sousage
ref3: out
ref2: burn sassage
3.56
ref3: put id
Attenttin
nett: spoon, mit Iit
0404
Atrention Score
0.2 0.3
atterdon scores (bext anelogy)
attention Scores Ortsual analogy)
Figure 3: Reasoning Attention Visualization over Analogy Pairs:
The bar charts shows attention scores in our reasoning module for textual (left chart) or visual (right chart) analogy pairs.
The quantitative and qualitative results are illustrated in Table I and Figure (in the appendix sec- tion), respectively. For new composition prediction, as shown in Table our model can significantly achieve 2.03% and 4.45% improvement over state-of-the-att baseline on Top-1 and Top-5 accuracy respectively. Such improvement results from the analogical reasoning based on relevant reference samples. For seen composition prediction, the proposed model is nearly unchanged. To under- stand what the analogical reasoning module has learned, we visualize the attention distribution over multimodal analogy pairs in ( analysis and more examples are in Figure and and para-

---

Under review as a conference paper at ICLR 2021
Composition Acquisitinn
Affordance Accuracy
Table Validity Test: Per-
Methods/Settings
Overall
New Comp
Seen Comp
formance comparison of va-
BERT (w/o vision)
81.67%
81.59%
81.72%
lidity accuracy.
Multimodal BERT
85. 75.
84.78%
86.36%
Proposed A ARTNET
86.48%
86.01%
86.75%
graphs in the appendix section). As shown in Figure 7 the model can successfully retrieve relevant reference samples that contain analogical pairs (detailed analysis is in the appendix section).
3.3 EXPERIMENTS ON DIFFERENT TRAINING SIZES
To evaluate our model low-data regime and simulate the language acquisition process of child, we train on different scales of data. Specifically, we consider 5 different training data size percent- ages (TPs) (100%, 80%, 60%, 40% and 20%), and plot the performance of our method compared to the stringest baseline in Figure 4 The ARTNET achieves 13.44% Top-5 accuracy with only 20% of training data, which has larger gap with the baseline, suggesting our stronger generalization ability.
Multimodal BERT
ARTNet
Correct Predictions
Performance (Top-5 Acc)
50
[peel, carrod]
Top 5
21. 5%
Top
[mmooinng garle]
(peaiigg eaded]
12.7%
hso plod
40
+3.5% on average
(elcee inft
9.2%
11.1%
paking onio)
3.0%
30
Jonil (oarrt)
[out, poacr)
10 8%
Sparil
(novs, chooad]
5.8%
20
Wrong Pradictions
10
Top 5
jaaks,
18.2%
[closs, cupboard 18.0%
jris anlrd]
15 5%
Espen tndgg))
14.7%
jes, spaghed]
2%
balo, drawed)
20
60
80
drain foco
12.0%
[sloiing, container) 10.0%,
Training Data Sizes (%)
Robustness: Performance Figure 5: Case Study: Top-5 prediction results for curve with different sizes of training compositions (highlight words are masked ground-truth data.
Figure 4:
compositions to be predicted).
3.4 VALIDITY TEST EVALUATION
We propose a way to aim at assessing whether the predicted results follow human commonsense in the real world, called "Validity Test", based the definition of affordance (Gibsonn 2014)). Affor- dance is a binary relationship between each noun and verb, which indicates whether certain verb typical involve a certain noun as an argument (Baber (2018)). For examples, "cutting pizza" and "cutting carrot" happen in our daily life, but we never see "cutting plate" or "cutting water". We annotate the validity accuracy for 8400 Top-1 verb-noun compositions (5704 of which appear in our dataset) predicted by each model given any test image. The type-miis mismatched compositions, such as "meat apple" (noun-noun) or "peel cut" (verb-verb), are also defined as invalid prediction. In Ta- 2 we show the Top-1 validity accuracy (the ratio of the number of affordable or valid predictions to all) for each model. Overall, ARTNET achieves higher validity than our baselines 86.5%. The validity of our new composition results outperforms Multimodal BERT/BERT with 1.23% improvements, which is significant. We attribute such gains composition validity to the novel components in model discovery of multimodal analogy reference samples and reasoning such samples to generate the language description. The method's emphasis on analogical relations to realistic reference samples helps improving the validity of the compositions predicted by the model.
3.5
CASE STUDY
To analyze the strengths and failure cases, select some examples shown in Figure Intuitively, vague visual clues such as ambiguous action and tiny objects, are hard to recognize. However, we observe ARTNET has unique ability to disambiguate such difficult cases, shown in the top row of Figure [ Nevertheless, the model fails in some cases, such as take/stir and close/open. This mainly due the fact that used still keyframes rather than dynamic videos, which discards

---

Under review as a conference paper at ICLR 2021
Target (ground-truth)
Prediction Results
Target (ground-fruth)
Prediction Results
ARTNet
Multimodall BERT
(slice) [squash)
[slice] [squash)
[remove]] the skin of [ginger)
[untaa] [opan] the skin of [garti]]
(slice] [carro]
[slice] [carro]]
[baka] the skin (ginger]
[our] (squesh)
[(lic]] [enifi]
(out] the skán [ginged]
[oper] the akin of [orion] [jníee] [peel] the skin [garlkc]
[salad]
(removingl
[ginner)
[peel]
[chop) [squash)
[slica] [potato]
[remaving] the skin of (packaging)
[chop] [squash]
[emove the skin of [garlic]
Figure 6: Case Study: The validity test results. The two words in brackets are masked and to be predicted the target composition. Green or red color indicates valid invalid prediction.
motion information. Future work should incorporate spatio-temporal features alleviate this lim- itation. Moreover, we provide several cases with Top-5 prediction results to show the affordance and rationality of our method's prediction. Considering Figure we observe that not only the Top-1 results, but also most of the Top-5 are in line with human commonsense, and superior to the baseline.
4 RELATED WORK
Compositional Generalization Compositional generalization is an important open reasoning challenge (Keysers et al. (2019); Chang et al. (2018); Loula et al. (2018) and a priority achieve human-like abilities (Bastings et al. (2018). regarding the ability to compose simple constituents into complex structures (Nikolaus et al. (2019). From the traditional view, numerous studies utilize linguistic principle to explore compositionaliy (Mitchell & Lapata (2008); Baroni & Zamparelli (2010). With recent advances, there are more attempts solve compositional generalization with neural networks for the tasks with synthetic commends or interactions (Lake (2019); Russin et al. 2019 Li et al.] (2019); Kato et al. (2018)). The current machine intelligence still lacks compo- sitional generalization ability, because they are prone to fail such tests in realistic or natural scenarios (Nikolaus et al. (2019): Loula et al. (2018); Keysers et al. (2019)). Morcover, several recent works also try to enhance compositional generalization for natural language (Nikolaus et al. (2019)) with pretrained representations or lexical knowledge. But there are few works addressing this emerging and valuable challenge for language acquisition in multimodal reasoning view.
Visually Grounded Language Acquisition (VLA) Like the language acquisition of kids, VLA is task of acquiring language constituents from seratch within visual-textual environment. Although the works of grounded language learning achieve the good progress in many tasks (such as visual captioning YYu & Siskind (2013); Kiela et al.) (2017): Ross et al.) (2018)) or robotics (Matuszek) )2018))), there are major differences between VLA and multimodal pretraining, including the goal of learned model, the data amount needed, and the the architecture condition (e.g., whether use the predefined language parsing). The former attempts to learn from scratch aspects of language models (e.g. compositional semantics) and effectively use limited samples, while the latter uses context from very large training corpora (Lu et al. (2019); Chen et al.] (2019)) to learn language representation by a data-hungry model. Several recent works further address acquiring concrete language including word representation (Kottur al. (2016); [Surss et al. (2019)), compositional semantics (Jin et al.] (2020)) from scratch via visual clues, which are more related to our task. Our work seeks improve, in the form of reasoning, the constituent generalization new compositions.
CONCLUSION
In this paper, take step towards visually grounded language acquisition, by studying the prob- lem of compositional generalization in the state-of-the-an multimedia language models. Inspired by the human brain's analogical reasoning process, we propose to form compositions by calling observed compositions and relating them to the new scenario through learned arithmetic method, Analogical Reasoning Transformer Net-
operations. Our proposed reasoning-augmente works, achieves superior compositional generalization capability compared to the state-of-the transformers, which results in sigmificant and stable performance gains in unseen compositions over a large-scale instructional video dataset.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chris Baber. Designing smart objects to support affording situations: Exploiting affordance through understanding of forms of engagement. Fronfiers in psychology, 9:292, 2018
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 121--121,40 2018.
Marco Baroni and Roberto Zamparelli. Nouns are vectors, adjectives are matrices: Representing adjective-nour constructions in semantic space. In Proceedings of the 2010 Conference on Em- pirical Methods in Natural Language Processing, pp. (183-1193 Association for Computational Linguistics, 2010.
Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: Scan both left and right. arXiv preprint arXiv: 2018.
Michael B Chang, Abhishek Gupta, Sergey Levine, and Thomas L Griffiths. Automatically composing representation transformations as a means for generalization. 187.004007.044400, 2018.
arXiv preprint
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint 1312.v3312.3000 2013.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXhv:1909.11728 117400 2019.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scal- ing egocentric vision: The epic-kitchens dataset. In European Conference Computer Vision (ECCV), 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Dedre Gentner and Linsey Smith. Analogical reasoning. Encyclopedia of human behavior, 130:
130, 2012.
James J Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014.
Lieve Hamers et al. Similarity measures in scientometric research: The jaccard index versus salton's cosine formula. Information Processing and Management, 25(3):315-18 1989.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Xisen Jin, Junyi Du, and Xiang Ren. Visually grounded continual learning of compositional seman- tics. arXiv preprint 20Xv:20005000885 2020.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reason- ing. Proceedings of the IEEE International Conference on Computer Vision, pp. 2989-29999 2017.
Keizo Kato, Yin Li. and Abhinav Gupta. Compositional learning for human object interaction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 234-251, 2018.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashu- bin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measur- ing compositional generalization: A comprehensive method on realistic data. arXiv preprint arXiv: 1rXiv:1912.09711 2019.

---

Under review as a conference paper at ICLR 2021
Douwe Kiela, Alexis Conneau, Allan Jabri, and Maximilian Nickel. Learning visually grounded sentence representations. arXiv preprint arXiv::707.0633 06320, 2017.
Satwik Kottur, Ramakrishna Vedantam, José MF Moura, and Devi Parikh. Visual word2vec (vis-
w2v): Learning visually grounded word embeddings using abstract scenes. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4985-4994, 2016.
Brenden M Lake, Compositional generalization through meta eqquence-t--sequenee learning. In Advances in Neural Information Processing Systems, pp. 9788 7788-9798, 2019.
Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compositional
skills of eqqeence-o-sequuee recurrent networks. arXiv preprint arXiv: 1.00000 2017.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills sequence-to-seeeee to-sequence recurrent networks. In Proceedings of the 35th International Con- ference Machine Learning (ICML), pp. 2879-28828 2018.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and bruin sciences, 40, 2017.
Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness. Compositional generalization for prim- itive substitutions. arXiv preprint vrXiv:1910.02612 2019.
Jeannette Littlemore. The relationship between associative thinking, analogical reasoning, image
formation and. Confronting metaphor in use: An applied linguistic approach, 173:199, 2008.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
170000171 2017.
Joao Loula, Marco Baroni, and Brenden M Lake. Rearranging the familiar: Testing compositional generalization in recurrent networks. arXiv preprint arXiv:1807.07545 2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and- aadoo---d--angugge tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Gary Marcus. Rethinking eliminative connectionism. Cognitive psychology, 37(3):243-282, 1998.
Cynthia Matuszek. Grounded language learning: Where robotics and nlp meet. In ISCAI. pp. 5687- 5691, 2018.
Marvin Minsky. Society of mind. Simon and Schuster, 1988.
Jeff Mitchell and Mirella Lapata. Vector-based models of semantic composition. In proceedings of ACL-08: HLT, pp. 236-244, 2008
Richard Montague. Universal grammar. Theoria, 36(3):373-398, 1970.
Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliott. Compo- sitional generalization in image captioning. arXiv preprint arXiv:1909. 0440044 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Candace Ross, Andrei Barbu, Yevgeni Berzak, Battushig Myanganbayar, and Boris Katz. Grounding language acquisition by training semantic parsers using captioned videos. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2647-2656, 2018.
Jake Russin, Jason Jo, Randall C O' O'RRilly, and Yoshua Bengio. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv: /00.0/:/99440000000 2019.
10

---

Under review as a conference paper at ICLR 2021
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pp. 4967-4976, 2017.
Dídac Surís, Dave Epstein, Heng Ji, Shih-Fu Chang, and Carl Vondrick. Learning to learn words from narrated video. arXiv preprint 19Iiv:991..
2019.
Dídac Surís, Dave Epstein, Heng Ji, Shih-Fu Chang, and Carl. Vondrick. Learning to learn words from visual scenes. arXiv preprint arXiv:1911.1122: 2019.
Andrew Trask, Felix Hill, Scott Reed, Jack Rac, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, pp. 8035-8044, 2018.
Xenia Vamvakoussi. The use of analogies in mathematics instruction: Affordances and challenges. In Cognitive Foundations for Improving Mathematical Learning, pp. 247-268. Elsevier, 2019.
Stella Vosniadou and Andrew Ortony. Similarity and analogical reasoning. Cambridge University Press, 1989.
Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in neural information processing systems, pp. 1473- 1480, 2006.
Haonan Yu and Jeffrey Mark Siskind. Grounded language learning from video described with sen- tences. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 53-63, 2013.
Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis Lau. A c-Istm neural network for text classification. arXiv preprint arXiv:1511008633 2015.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE international conference on computer vision, pp. 19 27, 2015.
11

---

Under review as a conference paper at ICLR 2021
APPENDIX
Implementation Details In all experiments, training is conducted on 2 GPUs for 200 epochs. In each mini-batch, 256 samples are drawn for each GPU and in each sample, image regions are cropped from the whole image and resized to 112 112. The transformer encoder in all models has the same configuration: 4 layers, a hidden size 384, and self-attention heads each layer. Besides,
AdamW optimizer Loshchilov & Hutter (207)) with the learning rate 3e-5. For A optimizer Loshchilov & Hutter (2017), we set the update coefficients, for averages of gradient and its square (/1,, (22), and on denominator 0.9, 0.999, le-4. During training, we mask out text tokens 3 of the time and image tokens 6 masking strategy with BERT Devlin et (2018).
of the time and follow the same setting of random
During training and testing, for each target sample, we randomly select 200 remaining samples as the corresponding reference In experiments, utilize Top-K (K = reference samples get involved in analogical reasoning.
In evaluation, calculate the accuracy based the same random mask strategy training process. An early stop strategy is utilized based on the Top-5 Acc. in validation that the training process will terminate the validation Top-5 Acc. doesn't increase again.
Marget-Reference Case Study From the two examples shown in Figure 7 it's easy to observe that the model can successfully retrieve relevant reference samples which contain analogical pairs, by computing their visual and language similarity the target composition. For the correct predic- tion "peel carrot", the model discovers "stir carrot", "peel potato" and "cut potato" for analogical reasoning, while for "wash knife", the model retrieves "wash plate", "take knife" and "rinse knife" as reference samples.
Target
References
peel carrot
stir carrot
peel polato
cut potato
Target
References
wash knife
wash plate
take knife
rinse knife
Figure 7: Case Study: The target sample and top samples discovered from the reference set.
Reasoning Attention Distribution over Multimodal Analogy Pairs In the four examples shown in Figure 7 we provide two examples for correct prediction and another for wrong predictions. From the bottom two examples, the model learns compositional semantics from both visual and textual constituents. For the correct prediction of new composition "put sausage". the model learns to acquire and approximate novel composition from our multimodal reasoning. The reasoning has more attention on the textual phrase of the first reference sample "put oil in pan" and visual regions "sausage, whole image" of the second reference sample. This implies that the model learns textual and visual semantics from reference samples and compose them under similar scenarios as context. A similar phenomenon also appears in the second prediction example for seen composition "chop onions". The model able to learn the phrase "chop onion" from different modalities. For wrong prediction results, the minor visual differences of several verbs will lead to wrong reasoning (e.g. "remove skin of garlic"), although the model can successfully retrieve relevant reference samples with aid by contextual information. While chopped garlic is not visually recognized by the adopted vision model, the attention distribution of the visual analogy pairs for the example seems to focus on the garlic but also be confused by "cream, whole image". Meanwhile, the accuracy of the reasoning
12

---

Under review as a conference paper at ICLR 2021
new compostiin predition: [pui] [aaueaee] onto pan terget: [oui] [seusage] omo pan reft: put al pan
reft: put
10.48
rift:
reft: oll
reft: rett,
10.02
reft: pan
reft: nets
ref?: burn sausage
raft: put
rez:: turn sausage
ref2: seusage,
rufts put lid
nett: attoritin
Altention Scores
attendonn scores (text analogy)
0.2 0.3 0.4
composiiion (chop) spring target: (chop) spring [amions] chop pesper
rell chop pepper
re22 chop onion
retli popper, ref_uuiimgg
renz: chop onion
ref3: shop dricd tamato
wef): chop
6.0
0.08
0.12
reft
Actandionn Sco
Atternien Scorm
0.
6.0
attention scones (test anstogy?
oitention scores (visuel enalogy)
new composition predicton: [enven] [garlic]
target: [remove]
reft: garli
rant:
ret:: crush garlc
refz: cream,
ret2: ramove cream
0.04
nelt peach, peet
ref2: ref3: peelings
refs: remove peetings
0.04
reits redd reta/uulliin
nets pocl, jin
0.4
attention scores (best anslogy)
attention (visual
new composilion prediction: [ohopping] [pan)
barget get (iry) [pen)
reft: turning meat sen
ref). tumling meat
0.84
reft par, onion
0.55
reft: mest
reft: pon
b.18
0.0
garlic
0.0
nett: onlon, rett, Jil.ing
0.0
gartic 0.0
Aitertion Scores
roft: pan, 1eft, Mill
Witention Scores
0.
0.8
02
0.5
atbention
Figure 8: Reasoning Attention Visualization over Multimodal Analogy Pairs (Correct or Wrong Predictions): The bar charts shows attention scores our reasoning module for textual (left chart) or visual (right chart) analogy pairs.
also impacted by the relevant samples (e.g., "fry in pan"). When the model didn't discover "fry"" in the relevant references and not distinguish the actions ("fry" and "chop") in target and reference sample, the reasoning would easier to get the wrong prediction.
13


Reviews:
[{"review_id": "QlbFmvUC7zQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model that takes in a keyframe from a video and emits the noun and verb best matching what is being done in the frame.\nAt inference time, the noun and verb have never been seen in combination with one another, but have each been seen paired with other nouns/verbs at training time.\nThe paper presents a complex, three part model (ArtNet) to tackle this challenge, as well as unimodal linguistic baselines.\nNotably, the evaluation does not include vision-only baselines or pretrained model toplines, making it difficult to assess exactly what ArtNet is learning and where its advantage lies.\n\nQuestions:\n\n- There are multiple references to \"creation\", e.g., \"create novel compositions\" which makes the method sound like it's performing generation. From my understanding, though, ArtNet is purely discriminative, taking in a keyframe and predicting a noun and verb. However, there is one line in the paper that says \"We also learn visual reconstruction via a regression task.\", which makes it sound like there's a formulation of ArtNet that maybe takes in a noun and verb and produces a keyframe image (using a GAN, maybe? Or a nearest neighbor lookup?), and so does \"create novel compositions\". If that's the case, it isn't described, and this image reconstruction task is never mentioned again in the paper or described in any equations.\n\n- Were pretrained ViLBERT/UNITER run alone as a topline? Establishing how much is lost in performance due to lack of pretraining + how this method addresses that with sparse data would make a much stronger argument, I feel. In particular, we would want to see pretrained ViLBERT/UNITER and then pretrained + ArtNET to give a sense of how performance will change as models have seen huge amounts of aligned data.\n- [Related] In Table 1 what's the intuition for language pretrained mBERT/UNITER/ViLBERT falling behind from scratch? Why use language pretraining and not vision pretraining (e.g., topline)?\n\n\n- In Eq (1), why use cosine similarity for visual embeddings but then back off to surface forms for words? Was cosine similarity for word representations computed by UNITER tried? What is the intuition for this not working, if it did not?\n\n- Not sure in Eq (4) what the sequence input to the LSTM is; does c range over some sequence? Doesn't the sum already collapse that?\n\n- \"To ensure the focus is on new compositions, rather than new words, we removed new compositions that contain new words not seen in the train set.\" All new words or just nouns/verbs? It's a big advantage/relaxation on the test set to have no OOV tokens.\n\n- Why no vision-only baseline? Strip word contexts at training time except noun/verb, then predict only noun/verb at test. A lot of this could be basically object recognition followed by activity recognition or strong priors on p(activity | object) (e.g., always \"open\" or \"close\" for cabinets).\n\n- \"outperforms Multimodal BERT/BERT with 1.23%/4.42% improvements, which is significant\"; what statistical significance test was used? How many random initializations were tried to establish the average performance numbers for comparison between performance populations?\n\nSuggestions for Improvement:\n\n- \"We call attention to a challenging problem, compositional generalization, in the context of machine language acquisition, which has seldom been studied.\" This is poorly worded, since compositional generalization is well and commonly studied, to the point that even in this paper there is a section in the related work about it. Major workshops also list compositionality as a topic of interest, so I don't think it's fair at all to say that this has \"seldom been studied\" [ https://sites.google.com/view/repl4nlp2020/home ]. In the context of language acquisition specifically, emergent communication work focuses heavily on composition [ https://sites.google.com/view/emecom2019/home ].\n\n- ViLBERT in intro, UNITER in description of method, \"Multimodal BERT\" (mBERT?) in Table 1. What was used? Needs to be consistent in presentation.\n\n- \"We discard the object labels due to strict constraints in the language acquisition scenario.\" Because Faster RCNN is trained on ImageNet, which is based on WordNet, the object categories still exist in the form of supervision. The model has a linguistically-motivated notion of what an \"object\" is that can be traced to the WordNet. This should be acknowledged; you can't actually \"get rid\" of linguistic information inherent in a pretrained Faster RCNN.\n\n- \"But there are few works addressing this emerging and valuable challenge for language acquisition in a multimodal reasoning view.\" This paper does not really tackle language acquisition, though? There's a restriction so that the test set has no OOV words, even. I think the claims and presentation of the paper need to be carefully re-scoped.\n\n- There is a lot of focus on \"learned arithmetic operations\" but no analysis as to what exactly this component ends up doing or learning.\n\nNits:\n- Typo Introduction \"a language model that generate\" S/V agreement.\n- Figure 1 doesn't feel like it communicates anything about the method, and does not seem tied to the caption. Are boxes (1, 2, 3) meant to represent the association, reasoning, and inference steps? What's happening in each?\n- \"The results show that ARTNET achieves significant performance improvements in terms of new composition accuracy, over a large-scale\nvideo dataset.\" strange wording makes it sound like ARTNet is outperforming a dataset, not a method.\n- \"We train the model to acquire words by directly predicting them.\" this sounds like the model will be predicting words unseen at training time, which is not so. In particular, \"acquire\" here sounds like the model will be exposed to the word at most once (at inference time) and then be able to memorize that exposure in sequence.\n- Typo? In 3.1 \"by running faster R-CNN too\" what is the \"too\" pointing to? Do you run Faster-RCNN somewhere?\n- Typo 3.3 \"stringest baseline\" strictest?\n- Figures 4 and 5 are so close together their captions bleed together and are really difficult to disentangle.\n- Typo \"than our baselines 86.5\" makes it sound like the baselines achieved 86.5.\n- Typo 4 \"the goal of learned model\" missing \"the\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "9U5Xx29qJj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper describes a model to predict unseen verb-noun compositions from a fixed set of verbs and nouns, given a training set of seen compositions. For an image with incomplete sentence, the model learns to select a set of relevant examples from the train set to serve as reference pairs of image-sentence mapping, and then uses these pairs to draw distribution over potentially unseen verb-noun pairs.\n\n\nStrengths\n- The problem is well-motivated\n- The model is shown to perform well both quantitatively and qualitatively, along with different experimental setups such as validity, low-data\n\n\nWeaknesses and concerns\n\n- Hanging notations and unexplained variables:  Unfortunately this is a bottleneck for me to understand the paper thoroughly\n    - Equation 2\n        - What is $a$?\n            - $a$ is used as a function within the softmax expression - $\\exp a()$, but $a$ is also used as a subscript $c_a$, and since I cannot find any reference for $a$, I’m unclear as to what it actually is\n        - The leftmost expression marginalizes out $j$ to yield $c_a^m$, but where does $i$ go? Are $a$ and $i$ related?\n        - What is $k$? Is $k$ the cardinality of the set of all analogy pairs?\n    - Equation 8 - Triplet loss\n        - What is $v_i$?\n- Results:\n    - Potential non-determinism\n        - You mentioned that for each test sample, you randomly pick 200 examples and then select top-K examples from them (ref: Section 2.2). Now if I understood this correctly, this introduces non-determinism in your results. Is it fair to say that your results should report standard deviations along with the numbers currently stated? Are the improvements statistically significant then?\n    - Lack of ablation\n        - Frequency bias in predictions? What’s the accuracy when you pick the most common verb, noun, and verb-noun pair?\n        - Choice of K in Top-K\n            - The Appendix (Implementation details) mentions that you pick K as 3. Why? What happens when you increase or decrease the value. Is there any insight to be gained from here?\n        - Accuracy numbers for nouns and verbs separately\n            - Again, this is an attempt to uncover biases in your predictions. You mark a prediction correct if both verb-noun pair gets correctly classified, but which one of them gets more wrong? Is there something to be learned here?\n    - Trends in Table 1\n        - BERT pre-trained v/s scratch: Why is there a reverse trend in top-1 and top-5 accuracy? Is there a conclusion to be drawn here?\n\n- Misc\n    - Number of objects (visual tokens) and words (lingual tokens) in an image?\n        - What is the distribution of number of object and words in the dataset for an image-sentence pair?\n    - Figure 3 - Attention over visual\n        - What is “ref1_full_img”, “ref2_full_img”? Are complete images also included in analogy pairs along with objects?\n    - Section 3.4\n        - “We annotate the validity accuracy for 8400 Top-1 verb-noun compositions” How many people were involved? What does the distribution look like?\n    - Reasoning module:\n        - (e.g. “washing carrot” = “washing apple’ + “cutting carrot” - “cutting apple”)\n            - Is this arithmetic observed in predictions?\n\n\nMinor concerns (suggestions, typos, etc.)\n- Section 3.4: grammatical errors\n\n\nPreliminary Rating and its justification\n\nThe current version of the paper seems to have a couple of loose ends in terms of clarity and completeness of experiments. However, I’m open to changing my rating if the above concerns are addressed to some extent.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9DmuLMNZETj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper explores the problem of generalizing to novel combinations of verbs and nouns in a task for captioning video stills from videos about cooking. The paper introduces a new dataset based off of EPIC-Kitchens (Damen et al. 2018) which masks out verbs and nouns and splits the evaluation data into seen combinations of verb/noun pairs and unseen combinations of verb/noun pairs, challenging a model to generate captions for pairs which were not seen during training. \n\nThe paper also proposed a model, ARTNet, to address this task. The model works by first finding nearest neighbor examples in the training data that are similar in (masked) caption and image, then embedding these similar examples into an \"analogical embedding\", which is used to condition a network that fills in the masked verb and noun in the target example.\n\n*************\n\nReason for score: \n\nThe proposed dataset is interesting, and the proposed model works well on it. However, I have some concerns about the dataset's construction and suggestions for additional experiments.\n\n*************\n\nStrengths:\n\n1. The proposed model performs well on the proposed task.\n1. The baselines seem solid.\n1. Includes experiments on different amounts of training data.\n\n*************\n\nWeaknesses:\n\n1. Some missing citations for compositional reasoning on novel images/visual contexts, including datasets like VQA-CP (Agrawal et al. 2017), CLEVR CoGent (Johnson et al. 2017), Room to Room (Anderson et al. 2018), and ALFRED (Shridhar et al. 2020).\n1. Would be good to see experiments with a differing number of images in the random pool -- how does this affect performance to have a larger or smaller pool? There seems there would be a tradeoff between accuracy and computing resources necessary to run inference, but what exactly is that tradeoff? Similarly, would be good to see a similar analysis for the number of top candidate examples that are passed to the ARN module from the AMM module.\n1. Many terms in the model section should be defined more formally, e.g., \"candidate analogy composition\", \"ordered constituents in a composition\", \"pars(ing) each sample into candidate analogy compositions\", \"multimodal resulting set of pairs / analogy pairs\", \"linguistic clues\".\n\n*************\n\nQuestions:\n\n1. How does the split for EPIC-Kitchens handle synonyms of verbs? It seems to split based off of verb/noun combinations, but what about things that are (at least partially) synonyms such as \"cut\", \"slice\", \"chop\"? This also seems to be relevant for evaluation -- is it strict for synonyms? (I am looking at the case study, which proposes \"slice\" squash, \"chop\" squash\", and \"cut\" squash, all of which seem reasonable to me.)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new dataset and compositional task based on the EPIC Kitchens dataset.  The goal is to test novel compositions and to build a transformer based network specifically for this inference (by analogy). Specifically, the analogy here references the use of nearest neighbors in the dataset.  There are a lot of concerns raised by reviewers which require a large number of changes to the presentation of the manuscript and they are not at present convinced by the current setup or experiments. Explicitly motivating which pretraining methods do or do not violate which aspects of composition and what role other factors like synonymy play in generalization is necessary. Several aspects of the claims made in the paper and in the discussion are big claims that require substantial discussion and analysis (e.g. the surprising weakness of pretrained models) which the reviewers do not feel can be so easily explained away (e.g. by domain shift).
Decision: Reject