Analogical Reasoning for Visually Grounded Compositional Generalization

Children acquire language subconsciously by observing the surrounding world and listening to descriptions . They can discover the meaning of words even without explicit language knowledge , and generalize to novel compositions effortlessly . In this paper , we bring this ability to AI , by studying the task of multimodal compositional generalization within the context of visually grounded language acquisition . We propose a multimodal transformer model augmented with a novel mechanism for analogical reasoning , which approximates novel compositions by learning semantic mapping and reasoning operations from previously seen compositions . Our proposed method , Analogical Reasoning Transformer Networks ( ARTNET ) , is trained on raw multimedia data ( video frames and transcripts ) , and after observing a set of compositions such as “ washing apple ” or “ cutting carrot ” , it can generalize and recognize new compositions in new video frames , such as “ washing carrot ” or “ cutting apple ” . To this end , ARTNET refers to relevant instances in the training data and uses their visual features and captions to establish analogies with the query image . Then it chooses a suitable verb and noun to create a new composition that describes the new image best . Extensive experiments on an instructional video dataset demonstrate that the proposed method achieves significantly better generalization capability and recognition accuracy compared to state-of-the-art transformer models . 1 INTRODUCTION . Visually grounded Language Acquisition ( VLA ) is an innate ability of the human brain . It refers to the way children learn their native language from scratch , through exploration , observation , and listening ( i.e. , self-supervision ) , and without taking language training lessons ( i.e. , explicit supervision ) . 2-year-old children are able to quickly learn semantics of phrases and their constituent words after repeatedly hearing phrases like “ washing apple ” , or “ cutting carrot ” and observing such situations . More interestingly , they will also understand new compositions such as “ washing carrot ” or “ cutting apple ” , even before experiencing them . This ability of human cognition is called compositional generalization ( Montague ( 1970 ) ; Minsky ( 1988 ) ; Lake et al . ( 2017 ) ) . It helps humans use a limited set of known components ( vocabulary ) to understand and produce unlimited new compositions ( e.g . verb-noun , adjective-noun , or adverb-verb compositions ) . This is also one of the long-term goals of Artificial Intelligence ( AI ) , e.g . in robotics , where it enables the robot to learn new instructions that they have never heard before . Nevertheless , contemporary machine intelligence needs to overcome several major challenges of the task . On one hand , learning compositional generalization can be difficult without using datahungry models . The power of existing language models mainly rely on large-scale language corpora ( Lake & Baroni ( 2017 ) ; Pennington et al . ( 2014 ) ; Devlin et al . ( 2018 ) ) . They are still inadequate at compositional generalization ( Marcus ( 1998 ) ; Lake & Baroni ( 2018 ) ; Surı́s et al . ( 2019 ) ) . Their goal is to recognize training examples rather than focusing on what is missing from training data . On the other hand , the designed model should close the paradigmatic gap ( Nikolaus et al . ( 2019 ) ) between seen compositions and new compositions . For instance , given seen verb-noun compositions “ 1A ” and “ 2B ” ( the digit indicates verb , the letter indicates noun ) , the model should be able to link seen compositions to new compositions ( like “ 1B ” or “ 2A ” ) in completely new cases . Different from previous work ( Johnson et al . ( 2017 ) ; Baradel et al . ( 2018 ) ; Santoro et al . ( 2017 ) ) , we bring the power of compositional generalization to state-of-the-art language models by incorporating Analogical Reasoning ( AR ) ( Gentner & Smith ( 2012 ) ; Littlemore ( 2008 ) ; Vosniadou & Ortony ( 1989 ) ) . An analogy is a comparison between similar concepts or situations , and AR is analogical semantic reasoning that relies upon an analogy . The human brain spontaneously engages in AR to make sense of unfamiliar situations in every day life ( Vamvakoussi ( 2019 ) ) . Inspired by the AR process in the human brain , we design the counterpart for machine language acquisition . To this end , we create a language model that generate appropriate novel compositions by relevant seen compositions , and forming analogies and appropriate arithmetic operations to express the new compositions ( e.g . “ washing carrot ” = “ washing apple ’ + “ cutting carrot ” - “ cutting apple ” ) . We describe this process in three steps : association , reasoning , and inference , as shown in Figure 1 . Given an image ( a video frame in our case ) and a narrative sentence describing it , we mask the main verb-noun composition from the sentence , and ask the model to guess the correct composition that completes the sentence , considering the provided image . To this end , we propose a novel self-supervised and reasoning-augmented framework , Analogical Reasoning Transformer Networks ( ARTNET ) . ARTNET adopts a multimodal transformer ( similar to ViLBERT ( Lu et al . ( 2019 ) ) ) as its backbone to represent visual-textual data in a common space . Then it builds three novel modules on top of the backbone that corresponds to the aforementioned AR steps : association , reasoning , and inference . First , we design Analogical Memory Module ( AMM ) , which discovers analogical exemplars for a given query scenario , from a reference pool of observed samples . Second , we propose Analogical Reasoning Networks ( ARN ) , which takes the retrieved samples as input , selects candidate analogy pairs from the relevant reference samples , and learns proper reasoning operations over the selected analogy pairs , resulting in an analogy context vector . Third , we devise a Conditioned Composition Engine ( CCE ) , which combines the analogy context vector with the representations of the query sample to predict the masked words and complete the target sentence with a novel composition . We show how ARTNET generalizes to new compositions and excels in visually grounded language acquisition by designing experiments in various evaluations : novel composition prediction , assessment of affordance , and sensitivity to data scarcity . The results on the ego-centric video dataset ( EPIC-Kitchens ) demonstrate the effectiveness of the proposed solution in various aspects : accuracy , capability , robustness , etc . The project code is publicly available at https : //github.com/XX . The main contributions of this paper include the following : • We call attention to a challenging problem , compositional generalization , in the context of machine language acquisition , which has seldom been studied . • We propose ideas supported by human analogical reasoning : approximating new verb-noun compositions by learned arithmetic operations over relevant compositions seen before . • We propose a novel reasoning-augmented architecture for visually grounded language acquisition , which addresses the compositional generalization problem through association and analogical reasoning . • We evaluate the proposed model in various aspects , such as composition prediction , validity test , and robustness against data scarcity . The results show that ARTNET achieves significant performance improvements in terms of new composition accuracy , over a large-scale video dataset . 2 ARTNET : ANALOGICAL REASONING TRANSFORMER NETWORKS . Our goal is to develop a framework that can support multimodal compositional generalization through learning in a visual-textual environment . The proposed framework learns to acquire the meaning of phrases and words from image-sentence pairs and to create novel compositions via reasoning . We call the framework Analogical Reasoning Transformer Networks ( ARTNET ) , due to its ability to establish analogies with the previously seen , relevant scenarios , and perform reasoning operations to generalize a composition for the new scenario . Figure 2 illustrates an overview of ARTNET , which is composed of a multimodal encoder backbone , followed by three main modules : Analogical Memory Module ( AMM ) , Analogical Reasoning Networks ( ARN ) , and Conditioned Composition Engine ( CCE ) . We elaborate each component in the rest of this section . 2.1 MULTIMODAL ENCODER BACKBONE . The backbone network is responsible for encoding image-sentence pairs into compositional semantic representations . To achieve this , we utilize the emerging multimodal transformers ( e.g . UNITER ( Chen et al . ( 2019 ) ) or ViLBERT ( Lu et al . ( 2019 ) ) ) , which have recently achieved great success in various vision-language tasks . These models take a set of visual and textual tokens ( e.g . objects and words ) , and extract a multimodal embedding for each token , which is contextualized by all the other tokens through layers of multi-head attention . We follow the architecture of UNITER , as it performs slightly better than ViLBERT and other similar models . Note that since our goal is language acquisition , we intentionally do not use the pretrained weights of UNITER , which are trained on a large-scale corpus . Instead , we train the backbone from scratch on our limited data . 2.2 ANALOGICAL MEMORY MODULE . AMM plays the role of analogical association . Like finding a useful puzzle piece , we propose AMM to discover the most useful reference samples for analogical reasoning in a target scenario . Given a target image-sentence pair ( query ) , where some tokens in the sentence are masked , we randomly select N ( N = 200 in our experiments ) sample image-sentence pairs from the training data to create a reference pool , and find the Top-K most relevant exemplars from that pool . To this end , we measure a multimodal relevance score between the query and each reference . Here , we use the initial embedding of each token on the query and reference samples as described in the Section 2.1 . Given a target and a reference sample , we define the multimodal relevance score as a combination of visual and text relevance between the corresponding sets of tokens . For visual tokens , we compute the mean cosine similarity of every pair of tokens from the query and reference token sets . For the language part , the contextual background words that are not masked can provide linguistic clues for semantic relevance . Thus , we compute the Jaccard Index ( Hamers et al . ( 1989 ) ) between two sentences as textual relevance . Specifically , the multimodal relevance score is svl = 1 2 · ( |WT ∩WR| |WT ∪WR| + 1 + ∑ i ∑ j cos ( vTi , vRj ) Nv 2 ) ( 1 ) whereWT andWR are the set of target words and reference words , Nv is the number of visual token pairs , and vTi and vRj represent the visual embeddings of the ith visual token of the query and the jth visual token of the reference . After computing the scores , AMM ranks reference samples with respect to their relevance scores and selects the Top-K most relevant samples for the given query . 2.3 ANALOGICAL REASONING NETWORKS . Given the retrieved analogical exemplars , we devise a neural network with reasoning ability to enrich the original representation of the masked compositions by making analogies with the seen compositions . The idea is to exploit the semantic relation mapping between the candidate analogy composition and the target composition . To this end , we represent the target masked composition as a query vector q , by concatenating the multimodal transformer embeddings of the masked words of that composition ( typically a verb and a noun from the target sentence ) and learning the representations of ordered constituents in a composition based on a Long Short-Term Memory ( LSTM ) ( Zhou et al . ( 2015 ) ) . Next , we apply the multimodal encoder backbone ( as mentioned above ) on the retrieved analogy samples , and parse each sample into candidate analogy compositions ( pairs of tokens ) . Since the goal is language acquisition , we do not rely on predefined grammar rules or pretrained models to generate the candidate compositions , such as applying part-of-speech tagging and taking each verb-noun pair . Instead , we enumerate all pairs of adjacent words from each retrieved sentence , and all pairs of detected image regions from each retrieved image . The multimodal resulting set of pairs are called analogy pairs hereafter . The core of ARN consists of three neural network modules for analogy attention , analogical reasoning , and analogy transformation . Analogical attention learns the importance of each pair of candidate analogy composition and query vector respectively and generates analogy aggregation from each modality independently . Analogical reasoning is designed to learn the appropriate arithmetic operations from analogy compositions for reasoning . It consists of modality-wise transformations and Neural Arithmetic Logic Units ( Trask et al . ( 2018 ) ) with multiple layers of Neural Accumulator ( NAC ) ( Trask et al . ( 2018 ) ) . NAC is a simple but effective operator that supports the ability to learn addition and subtraction . This module is applied on the analogy pairs , and computes a single vector that represents the output of some reasoning operations , optimized for our task through gradient descent . Through the analogy transformation , ARN generates the sequential representations of final analogy context vector . Specifically , ARN can be denoted as cma = ∑ j αmijh m j , αij = exp a ( [ rmi ; r m i+1 ; q ] , [ r m j ; r m j+1 ; q ] ) ∑ k exp a ( [ r m i ; r m i+1 ; q ] , [ r m k ; r m k+1 ; q ] ) Analogical Attention , ( 2 ) hc = fNAC ( [ gv ( c v a ) , gl ( c l a ) ] T ) Analogical Reasoning , ( 3 ) c = LSTM ( hc ) Analogy Transformation , ( 4 ) where v and l represent the vision and language modalities , rmi and r m i+1 ( m is modality indicator ) are the image regions or text words of candidate analogical compositions . gv and gl are modality transformations that contains two-layer fully connected networks with ReLU activation and dropout , and T represents matrix transpose . The output of ARN is the vector c , which is called analogical context vector , and will be used to augment the composition representations .

Summary: This paper explores the problem of generalizing to novel combinations of verbs and nouns in a task for captioning video stills from videos about cooking. The paper introduces a new dataset based off of EPIC-Kitchens (Damen et al. 2018) which masks out verbs and nouns and splits the evaluation data into seen combinations of verb/noun pairs and unseen combinations of verb/noun pairs, challenging a model to generate captions for pairs which were not seen during training.