Unsupervised Vision-Language Grammar Induction with Shared Structure Modeling

1 INTRODUCTION . Humans are amazing at extracting knowledge efficiently from our complicated and multimodal world , leveraging both redundant and complementary information from visual , acoustic , or tactile cues . Investigating into such behavior , neuroimaging and neuroanatomical studies suggested that specific brain regions are dispatched to support the convergence of auditory and visual word comprehension ( Calvert et al. , 1997 ; Calvert & Campbell , 2003 ; Campbell , 2008 ; Keitel et al. , 2020 ) . For example , auditory regions are involved in lip reading by receiving signals from visual cortices ( Bourguignon et al. , 2020 ) . These observations imply a mysterious “ shared world ” when perceiving multimodal signals , functioning as a centralized processor for understanding fused information . In contrast , such phenomena have not been revealed in modern state-of-the-art VL models , most of which process visual and language signals in two separate streams and fuse the results only in the final stage ( Ma et al. , 2015 ; You et al. , 2018 ; Shi et al. , 2019 ; Kojima et al. , 2020 ; Zhao & Titov , 2020b ) . In this work , we dive into the “ shared world ” for vision-language ( VL ) representations and introduce a new challenge – unsupervised VL grammar induction – aiming at extracting the shared hierarchical structure for both vision and language simultaneously . As a brief introduction , conventional grammar induction ( Figure 1 ( a ) ) , specifically constituency grammar induction , captures syntactic information in the form of constituency trees , which provide extra interpretability to downstream tasks , e.g. , semantic role labeling ( Strubell et al. , 2018 ) , sentence completion ( Zhang et al. , 2016 ) and word representation ( Kuncoro et al. , 2020 ) . It is commonly formulated as a self-contained system that relies solely on language corpora ( Kim et al. , 2019a ; Drozdov et al. , 2019 ; 2020 ; Shen et al. , 2018 ; 2019 ) . On top of this , Shi et al . ( 2019 ) proposes visually-grounded grammar induction , focusing on enhancing language grammar induction performance by leveraging additional visual information . Similar benefits of multi-modality on grammar induction have also been demonstrated by Zhao & Titov ( 2020b ) ; Zhang et al . ( 2021 ) ; Hong et al . ( 2021 ) . These works , however , fail to consider a unified VL structure , nor have they demonstrated impact on visual understanding . Different from prior arts , unsupervised VL grammar induction aims to construct a shared constituency structure at a fine-grained level for both the input image and the corresponding language caption ; see Figure 1 ( b ) . It requires capabilities of structured prediction for a single-modality ( language constituency structure ) along with fine-grained alignment with heterogeneous modalities , while only having access to associated image-caption pairs for training ( no human-generated ground truth ) . Besides the general challenge of the unsupervised setting existing in conventional visually-grounded grammar induction tasks , we highlight two main challenges specific to our proposed new task : 1 . Context-dependent semantic representation learning . The non-terminal symbol of a conventional constituency structure is a category label from a limited set ( Hopcroft et al. , 2001 ) . ( 1 ) Such limitation leads to tractable learning but limited expressive power as it lacks semantics from words . In particular , sharing the same representation in the tree structure leads to semantic ambiguities . For example , two NP nodes in the tree represent two different phrases , but may have the same embedding . ( 2 ) Apart from lacking semantics from words , it also lacks rich contextual encoding of the phrases . Simply associating each symbol with specific words or contexts , as done in Zhu et al . ( 2020 ) , would lead to memory explosion . ( 3 ) Besides textual data as context , exploiting the visual data as context information also remains a challenge . 2 . Fine-grained vision-language alignment for all levels of the hierarchical structure . Instead of fusing information of the entire image into the phrase representations , as done in visually-grounded grammar induction ( Shi et al. , 2019 ; Zhao & Titov , 2020b ) , VL grammar induction requires each phrase in the structure to align with a specific Region of Interest ( RoI ) in the image . Such finegrained alignment enables the model to have a thorough understanding of the image ( Anderson et al. , 2018 ; Zheng et al. , 2019 ) . However , the fine-grained alignment is difficult to deal with because the feature sets generated from different modalities are different in nature . To address these challenges , we propose a potential approach , namely Contrastive LanguageImage inside-Outside Recursive Autoencoder ( CLIORA ) . It leverages the previous success of DIORA ( Drozdov et al. , 2019 ) on context-dependent grammar induction for language and extends it in a multimodal scenario . Specifically , as sketched in Figure 2 , it first extracts features from both modalities , then incorporates the inside-outside algorithm to compute the constituents and construct the constituency structure . Already at this stage , we combine the two modalities , by recursively having the language span embeddings attend to the visual features . We refer to this as feature-level fusion . This makes the phrases aware of the visual context , effectively exploiting the visual context as well as the textual semantics as context information , addressing the first challenge . On top of that , we compute a matching score between each constituent and image region . This score is used to promote the cross-modal fine-grained correspondence , leveraging the supervisory signal of the image-caption pairs via a contrastive learning strategy . Here , we further fuse the two modalities , by weighting the cross-modal matching score with the constituent ’ s score given by the induced grammar . We refer to this as score-level fusion . This ensures fine-grained alignment in every level of the tree structure , addressing the second challenge . In summary , our contributions are three-fold : ( i ) A new challenging task – unsupervised VL grammar induction for better cross-modal understanding , and a new metric to evaluate it ; ( ii ) A novel method , CLIORA , to build the VL structure by context encoding with a multi-level fusion strategy ; ( iii ) New state-of-the-art performance on MSCOCO and Flickr30k Entities benchmarks . 2 TASK DEFINITION . VL Structure Formulation We introduce the shared VL constituency tree structure ( VL structure ) in Figure 1 to represent the shared semantics for cross-modality . In detail , given a sentence x “ tx1 , x2 , ... , xnu with n words and an associated image I , the VL structure y , is formed in a phrasestructure tree similar to Chomsky Normal Form ( CNF ) ( Chomsky , 1959 ) , where each non-terminal node in the tree will have exactly two children . Formally , a VL structure y is a set of constituents1 tpci , j , bi , jqu that forms a tree structure of x . Each non-terminal node of the tree contains a language span ( said informally , phrase ) ci , j corresponding to a sequence of words txi , xi ` 1 , ... , xju , and an aligned box region bi , j P R4 in the image I . ci , j is said to be grounded to bi , j . Different from the usual linguistic setting in context-free grammars ( Hopcroft et al. , 2001 ) , the features of our non-terminal nodes contain rich context-dependent semantics instead of a category label . This results in a structure with powerful expressive ability , but simultaneously a high computational complexity if using conventional approaches ( e.g. , Kim et al . ( 2019a ) ) to model it , as claimed in Yang et al . ( 2021 ) ; Han et al . ( 2017 ) . On the image side , the structure provides explainable scene understanding . Different from the flat form of the structure typically used in phrase grounding ( Wang et al. , 2020a ) , each node in the hierarchical structure is associated with an image region . Different from scene parsing ( Zhao et al. , 2017 ) , regions of different nodes are allowed to overlap . Task Formulation In unsupervised VL grammar induction , the goal is to induce phrase-structure grammars from only image-caption pairs without tree structure annotations nor phrase-region correspondence annotations for training2 . Formally , we aim to learn a model M , which takes an image I and a language description x as input and predicts the VL structure y , i.e. , y “ MpI , xq . Note that different from Wang et al . ( 2020a ) who predict the corresponding regions for a given set of noun phrases , noun phrases in VL grammar induction are unknown and all spans in the VL structure are aligned to corresponding regions in the image . Evaluation Metrics Due to lacking annotations of VL structure , we indirectly assess our model by two derived tasks from each modality ’ s perspective , i.e . language grammar induction and phrase grounding . Furthermore , we propose a new evaluation metric and conduct a frontal evaluation for the VL structure we obtained . Lateral Evaluation For language grammar induction , we use two widely-used metrics : the averaged corpus-level F1 and averaged sentence-level F1 numbers along with the unbiased standard deviations following Zhao & Titov ( 2020a ) . For visual grounding , we report the grounding accuracy . We consider a noun phrase correctly grounded if its predicted bounding box has at least 0.5 IoU ( Intersection over Union ) with the ground-truth location . The grounding accuracy ( ACC ) is the fraction of correctly grounded noun phrases ( from a given set of noun phrases ) . Frontal Evaluation We propose a new metric , critical concept recall rate ( CCRR ) , to explicitly evaluate the VL structure . A critical concept is a noun phrase found in visual grounding annotations . We say a critical concept is recalled when it is retrieved in the parsed constituency tree structure and correctly grounded in the image . CCRR is the recall rate of all the critical concepts . 3 CONTRASTIVE LANGUAGE-IMAGE INSIDE-OUTSIDE RECURSIVE AUTOENCODER . In this section , we design a novel VL grammar induction method , CLIORA , to construct a shared constituency structure for paired vision-language inputs . We start by briefly introducing our basis model DIORA ( Drozdov et al. , 2019 ) in Section 3.1 . Then we present in detail our proposed CLIORA from modeling in Section 3.2 , through inference in Section 3.3 to objective and learning in Section 3.4 . 3.1 BACKGROUND . Detailed Formulation of VL Structure Following Lafferty ( 2000 ) ; Drozdov et al . ( 2020 ) , we adopt an indexing scheme for the constituency VL structure , and use a two-dimensional nˆn chart T storing the intermediate representations while computing the spans of the VL structure . Cell pi , jq in the chart contains all scores and vectors of span ci , j . For each 1 ď i ă k ă j ď n , span pi , jq can 1In a usual grammar induction setting , the constituent ( span ) corresponds to a sequence of words . Here we reuse the conventional name but expand it to a pair of language span and aligned visual region . 2We do use a pre-trained object detector to obtain box regions , as is common in the phrase grounding literature . For a fully unsupervised setting , this could be replaced by a generic object proposal method ( e.g. , Uijlings et al . ( 2013 ) ) , combined with features trained with self-supervision ( Jaiswal et al. , 2021 ) . be decomposed in spans pi , kq and pk ` 1 , jq . Each span ci , j in the tree is associated with an inside score sinij and inside vector h in ij P RD computed from bottom up ( inside pass in next paragraph ) ; as well as an outside score soutij and outside vector h out ij P RD from top down ( outside pass ) , with D the feature dimension . The inside vector captures information of the inner content of the span , while the inside score assesses to what extent the span forms a phrase with complete semantics . Likewise , the outside score soutij and outside vector h out ij represent the contextual cues not in span ci , j . DIORA Deep Inside-Outside Recursive Autoencoder ( DIORA ) ( Drozdov et al. , 2019 ) aims to induce the language grammar and produce the constituency tree parser in an encoder-decoder framework . Different from context-free grammars , DIORA mitigates the strong context-freeness assumption by computing each span ’ s representation and spans ’ composition possibility dependent on the context . DIORA operates like a masked language model since it models the context of a missing word and then reconstructs this missing word using the context as clue . Formally , DIORA encodes the input sentence x in the shape of a constituency tree . Since the groundtruth tree structure is not given , all possible valid trees are considered simultaneously , with weights , using dynamic programming similar to the inside-outside algorithm ( Baker , 1979 ) . In the bottom-up inside pass , the encoder recursively runs an inside pass through all spans and computes the inside vector hini , j and inside score s in i , j for each constituent ci , j . The combined constituent is obtained by weighted summation of all possible pairs split by k P ri , jq . Similarly , the decoder performs a top-down outside pass , recursively computing the outside score souti , j and outside vector h out i , j with k P p1 , i´ 1s Y rj ` 1 , nq . In this way , the bottom-most vectors in the outside pass houti , i encode the context of the entire sentence x except for the i-th word . During inference , the predicted tree with the maximum inside scores is obtained with the CKY algorithm ( Kasami , 1966 ; Younger , 1967 ) .

Summary: This paper presents a new model for grammar induction for text, with help from the coupled images. The model was built on top of an existing unsupervised grammar induction model used for text without image information. The experimental results show the approach was effective. The work essentially demonstrates some effective ways of leveraging the additional image information for improving the grammar induction task. The paper also discussed some weaknesses of the approach and future work.