Unsupervised Vision-Language Grammar Induction with Shared Structure Modeling
Authors: Bo Wan
Wenjuan Han
Zilong Zheng
Tinne Tuytelaars
Venue: ICLR 2022 Oral (2022)
Keywords: Grammar Induction
Vision-Language Matching
Unsupervised Learning

Abstract: We introduce a new task, unsupervised vision-language (VL) grammar induction. Given an image-caption pair, the goal is to extract a shared hierarchical structure for both image and language simultaneously.  We argue that such structured output, grounded in both modalities, is a clear step towards the high-level understanding of multimodal information. Besides challenges existing in conventional visually grounded grammar induction tasks, VL grammar induction requires a model to capture contextual semantics and perform a fine-grained alignment. To address these challenges, we propose a novel method, CLIORA, which constructs a shared vision-language constituency tree structure with context-dependent semantics for all possible phrases in different levels of the tree. It computes a matching score between each constituent and image region, trained via contrastive learning.  It integrates two levels of fusion, namely at feature-level and at score-level, so as to allow fine-grained alignment. We introduce a new evaluation metric for VL grammar induction, CCRA, and show a 3.3% improvement over a strong baseline on Flickr30k Entities. We also evaluate our model via two derived tasks, i.e., language grammar induction and phrase grounding, and improve over the state-of-the-art for both.

Published as a conference paper at ICLR 2022
UNSUPERVISED VISION-LANGUAGE GRAMMAR INDUCTION WITH SHARED STRUCTURE MODELING
Bo Wan', Wenjuan H Zilong Zheng?, Tinne Tuytelaars'
1. Department of Electrical Engineering, KU Leuven;
2. Beijing Institute for General Artificial Intelligence, Beijing, China (bwan; Tuyte /eeaaa (hanwenjuan; z zbzhengjabigai.
fesat. wullevenn.be be;
ABSTRACT
We introduce a new task, unsupervised vision-language (VL) grammar induction. Given an image-caption pair, the goal is extract shared hierarchical struc- ture for both image and language simultaneously. We argue that such structured output, grounded in both modalities, a clear step towards the high-level under- standing of multimodal information. Besides challenges existing in conventional visually grounded grammar induction tasks, VL grammar induction requires a model to capture contextual semantics and perform fine fineggrinedd alignment. To address these challenges, we propose a novel method, CLIORA, which constructs shared vision-language constituency tree structure with context-dependent se- mantics for all constituents in different levels of the tree. It computes a matching score between each constituent and image region, trained via contrastive learning. It integrates two levels of fusion, namely at feature-level and at score-level, so as allow fine-grained alignment. We introduce new evaluation metric: Critical Concept Recall Rate (CCRR) to explicitly evaluate VI. grammar induction, and show 2.6% improvement over a strong baseline on Flickr30k Entities. We also evaluate our model via two derived tasks, i.e., language grammar induction and phrase grounding, and improve over the state-of-the-ar for both.
INTRODUCTION
Humans are amazing at extracting knowledge efficiently from our complicated and multimodal world, leveraging both redundant and complementary information from visual, acoustic, or tac- tile cues. Investigating into such behavior, neuroimaging and neuroanatomical studies suggested that specific brain regions are dispatched support the convergence of auditory and visual word comprehension (Calvert et al. 1997, Campbell 2008 Keitel et al., 2020). For example, auditory regions are involved in lip reading by receiving signals from visual cortices (Bourguignon et al.) 2020). These observations imply a mysterious "shared world" when perceiving multimodal signals, functioning centralized processor for understanding fused information. In contrast, such phe- nomena have not been revealed in modern state-of-the-art VI. models, most of which process visual and language signals in two separate streams and fuse the results only in the final stage (Ma al.,
2015 You et al. 2018 Shi et al. 2019 Kojima et al. 2020 Zhao & Titov 2020b).
In this work, we dive into the "shared world" for vision-language (VL) representations and introduce new challenge- unsupervised VL grammar induction aiming at extracting the shared hierarchi- cal structure for both vision and language simultaneously. As a brief introduction, conventional grammar induction (Figure [a)), specifically constituency grammar induction, captures syntactic information in the form of constituency trees, which provide extra interpretability to downstream tasks, e.g., semantic role labeling (Strubell et al. 2018), sentence completion (Zhang et al., (016) and word representation (Kuncoro et al., 2020). It is commonly formulated as self-contained sys- tem that relies solely on language corpora (Kim et al., 2019a| Drozdov al.) 2019 2020 Shen et al. 2018 2019). On top of this, Shh et al. (2119) proposes risually-grounde grammar induction, focusing enhancing language grammar induction performance by leveraging additional visual information. Similar benefits of multi-modality on grammar induction have also been demonstrated
$ *Corresooding author. Code is available at http://gthha.ccmm/oowwa111/////9 http:///oooooao19////ccor...gg

---

Published as a conference paper at ICLR 2022
by Zhao & Titov (2020b): Zhang al. 2021 l ; Hong et al. (2021). These works, however, fail to consider have they demonstrated
(i)
Language A man pushes a boy Grammar on zip-lime Induction
unified VL structure, impact on visual understanding.
paeso buy spiinc
(b)
Different from prior arts, unsupervised VL grammar induction aims to construct a shared constituency structure at fine-grained level for both the input image and the corresponding language caption; see b)). requires capabilities of structured pre- diction for single-modality (language constituency structure) along with fine-grained alignment with heterogeneous modalities, while only having access to associated image-caption pairs for training (no human-generater ground truth). Besides the general challenge of the unsupervised setting existing in conventional Assuall--groudded grammar induction tasks, highlight two main challenges specific to our proposed new task:
Vision-
Language
Grammar
Induction
man pushes bay on a zip-line
Ammun
aipline
Figure 1: Task illustration of (a) conven- tional grammar induction for natural language and (h) VI. grammar induction.
1. Context-dependert semantic representation learning. The non-ter con-terminal symbol of a conventional constituency structure is category label from a limited set (Hopcroft et al. 200T). (1) Such lim- itation leads to tractable learning but limited expressive power as it lacks semantics from words. In particular, sharing the same representation in the tree structure leads to semantic ambiguities. For example, two NP nodes in the tree represent two different phrases, but may have the same embedding. (2) Apart from lacking semantics from words, also lacks rich contextual encoding of the phrases. Simply associating each symbol with specific words or contexts, as done in Zhu al. (2020). would lead to memory explosion. (3) Besides textual data as context, exploiting the visual data context information also remains challenge.
Fine-gruined sisiio-langagee alignment for all levels of the hierarchical structure. Instead of fus- ing information of the entire image into the phrase representations, done in visually-grounded grounded grammar induction (Shi et al. 2019 Zhao & Titov] 2020b), VL grammar induction requires each phrase in the structure align with specific Region of Interest (Rol) in the image. Such fine- grained alignment enables the model to have a thorough understanding of the image (Anderson let al. 2018 [Zeng et al. 2019). However, the fine-grate mne-grained alignment is difficult to deal with because the feature sets generated from different modalities are different in nature.
To address these challenges, Image inside-Outside Recursive Autoencoder (CLIORA). It leverages the previous success of DIORA (Drozdov et al. 2219] on context-dependent grammar induction for language and extends it in a multimodal scenario. Specifically, as sketched in Figure 2 it first extracts features from both modalities, then incorporates the inside-outside algorithm compute the constituents and construct the constituency structure. Already at this stage, we combine the two modalities, by recursively having the language span embeddings attend to the visual features. We refer to this as feature-level fusion. This makes the phrases aware of the visual context, effectively exploiting the visual context as well as the textual semantics as context information, addressing the first challenge. On top of that, we compute matching score between each constituent and image region. This score is used to promote the cross- modal fine-gra grained correspondence, leveraging the supervisory signal of the image-caption pairs via contrastive learning strategy. Here, further fuse the two modalities, by weighting the cross- matching score with the constituent's 's score given by the induced gram- mar. We refer this as score-level fusion. This ensures fine-grained alignment in every level of the tree structure, addressing the second challenge.
propose a potential approach, namely Contrastive Language-
In summary, our contributions three-fold: (i) A new challenging task unsupervised VL gram- mar induction for better cross-modal understanding, and new metric to evaluate it; (ii) A novel method, CLIORA, build the VL structure by context encoding with a multi-level fusion strategy; (iii) New state-of-the-ae performance on MSCOCO and Flickr30k Entities benchmarks.
TASK DEFINITION
VL Structure Formulation We introduce the shared VL constituency tree structure (VL structure) in Figure T to represent the shared semantics for cross-modality. In detail, given sentence
2

---

Published as a conference paper at ICLR 2022
{21, 22,, .2,,...... [nn) with words and associated image I, the VL structure y, is formed in phrase- structure tree similar to Chomsky Normal Form (CNF) (Chomsky) 1959). where each non-tent non-terminal node in the tree will have exactly two children. Formally, VL structure y is a set of constituent?
[(G1,5 ) that forms a tree structure of x. Each non-terminal node of the tree contains a language span (said informally, phrase) Cr corresponding to a sequence of words (2i, (it1:i (.:,,::,1))))) Ty., and an aligned box region brg R in the image I. Cijj is said to be grounded bi.j
Different from the usual linguistic setting in context-free grammars (Hopcroft et al. 2001). the fea- tures of our non non-terminal nodes contain rich coneext-dependent semantics instead of category label. This results in a structure with powerful expressive ability. but simultancously a high computational complexity if using conventional approaches (e.g.. Kim et al. (2019a)) model it, as claimed in Yang et al. (202T): Han et al. (2017). On the image side, the structure provides explainable scene understanding. Different from the flat form of the structure typically used in phrase grounding (Wang it al. 2020a), each node in the hierarchical structure is associated with an image region. Different from scene parsing (Zhao al. 2017). regions of different nodes are allowed to overlap.
Task Formulation In unsupervised VL grammar induction, the goal is to induce phrase-structue grammars from only image-caption pairs without tree structure annotations nor phrase-region corre- spondence annotations for training Formally, we aim to learn model M,, which takes an image T and language description x as input and predicts the VI. structure y, i.e., y = MM x). Note that different from Wang et al. (2020a) who predict the corresponding regions for a given set of noun phrases, noun phrases in VL grammar induction are unknown and all spans in the VL structure are aligned to corresponding regions in the image.
Evaluation Metries Due to lacking annotations of VL structure, we indirectly assess our model by two derived tasks from each modality's 's perspective, i.e. language grammar induction and phrase grounding. Furthermore, we propose new evaluation metric and conduct a frontal evaluation for the VL structure obtained.
Lateral Evaluation For language grammar induction, we use two widely-used metrics: the av- eraged corpus-level F1 and averaged sentence-level F1 numbers along with the unbiased standard deviations following Zhao & Titov (2020a). For visual grounding, we report the grounding accu- racy. We consider a noun phrase correctly grounded if its predicted bounding box has least 0.5 IoU (Intersection over Union) with the ground-trute location. The grounding accuracy (ACC) is the fraction of correctly grounded noun phrases (from given set of noun phrases).
Frontal Evaluation We propose new metric, critical concept recall rate (CCRR), to explicitly evaluate the VL structure. critical concept is noun phrase found in visual grounding annotations. We say a critical concept is recalled when it is retrieved in the parsed constituency tree structure and correctly grounded in the image. CCRR is the recall rate of all the critical concepts.
3 CONTRASTIVE LANGUAGE-IMAGE INSIDE-OUTSIDE RECURSIVE AUTOENCODER
In this section, we design a novel VI. grammar induction method, CLIORA, to construct a shared constituency structure for paired vision-language inputs. We start by briefly introducing our ba- sis model DIORA (Drozdov et al. 2019) in Section 3.1 Then we present in detail our proposed
CLIORA from modeling in Section 3.2 through inference Section 3.3 objective and learning in Section 3.4
3.1 BACKGROUND
Detailed Formulation of VL Structure Following Lafferty (2000); Drozdov et al.) (2020), we adopt an indexing scheme for the constituency VI. structure, and use a two-dimensional n chart
In usual grammar induction setting. the constituent (span) corresponds a sequence of words. Here reuse the conventional name but expand to pair of language span and aligned visual region.
We do use a pre-trained object detector to obtain box regions, as is common in the phrase grounding litera- ture. For fully unsupervised setting, this could be replaced by generic object proposal method (e.g.. Uijlings 2013). combined with features trained with self-supervision (Jaiswal et al. 202T
3

---

Published as a conference paper at ICLR 2022
Language
Feature Extraction
Structure Construction
I Loss
Vision
Word Embedding
Vision-I language
Faden
Recoustructio
Rec.
GT
Caption
A
man pushes a bay on a zip line-
LM
Wa
We
Chitaddo Pars
pokke
Contrastive Loss
Finc-Grained Fastare
Image
man
MG
Vis
Featerekked Arre
Scarm-tasel
Faster
Iniide Piosi
CAN
V2
pallo
ALL Fg)
Figure 2: Diagram of CLJORA. The Feature Extraction module prepares the language and image fcatures. The feature-level fusion enhances the language feature with visual cues. The Structure Construction module recursively computes the VL representation for constituents with feature-level fusion. Finally, the score-level fusion promotes the cross-modal fine-grained correspondence, leveraging the supervisory signal of the image- caption pairs.
T storing the intermediate representations while computing the spans of the VL structure. Cell (i,j) in the chart contains all scores and vectors of span Cij. For each 1 & i = n, span (i,j) can be decomposed in spans (z, k) and (k: + 1. Each span Ci in the tree is associated with an inside score sf and inside vector h: E IRD computed from bottom up (inside pass in next paragraph); as well as an outside score qut and outside vector h R from top down (outside pass), with D the feature dimension. The inside vector captures information of the inner content of the span, while the inside score assesses to what extent the span forms phrase with complete semantics. Likewise, the outside score $1
gout and outside vector hg represent the contextual cues not in span Gijj-
DIORA Deep Outiid- Recursive Autoencoder (DIORA) (Drozdov et al. 2019) aims to induce the language grammar and produce the constituency tree parser in an encoder-decodee frame- work. Different from context-free grammars, DIORA mitigates the strong context-freeness assump- tion by computing each span's representation and spans' composition possibility dependent on the context. DIORA operates like a masked language model since models the context of a missing word and then reconstructs this missing word using the context as clue.
Formally, DIORA encodes the input sentence x in the shape of a constituency tree. Since the ground- truth tree structure is not given, all possible valid trees are considered simultaneously. with weights, using dynamic programming similar to the onside-outside algorithm (Baker) 1979). In the bottom-up inside pass, the encoder recursively runs an inside pass through all spans and computes the inside vector h and inside score 2 for each constituent Ci.j- The combined constituent is obtained by weighted summation of all possible pairs split by k E [(,)). Similarly, the decoder performs a top-down outside pass, recursively computing the outside score sout and outside vector h with ke (1, - 1] U [j + n). In this way, the bottom-most vectors in the outside pass houe encode the context of the entire sentence x except for the i-th word. During inference, the predicted tree with the maximum inside scores is obtained with the CKY algorithm (Kasami) 1966 Younger 1967
3.2 MODELING
Figure 2 illustrates the overall workflow, including visual/textual feature extraction, feature-level fusion, structure construction, score-level fusion, and the loss function module. The whole fusion process can be classified in feature-level (combining features vectors from different modalities) and score-level (combining the scores).
Feature Extraction For visual feature extraction, previous works on visually-grounded grammar induction often use only the global image feature (e,g., global output feature in Shi al. (2019); Zhao & Titoy| (2020b); Zhang et al. (202T)) as additional information. The detailed information from the image at region level, which is helpful in building the fine- grained VL correspondence, is ignored. Instead, we explore different aspects of visual features. In particular, we use an external
object detector to generate set of object proposals O 3 m= where Om RRd denotes an
M
4

---

Published as a conference paper at ICLR 2022
image region (Rol). Similar to Wang et al. (2020b). for each Om, we compute its conv-features and project it to a vector Vm E R with the same dimension D as the span features. For textual features, following Drozdov et al. 2019), we use pretrained word embedding w; RP to initialize all the words 24 in sentence x. For more details c.f. Experiments Section.
Feature-Ievel Fusion The feature-level fusion strategy fuses the visual and language features, different stages in our pipeline at the very start using the word embeddings w, as well as during the recursive structure construction (described below), using the inside vectors h Denote Va,i as the visual representation corresponding to span Cij- This is obtained by adopting an attention mechanism on the object features [VmJm= In detail, we first compute the attention scores between hi and each Vm, then normalize the scores and obtain Vi.j by weighted summation over all object features. Finally, we enhance the span features by by fusing the visual information Vi.j-
Att Soft max {Vvm) "
Vi. Σ Att.j.m Vms
B norm (h + +V.
m
(1)
where y is a weight value for fusion and norm is L2 normalization.
Structure Construction Inspired by DIORA,
use autoencoder model to integrate the visual information and employ the inside pass and outside pass to fill in the chart T. During the inside pass, for the leaf nodes, the scores sie set to 0 and the span features h are initialized with the normalized word embeddings (orm((;), During the outside pass, for the root node, the outside score 1nn is set to 0 and the language feature h is initialized randomly independent x. Then the onside-outside algorithm is employed to recursively calculate the scores s Si and vectors h hoyt.
sout
Inside Pass The encoder is a bottom-up flow running an inside pass through all spans in the input sentence It computes hy and si for each span Cr2j fill up each cell (i,j) in the chart T as We start by computing items for each decomposition (2, k) and (k + 1,5).
shown in Figure 2
in by f(h
in hg.)
in s
(h))) W (h + si in sk+1J
(2)
where f is a composition function to merge the two spans (we use two linear layers with ReLU activation funetion) and W are trainable parameters compute the composition scores. bi is obtained by weighted summation over all possible pairs with the normalized ijib
Soft max, (
in $.
=
Σ
j.j.k
fij.k
(3)
Then we fuse the visual information into hi to obtain h using Equation hig in turn acts sub-span to recursively produce bigger spans, e.g., h h etc., until h
gou Outside Pass The decoder performs a top-down outside pass, computing the outside scores s and the outside vectors hgy by aggregating the inside representation and outside representation with the outside algorithm. k E (1, 1 [j 1,m.. Take k > j as example
hout f(h 1,))
S..jk
(hep*) T W (h))11,)) + ouu 8jt1EK
(4)
We use the same composition function f and score weight W as the inside pass. The computation of the outside score sout and outside feature representation hout is similar to the inside pass with gout Soft max; (s J.jkk
h hg
Σ bog.k
$
In this way, the bottom-met vectors in the outside pass h encode the context of entire sentence x except for the i-th word.
We explain it here for the latter case only. For the word embeddings, the procedure is basically the same. *If & < j. the procedure is similar. Please refer to the Appendix D for more details.
5

---

Published as a conference paper at ICLR 2022
Score-level Fusion Finally, we fuse the span scores with visual similarity scores to obtain the input for our contrastive learning (detailed in Section 3.4). For this, we multiply the span score ((C... (calculated using inside scores and outside scores, below) with a similarity score sim T. G.g)) indicating the semantic similarity between span Cij and the image J. This results in distance between image 1 and span Cij- that incorporates the VL structure information: d(I, cx.)) = sim(l, Ci,)) x ((C.,)
Similarity Score
Following Wang et (2020b), we first compute the similarity score aijim be- tween each span çitj and each image region Om, then we select the image region with the highest similarity score, and use that score as similarity for the entire image:
aj..m T (fi)) + hgy?)
sim(I, cs.g) max(as..m)
m
Note how a..j. is calculated using both inside and outside feature vectors.
Similar to sim(I, I c.,)), which computes similarity at the span-level, we can also compute simi- larity scores at word level based on the original word embeddings: a. = v w,, Analogically, sim (1, x.) ) maxm (aj.))) then represents the similarity between a word and the entire image.
Span Score g(os,j= x) reflects how likely the span G.jj exists given x. PCFG (Michael 2011). since each span in the sentence is assigned to non-terminal symbol, the marginal of the span C.jj is the summation over all non-terminal symbols. In CLIORA, we do not sout
conventional
work with discrete symbols for the spans. Instead, we define q(Gi,;: x) = o 1. / This is inspired by PCFG (please refer to the Appendix E for more details).
3.3 INFERENCE
In the inference stage, the language tree structure (c is predicted with the maximum inside scores using the CKY algorithm (Drozdov et al.) 2019) on chart T, and then the VL structure y [[c bt,,) } is obtained by aligning each span cf to image region bi.
When a meaningful phrase (ground-truth constituent)
{Ig k [i,j]} is given (e.g., for the visual grounding task), we predict its image region {om_, with a voting mechanism. We use ak.m = k. + Am (where "m and @& k.m are defined in the Similarity Score paragraph above) as the matching score to select the most representative word SAs in p with the maximum ag.m> and take the box region Om!., based on the Tg with maximum matching score p's grounding result:
h arg max max Ok,m KEKE
m arg max 02 .m
When the meaningful phrase is not given, can still predict the grounding region Ong,, of an arbitrary span C.y. In that case, apart from 0g* in Equation 7 we consider the Rol-span matching scores ai.j. (defined in Similarity Score paragraph) to predict the aligned box Onl This allows to assess how meaningful this span is given the visual information. The box region of G, is then:
m arg maxx(aa.mm + ai...m)
In Experiments Section, we adopt the first case in "Lateral Evaluation on Grounding". In "Frontal Evaluation on VL Grammar Induction", we use the second case and obtain the grounding region bü Omgy for each predicted span cf since we do not have given ground- truth phrases as supervision.
Visual
3.4 OBJECTIVE AND LEARNING
Loss for Structure Construction As mentioned in Structure Construction module, the bottom- most vectors the outside pass heat encode the context of the entire sentence x except for the i-th word. With the hypothesis that visual cues help to predict the missing words, CLIORA takes self-supervised blank-filling filling objective as follows:
Lree =
Σ z,Ex
log P( (x P(z//hg out
(9)
n

---

Published as a conference paper at ICLR 2022
Loss for Contrastive Learning We design contrastive learning strategy based maximizing the matching score between paired Rol-span elements instead of coarse Image-span matching Zhaoo & Titov (2020a) Rol-sentence matching (Wang al. 2020b). Behind this design, motivation is that compared with a whole sentence, a short span is more likely to find a corresponding Rol that construct strong negative pair enhance contrastive learning. The same holds for the Rol compared with the whole image. Particularly, for each span C,2 in a caption sentence, all the other images except for the corresponding image 1 in the current batch are negative examples. We can define the following Ispan (I, Ci.)) and worrd(I (I, xz)) in the batch:
Lapan Ca.) = [d[[1 cg.g) - d(I, + e]+ + [[([I', d(I, +c+
(10)
e (1,a.)
wword [1) log
(11)
Eiebatch esim"
where 1+++ = max e is the positive margin, and variables with are negative examples in batch. small (span means an aligned span-image pair that is closer than any unaligned ones in batch. Then we obtain the contrastive learning loss:
La Eptyyx)
Σ C.
Ispan (ppan(1,cmm cig)) +
Σ SEEX
LaworalI (1:)
Σ G
Ispan (T, (1.5)) +
Σ hhord (I, xi
(12)
where y is a valid tree for the sentence x. Finally, the total loss defined as:
LCLIORA = A Free + +11 (1- - La
(13)
where À is a hyper representing the trade-off between the two losses.
EXPERIMENTS
4.1
EXPERIMENTAL SETTING
Datasets We evaluate our method on the Flickr30k Entities (Plummer et al., 2017) and MSCOCO (Lin cet al. 2014) datasets. Flickr30K Entities contains 29783 images for training, 1,000 images for valida- tion and 1,000 for test. We use the same split of MSCOCO as Zhao & Titov (2020),, which contains 82,783 training images, 1,000 validation images, and 1,000 test images. Each image is associated with 5 sentence descriptions for both Flickr30k Entities and MSCOCO datasets. Following Shi et al.) (2019);
MSCOCO Flickr30k
Only Language
Left branch
15.1
13.4
Right branch
51.0
1.7
Random
24.2+0.3
15.4+0.2
C-PCFG"
53.6+4.7
25.7+2.6
DIORA
53.4+0.6
46.420.9
DIORA
58.0+0.7
54.342.0
Visnally-Gounded
VG-NSL
50.4+0.3
Zhao & Titov (2020b), we filter words using the oc- curring frequency the training set and build vo- cabulary with size 10,000 for both Flickr30K Enti- ties and MSCOCO. All sentences are converted to lowercase, discarding non-alphanumeric characters.
VG-NSL+HI
53.3+0.2
VC-PCFG"
59.3+8.2
26.3+2.1
Vision- Language (VL)
Vision-Language
CLIORAt
56.2+0.7
53. 11.8
CLIORA
60.8
56.6+1.7
Preprocessing Following MAF (Wang et al. 2020b), for an input image, we use an external object detector, Faster R-CNN (Ren et al. 2015). to gener- ate object proposals and extract their visual features. Following Wang et al. (2020b), for each proposal, we use Rol-Align (He et al.) 2017) and global aver- age pooling to compute its feature and embed it to a vector. We follow Drozdov et al. (2019) and Wang et al. 2020a) and use ELMo PPterr al. 2018), and Glove embedding (Pennington et al. 2014)) for MSCOCO and Flickr30k Entities, respectively. For the ground-truth structure of the captions, we follow Shhi et al. (2019) and Zhao & Titov (2020a) to use predictions produced by Benepar (Kitaev & Klein 2018)..
Table 1: Grammar induction for language. Corpus-level Fl on the Test Datasets. * means re-implement the results on Flickr30k dataset as Zhao Titov (2020b). indicates we use ran- domly initialized word embedding.
Setting For all experiments, report the average score along with the unbiased standard devia- tions on four rums with different random seeds. We load DIORA as an initialization for CLIORA.
Other detailed parameess are provided in Appendix F
7

---

Published as a conference paper at ICLR 2022
4.2 QUANTITATIVE RESULTS
Lateral Evaluation on Language Grammar Induction As shown in Table we compare our approach with two classes of strong baselines: language grammar induction and visually grounded grammar induction. We use the corpus Fl as the metrie. Our model outperforms the previous SOTA approaches that use only language information and those incorporating visual cucs. In particular, CLIORA outperforms VC-PCFG by 1.5% corpus Fl score MSCOCO dataset and 30.3% corpus F1 score on Flickr30k dataset.
Notably, PCFG-based models (C-PCFG and VC-PCFG) perform poorly Flickr30k, while they work well on MSCOCO. Meanwhile, two classic baselines "Right branching" and "Left branch- due
ing" perform very differently MSCOCO and Flickr30k. We posit these phenomena to the different data distributions of MSCOCO and Flickr30k (See Figure [T] in Appendix). con- trast, CLIORA displays a more even performance across different corpora than PCFG-based models, which reveals its robustness across corpora.
Lateral Evaluation Weakly-supervised Visual Grounding We evaluate our approach on weakly- ing task and compare CLIORA with current SOTA method MA MAF only considers the ROI-word matching score d.., as in Equa- tion 7 while CLIORA leverages this ROI-word matching score as well as the context-aware Rol-span matching score aq,im in score- level fusion strategy. Not surprisingly, we observe a significant im- provement by 2.1% ACC on Flickr30k, as shown in Table
ACC
visual ground-
MAF
50.4
CLIORA
52.5+0.7
Table 2: ACC on Flickr30k test set.
CCRR
Baseline
47.0#0.4
CLIORA
49.6+0.9
addition to the indirect lateral evaluations described above, we also directly as- sess the VL structure we obtained on the frontal evaluation met- ric CCRR. Our baseline adopts DIORA for grammar induction and
Frontal Evaluation on VL Grammar Induction
Table 3: CCRR on Flickr30k
test set.
MAF for span grounding (as described in Equation [7) CLIORA incorporates the cross-modal matching from the original word-region similarities and hierarchical RoI-span correspondence, Ta-
ble 3 shows that our jointly learned VL structure outperforms the baseline with a large margin (2.6% on CCRR).
4.3 ABLATION STUDY ON FUSION STRATEGY
In this section we study the effectiveness of our two fusion strategies on language grammar induction: the feature-level fusion (FI.F) to enhance feature representation and score-level fusion (SI.F) to build structural cross-modal correspondence and provide regularization on meaningful spans. We leave out hoord because the empirical study shows that it hardly affects the F1 score. As show in Table FLF. which augments the language representation with vi- sual context cues, is capable of improving the constituency parsing result with 1.8% on Corpus-F1. In addition, SLF takes advantage of Rol-span matching signal supervise the learning of VL struc- ture and benefits the grammar induction result with 1.6% on Corpus-F1. When simultancously combining FLF and SLF, we observe further improvement on the final result.
FLF SLF
C-F1
54.3+2.0
56.1+2.1
55.9+2.4
1.7
Table 4: Results Flickr30k test set. FLF: Feature-Lovel Fusion. SLF: Score-Level Fusion.
4.4 ANALYSIS ON SPAN LENGTHS AND LABELS
In details, we show the recall rate on six frequent constituent labels on Flickr30k Dev set, well as Corpus Fl (CF1) and Sentence Fl (SF1) in Figure 3 Our model exhibits general boost across constituent labels. Specifically, it is most accurate on ADJPs and ADVPs and works fairly well on PPs. We contribute this improvement to the benefits of using contextual semantics.
MMAF* indicates we use the same feature extractor and learning strategy as MAF As the ground- noun phrase is only given in the inference stage, we the corresponding region of the most representative word as the prediction of the phrase, which results in lower number than the original paper.
8

---

Published as a conference paper at ICLR 2022
Next, analyze model performance on Flickr30k Dev set for constituents of different lengths in Figure Surprisingly, CLIORA enhanced by image generally its non non-visual version (i.e., DIORA) on not only short phrases but also longer phrases. As claimed in VC-PCFG, visual information beneficial for short spans but performs poorly on longer constituents(> 5). Our model goes a step further in multi-modal fusion, proving that multi-modal fusion is still promising.
100
Branching
Branching
Random
80
DIDRA
CLIORA
RECAL
RECALL
60
1_
40
MF
V2
PP
STAR
AD]1
ADVF
CF
SF1
BLotBranching
5.4
8.27
10.47
8.35
16.23
7.11
12.63
13.85
Branahing
235
2.23
2.54
1.31
2.14
RRndaam
10.05
05.5
PRIDSSE LAGEL
CONSTITUENT LENGHTH
Figure 3: Recall on different constituent label.
Figure 4: Recall on different constituent length.
Correct Span
Wrong Spon
Wrong Grounding
Correct Grounding
Medified Grounding
woman walk the sand (a) Good Case
she carrics her shoes
shirt is working under's blue vchicle
(b) Bad Case
Figure 5: Case Study. RoIs and spans without special marks predicted correctly.
LIMITATION AND FUTURE WORK
Although inspiring results have been achieved by CLIORA (See Figure 51a and Appendix for success examples), the unsupervised VL grammar induction is far from satisfaction. Figure (]b) demonstrates a typical failure case that wrongly predicts a span "a green" that should be "green shirt" in the ground-truth tree. This erroneous span is wrongly grounded to large region containing a "green building". Interestingly, after combining more contexts and constituent a bigger span "a green shirt", the grounding result is modified correct region (green rectangle). This failure reveals the importance of appropriately modeling contextual semantics, which highlights the major challenge of our proposed task and is calling for future research for better contextual models.
Moving forward, what is the best way of modeling such shared structure for VL grammar induction? promising extension could be to explore the visual structure to regularize the shared VL grammar. It is worth noting that the visual images also contains enriched spatial and semantic structures (Si & Zhu 2011 Zhao et al. 2017) which could be aligned with the constituency trees. Leveraging such structures may also be beneficial in producing a more meaningful shared structure.
Going back to the motivation of our work, how do humans model and process multimodal infor- mation with such shared space? This work provides potential answer with respect to grammar induction and phrase grounding. Nevertheless, the debate between using dense vectors and sym- bolic structures in human's cognitive computational models has never been stopped (Tang et al.) 2019). This mystery also leaves us a wide space to explore other potential explanations in modeling human's 's multimodal "shared world".
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
We acknowledge funding from Flemish Government under the Onderzoeksprogrammaa Artificiele Intelligentie (AI) Vlaanderen programme. This work has also been supported by the ERC project 101021347 KeepOnl.earning We thank Yanpeng Zhao for helpful suggestions on this work.
ETHICS STATEMENT
Hereby, we consciously assure that our study is original work which has not been previously pub- lished elsewhere, and is not currently being considered for publication elsewhere, Our study doesn't 't have any threats to health, safety, personal security, and privacy. We do not have ethics risks as mentioned in the author guidelines.
REPRODUCIBILITY STATEMENT
All the codes, processed data, and the trained model in this paper are publicly released at https://gitub..co//bbbaaa www.m//baaan9995/ciiira... itt We use two public datasets: MSCOCO and Flickr30k Entities. We use the same split of MSCOCO dataset as VC-PCFG (Zhao & Titov 2020b) and use their ground-truth tree structure annotation. We generate the ground-truth tree structure as VC- PCFG with Kitaev & Klein (2018) for Flickr30k Entities dataset. We re-implement the experiment results of CPCHG and VC-PCFO on Flickr30k Entities dataset Table with the same codes as VC-PCFG (Zhao & Titov 2020b).
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6077-6086, 2018.
James K Baker. Trainable grammars for speech recognition. The Journal of the Acoustical Society of America, 65(S1):S132 $132, 1979.
Mathicu Bourguignon, Martijn Baart, Efthymia C Kapnoula, and Nicola Molinaro. Lip-reading ables the brain to synthesize auditory features of unknown silent speech. Journal of Neuroscience, 40(5):1053-1065 2020.
Gemma A Calvert, Edward T Bullmore, Michael J Brammer, Ruth Campbell, Steven CR Williams, Philip McGuire, Peter WR Woodruff, Susan D Iversen, and Anthony S David. Activation of auditory cortex during silent lipreading- science, 276(5312):593
1997.
Ruth Campbell. The processing of audio-visual speech: empirical and neural bases. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1493):1001-1110 2008
Kan Chen, Jiyang Gao, and Ram Nevatia. Knowledge aided consistency for weakly supervised phrase grounding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Noam Chomsky, On certain formal properties of grammars. Information and control, 2(2):137-167, 1959.
Andrew Drozdov, Pat Verga, Mohit Yadav, Mohit lyyer, and Andrew McCallum. Unsupervised latent tree induction with deep inside-outside recursive autoencoders. In Proceedings of the An- nual Conference of the North American Chapter the Association for Computational Linguisties (NAACL), 2019.
Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman, Mohit Iyyer, and Andrew Mc- Callum. Unsupervised parsing with s-diora: Single tree encoding for deep inside-outside recur- sive autoencoders. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
10

---

Published as a conference paper at ICLR 2022
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang. Jan Kautz, and Derek Hoiem. Con- trastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision (ECCV), 2020.
Wenjuan Han, Yong Jiang, and Kewei Tu. Dependency grammar induction with neural lexicalization and big training data. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1683-1688, 2017.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE International Conference on Computer Vision (ICCV), 2017.
Yining Hong. Qing Li, Song-Chun Zhu, and Siyuan Huang. Vigrammar: Grounded grammar in- duction vision and language. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
John E. Hoperoft, Rajeev Motwani, and Jeffrey D Ullman. Introduction to automata theory, lan- guages, and computation. Acm Sigact News, 32(())60-653 65, 2001.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Dehapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9, 2021.
Tadao Kasami. An efficient recognition and syntax-analysis algorithm for context-free languages. Coordinated Science Laboratory Report no. R-257, 1966.
Anne Keitel, Joachim Gross, and Christoph Kayser. Shared and modality-specific brain regions that mediate auditory and visual word comprehension. ELife, 9re56972, 2020.
Yoon Kim, Chris Dyer, and Alexander Rush. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the Annual Meeting of the Association for Computational Lingnistics (ACL), 2019a.
Yoon Kim, Alexander M. Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. Unsu- pervised recurrent neural network grammars. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). pp. 105--117, Association for Computational Linguis- tics, 2019b.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic wtwps/eemmaniicmmmbiiiiningg with multimodal neural language models. arXiv, 2014. URL http:///rxxxii.rrr/// /
1411. 2539
Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2676-2686, 2018.
Noriyuki Kojima, Hadar Averbuch-Elor, Alexander Rush, and Yoav Artzi What is learned in visu- ally grounded neural syntax acquisition. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2615-2635, 2020.
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, and Phil Blunsom. Syntactic structure distillation pretraining for bidirectional encoders. Transactions of the Association for Computational Linguistics (TACL), 8:776-794, 2020.
John D Lafferty. A derivation of the inside-ousside algorithm from the EM algorithm. IBM TJ Watson Research Center, 2000.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Yongfei Liu, Bo Wan, Xiaodan Zhu, and Xuming He. Learning cross- modal context graph for visual grounding. In Proceedings the AAAI Conference on Artificial Intelligence (AAAI), 2020.
11

---

Published as a conference paper at ICLR 2022
Yongfei Liu, Bo Wan, Lin Ma, and Xuming He. Relation-aware instance refinement for weakly supervised visual grounding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. Multimodal convolutional neural networks for matching image and sentence. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2623-2631, 2015.
Collins Michael. Probabilistic context-free grammars. In NLP course note, 2011.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Pro- cessing (EMNLPP. pp- 1532-1543, 2014. URL http:/ /www c web. org werrg/anntan anthoothoooooooooo ogy/
D14- 1162
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics (NAACL). pp. 2227-2237, As- sociation for Computational Linguistics, 2018. doi: 10.18653/v1/n1-- 12022
URL https:
//doi. 11t.1/811//111112212
A. Bryan Plummer, Liwei Wang, M. Christopher Cervantes, C. Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrast correspondences for richer image-to-sentene models. I/C,, 123:74-93, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Workshop on Advances in Neural Information Processing Systems (NIPS), 2015.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Computer Vision (ECCV), 2016.
Yikang Shen, Zhouhan Lin, Chin-wei Huang, and Aaron Courville. Neural language modeling by jointly learning syntax and lexicon. In International Conference on Learning Representations (ICLR), 2018
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. International Conference on Learning Representa- tions (ICLR), 2019.
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. Visually grounded neural syntax ac- quisition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1842-1861, 2019.
Zhangzhang Si and Song-Chun Zhu. Unsupervised learning of stochastic and-or templates for ob- ject modeling. In 2011 IEEE International Conference Computer Vision Workshops (ICCV Workshops). pp.
IEEE, 2011.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. Linguistically- informed self-attention for semantic role labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5027-5038, 2018.
Evelyn Tang, Marcelo G Mattar, Chad Giusti, David M Wydon-Staley, Sharon L Thompson Schill, and Danielle S Bassett. Effective learning is accompanied by high-dimensional and efficient representations of neural activity. Nature neuroscience, 2206):000- 2019.
J. R. Uijlings, K. E. Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. International Journal of Computer Vision (IJCV), 2013.
Liwei Wang, Jing Huang, Yin Li, Kun Xu, Zhengyuan Yang, and Dong Yu. Improving weakly supervised visual grounding by contrastive knowledge distillation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),202
12

---

Published as a conference paper at ICLR 2022
Qinxin Wang, Hao Tan, Sheng Shen, Michael Mahoney, and Zhewei Yao. MAF: Multi- modal alignment framework for weakly weakly-supervised phrase grounding. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp- 2030-2038, Online, November 2020a. Association for Computational Linguisticss. doi: 10. 00:35///2220.emmmpmmmmm... emn1p mami..nn5
htt.:tt::: /www ac web org/ wwtpg//naooooo/////no/o
Qinxin Wang, Hao Tan, Sheng Shen, Michael W Mahoney, and Zhewei Yao. Maf: Multimodal phrase grounding. In Proceedings of the Conference
alignment framework for on Empirical Methods in Natural Language Processing (EMNLP), 2020b.
Songlin Yang, Yanpeng Zhao, and Kewei Tu. Neural bi-lexicalized PCFG induction. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference 2688 2898, Online, August 2021, Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl- wow..llllooo.9..... 209.
Natural Language Processing (Volume 1: Long Papers), pp-
ac http:////aaaaaaooooooooor///////
Quanzeng You, Jiebo Luo, and Zhengyou Zhang. End-to-end convolutional semantic embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5735- 5744, 2018.
Daniel H Younger. Recognition and parsing of context-free languages in time n3. Information and control, 10(2):189-2M) -0020990111-000000 1967.
Songyang Zhang, Linfeng Song, Lifeng Jin, Kun Xu, Dong Yu, and Jiebo Luo. Video-aided un- supervised grammar induction. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 1513-1524, 2021.
Xingxing Zhang, Liang Lu, and Mirella Lapata. Top-down tree long short-term memory networks. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 310-320, 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2881 12900 2017.
Yanpeng Zhao and Ivan Titov. Visually grounded compound PCFGs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4369-4379, Online, November 2020a. Association for Computational Linguistics. doi: 10. 18653/v1/2020.emm// http:////22000mmmmmmmm.......... URL htttps://www.c.. awwweb.wwwwb.co................. www.eeeee..g//aatooooo//////////
em..p-mmin..3 354
Yanpeng Zhao and Ivan Titov. Visually grounded compound pcfgs. In Proceedings the Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 4369-4379, 2020b.
Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chur Zhu. Reasoning visual dialogs with structural and partial observations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6669-6678, 2019.
Hao Zhu, Yonatan Bisk, and Graham Neubig. The lexical dependencies: Neural lexicalized pefgs. Transactions of the Association for Computational Linguistics (TACL., 8:647-661, 2020.
A RELATED WORK
Grammar Induction for Language Grammar Induction for Language, especially unsupervised constituent parsing, a fundamental task in Natural Language Processing that aims to capture syn- tactic information in sentences in the form of phrase-structure tree. The constituency structure shows the process of analyzing the sentences by breaking down it into constituents. It finds its ap- plications in semantic role labeling (Strubell et al. 2018) and word representation (Kuncoro al.) 2020) and many other tasks. While grammar induction has a long history in computational linguis- ties, previous studies are mostly limited language domain with unannotated language corpora,
13

---

Published as a conference paper at ICLR 2022
e.g., Michael 12011: Kim al. (2019)); Drozdov al. (2019) 2020); Shen et al. 2019
2018);
Kim et al. (2019)). Recently, visually-grouma grammar induction, using visual images as percep- tual experience of language, obtains increasing attention. [Shi et al. (2019) first propose visually grounded neural syntax learner, and Kojima et al. (2020) find such model is potentially biased to- wards concrete noun phrases. Then, more works (Zhao & Titov 2020b Zhang et al. 2021 Hong itt al. 2021) further exploit visual semantics derived from images to improve continuous vector rep- resentations of language. Although these works get superior results on grammar induction, they don't 't consider building a unified vision and language structure.
wreakly-uuperiseer Visual Grounding Weakly-supervisee Visual Grounding aims build the fine-grained correspondence between language phrases and image regions with only image-caption pains as supervision. Current works (Rohrbach et al. 2016 Chen al.) 2018 Gupta et al.) 2020 Wang et al. 2021 Liu et al. 2021 Wang et al. 2020b) typically start with generating a set of region proposals from the input image by adopting an off-the-shelf object detector, and then formulate the grounding task as weakly-supervisee matching problem between the phrases and the proposals. GroundR (Rohrbach et al., 2016) built the cross-modal matching by reconstructing the input phrases with attention mechanism the visual features of the proposals. KAC-Net (Chen et al. 2018) took a similar pipeline but further utilized object categories as additional constraints and exploited visual consistency for supervision. Besides, some approaches (Gupta et al., 2020 Wang et al., 2021] exploited knowledge distillation from the language models and visual models pre-trained on an external corpus. RIR (Liu et al., 202I) explored the language structure from an external language parser that provided additional relation constrain on the noun phrases (Liu et al. 2020).
and adopts a self-taught regression strategy to refine the box locations. MAF (Wang et al. 2020b)
used simple but effective contrastive learning strategy for visual representation learning (Kiros al. 2014) alignment between the constituents of an image structure and language structure. Due to the separate image grammar, segmentation parts should provided in advance which hinders application in broader domain.
20144 and
visual grounding. VLGrammar (Hong et al. (202I) represents an
B VL STRUCTURE FORMULATION
The VL structure is shown in Figure 0 (left). shared vision-language constituency structure can be decoupled into language constituency tree for grammar induction (Figure ( (bottom right)) and grounding results for visual grounding (Figure 6 (top right)). From the visual angle, the structure provides a hierarchy scene understanding.
During this natural decoupling process, no training or extra parameters are needed. It reveals that building this VL structure requires the capability of structured prediction for single-modality along with fine-grained alignment with heterogeneous modalities. Due to this reason, this task is chal- lenging. Nevertheless, building this VL structure benefits each separate task and obtains superior performance without using any extra annotations.
C TASK FORMULATION
Unsupervised VL grammar induction aims to induce a constituency grammar for both the caption and the image that the caption describes simultaneously. The VL structure y given the caption based on the constituency relations and defined as a 3-tuple G = {N, RJ, where N is the non-terminal nodes; E is finite set of terminal nodes, namely words; R represents the production rules including two classes of rules:
BC; A, A,B,CEN,
A - I A E.
Each node A E N located in cell (i is associated with inside score sty. outside vector hy for the inside pass and outside score sout outside vector h for the outside pass.
14

---

Published as a conference paper at ICLR 2022
Visual Grounding
man man boy a boy zip-line a zip-line on a zip-line
Grammar Induction
A man pushes a boy on zip-line
Figure 6: Illustration of the VL structure. Left: VL structure. Left-Right: decoupling of the VL structure. Bottom right: language constituency tree. Top right:grounding results. Different phrase are allowed to be aligned to the same region.
Inside Pass
Outside Pass
Inside Soare
Isade Veier
Socre
h +
Carezle Vator
2.
8 to
to
2,1
Arm
(vn)
$ 2.A
h
not
1,k
i=j
j+l
k
k+1
IEK'
I++1
Figure 7: Detailed diagram of structure construction. The inside pass (left) and outside pass (right) illustrated. Structure construction is the process of filling the chart T. Rounds denote scores and rectangles denote vectors. Orange and blue shows variables the inside pass. Yellow and green shows variables in the outside pass. Arrows refer operations data.
D STRUCTURE CONSTRUCTION
We follow Lafferty (2000): Drozdov al. (2020) use an indexing scheme for the constituency tree structure as shown in Figure We use autoencoder model to integrate the visual information
Outside Pass
Inside Pass
1++
2-7
j+7
k+7
Figure 8: Inside pass and outside pass using the indexing scheme. For the inside pass (Left) two spans 6. and Gl+1,5 1.j composed as bigger span. For the outside pass (Right), representation of a target span G, is recursively computed from the inside representation of C1 and outside representation of 1,+1,k and Cx+1,n- can appear to the left or right the target span C₁ Here we show & on the right. If & on the left, the indexing is adjusted.
15

---

Published as a conference paper at ICLR 2022
and employ the inside pass and outside pass fill in the chart T. For leaf nodes, s/ and fig = norm(ws (w)) + +. It, with a normalization layer. For Root node, the outside score sp, is initialized as and hi is initialized randomly independent of x. Then inside-outside algorithm is employed to recursively calculate the scores and vectors as shown in Figure 7
Inside Pass The encoder is bottomm-p flow by running an inside pass through all spans in the input sentence x and computes the inside vector hy and inside score s for each constituent C, to fill up each cell (i in the chart T as shown in Figure 2 for each decomposition (i, k) and (k 1,
More formally, we first compute items
h i.j.k f( bet1,))
" (h.tk) w(hg sit sk+l.j
(14)
where the matrix W and the weights in f are trainable parameters. Then the combined constituent obtained by weighted summation of all possible pairs with the normalized 8.
gin k
Soft max ( k
Σ hi
Σ
(15)
Outside Pass In contrast, the decoder performs top-down outside pass, computing the outside score Si
yuut and the outside vector h by aggregating the inside representation and outside repre-
sentation with the outside algorithm.:
f(hrue (hoou h.41,k)
k>
h out biigh =
(16)
[(hgu hg k <
(hou
W W(h ( si
+ sin
k>
out J.k
1.+1k
(17)
(haut)
T
W (h in
k + sout
+ + k
in
k<i
The computation of outside score st.y and outside feature representation h similar to the inside pass:
gond
Soft max (sk.k)
hout
Σ
h hou J.
gout
Σ
out Siik Sig.
(18)
In this way, bottom-most vectors in the outside pass hove encodes the context of entire sentence except for i-th word.
E SPAN SCORE
In PCFG, the marginal of the span C.j is the conditional probability that is assigned after x is taken into account. Inspired by this, we define q(Grj x) measure how likely the span C.2 exists given x. Since each span in PCFG is assigned to a non-terminal symbol, the marginal of the span Gi can be calculated using the summarization over all non-terminal symbols. However, due to the continuous representation for each span for CLIORA, we define q(G.;: x) similar to PCFG. As shown in Figure 9 for PCFG since each span (namely, the string of words (x,...y} in the sentence XI...X, is assigned to non-terminal symbol A. The marginal (ie., posterior) of span [x) is the summarization over all non-terminal symbols
N (N is the set of all AC*) can be factorized inside scores a (.+1, outside score (B,((),, production rule p(B + AC). The product of these three parts is finally divided by the partition function (the inside score of the whole sentence
non-terminal symbols). The numerator of p(ct,,,(B
three parts:
a1,n.):
P.G.AA - AC)(x) = (A)(A) aa+i,((C)) p(B - AC). (,,(())
(19)
01,n
16

---

Published as a conference paper at ICLR 2022
Chorklo Sane
PCRG
CLIORA
5
B
1-1
j+/
k+1
6-1
j+1
K+1
Figure 9: Calculation of the marginal and the span score. Right: marginal p(c,)lx) of PCFG. The marginal of span G.j is fraction and the numerator consists of three parts: inside scores (. (f,,,,((C))) (shaded in blue), outside score (4.x (shaded in green), production rule p( - AC). Left: span score q(cr,, x) of CLIORA. The numerator of q(cs.), x) can be factorized as two sout (shaded in green). The product of parts: inside scores st. (shaded in blue) and outside score 1.j
these two parts finally divided by s
0.5
0.5
Epoch
10
Learning rate
le -
Batch Size
64
Table 5: Hyper-parameters for CLIORA.
However, the non-terminal symbol is not assigned to spans for CLIORA. Hence, the span score of span Grj defined as:
8in out
x(C.j.,)=
(20)
where is the length of the sentence i and represent the begin and end positions of the span in the sentence x, respectively. The numerator of (( x) can be factorized as two parts: inside scores sig and outside score out The product of these two parts is finally divided by sfm. i.j
F
SETTING
The model selection is based on the loss on the development dataset. More concretely, we compute the average loss on the development set after each training epoch and select the model before the loss starts to increase. We list the main prreepparameters in Table and others are default as DIORA.
G DETAILED CASE STUDY
We visualize parse trees predicted by CLIORA in Figure Left trees illustrate success cases and right trees are failure cases. For the top right tree of the failed cases, CLIORA wrongly predicts two spans "a green" and "a blue" that should be "green shirt" and "blue vehicle" in the ground-truie The erroneous span "a green" is wrongly grounded to green building. However, because there exists only one blue region, the erroneous span blue" is correctly grounded to the blue vehicle. Interestingly, after combining more contexts and constituent bigger span "in green shirt", the grounding result modified to a correct region. This fail case reveals the importance of contextual semantics, which is essential motivation of our task and model.
For the bottom right tree of the failed cases, CLIORA wrongly grounded the span "a boy and a girl" to the region referring to only the boy. We speculate that this enror is due the binary trait of the structure. "A boy" "and" and "a girl" should be combined simultancously instead of wrongly combined the first two phrases then combine the last We will leave this for further study.
17

---

Published as a conference paper at ICLR 2022
Success Case
Failure Case
Georsting
Cunect Grousling
bey
in red
sweatshirt presends
80
deive
tractor Tor
bay
and
swinging
OE
barrel
Squn
Figure 10: Case Study. Rols without rectangles and spans without special marks are predicted correctly.
5.079
8.356
0.048
0037
8.000
0.090
OUTO
0.250
0.100
0/080
0.018
0470
5,010
nC10
SP
3.041
O018
0.013
0030
B.22
10
8,003
0.000
0.30
0.000
0/00
890
0.20
0630
8C10
5848
2/34
0.244
0011
0001
OU25
5,000
0.00
0000
0000
0030
0.00
0.00
8,022
0.364
5,003
0031
0.000
0.000
0,90
8,000
0.000
0,000
0.000
0080
0.000
0.000
Figure 11: Label distribution over constituent length. The values in the cell denote frequencies of different constituent lengths and phrase types. Left: Flickr30k. Right: MSCOCO.
18

---

Published as a conference paper at ICLR 2022
H ABLATION STUDY OF LANGUAGE AND IMAGE
We conduct an ablation study for contextual information and investigate the impact of image and text. After gradually removing the components, the model exhibit consistent degradation.
We use different forms of visual information in score-level fusion: global image information and fine-grained ROI information. If we remove the
C-F1
All
54.3
fine-grained information, which means the atten-
-Fine- uine-grained ROI information
54.2
tion to the different ROIs are ignored and the av- erage weight are used, the score are even. demonstrates that for score-level fusion, the global
Globbll image information
51.8
Language Context
48.0
Table 6: F1 on Flickr30k development dataset.
image information enough. Additionally, to un- derstand if the contribution of the visual compo- nent to the overall model is analogy with the language context, we add the representation of the whole sentence after removal of all visual information including fine- aineggrined ROIs and the whole image. As for this replacement, the performance decreases, demonstrating that visual information really helps in compensation way, that is restricted to providing contextual information like language.
I ROBUSTNESS ACROSS CORPORA
As shown in Table I different models perform very differently on MSCOCO and Flickr30k. PCFG- based models perform poorly on Flickr30k, while works well MSCOCO. Two classic baselines "Right branching" and "Left branching" perform completely opposite on MSCOCO and Flickr30k. "Right branch" obtains superior 51.0 1.0FF F1 while "Left branch" only gets 15.1 on MSCOCO. "Right branch" performs worse than "Right branch" on Flickr30k. We posit this phenomenon is due to the diverse data distribution MSCOCO and Flickr30k. To study this, we plot the detailed distribution over constituent length for different phrase types in Figure 11 Flickr30k contains various phrase types, e.g., PP, SBAR and ADJP and longer constituents. In contrast, CLIORA displays more even performance across different corpora than PCFG-based models, which reveals its robustness across corpora.
19


Reviews:
[{"review_id": "yOUXkMD_Xds", "reviewer": "Reviewer_cvGw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea of jointly inducing structure in natural language and grounding the constituents with real-world images is intuitively correct.\nThe ablation study also shows that both feature-level fusion and score-level fusion (including the contrastive loss, if I understand correctly) helps in improving the parsing results.\n\nWeakness:\n1) The image features are only used for computing the inside pass. The image feature should contain information that can help predict the missing word, such that it could be used in the outside pass too. Selecting the best image region for predicting the missing word is also an intuitively correct way to build the vision-language alignment.\n2) The compute of sim(I, c_ij) includes a max operator, this could lead to a biased gradient.\n3) As the author mentioned in the discussion section, the model doesn't consider the latent hierarchical structure of the image. For example, the sentence describes the entire image, while each phrase describes part of the image.", "summary_of_the_paper": "The paper proposed a new method CLIORA to do unsupervised parsing and vision-language grounding. \nCLIORA is based on DIORA model. \nBut different from previous unsupervised parsing methods, CLIORA also induces alignment between constituents and image regions. \nIn order to train the model, the author introduces a contrastive loss.\nExperiment results show that the proposed method outperforms baseline unsupervised parsing methods and it also induces meaningful alignment between image regions and constituents.", "summary_of_the_review": "Overall, the proposed method is interesting and inspiring. \nThe idea should be interesting to both unsupervised parsing and multimodel communities.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4o5--jl8W9T", "reviewer": "Reviewer_3QG1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of grammar induction has been there for a very long time in NLP and is a very fundamental topic.  The model was largely built based on an existing model for purely text-based grammar induction. The model essentially makes use of neural networks to learn good latent representations (using a reconstruction loss) where the latent representation is defined with neural networks which yield scores for constituents and vector representations of them. The approach adopts the classical inside-outside process for the computing of the scores.\n\nThe paper essentially investigates what might be the effective methods for integrating image information into text for improved grammar induction. The execution of the paper was quite good, and the results are convincing. However, I feel the overall model is essentially a way to use image information to regularize the grammar induction process. Little can be said about in what precise manner the image is actively contributing to the induction process. Indeed, the authors also acknowledged something along with what I thought in the final section. Nevertheless, I think it is an interesting piece that might inspire future research on multimodal processing (for image + language).", "summary_of_the_paper": "This paper presents a new model for grammar induction for text, with help from the coupled images. The model was built on top of an existing unsupervised grammar induction model used for text without image information. The experimental results show the approach was effective. The work essentially demonstrates some effective ways of leveraging the additional image information for improving the grammar induction task. The paper also discussed some weaknesses of the approach and future work.", "summary_of_the_review": "I think this is a reasonable piece, with good writing and a nice set of experiments. It would be helpful for future research in this domain.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-x0_r3p_rm", "reviewer": "Reviewer_hzHn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths \n\n- As far as I know, this is the first paper that proposes joint visual-linguistic grammar induction in a real-world setting (in contrast to synthetic settings; Hong et al., 2021).\n\n- The approach and the evaluation process are solid and make a lot of sense to me.  \n\n- The visually grounded parsing results are quite impressive. \n\n### Weakness\n\n- My major concern is about the model selection process and the potential unfair comparisons to existing work. \n    - Model selection: If I understood correctly, for text parsing, the best models are selected w.r.t. to the parsing performance on a 1000-example dev set (Appendix F). \\\nThis is an unrealistic setting (see https://aclanthology.org/2020.emnlp-main.614.pdf for discussions; in short, for any fancy unsupervised parsing model that uses a labeled development set, a supervised model trained on these development examples should be considered as a strong baseline) -- introducing unsupervised criteria for model selection is more important than our initial impression.\n\n    - Unfair comparison: CLIORA, the model proposed in this paper, uses DIORA as initialization, which uses ELMo to initialize word embeddings and the PTB labeled development set for model selection. This means that CLIORA has seen far more text than other baselines (VG-NSL, C-PCFG, VC-PCFG, and so on) and human language learners. \\\nThis issue also undermines the authors' arguments about potential links to how humans learn language. I expect either a CLIORA trained from scratch (without DIORA initialization) or weakened arguments about the relationship between the current CLIORA and human language learning.\n\n- There seem to be some confusion on basic linguistic concepts, e.g., nonterminal vs. terminal symbols, and a few typos that affects smooth understanding (please see also detailed comments below). \n\n### Other comments and questions\n\n- Introduction: \"These works, however, fail to consider a unified VL structure, nor have they demonstrated impact on visual understanding.\"  \\\nI don't think I necessarily agree with this statement, especially regarding Hong et al. (2021). Despite that there is a clear gap between their dataset and the real world settings, they are aligning the \"visual grammars\" to language grammars, yielding an arguably unified VL structure. \n\n- Introduction: \"The non-terminal symbol of a conventional constituency structure is a category label from a limited set (e.g., the set of part-of-speech (POS) tags) (Hopcroft et al., 2001). \" \\\nDo you mean *terminal* symbols here? We usually refer to POS tags (to clarify, phrase tags are not POS tags) by preterminal or terminal (depending on whether the phrase-structure grammar is lexicalized, i.e., whether it's considering real words or just POS tags), and refer to the phrase nodes by nonterminal nodes/symbols (e.g., NP, PP). It seems that this is not a typo -- I have the same questions for the following task definition section on page 3.\n\n- Task definition, evaluation metrics: if I understood correctly, CCRA requires some extra annotation of critical concepts -- how did you collect such annotations to determine which NPs are critical? \\\n(Very minor) based on the full name, CCRA should really be CCRR -- what does A stand for here? \n\n- Section 3.2, feature extraction: the Yoon Kim et al. (2019b) paper is not relevant to image features at all -- did you mean Shi et al., (2019)? \n\n- Table 1: what is the dagger after VGNSL-HI? \n\n- Section 4.3: did you mean \"augments\" by \"arguments\"? \n\n- Some more thoughts regarding motivation limitations: humans arguably learns how to parse concrete sentences first, and can then generalize to abstract domains that are not visually groundable. In this work, it seems that the model only works when both the text and image are available, as there is a need to infuse visual features into text spans. Do you have any thoughts on enabling a trained CLIORA model to parse pure text without grounding signals?\n\n### Missing Reference\n\nKojima et al. [1] has strengthened the VG-NSL model by simplifying the architecture, and argued that such visually grounded models are potentially biased towards concrete noun phrases. However, the paper neither cited it nor discussed the relevant issues. \n\n[1] https://aclanthology.org/2020.acl-main.234.pdf \n\nThere have been a lot of relevant work earlier than 2019 on visual-semantic embeddings or structured visual understanding with text. To name a few, \n\n**Older work on structured image-text representations**\n\n[2] https://openaccess.thecvf.com/content_iccv_2015/papers/Ma_Multimodal_Convolutional_Neural_ICCV_2015_paper.pdf\n\n[3] https://openaccess.thecvf.com/content_cvpr_2018/papers/You_End-to-End_Convolutional_Semantic_CVPR_2018_paper.pdf\n\n**Contrastive loss for visual-semantic embeddings**\n\n[4] https://arxiv.org/pdf/1411.2539.pdf\n\n\n### Minor Editing Comments\n- I was confused about what CCRA is when reading the abstract -- would be good to include the full name and give an intuitive description of the metric. \n\n- Yoon et al. $\\rightarrow$ Kim et al. \n\n- Shi et al. (2019) proposes $\\rightarrow$ Shi et al. (2019) propose\n\n- In my opinion, putting Section 3.4 before 3.3 would better streamline the paper. \n", "summary_of_the_paper": "This paper introduces a task of joint visual-linguistic grammar induction from parallel image-text data, presents models and metrics for the task, and shows strong empirical results.\n", "summary_of_the_review": "This paper introduces the task of joint visual-linguistic grammar induction, and presents models, metrics and empirical results on it. While I appreciate the impressive results, I am concerned about the unrealistic model selection process (comparing model outputs to a large set of ground-truth parse trees) and the unfair comparison (the proposed model has access to much more unlabeled text data than baselines).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to perform unsupervised grammar induction over image-text pairs and used shared structure between the modalities to improve grammar induction on both sides. Authors find the paper clear, creative and interesting and recommend acceptance without hesitation.
Decision: Accept (Oral)