Venue: NAACL 2021

# Video-aided Unsupervised Grammar Induction

 Songyang Zhang\({}^{1}\), Linfeng Song\({}^{2}\), Liffeng Jin\({}^{2}\), Kun Xu\({}^{2}\), Dong Yu\({}^{2}\)  and Jiebo Luo\({}^{1}\)

\({}^{1}\)University of Rochester, Rochester, NY, USA

szhang83@ur.rochester.edu, jluo@cs.rochester.edu

\({}^{2}\)Tencent AI Lab, Bellevue, WA, USA

{lfsong,lifengjin,kxkunxu,dyu}@tencent.com

This work was done when Songyang Zhang was an intern at Tencent AI Lab.

###### Abstract

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (_e.g._ action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model Kim et al. (2019) as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, _i.e._ DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction.

## 1 Introduction

Constitency parsing is an important task in natural language processing, which aims to capture syntactic information in sentences in the form of constituency parsing trees. Many conventional approaches learn constituency parser from human-annotated datasets such as Penn Treebank Marcus et al. (1993). However, annotating syntactic trees by human language experts is expensive and time-consuming, while the supervised approaches are limited to several major languages. In addition, the treebanks for training these supervised parsers are small in size and restricted to the newswire domain, thus their performances tend to be worse when applying to other domains Fried et al. (2019). To address these issues, recent approaches Shen et al. (2018); Jin et al. (2018); Drozdov et al. (2019); Kim et al. (2019) design unsupervised constituency parsers and grammar inducers, since they can be trained on large-scale unlabeled data. In particular, there has been growing interests in exploiting visual information for unsupervised grammar induction because visual information can capture important knowledge required for language learning that is ignored by text Gleitman (1990); Pinker and MacWhinney (1987); Tomasello (2003). This task aims to learn a constituency parser from raw unlabeled text aided by its visual context.

Previous methods Shi et al. (2019); Kojima et al. (2020); Zhao and Titov (2020); Jin and Schuler (2020) learn to parse sentences by exploiting object information from images. However, images are static and cannot present the dynamic interactions among visual objects, which usually correspond to verb phrases that carry important information. There

Figure 1: Examples of video aided unsupervised grammar induction. We aim to improve the constituency parser by leveraging aligned video-sentence pairs.



fore, images and their descriptions may not be fully-representative of all linguistic phenomena encountered in learning, especially when action verbs are involved. For example, as shown in Figure 1(a), when parsing a sentence "A squirrel **jumps on stump**", a single image cannot present the verb phrase "**jumps on stump**" accurately. Moreover, as shown in Figure 1(b), the guitar sound and the moving fingers clearly indicate the speed of music playing, while it is impossible to present only with a static image as well. Therefore, it is difficult for previous methods to learn these constituents, as static images they consider lack dynamic visual and audio information.

In this paper, we address this problem by leveraging video content to improve an unsupervised grammar induction model. In particular, we exploit the current state-of-the-art techniques in both video and audio understanding, domains of which include object, motion, scene, face, optical character, sound, and speech recognition. We extract features from their corresponding state-of-the-art models and analyze their usefulness with the VC-PCFG model Zhao and Titov (2020). Since different modalities may correlate with each other, independently modeling each of them may be sub-optimal. We also propose a novel model, Multi-Modal Compound Probabilistic Context-Free Grammars (MMC-PCFG), to better model the correlation among these modalities.

Experiments on three benchmarks show substantial improvements when using each modality of the video content. Moreover, our MMC-PCFG model that integrates information from different modalities further improves the overall performance. Our code is available at [https://github.com/Sy-Zhang/MMC-PCFG](https://github.com/Sy-Zhang/MMC-PCFG).

The main contributions of this paper are:

* We are the first to address video aided unsupervised grammar induction and demonstrate that verb related features extracted from videos are beneficial to parsing.
* We perform a thorough analysis on different modalities of video content and propose a model to effectively integrate these important modalities to train better constituency parsers.
* Experiments results demonstrate the effectiveness of our model over the previous state-of-the-art methods.

## 2 Background and Motivation

Our model is motivated by C-PCFG Kim et al. (2019) and its variant of the image-aided unsupervised grammar induction model, VC-PCFG Zhao and Titov (2020). We will first review the evolution of these two frameworks in Sections 2.1-2.2, and then discuss their limitations in Section 2.3.

### Compound PCFGs

A probabilistic context-free grammar (PCFG) in Chomsky normal form can be defined as a 6-tuple \((S,{\cal N},{\cal P},\Sigma,{\cal R},\Pi)\), where \(S\) is the start symbol, \({\cal N},{\cal P}\text{ and }\Sigma\) are the set of nonterminals, preterminals and terminals, respectively. \({\cal R}\) is a set of production rules with their probabilities stored in \(\Pi\), where the rules include binary nonterminal expansions and unary terminal expansions. Given a certain number of nonterminal and preterminal categories, a PCFG induction model tries to estimate rule probabilities. By imposing a sentence-specific prior on the distribution of possible PCFGs, the compound PCFG model Kim et al. (2019) uses a mixture of PCFGs to model individual sentences in contrast to previous models Jin et al. (2018) where a corpus-level prior is used. Specifically in the generative story, the rule probability \(\pi_{r}\) is estimated by the model \(g\) with a latent representation \(\mathbf{z}\) for each sentence \(\sigma\), which is in turn drawn from a prior \(p(\mathbf{z})\):

\[\pi_{r}=g_{r}(\mathbf{z};\theta),\qquad\mathbf{z}\sim p(\mathbf{z}). \tag{1}\]

The probabilities for the CFG initial expansion rules \(S\to A\), nonterminal expansion rules \(A\to B~{}C\) and preterminal expansion rules \(T\to w\) can be estimated by calculating scores of each combination of a parent category in the left hand side of a rule and all possible child categories in the right hand side of a rule:

\[\begin{split}\pi_{S\to A}&=\frac{\exp(\mathbf{u}^{ \top}_{A}f_{s}([\mathbf{w}_{S};\mathbf{z}]))}{\sum_{A^{\prime}\in{\cal N}} \exp(\mathbf{u}_{A^{\prime}}f_{s}([\mathbf{w}_{S};\mathbf{z}]))},\\ \pi_{A\to BC}&=\frac{\exp(\mathbf{u}^{\top}_{BC}[ \mathbf{w}_{A};\mathbf{z}])}{\sum_{B^{\prime},C^{\prime}\in{\cal N}\cup{\cal P }}\exp(\overline{\mathbf{u}^{\top}_{B^{\prime}C^{\prime}}[\mathbf{w}_{A}; \mathbf{z}])})},\\ \pi_{T\to w}&=\frac{\exp(\mathbf{u}^{\top}_{v} f_{t}([\mathbf{w}_{T};\mathbf{z}]))}{\sum_{w^{\prime}\in\Sigma}\exp( \mathbf{u}^{\top}_{w^{\prime}}f_{t}([\mathbf{w}_{T};\mathbf{z}]))},\end{split} \tag{2}\]

where \(A,B,C\in{\cal N}\) , \(T\in{\cal P}\), \(w\in\Sigma\), \(\mathbf{w}\) and \(\mathbf{u}\) vectorial representations of words and categories, and \(f_{t}\) and \(f_{s}\) are encoding functions such as neural networks 

Optimization of the PCFG induction model usually involves maximizing the marginal likelihood of a training sentence \(p(\sigma)\) for all sentences in a corpus. In the case of compound PCFGs:

\[\log p_{\theta}(\sigma)=\log\int_{\mathbf{z}}\sum_{t\in\mathcal{T}_{\mathcal{G}} (\sigma)}p_{\theta}(t|\mathbf{z})p(\mathbf{z})d\mathbf{z}, \tag{3}\]

where \(t\) is a possible binary branching parse tree of \(\sigma\) among all possible trees \(\mathcal{T}\) under a grammar \(\mathcal{G}\). Since computing the integral over \(\mathbf{z}\) is intractable, \(\log p_{\theta}(\sigma)\) can be optimized by maximizing its evidence lower bound \(\mathrm{ELBO}(\sigma;\phi,\theta)\):

\[\begin{split}\mathrm{ELBO}(\sigma;\phi,\theta)&= \mathbb{E}_{q_{\phi}(\mathbf{z}|\sigma)}[\log p_{\theta}(\sigma|\mathbf{z})]\\ &-\mathrm{KL}[q_{\phi}(\mathbf{z}|\sigma)||p(\mathbf{z})],\end{split} \tag{4}\]

where \(q_{\phi}(\mathbf{z}|\sigma)\) is a variational posterior, a neural network parameterized with \(\phi\). The sample log likelihood can be computed with the inside algorithm, while the KL term can be computed analytically when both prior \(p(\mathbf{z})\) and the posterior approximation \(q_{\phi}(\mathbf{z}|\sigma)\) are Gaussian Kingma and Welling (2014).

### Visualy Grounded Compound PCFGs

The visually grounded compound PCFGs (VCPCFG) extends the compound PCFG model (C-PCFG) by including a matching model between images and text. The goal of the vision model is to match the representation of an image \(\mathbf{v}\) to the representation of a span \(\mathbf{c}\) in a parse tree \(t\) of a sentence \(\sigma\). The word representation \(\mathbf{h}_{i}\) for the \(i\)th word is calculated by a BiLSTM network. Given a particular span \(c=w_{i},\dots,w_{j}(0<i<j\leq n)\)], we then compute its representation \(\mathbf{c}\). We first compute the probabilities of its phrasal labels \(\{p(k|c,\sigma)|1\leq k\leq K,K=|\mathcal{N}|\}\), as described in Section 2.1. The representation \(\mathbf{c}\) is the sum of all label-specific span representations weighted by the probabilities we predicted:

\[\mathbf{c}=\sum_{k=1}^{K}p(k|c,\sigma)f_{k}(\frac{1}{j-i+1}\sum_{l=i}^{j} \mathbf{h}_{l}), \tag{5}\]

Finally, the matching loss between a sentence \(\sigma\) and an image representation \(\mathbf{v}\) can be calculated as a sum over all matching losses between a span and the image representation, weighted by the marginal of a span from the parser:

\[s_{img}(\mathbf{v},\sigma)=\sum_{c\in\sigma}p(c|\sigma)h_{img}(\mathbf{c}, \mathbf{v}), \tag{6}\]

where \(h_{img}(\mathbf{c},\mathbf{v})\) is a hinge loss between the distances from the image representation \(\mathbf{v}\) to the matching and unmatching (_i.e._ sampled from a different sentence) spans \(\mathbf{c}\) and \(\mathbf{c}^{\prime}\), and the distances from the span \(\mathbf{c}\) to the matching and unmatching (_i.e._ sampled from a different image) image representations \(\mathbf{v}\) and \(\mathbf{v}^{\prime}\):

\[h_{img}(\mathbf{c},\mathbf{v})=\mathbb{E}_{\mathbf{c}^{\prime}}[ \cos(\mathbf{c}^{\prime},\mathbf{v})-\cos(\mathbf{c},\mathbf{v}))+\epsilon]_{+} \\ +\mathbb{E}_{\mathbf{v}^{\prime}}[\cos(\mathbf{c},\mathbf{v}^{ \prime})-\cos(\mathbf{c},\mathbf{v})+\epsilon]_{+}, \tag{7}\]

where \(\epsilon\) is a positive margin, and the expectations are approximated with one sample drawn from the training data. During training, ELBO and the image-text matching loss are jointly optimized.

### Limitation

VC-PCFG improves C-PCFG by leveraging the visual information from paired images. In their experiments Zhao and Titov (2020), comparing to C-PCFG, the largest improvement comes from NPs (\(+11.9\%\) recall), while recall values of other frequent phrase types (VP, PP, SBAR, ADJP and ADVP) are fairly similar. The performance gain on NPs is also observed with another multi-modal induction model, VG-NSL Shi et al. (2019); Kojima et al. (2020). Intuitively, image representations from image encoders trained on classification tasks very likely contain accurate information about objects in images, which is most relevant to identifying NPs1. However, they provide limited information for phrase types that mainly involve action and change, such as verb phrases. Representations of dynamic scenes may help the induction model to identify verbs, and also contain information about the argument structure of the verbs and nouns based on features of actions and participants extracted from videos. Therefore, we propose a model that induces PCFGs from raw text aided by the multi-modal information extracted from videos, and expect to see accuracy gains on such places in comparison to the baseline systems.

Footnote 1: Jin and Schuler (2020) reports no improvement on English when incorporating visual information into a similar neural network-based PCFG induction model, which may be because Zhao and Titov (2020) removes punctuation from the training data, which removes a reliable source of phrasal boundary information. This loss is compensated by the induction model with image representations. We leave the study of evaluation configuration on induction results for future work.

## 3 Multi-Modal Compound PCFGs

In this section, we introduce the proposed multi-modal compound PCFGs (MMC-PCFG). Instead 

of purely relying on object information from images, we generalize VC-PCFG into the video domain, where multi-modal video information is considered. We first introduce the video representation in Section 3.1. We then describe the procedure for matching the multi-modal video representation with each span in Section 3.2. After that we introduce the training and inference details in Section 3.3.

### Video Representation

A video contains a sequence of frames, denoted as \(V=\{v_{i}\}_{i=1}^{L^{0}}\), where \(v_{i}\) represents a frame in a video and \(L^{0}\) indicates the total number of frames. We extract video representation from \(M\) models trained on different tasks, which are called _experts_. Each expert focuses on extracting a sequence of features of one type. In order to project different expert features into the same dimension, their feature sequences are feed into linear layers (one per expert) with same output dimension. We denote the outputs of the \(m\)th expert after projection as \(\mathbf{F}^{m}=\{\mathbf{f}_{i}^{m}\}_{i=1}^{L^{m}}\), where \(\mathbf{f}_{i}^{m}\) and \(L^{m}\) represent the \(i\)th feature and the total number of features of the \(m\)th expert, respectively.

A simple method would average each feature along the temporal dimension and then concatenating them together. However, this would ignore the relations among different modalities and the temporal ordering within each modality. In this paper, we use a multi-modal transformer to collect video representations Gabeur et al. (2020); Lei et al. (2020).

The multi-modal transformer expects a sequence as input, hence we concatenate all feature sequences together and take the form:

\[\mathbf{X}=[\mathbf{f}_{avg}^{1},\mathbf{f}_{1}^{1},...,\mathbf{f}_{L_{1}}^{ L},...\mathbf{f}_{avg}^{M},\mathbf{f}_{1}^{M},...,\mathbf{f}_{L_{M}}^{M}], \tag{8}\]

where \(\mathbf{f}_{avg}^{m}\) is the averaged feature of \(\{\mathbf{f}_{i}^{m}\}_{i=1}^{L_{m}}\). Each transformer layer has a standard architecture and consists of multi-head self-attention module and a feed forward network (FFN). Since this architecture is permutation-invariant, we supplement it with expert type embeddings \(\mathbf{E}\) and positional encoding \(\mathbf{P}\) that are added to the input of each attention layer. The expert type embeddings indicate the expert type for input features and take the form:

\[\mathbf{E}=[\mathbf{e}^{1},\mathbf{e}^{1},...,\mathbf{e}^{1},...,\mathbf{e}^{ M},\mathbf{e}^{M},...,\mathbf{e}^{M}], \tag{9}\]

where \(\mathbf{e}^{m}\) is a learned embedding for the \(m\)th expert. The positional encodings indicate the location of each feature within the video and take the form:

\[\mathbf{P}=[\mathbf{p}_{0},\mathbf{p}_{1},...,\mathbf{p}_{L_{1}},...,\mathbf{ p}_{0},\mathbf{p}_{1},...,\mathbf{p}_{L_{M}}], \tag{10}\]

where fixed encodings are used Vaswani et al. (2017). After that, we collect the output of transformer that corresponds to the averaged features as the final video representation, _i.e._, \(\boldsymbol{\Psi}=\{\mathbf{\psi}_{avg}^{i}\}_{i=1}^{M}\). In this way, we can learn more effective video representation by modeling the correlations of features from different modalities and different timestamps.

### Video-Text Matching

To compute the similarity between a video \(V\) and a particular span \(c\), a span representation \(\mathbf{c}\) is obtained following Section 2.2 and projected to \(M\) separate expert embeddings via gated embedding modules (one per expert) Miech et al. (2018):

\[\begin{split}&\mathbf{\xi}_{1}^{i}=\mathbf{W}_{1}^{i}\mathbf{c}+\mathbf{b}_{1}^{ i},\\ &\mathbf{\xi}_{2}^{i}=\mathbf{\xi}_{1}^{i}\circ sigmoid(\mathbf{W}_{2}^{i} \mathbf{\xi}_{1}^{i}+\mathbf{b}_{2}^{i}),\\ &\mathbf{\xi}^{i}=\frac{\mathbf{\xi}_{2}^{i}}{\|\mathbf{\xi}_{2}^{i}\|_{2}} ,\end{split} \tag{11}\]

where \(i\) is the index of expert, \(\mathbf{W}_{1}^{i}\), \(\mathbf{W}_{2}^{i}\), \(\mathbf{b}_{1}^{i}\), \(\mathbf{b}_{2}^{i}\) are learnable parameters, \(sigmoid\) is an element-wise sigmoid activation and \(\circ\) is the element-wise multiplication. We denote the set of expert embeddings as \(\mathbf{\Xi}=\{\mathbf{\xi}^{i}\}_{i=1}^{M}\). The video-span similarity is computed as following,

\[\begin{split}&\omega_{i}(\mathbf{c})=\frac{\exp(\mathbf{u}_{i}^{ \top}\mathbf{c})}{\sum_{j=1}^{M}\exp(\mathbf{u}_{j}^{\top}\mathbf{c})},\\ & o(\mathbf{\Xi},\boldsymbol{\Psi})=\sum_{i=1}^{M}\omega_{i}( \mathbf{c})\mathrm{cos}(\mathbf{\xi}^{i},\mathbf{\psi}^{i}),\end{split} \tag{12}\]

where \(\{\mathbf{u}_{i}\}_{i=1}^{M}\) are learned weights. Given \(\mathbf{\Xi}^{\prime}\), an unmatched span expert embeddings of \(\boldsymbol{\Psi}\), and \(\boldsymbol{\Psi}^{\prime}\), an unmatched video representation of \(\mathbf{\Xi}\), the hinge loss for video is given by:

\[h_{vid}(\mathbf{\Xi}, \boldsymbol{\Psi})=\mathbb{E}_{\mathbf{c}^{\prime}}[o(\mathbf{\Xi}^{ \prime},\boldsymbol{\Psi})-o(\mathbf{\Xi},\boldsymbol{\Psi}))+\epsilon]_{+}\] \[+\mathbb{E}_{\boldsymbol{\Psi}^{\prime}}[o(\mathbf{\Xi},\boldsymbol{ \Psi}^{\prime})-o(\mathbf{\Xi},\boldsymbol{\Psi})+\epsilon]_{+}, \tag{13}\]

where \(\epsilon\) is a positive margin. Finally the video-text matching loss is defined as:

\[s_{vid}(V,\sigma)=\sum_{c\in\sigma}p(c|\sigma)h_{vid}(\mathbf{\Xi},\boldsymbol{ \Psi}). \tag{14}\]

Noted that \(s_{vid}\) can be regarded as a generalized form of \(s_{img}\) in Equation 6, where features from different timestamps and modalities are considered 



### Training and Inference

During training, our model is optimized by the \({\rm ELBO}\) and the video-text matching loss:

\[\mathcal{L}(\phi,\theta)=\sum_{(V,\sigma)\in\Omega}-{\rm ELBO}(\sigma;\phi, \theta)+\alpha s_{vid}(V,\sigma), \tag{15}\]

where \(\alpha\) is a hyper-parameter balancing these two loss terms and \(\Omega\) is a video-sentence pair.

During inference, we predict the most likely tree \(t^{*}\) given a sentence \(\sigma\) without accessing videos. Since computing the integral over \(\mathbf{z}\) is intractable, \(t^{*}\) is estimated with the following approximation,

\[\begin{split} t^{*}&=\operatorname*{arg\,max}_{t} \int_{\mathbf{z}}p_{\theta}(t|\mathbf{z})p_{\theta}(\mathbf{z}|\sigma)d\mathbf{ z}\\ &\approx\operatorname*{arg\,max}_{t}p_{\theta}(t|\sigma,\mathbf{\mu} _{\phi}(\sigma)),\end{split} \tag{16}\]

where \(\mathbf{\mu}_{\phi}(\sigma)\) is the mean vector of the variational posterior \(q_{\phi}(\mathbf{z}|\sigma)\) and \(t^{*}\) can be obtained using the CYK algorithm Cocke et al. (2019); Younger (1967); Kasami (1966).

## 4 Experiments

### Datasets

**DiDeMo Hendricks et al. (2017)** collects \(10\)K uncided, personal videos from Flickr with roughly \(3-5\) pairs of descriptions and distinct moments per video. There are \(32\,994\), \(4\,180\) and \(4\,021\) video-sentence pairs, validation and testing split.

**YouCook2 Zhou et al. (2018)** includes \(2\)K long untrimmed videos from \(89\) cooking recipes. On average, each video has \(6\) procedure steps described by imperative sentences. There are \(8\,713\), \(969\) and \(3\,310\) video-sentence pairs in the training, validation and testing sets.

**MSRVTT Xu et al. (2016)** contains \(10\)K videos sourced from YouTube which are accompanied by \(200\)K descriptive captions. There are \(130\,260\), \(9\,940\) and \(59\,794\) video-sentence pairs in the training, validation and testing sets.

### Evaluation

Following the evaluation practice in Zhao and Titov (2020), we discard punctuation and ignore trivial single-word and sentence-level spans at test time. The gold parse trees are obtained by applying a state-of-the-art constituency parser, Benepar Kitaev and Klein (2018), on the testing set. All models are run \(4\) times for \(10\) epochs with different random seeds. We evaluate both averaged corpus-level F1 (C-F1) and averaged sentence-level F1 (S-F1) numbers as well as their standard deviations.

### Expert Features

In order to capture the rich content from videos, we extract features from the state-of-the-art models of different tasks, including object, action, scene, sound, face, speech, and optical character recognition (OCR). For object and action recognition, we explore multiple models with different architectures and pre-trained dataset. Details are as follows:

**Object features** are extracted by two models: ResNeXt-\(101\) Xie et al. (2017), pre-trained on Instagram hashtags Mahajan et al. (2018) and fine-tuned on ImageNet Krizhevsky et al. (2012), and SENet-\(154\)Hu et al. (2018), trained on ImageNet. These datasets include images of common objects, such as, "cock", "kite", and "goose", etc. We use the predicted logits as object features for both models, where the dimension is \(1000\).

**Action features** are extracted by three models: I3D trained on Kinetics-\(400\)Carreira and Zisserman (2017), R2P1D Tran et al. (2018) trained on IG-65M Ghadiyaram et al. (2019) and S3DG Miech et al. (2020) trained on HowTo100M Miech et al. (2019). These datasets include videos of human actions, such as "playing guitar", "ski jumping", and "jogging", etc. Following the same processing steps in their original work, we extract the predicted logits as action features, where the dimension is \(400\) (I3D), \(359\) (R2P1D) and \(512\) (S3DG), respectively.

**Scene features** are extracted by DenseNet-\(161\) Huang et al. (2017) trained on Places\(365\)Zhou et al. (2017). Places\(365\) contains images of different scenes, such as "library", "valley", and "rainforest", etc. The predicted logits are used as scene features, where the feature dimension is \(365\).

**Audio features** are extracted by VGGish trained on YouTube-\(8M\) Hershey et al. (2017), where the feature dimension is \(128\). YouTube-\(8M\) is a video dataset where different types of sound are involved, such as "piano", "drum", and "violin".

**OCR features** are extracted by two steps: characters are first recognized by combining text detector Pixel Link Deng et al. (2018) and text recognizer SSFL Liu et al. (2018). The characters are then converted to word embeddings through \(word2vec\) Mikolov et al. (2013) as the final OCR features, where the feature dimension is \(300\).

**Face features** are extracted by combining face de

tector SSD Liu et al. (2016) and face recognizer ResNet50 He et al. (2016). The feature dimension is \(512\).

**Speech features** are extracted by two steps: transcripts are first obtained via Google Cloud Speech to Text API. The transcripts are then converted to word embeddings through \(word2vec\)Mikolov et al. (2013) as the final speech features, where the dimension is \(300\).

### Implementation Details

We keep sentences with fewer than \(20\) words in the training set due to the computational limitation. After filtering, the training sets cover \(99.4\%\), \(98.5\%\) and \(97.1\%\) samples of their original splits in DiDeMo, YouCook2 and MSRVTT.

We train baseline models, C-PCFG and VC-PCFG, with same hyper parameters suggested in Kim et al. (2019); Zhao and Titov (2020). Our MMC-PCFG is composed of a parsing model and a video-text matching model. The parsing model has the same parameters as VC-PCFG (please refer to their paper for details). For video-text matching model, all extracted expert features are projected to \(512\)-dimensional vectors. The transformer has \(2\) layers, a dropout probability of \(10\%\), a hidden size of \(512\) and an intermediate size of \(2048\). We select the top-\(2000\) most common words as vocabulary for all datasets. All the baseline methods and our models are optimized using Adam Kingma and Ba (2015) with the learning rate set to \(0.001\), \(\beta_{1}=0.75\) and \(\beta_{2}=0.999\). All parameters are initialized with Xavier uniform initializer Glorot and Bengio (2010). The batch size is set to \(16\).

Due to the long video durations, it is infeasible to feed all features into the multi-modal transformer. Therefore, each feature from object, motion and scene categories is partitioned into \(8\) chunks and then average-pooled within each chunk. For features from other categories, global average pooling is applied. In this way, the coarse-grained temporal information is preserved. Noted that some videos do not have audio and some videos do not have detected faces or text characters. For these missing features, we pad them with zeros. All the aforementioned expert features are obtained from Albanie et al. (2020).

### Main Results

We evaluate the proposed MMC-PCFG approach on three datasets, and compare it with recently proposed state-of-the-art methods, C-PCFG Kim et al. (2019) and VC-PCFG Zhao and Titov (2020). The results are summarized in Table 1. The values high-lighted by **bold** and _italic_ fonts indicate the top-\(2\) methods, respectively. All results are reported in percentage (\(\%\)). LBranch, RBranch and Random represent left branching trees, right branching trees and random trees, respectively. Since VC-PCFG is originally designed for images, it is not directly comparable with our method. In order to allow VC-PCFG to accept videos as input, we average video features in the temporal dimension first and then feed them into the model. We evaluate VC-PCFG with \(10\), \(7\), and \(10\) expert features for DiDeMo, YouCook2 and MSRVTT, respectively. In addition, we also include the concatenated averaged features (Concat). Since object and action categories involve more than one expert, we directly use experts' names instead of their categories in Table 1.

**Overall performance comparison.** We first compare the overall performance, _i.e._, C-F1 and S-F1, among all models, as shown in Table 1. The right branching model serves as a strong baseline, since English is a largely right-branching language. C-PCFG learns parsing purely based on text. Compared to C-PCFG, the better overall performance of VC-PCFG demonstrates the effectiveness of leveraging video information. Compared within VC-PCFG, concatenating all features together may not even outperform a model trained on a single expert (R2P1D _v.s._ Concat in DiDeMo and MSRVTT). The reason is that each expert is learned independently, where their correlations are not considered. In contrast, our MMC-PCFG outperforms all baselines on C-F1 and S-F1 in all datasets. The superior performance indicates that our model can leverage the benefits from all the experts2. Moreover, the superior performance over Concat demonstrates the importance of modeling relations among different experts and different timestamps.

Footnote 2: The larger improvement on DiDeMo may be caused by the diversity of the video content. Videos in DiDeMo are more diverse in scenes, actions and objects, which provide a great opportunity for leveraging video information.

**Performance comparison among different phrase types.** We compare the models' recalls on top-\(3\) frequent phrase types (NP, VP and PP). These three types cover \(77.4\%\), \(80.1\%\) and \(82.4\%\) spans of gold trees on DiDeMo, YouCook2 and MSRVTT, respectively. In the following, we compare their performance on DiDeMo, as shown in Table 1. Comparing 

with a single expert, we find that object features (ResNeXt and SENet) achieve top-\(2\) recalls on NPs, while action features (I3D, R2P1D and S3DG) achieve the top-\(3\) recalls on VPs and PPs. It indicates that different experts help parser learn syntactic structures from different aspects. Meanwhile, action features improve C-PCFG3 on VPs and PPs by a large margin, which once again verifies the benefits of using video information.

Footnote 3: The low performance of C-PCFG on DiDeMo in terms of VP recall may be caused by it attaching a high attaching PP to the rest of the sentence instead of the rest of the verb phrase, which breaks the whole VP. For PPs, C-PCFG attaches prepositions to the word in front, which may be caused by confusion between prepositions in PPs and phrasal verbs.

Comparing our MMC-PCFG with VC-PCFG, our model achieves the top-\(2\) recall and is smaller in variance in NP, VP and PP. It demonstrates that our model can take the advantages of different experts and learn consistent grammar induction.

### Ablation Study

In this section, we conduct several ablation studies on DiDeMo, shown in Figures 2-4. All results are reported in percentage (\(\%\)).

**Performance comparison over constituent length.** We first demonstrate the model performance for constituents at different lengths in Figure 2. As constituent length becomes longer, the recall of all models (except RBranch) decreases as expected Kim et al. (2019); Zhao and Titov (2018)

Figure 3: Label distributions over the constituent length on DiDeMo. _All_ represent frequencies of constituent lengths.

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{5}{c}{DiDeMo} & \multicolumn{5}{c}{YouCook2} & MSRVTT \\ \cline{2-10}  & NP & VP & pp & C-F1 & S-F1 & C-F1 & S-F1 & C-F1 & S-F1 \\ \hline LBranch & \(41.7\) & \(0.1\) & \(0.1\) & \(16.2\) & \(18.5\) & \(6.8\) & \(5.9\) & \(14.4\) & \(16.8\) \\ RBranch & \(32.8\) & \(\mathbf{91.5}\) & \(\mathbf{66.5}\) & _53.6_ & _57.5_ & \(35.0\) & \(41.6\) & \(54.2\) & \(58.6\) \\ Random & \(36.5_{+0.6}\) & \(30.5_{+0.5}\) & \(30.1_{+0.5}\) & \(29.4_{+0.3}\) & \(32.7_{+0.5}\) & \(21.2_{+0.2}\) & \(24.0_{+0.2}\) & \(27.2_{+0.1}\) & \(30.5_{+0.1}\) \\ C-PCFG & \(\mathbf{72.9}_{+5.5}\) & \(16.5_{+6.2}\) & \(23.4_{+16.9}\) & \(38.2_{+5.0}\) & \(40.4_{+1.1}\) & \(37.8_{+6.7}\) & \(41.4_{+6.6}\) & \(50.7_{+3.2}\) & \(55.0_{+3.2}\) \\ \hline ResNeXt & \(\mathbf{64.4}_{+1.2}\) & \(25.7_{+11.7}\) & \(34.6_{+2.9}\) & \(40.0_{+13.7}\) & \(41.8_{+14.0}\) & \(38.2_{+3.4}\) & \(24.8_{+8.4}\) & \(50.7_{+1.7}\) & \(54.9_{+2.2}\) \\ SENet & \(70.5_{+15.3}\) & \(25.7_{+11.9}\) & \(36.5^{+2.4}_{+2.6}\) & \(42.6_{+10.4}\) & \(44.0_{+10.4}\) & \(39.9_{+8.7}\) & \(44.9_{+8.3}\) & \(52.2_{+1.6}\) & \(56.0_{+1.6}\) \\ I3D & \(57.9^{+1.5}_{+13.5}\) & \(45.7_{+14.1}\) & \(45.8_{+17.2}\) & \(45.1_{+6.0}\) & \(49.2_{+6.0}\) & \(40.6_{+3.6}\) & \(45.7_{+3.2}\) & \(54.5_{+1.6}\) & \(59.1_{+1.7}\) \\ R2P1D & \(61.2_{+8.5}\) & \(38.1_{+5.4}\) & \(62.1_{+4.1}\) & \(48.1_{+4.4}\) & \(50.7_{+4.2}\) & \(39.4_{+8.1}\) & \(44.4_{+8.3}\) & \(54.0_{+2.5}\) & \(58.0_{+2.3}\) \\ S3DG & \(61.3_{+13.4}\) & \(31.7_{+16.7}\) & \(51.8_{+8.0}\) & \(44.0_{+2.7}\) & \(46.5_{+1.9}\) & \(39.6_{+5.4}\) & \(41.6_{+6.0}\) & \(50.7_{+3.2}\) & \(54.7_{+2.9}\) \\ Scene & \(62.2_{+9.6}\) & \(30.6_{+12.3}\) & \(41.1_{+2.8}\) & \(41.7_{+6.5}\) & \(44.9_{+7.4}\) & – & – & \(54.6_{+1.5}\) & \(58.4_{+1.3}\) \\ Audio & \(64.2_{+18.6}\) & \(21.3_{+26.5}\) & \(34.7_{+11.0}\) & \(38.7_{+3.7}\) & \(39.5_{+5.2}\) & \(39.2_{+4.7}\) & \(43.3_{+4.9}\) & \(52.8_{+1.3}\) & \(56.7_{+1.4}\) \\ OCR & \(64.4_{+15.0}\) & \(27.4_{+19.5}\) & \(42.8_{+31.2}\) & \(41.9_{+16.9}\) & \(44.6_{+17.5}\) & \(38.6_{+5.5}\) & \(43.2_{+5.6}\) & \(51.0_{+3.0}\) & \(55.5_{+3.0}\) \\ Face & \(60.8_{+16.0}\) & \(31.5_{+17.0}\) & \(52.8_{+9.8}\) & \(43.9_{+5.5}\) & \(46.3_{+5.5}\) & — & — & \(50.5_{+2.6}\) & \(54.5_{+2.6}\) \\ Speech & \(61.8_{+12.8}\) & \(26.6_{+17.6}\) & \(43.8_{+3.4}\) & \(40.9_{+16.0}\) & \(43.1_{+16.1}\) & — & — & \(51.7_{+2.6}\) & \(56.2_{+2.5}\) \\ Concat & \(68.6_{+8.6}\) & \(24.9_{+19.9}\) & \(37.19_{+5.9}\) & \(42.2_{+12.3}\) & \(43.2_{+14.2}\) & \(42.3_{+5.7}\) & \(47.0_{+5.6}\) & \(49.8_{+4.1}\) & \(54.2_{+4.0}\) \\ \hline
**MMC-PCFG** & \(67.9_{+9.8}\) & _52.3_{+9.0}\) & _63.5_±8.6 & \(\mathbf{55.0}_{+3.7}\) & \(\mathbf{58.9}_{+3.4}\) & \(\mathbf{44.7}_{+5.2}\) & \(\mathbf{48.9}_{+5.7}\) & \(\mathbf{56.0}_{+1.4}\) & \(\mathbf{60.0}_{+1.2}\) \\ \hline \end{tabular}
\end{table}
Table 1: Performance comparison on three benchmark datasets.

Figure 2: Recall comparison over constituent length on DiDeMo. Methods are differentiated with colors. For easy comparison, we additional draw gray lines in each figure to indicate the average recall shown by other figures.

 

2020). MMC-PCFG outperforms C-PCFG and VC-PCFG under all constituent lengths. We further illustrate the label distribution over constituent length in Figure 3. We find that approximately \(98.1\%\) of the constituents have fewer than \(9\) words and most of them are NPs, VPs and PPs. This suggests that the improvement on NPs, VPs and PPs can strongly affect the overall performance.

**Consistency between different models.** Next, we analyze the consistency of these different models. The consistency between two models is measured by averaging sentence-level F1 scores over all possible pairings of different runs4[Williams et al., 2018]. We plot the consistency for each pair of models in Figure 4 and call it consistency matrix. Comparing the self F1 of all the models (the diagonal in the matrix), R2P1D has the highest score, suggesting that R2P1D is the most reliable feature that can help parser to converge to a specific grammar. Comparing the models trained with different single experts, ResNeXt _v.s._ SENet reaches the highest non-self F1, since they are both object features trained on ImageNet and have similar effects to the parser. We also find that the lowest non-self F1 comes from Audio _v.s._ I3D, since they are extracted from different modalities (video _v.s._ sound). Compared with other models, our model is most consistent with R2P1D, indicating that R2P1D contributes most to our final prediction.

Footnote 4: Different runs represent models trained with different seeds.

**Contribution of different modalities.** We also evaluate how different modalities contribute to the performance of MMC-PCFG. We divide current experts into three groups, video (objects, action, scene and face), audio (audio) and text (OCR and ASR). By ablating one group during training, we find that the model without video experts has the largest performance drops (see Table 2). Therefore, videos contribute most to the performance among all modalities.

### Qualitative Analysis

In Figure 5, we visualize a parse tree predicted by the best run of SENet154, I3D and MMC-PCFG. We can observe that SENet identifies all NPs but fails at the VP. I3D correctly predicts the VP but fails at recognizing a NP, "the man". Our MMC-PCFG can take advantages of all experts and produce the correct prediction.

## 5 Related Work

**Grammar Induction** Grammar induction and unsupervised parsing has been a long-standing problem in computational linguistics Carroll and Charniak (1992). Recent work utilized neural networks in predicting constituency structures with no supervision Shen et al. (2018a); Drozdov et al. (2019); Shen et al. (2018b); Kim et al. (2019); Jin et al. (2019) and showed promising results. In addition to learning purely from text, there is a growing interest to use image information to improve accuracy of induced constituency trees Shi et al. (2019); Kojima et al. (2020); Zhao and Titov (2020); Jin and Schuler (2020). Different from previous work, our work improves the constituency parser by using videos containing richer information than images.

**Video-Text Matching** Video-text matching has been widely studied in various tasks, such as video retrieval Liu et al. (2019); Gabeur et al. (2020), moment localization with natural language Zhang

Figure 4: Consistency scores for different models on DiDeMo.

Figure 5: Parse trees predicted by different models for the sentence _The man falls to the floor_.

 

et al., 2019, 2020) and video question and answering Xu et al. (2017); Jin et al. (2019). It aims to learn video-semantic representation in a joint embedding space. Recent works Liu et al. (2019); Gabeur et al. (2020); Chen et al. (2020) focus on learning video's multi-modal representation to match with text. In this work, we borrow this idea to match video and textual representations.

## 6 Conclusion

In this work, we have presented a new task referred to as video-aided unsupervised grammar induction. This task aims to improve grammar induction models by using aligned video-sentence pairs as an effective way to address the limitation of current image-based methods where only object information from static images is considered and important verb related information from vision is missing. Moreover, we present Multi-Modal Compound Probabilistic Context-Free Grammars (MMC-PCFG) to effectively integrate video features extracted from different modalities to induce more accurate grammars. Experiments on three datasets demonstrate the effectiveness of our method.

## 7 Acknowledgement

We thank the support of NSF awards IIS-1704337, IIS-1722847, IIS-1813709, and the generous gift from our corporate sponsors.

## References

* Albanie et al. (2020) Samuel Albanie, Yang Liu, Arsha Nagrani, Antoine Miech, Ernesto Coto, Ivan Laptev, Rahul Sukthankar, Bernard Ghanem, Andrew Zisserman, Valentin Gabeur, et al. 2020. The end-of-end-to-end: A video understanding pentathlon challenge (2020). _arXiv preprint arXiv:2008.00744_.
* Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In _CVPR_.
* Carroll and Charniak (1992) Glenn Carroll and Eugene Charniak. 1992. _Two experiments on learning probabilistic dependency grammars from corpora_. Department of Computer Science, Univ.
* Chen et al. (2020) Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. 2020. Learning modality interaction for temporal sentence localization and event captioning in videos. In _ECCV_.
* Cocke (1969) John Cocke. 1969. _Programming languages and their compilers: Preliminary notes_. New York University.
* Deng et al. (2018) Dan Deng, Haifeng Liu, Xuelong Li, and Deng Cai. 2018. Pixellink: Detecting scene text via instance segmentation. In _AAAI_.
* Drozdov et al. (2019) Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In _NAACL_.
* Fried et al. (2019) Daniel Fried, Nikita Kitaev, and Dan Klein. 2019. Cross-domain generalization of neural constituency parsers. In _ACL_.
* Gabeur et al. (2020) Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In _ECCV_.
* Ghadiyaram et al. (2019) Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. 2019. Large-scale weakly-supervised pre-training for video action recognition. In _CVPR_.
* Gleitman (1990) Lila Gleitman. 1990. The Structural Sources of Verb Meanings. _Language Acquisition_, 1(1):3-55.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _AISTATS_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In _ECCV_.
* Hendricks et al. (2017) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In _ICCV_.
* Hershey et al. (2017) Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In _ICASSP_.
* Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In _CVPR_.
* Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In _CVPR_.
* Jin et al. (2018) Lifeng Jin, Finale Doshi-Velez, Timothy Miller, William Schuler, and Lane Schwartz. 2018. Unsupervised grammar induction with depth-bounded pcfg. _TACL_.
* Jin et al. (2019) Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019a. Unsupervised learning of PCFGs with normalizing flow. In _ACL_.
* Jin and Schuler (2020) Lifeng Jin and William Schuler. 2020. Grounded pcfg induction with images. In _AACL-IJCNLP_.
* Jin et al. (2019) Weike Jin, Zhou Zhao, Mao Gu, Jun Yu, Jun Xiao, and Yueting Zhuang. 2019b. Multi-interaction network with object relation for video question answering. In _ACM Multimedia_ 

Tadao Kasami. 1966. An efficient recognition and syntax-analysis algorithm for context-free languages. Coordinated Science Laboratory Report no. R-257. Cited by: SS4.
* Y. Kim, C. Dyer, and A. M. Rush (2019)Compound probabilistic context-free grammars for grammar induction. In ACL, Cited by: SS3.2.
* Diederik P Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In ICLR, Cited by: SS4.
* Diederik P Kingma and Max Welling (2014)Auto-encoding variational bayes. In ICLR, Cited by: SS4.
* N. Kitaev and D. Klein (2018)Constitency parsing with a self-attentive encoder. In ACL, Cited by: SS4.
* N. Kojima, H. Averbuch-Elor, A. M. Rush, and Y. Artzi (2020)What is learned in visually grounded neural syntax acquisition. In ACL, Cited by: SS4.
* A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012)Imagenet classification with deep convolutional neural networks. In NIPS, Cited by: SS4.
* J. Lei, L. Wang, Y. Shen, D. Yu, T. L. Berg, and M. Bansal (2020)MART: memory-augmented recurrent transformer for coherent video paragraph captioning. In ACL, Cited by: SS4.
* W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg (2016)SSD: single shot multibox detector. In ECCV, Cited by: SS4.
* Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman (2019)Use what you have: video retrieval using representations from collaborative experts. In arXiv preprint arxiv:1907.13487, Cited by: SS4.
* Y. Liu, Z. Wang, H. Jin, and I. Wassell (2018)Synthetically supervised feature learning for scene text recognition. In ECCV, Cited by: SS4.
* D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten (2018)Exploring the limits of weakly supervised pretraining. In ECCV, Cited by: SS4.
* M. P. Marcus, B. Santorini, and M. Ann Marcinkiewicz (1993)Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics19 (2), pp. 313-330. External Links: ISSN 0021-909X, Document, Link Cited by: SS4.
* A. Miech, J. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman (2020)End-to-end learning of visual representations from uncurated instructional videos. In CVPR, Cited by: SS4.
* A. Miech, I. Laptev, and J. Sivic (2018)Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.02516, Cited by: SS4.
* A. Miech, D. Zhukov, J. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic (2019)HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, Cited by: SS4.
* T. Mikolov, K. Chen, G. Corrado, and J. Dean (2013)Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, pp. . Cited by: SS4.
* S. Pinker and B. MacWhinney (1987)The bootstrapping problem in language acquisition. Mechanisms of language acquisition, pp. 399-441. Cited by: SS4.
* Y. Shen, Z. Lin, C. w. Huang, and A. Courville (2018a)Neural language modeling by jointly learning syntax and lexicon. In ICLR, Cited by: SS4.
* Y. Shen, S. Tan, A. Sordoni, and A. Courville (2018b)Ordered neurons: integrating tree structures into recurrent neural networks. In ICLR, Cited by: SS4.
* H. Shi, J. Mao, K. Gimpel, and K. Livescu (2019)Visually grounded neural syntax acquisition. In ACL, Cited by: SS4.
* M. Tomasello (2003)Constructing a language: a usage-based theory of language acquisition. Harvard University Press, Cambridge, MA, US. Cited by: SS4.
* D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri (2018)A closer look at spatiotemporal convolutions for action recognition. In CVPR, Cited by: SS4.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In NIPS, Cited by: SS4.
* A. Williams, A. Drozdov*, and S. R. Bowman (2018)Do latent tree learning models identify meaningful structure in sentences?. TACL. Cited by: SS4.
* S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He (2017)Aggregated residual transformations for deep neural networks. In CVPR, Cited by: SS4.
* D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang (2017)Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, Cited by: SS4.
* J. Xu, T. Mei, T. Yao, and Y. Rui (2016)Msr- vtt: a large video description dataset for bridging video and language. In CVPR, Cited by: SS4.
* D. H. Younger (1967)Recognition and parsing of context-free languages in time n3. Information and control10 (2), pp. 189-208. Cited by: SS4.
* S. Zhang, H. Peng, J. Fu, and J. Luo (2020)Learning 2d temporal adjacent networks formment localization with natural language.. In AAAI, 

Songyang Zhang, Jinsong Su, and Jiobe Luo. 2019. Exploiting temporal relationships in video moment localization with natural language. In _ACM Multimediaa_.
* Zhao and Titov (2020) Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In _EMNLP_.
* Zhou et al. (2017) Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. _TPAMI_.
* Zhou et al. (2018) Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018. Towards automatic learning of procedures from web instructional videos. In _AAAI_.

 



## Appendix A Performance Comparison - Full Tables

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline Method & NP & VP & PP & SBAR & ADIP & ADVP & C-F1 & S-F1 \\ \hline LBranch & \(1.7\) & \(42.8\) & \(0.4\) & \(8.1\) & \(1.5\) & \(0.0\) & \(6.8\) & \(5.9\) \\ RBranch & \(35.6\) & _47.5_ & \(67.0\) & **88.9** & **33.9** & **65.0** & 35.0 & 41.6 \\ Random & \(27.2_{\pm 0.3}\) & \(27.1_{\pm 1.4}\) & \(29.9_{\pm 0.5}\) & \(31.3_{\pm 5.2}\) & \(26.9_{\pm 7.7}\) & \(26.2_{\pm 11.9}\) & \(21.2_{\pm 0.2}\) & \(24.0_{\pm 0.2}\) \\ C-PCFG & \(47.4_{\pm 18.4}\) & **49.4**\(\pm 11.9\) & \(58.0_{\pm 22.6}\) & \(45.7_{\pm 6.0}\) & \(27.7_{\pm 15.1}\) & \(36.2_{\pm 7.4}\) & \(37.8_{\pm 6.7}\) & \(41.4_{\pm 6.6}\) \\ \cline{2-10}  & ResNeXt & \(46.5_{\pm 13.7}\) & \(40.8_{\pm 9.8}\) & \(67.9_{\pm 12.7}\) & \(50.5_{\pm 13.3}\) & \(22.3_{\pm 6.7}\) & \(38.8_{\pm 21.3}\) & \(38.2_{\pm 8.3}\) & \(42.8_{\pm 8.4}\) \\  & SENet & \(48.3_{\pm 14.4}\) & \(40.7_{\pm 9.2}\) & \(73.6_{\pm 11.2}\) & \(45.5_{\pm 17.0}\) & \(26.9_{\pm 13.6}\) & \(41.2_{\pm 17.5}\) & \(39.9_{\pm 8.7}\) & \(44.9_{\pm 8.3}\) \\  & I3D & \(48.1_{\pm 10.7}\) & \(39.0_{\pm 8.0}\) & _79.4_{\pm 8.4}\) & \(50.0_{\pm 14.9}\) & \(18.5_{\pm 7.0}\) & \(41.2_{\pm 4.1}\) & \(40.6_{\pm 3.6}\) & \(45.7_{\pm 3.2}\) \\  & R2PID & \(52.4_{\pm 10.9}\) & \(33.7_{\pm 16.4}\) & \(66.7_{\pm 10.7}\) & \(49.5_{\pm 13.8}\) & \(25.8_{\pm 10.6}\) & \(33.8_{\pm 12.4}\) & \(39.4_{\pm 8.1}\) & \(44.4_{\pm 8.3}\) \\  & S3DG & \(50.4_{\pm 13.1}\) & \(32.6_{\pm 16.3}\) & \(71.7_{\pm 7.5}\) & \(33.3_{\pm 5.9}\) & \(30.8_{\pm 17.5}\) & \(40.0_{\pm 7.1}\) & \(39.3_{\pm 6.5}\) & \(44.1_{\pm 6.6}\) \\  & Audio & \(51.2_{\pm 3.1}\) & \(42.0_{\pm 7.2}\) & \(61.5_{\pm 18.0}\) & \(51.0_{\pm 14.8}\) & \(23.5_{\pm 16.8}\) & \(48.8_{\pm 8.8}\) & \(39.2_{\pm 4.7}\) & \(43.3_{\pm 4.9}\) \\ OCR & \(48.6_{\pm 8.1}\) & \(41.5_{\pm 4.1}\) & \(65.5_{\pm 17.4}\) & \(39.9_{\pm 4.4}\) & \(18.5_{\pm 6.6}\) & \(53.8_{\pm 14.7}\) & \(38.6_{\pm 5.5}\) & \(43.2_{\pm 5.6}\) \\  & Concat & \(50.3_{\pm 10.3}\) & \(42.3_{\pm 2.9}\) & \(\textbf{81.6}_{\pm 8.7}\) & \(40.1_{\pm 3.9}\) & \(17.7_{\pm 8.2}\) & \(52.5_{\pm 5.6}\) & \(\textit{42.3}_{\pm 5.7}\) & \(\textit{47.0}_{\pm 5.6}\) \\ \hline
**MMC-PCFG** & \(\textbf{62.7}_{\pm 9.8}\) & \(45.3_{\pm 2.8}\) & \(63.4_{\pm 17.7}\) & \(43.9_{\pm 4.8}\) & \(26.2_{\pm 7.5}\) & \(35.0_{\pm 3.5}\) & \(\textbf{44.7}_{\pm 5.2}\) & \(\textbf{48.9}_{\pm 5.7}\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance Comparison on YouCook2.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline Method & NP & VP & PP & SBAR & ADIP & ADVP & C-F1 & S-F1 \\ \hline LBranch & \(41.7\) & \(0.1\) & \(0.1\) & \(0.7\) & \(7.2\) & \(0.0\) & \(16.2\) & \(18.5\) \\ RBranch & \(32.8\) & \(\textbf{91.5}\) & \(\textbf{66.5}\) & \(\textbf{88.2}\) & \(\textbf{36.9}\) & \(\textbf{63.6}\) & _53.6_ & _57.5_ \\ Random & \(36.5_{\pm 0.6}\) & \(30.5_{\pm 0.5}\) & \(30.1_{\pm 0.5}\) & \(25.7_{\pm 2.8}\) & \(29.5_{\pm 2.3}\) & \(28.5_{\pm 4.8}\) & \(29.4_{\pm 0.3}\) & \(32.7_{\pm 0.5}\) \\ C- PCFG & \(\textbf{72.9}_{\pm 5.5}\) & \(16.5_{\pm 6.2}\) & \(23.4_{\pm 16.9}\) & \(26.6_{\pm 15.9}\) & \(25.0_{\pm 11.6}\) & \(14.7_{\pm 12.8}\) & \(38.2_{\pm 5.0}\) & \(40.4_{\pm 4.1}\) \\ \hline \multirow{8}{*}{C-PCFG} & ResNetAt & \(64.4_{\pm 1.4}\) & \(25.7_{\pm 17.7}\) & \(34.6^{\pm 25.0}_{\pm 25.0}\) & \(40.5_{\pm 26.3}\) & \(16.7_{\pm 9.5}\) & \(38.4_{\pm 21.3}\) & \(40.0_{\pm 13.7}\) & \(41.8_{\pm 14.0}\) \\  & SENet & \(70.5_{\pm 15.3}\) & \(25.7_{\pm 15.9}\) & \(36.5_{\pm 24.6}\) & \(36.8_{\pm 25.9}\) & \(21.2_{\pm 12.5}\) & \(23.6_{\pm 16.8}\) & \(42.6_{\pm 10.4}\) & \(44.0_{\pm 10.4}\) \\  & I3D & \(57.9_{\pm 13.5}\) & \(45.7_{\pm 14.1}\) & \(45.8_{\pm 17.2}\) & \(38.2_{\pm 14.8}\) & \(28.4_{\pm 9.2}\) & \(22.0_{\pm 9.3}\) & \(45.1_{\pm 6.0}\) & \(49.2_{\pm 6.0}\) \\  & R2PID & \(61.2_{\pm 8.5}\) & \(38.1_{\pm 5.4}\) & \(62.1_{\pm 4.1}\) & \(61.5_{\pm 5.1}\) & \(21.4_{\pm 11.4}\) & \(40.8_{\pm 7.3}\) & \(48.1_{\pm 4.4}\) & \(50.7_{\pm 4.2}\) \\  & S3DG & \(6