SlowGuess/NAACL_2021_OCR huggingface dataset:
Speech features are extracted by two steps: transcripts are first obtained via Google Cloud Speech to Text API. The transcripts are then converted to word embeddings through word2vec (Mikolov et al., 2013) as the final speech features, where the dimension is 300.
We keep sentences with fewer than 20 words in the training set due to the computational limitation. After filtering, the training sets cover  $99.4\%$ ,  $98.5\%$  and  $97.1\%$  samples of their original splits in DiDeMo, YouCook2 and MSRVTT.
We train baseline models, C-PCFG and VC-PCFG, with same hyper parameters suggested in Kim et al. (2019); Zhao and Titov (2020). Our MMC-PCFG is composed of a parsing model and a video-text matching model. The parsing model has the same parameters as VC-PCFG (please refer to their paper for details). For video-text matching model, all extracted expert features are projected to 512-dimensional vectors. The transformer has 2 layers, a dropout probability of  $10\%$ , a hidden size of 512 and an intermediate size of 2048. We select the top-2000 most common words as vocabulary for all datasets. All the baseline methods and our models are optimized using Adam (Kingma and Ba, 2015) with the learning rate set to 0.001,  $\beta_{1} = 0.75$  and  $\beta_{2} = 0.999$ . All parameters are initialized with Xavier uniform initializer (Glorot and Bengio, 2010). The batch size is set to 16.
Due to the long video durations, it is infeasible to feed all features into the multi-modal transformer. Therefore, each feature from object, motion and scene categories is partitioned into 8 chunks and then average-pooled within each chunk. For features from other categories, global average pooling is applied. In this way, the coarse-grained temporal information is preserved. Noted that some videos do not have audio and some videos do not have detected faces or text characters. For these missing features, we pad them with zeros. All the aforementioned expert features are obtained from Albanie et al. (2020).
We evaluate the proposed MMC-PCFG approach on three datasets, and compare it with recently proposed state-of-the-art methods, C-PCFG (Kim
et al., 2019) and VC-PCFG (Zhao and Titov, 2020). The results are summarized in Table 1. The values high-lighted by bold and italic fonts indicate the top-2 methods, respectively. All results are reported in percentage (\%). LBranch, RBranch and Random represent left branching trees, right branching trees and random trees, respectively. Since VC-PCFG is originally designed for images, it is not directly comparable with our method. In order to allow VC-PCFG to accept videos as input, we average video features in the temporal dimension first and then feed them into the model. We evaluate VC-PCFG with 10, 7, and 10 expert features for DiDeMo, YouCook2 and MSRVTT, respectively. In addition, we also include the concatenated averaged features (Concat). Since object and action categories involve more than one expert, we directly use experts' names instead of their categories in Table 1.
Overall performance comparison. We first compare the overall performance, i.e., C-F1 and S-F1, among all models, as shown in Table 1. The right branching model serves as a strong baseline, since English is a largely right-branching language. C-PCFG learns parsing purely based on text. Compared to C-PCFG, the better overall performance of VC-PCFG demonstrates the effectiveness of leveraging video information. Compared within VC-PCFG, concatenating all features together may not even outperform a model trained on a single expert (R2P1D v.s. Concat in DiDeMo and MSRVTT). The reason is that each expert is learned independently, where their correlations are not considered. In contrast, our MMC-PCFG outperforms all baselines on C-F1 and S-F1 in all datasets. The superior performance indicates that our model can leverage the benefits from all the experts<sup>2</sup>. Moreover, the superior performance over Concat demonstrates the importance of modeling relations among different experts and different timestamps.
Performance comparison among different phrase types. We compare the models' recalls on top-3 frequent phrase types (NP, VP and PP). These three types cover  $77.4\%$ ,  $80.1\%$  and  $82.4\%$  spans of gold trees on DiDeMo, YouCook2 and MSRVTT, respectively. In the following, we compare their performance on DiDeMo, as shown in Table 1. Comparing VC-PCFG trained
2The larger improvement on DiDeMo may be caused by the diversity of the video content. Videos in DiDeMo are more diverse in scenes, actions and objects, which provide a great opportunity for leveraging video information.
Method	DiDeMo	YouCook2	MSRVTT
NP	VP	PP	C-F1	S-F1	C-F1	S-F1	C-F1	S-F1
LBranch	41.7	0.1	0.1	16.2	18.5	6.8	5.9	14.4	16.8
RBranch	32.8	91.5	66.5	53.6	57.5	35.0	41.6	54.2	58.6
Random	36.5±0.6	30.5±0.5	30.1±0.5	29.4±0.3	32.7±0.5	21.2±0.2	24.0±0.2	27.2±0.1	30.5±0.1
C-PCFG	72.9±5.5	16.5±6.2	23.4±16.9	38.2±5.0	40.4±4.1	37.8±6.7	41.4±6.6	50.7±3.2	55.0±3.2
VC-PCFG	ResNeXt	64.4±21.4	25.7±17.7	34.6±25.0	40.0±13.7	41.8±14.0	38.2±8.3	42.8±8.4	50.7±1.7	54.9±2.2
SENet	70.5±15.3	25.7±15.9	36.5±24.6	42.6±10.4	44.0±10.4	39.9±8.7	44.9±8.3	52.2±1.2	56.0±1.6
I3D	57.9±13.5	45.7±14.1	45.8±17.2	45.1±6.0	49.2±6.0	40.6±3.6	45.7±3.2	54.5±1.6	59.1±1.7
R2P1D	61.2±8.5	38.1±5.4	62.1±4.1	48.1±4.4	50.7±4.2	39.4±8.1	44.4±8.3	54.0±2.5	58.0±2.3
S3DG	61.3±13.4	31.7±16.7	51.8±8.0	44.0±2.7	46.5±5.1	39.3±6.5	44.1±6.6	50.7±3.2	54.7±2.9
Scene	62.2±9.6	30.6±12.3	41.1±24.8	41.7±6.5	44.9±7.4	-	-	54.6±1.5	58.4±1.3
Audio	64.2±18.6	21.3±26.5	34.7±11.0	38.7±3.7	39.5±5.2	39.2±4.7	43.3±4.9	52.8±1.3	56.7±1.4
OCR	64.4±15.0	27.4±19.5	42.8±31.2	41.9±16.9	44.6±17.5	38.6±5.5	43.2±5.6	51.0±3.0	55.5±3.0
Face	60.8±16.0	31.5±17.0	52.8±9.8	43.9±4.5	46.3±5.5	-	-	50.5±2.6	54.5±2.6
Speech	61.8±12.8	26.6±17.6	43.8±34.5	40.9±16.0	43.1±16.1	-	-	51.7±2.6	56.2±2.5
Concat	68.6±8.6	24.9±19.9	39.7±19.5	42.2±12.3	43.2±14.2	42.3±5.7	47.0±5.6	49.8±4.1	54.2±4.0
MMC-PCFG	67.9±9.8	52.3±9.0	63.5±8.6	55.0±3.7	58.9±3.4	44.7±5.2	48.9±5.7	56.0±1.4	60.0±1.2
Figure 2: Recall comparison over constituent length on DiDeMo. Methods are differentiated with colors. For easy comparison, we additional draw gray lines in each figure to indicate the average recall shown by other figures.
Figure 3: Label distributions over the constituent length on DiDeMo. All represent frequencies of constituent lengths.
with a single expert, we find that object features (ResNeXt and SENet) achieve top-2 recalls on NPs, while action features (I3D, R2P1D and S3DG) achieve the top-3 recalls on VPs and PPs. It indicates that different experts help parser learn syntactic structures from different aspects. Meanwhile, action features improve C-PCFG<sup>3</sup> on VPs and PPs by a large margin, which once again verifies the benefits of using video information.
Comparing our MMC-PCFG with VC-PCFG, our model achieves the top-2 recall and is smaller in variance in NP, VP and PP. It demonstrates that our model can take the advantages of different experts and learn consistent grammar induction.
Table 1: Performance comparison on three benchmark datasets.
NP-	26.5	9.1	3.6	2.1	1.9	0.8	0.4	0.2	0.1	0.1	0.1	0.0	45.0
VP-	5.9	5.8	6.4	4.4	3.2	2.1	1.6	1.0	0.8	0.3	0.2	0.2	32.0
PP-	6.9	9.3	3.8	1.2	0.8	0.5	0.1	0.1	0.0	0.0	0.0	0.0	22.8
All-	39.3	24.2	13.9	7.8	5.8	3.4	2.2	1.4	0.9	0.4	0.3	0.2	100.0
2	3	4	5	6	7	8	9	10	11	12	13	All
Constituent Length
In this section, we conduct several ablation studies on DiDeMo, shown in Figures 2-4. All results are reported in percentage  $(\%)$ .
Performance comparison over constituent length. We first demonstrate the model performance for constituents at different lengths in Figure 2. As constituent length becomes longer, the recall of all models (except RBranch) decreases as expected (Kim et al., 2019; Zhao and Titov,
3The low performance of C-PCFG on DiDeMo in terms of VP recall may be caused by it attaching a high attaching PP to the rest of the sentence instead of the rest of the verb phrase, which breaks the whole VP. For PPs, C-PCFG attaches prepositions to the word in front, which may be caused by confusion between prepositions in PPs and phrasal verbs.
Figure 4: Consistency scores for different models on DiDeMo.
2020). MMC-PCFG outperforms C-PCFG and VC-PCFG under all constituent lengths. We further illustrate the label distribution over constituent length in Figure 3. We find that approximately  $98.1\%$  of the constituents have fewer than 9 words and most of them are NPs, VPs and PPs. This suggests that the improvement on NPs, VPs and PPs can strongly affect the overall performance.
Consistency between different models. Next, we analyze the consistency of these different models. The consistency between two models is measured by averaging sentence-level F1 scores over all possible pairings of different runs $^4$  (Williams et al., 2018). We plot the consistency for each pair of models in Figure 4 and call it consistency matrix. Comparing the self F1 of all the models (the diagonal in the matrix), R2P1D has the highest score, suggesting that R2P1D is the most reliable feature that can help parser to converge to a specific grammar. Comparing the models trained with different single experts, ResNeXt v.s. SENet reaches the highest non-self F1, since they are both object features trained on ImageNet and have similar effects to the parser. We also find that the lowest non-self F1 comes from Audio v.s. I3D, since they are extracted from different modalities (video v.s. sound). Compared with other models, our model is most consistent with R2P1D, indicating that R2P1D contributes most to our final prediction.
Contribution of different modalities. We also evaluate how different modalities contribute to the performance of MMC-PCFG. We divide current experts into three groups, video (objects, action, scene and face), audio (audio) and text (OCR and ASR). By ablating one group during training, we find that the model without video experts has the
Model	NP	VP	PP	C-F1	S-F1
full	68.0±9.9	52.7±9.0	63.8±8.7	55.3±3.3	59.0±3.4
w/o audio	69.3±8.0	41.8±11.0	45.3±20.2	48.7±6.2	52.0±6.5
w/o text	68.5±13.7	38.8±16.9	57.0±20.4	49.6±10.4	52.0±11.1
w/o video	64.3±4.4	28.1±7.5	38.9±25.6	41.4±6.0	44.8±5.9
Table 2: Performance comparison over modalities on MMC-PCFG on DiDeMo.
Figure 5: Parse trees predicted by different models for the sentence The man falls to the floor.
largest performance drops (see Table 2). Therefore, videos contribute most to the performance among all modalities.
In Figure 5, we visualize a parse tree predicted by the best run of SENet154, I3D and MMC-PCFG. We can observe that SENet identifies all NPs but fails at the VP. I3D correctly predicts the VP but fails at recognizing a NP, "the man". Our MMC-PCFG can take advantages of all experts and produce the correct prediction.
Grammar Induction Grammar induction and unsupervised parsing has been a long-standing problem in computational linguistics (Carroll and Charniak, 1992). Recent work utilized neural networks in predicting constituency structures with no supervision (Shen et al., 2018a; Drozdov et al., 2019; Shen et al., 2018b; Kim et al., 2019; Jin et al., 2019a) and showed promising results. In addition to learning purely from text, there is a growing interest to use image information to improve accuracy of induced constituency trees (Shi et al., 2019; Kojima et al., 2020; Zhao and Titov, 2020; Jin and Schuler, 2020). Different from previous work, our work improves the constituency parser by using videos containing richer information than images. Video-Text Matching Video-text matching has been widely studied in various tasks, such as video retrieval (Liu et al., 2019; Gabeur et al., 2020), moment localization with natural language (Zhang
Different runs represent models trained with different seeds.
et al., 2019, 2020) and video question and answering (Xu et al., 2017; Jin et al., 2019b). It aims to learn video-semantic representation in a joint embedding space. Recent works (Liu et al., 2019; Gabeur et al., 2020; Chen et al., 2020) focus on learning video's multi-modal representation to match with text. In this work, we borrow this idea to match video and textual representations.
In this work, we have presented a new task referred to as video-aided unsupervised grammar induction. This task aims to improve grammar induction models by using aligned video-sentence pairs as an effective way to address the limitation of current image-based methods where only object information from static images is considered and important verb related information from vision is missing. Moreover, we present Multi-Modal Compound Probabilistic Context-Free Grammars (MMC-PCFG) to effectively integrate video features extracted from different modalities to induce more accurate grammars. Experiments on three datasets demonstrate the effectiveness of our method.
We thank the support of NSF awards IIS-1704337, IIS-1722847, IIS-1813709, and the generous gift from our corporate sponsors.
Samuel Albanie, Yang Liu, Arsha Nagrani, Antoine Miech, Ernesto Coto, Ivan Laptev, Rahul Sukthankar, Bernard Ghanem, Andrew Zisserman, Valentin Gabeur, et al. 2020. The end-of-end-to-end: A video understanding pentathlon challenge (2020). arXiv preprint arXiv:2008.00744.
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR.
Glenn Carroll and Eugene Charniak. 1992. Two experiments on learning probabilistic dependency grammars from corpora. Department of Computer Science, Univ.
Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. 2020. Learning modality interaction for temporal sentence localization and event captioning in videos. In ECCV.
John Cocke. 1969. Programming languages and their compilers: Preliminary notes. New York University.
Dan Deng, Haifeng Liu, Xuelong Li, and Deng Cai. 2018. Pixelink: Detecting scene text via instance segmentation. In AAAI.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In *NAACL*.
Daniel Fried, Nikita Kitaev, and Dan Klein. 2019. Cross-domain generalization of neural constituency parsers. In ACL.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In ECCV.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. 2019. Large-scale weakly-supervised pre-training for video action recognition. In CVPR.
Lila Gleitman. 1990. The Structural Sources of Verb Meanings. Language Acquisition, 1(1):3-55.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In ECCV.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In ICCV.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for largescale audio classification. In ICASSP.
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-andexcitation networks. In CVPR.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In CVPR.
Lifeng Jin, Finale Doshi-Velez, Timothy Miller, William Schuler, and Lane Schwartz. 2018. Unsupervised grammar induction with depth-bounded pcfg. TACL.
Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019a. Unsupervised learning of PCFGs with normalizing flow. In ACL.
Lifeng Jin and William Schuler. 2020. Grounded pcfg induction with images. In AACL-IJCNLP.
Weike Jin, Zhou Zhao, Mao Gu, Jun Yu, Jun Xiao, and Yueting Zhuang. 2019b. Multi-interaction network with object relation for video question answering. In ACM Multimedia.
Tadao Kasami. 1966. An efficient recognition and syntax-analysis algorithm for context-free languages. Coordinated Science Laboratory Report no. R-257.
Yoon Kim, Chris Dyer, and Alexander M Rush. 2019. Compound probabilistic context-free grammars for grammar induction. In ACL.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In ICLR.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In ACL.
Noriyuki Kojima, Hadar Averbuch-Elor, Alexander M Rush, and Yoav Artzi. 2020. What is learned in visually grounded neural syntax acquisition. In ACL.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS.
Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L Berg, and Mohit Bansal. 2020. Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning. In ACL.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016. SSD: Single shot multibox detector. In ECCV.
Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. 2019. Use what you have: Video retrieval using representations from collaborative experts. In arXiv preprint arxiv:1907.13487.
Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell. 2018. Synthetically supervised feature learning for scene text recognition. In ECCV.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. 2018. Exploring the limits of weakly supervised pretraining. In ECCV.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. In CVPR.
Antoine Miech, Ivan Laptev, and Josef Sivic. 2018. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.02516.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Steven Pinker and B MacWhinney. 1987. The bootstrapping problem in language acquisition. Mechanisms of language acquisition, pages 399-441.
Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville. 2018a. Neural language modeling by jointly learning syntax and lexicon. In ICLR.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2018b. Ordered neurons: Integrating tree structures into recurrent neural networks. In ICLR.
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019. Visually grounded neural syntax acquisition. In ACL.
Michael Tomasello. 2003. Constructing a language: A usage-based theory of language acquisition. Harvard University Press, Cambridge, MA, US.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018. A closer look at spatiotemporal convolutions for action recognition. In CVPR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Adina Williams, Andrew Drozdov*, and Samuel R Bowman. 2018. Do latent tree learning models identify meaningful structure in sentences? TACL.
Saining Xie, Ross Girshick, Piotr Dolkar, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In CVPR.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In CVPR.
Daniel H Younger. 1967. Recognition and parsing of context-free languages in time n3. Information and control, 10(2):189-208.
Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020. Learning 2d temporal adjacent networks form moment localization with natural language. In AAAI.
Songyang Zhang, Jinsong Su, and Jiebo Luo. 2019. Exploiting temporal relationships in video moment localization with natural language. In ACM Multimedia.
Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In EMNLP.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. TPAMI.
Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018. Towards automatic learning of procedures from web instructional videos. In AAAI.
A Performance Comparison - Full Tables
Method	NP	VP	PP	SBAR	ADJP	ADVP	C-F1	S-F1
LBranch	41.7	0.1	0.1	0.7	7.2	0.0	16.2	18.5
RBranch	32.8	91.5	66.5	88.2	36.9	63.6	53.6	57.5
Random	36.5±0.6	30.5±0.5	30.1±0.5	25.7±2.8	29.5±2.3	28.5±4.8	29.4±0.3	32.7±0.5
C-PCFG	72.9±5.5	16.5±6.2	23.4±16.9	26.6±15.9	25.0±11.6	14.7±12.8	38.2±5.0	40.4±4.1
VC-PCFG	ResNeXt	64.4±21.4	25.7±17.7	34.6±25.0	40.5±26.3	16.7±9.5	28.4±21.3	40.0±13.7	41.8±14.0
SENet	70.5±15.3	25.7±15.9	36.5±24.6	36.8±25.9	21.2±12.5	23.6±16.8	42.6±10.4	44.0±10.4
I3D	57.9±13.5	45.7±14.1	45.8±17.2	38.2±14.8	28.4±9.2	22.0±9.3	45.1±6.0	49.2±6.0
R2P1D	61.2±8.5	38.1±5.4	62.1±4.1	61.5±5.1	21.4±11.4	40.8±7.3	48.1±4.4	50.7±4.2
S3DG	61.3±13.4	31.7±16.7	51.8±8.0	50.3±6.5	18.0±4.5	35.2±11.4	44.0±2.7	46.5±5.1
Scene	62.2±9.6	30.6±12.3	41.1±24.8	35.2±21.9	21.4±14.0	27.6±17.1	41.7±6.5	44.9±7.4
Audio	64.2±18.6	21.3±26.5	34.7±11.0	37.3±19.6	26.1±4.9	18.2±11.6	38.7±3.7	39.5±5.2
OCR	64.4±15.0	27.4±19.5	42.8±31.2	35.9±20.7	14.6±1.7	23.2±24.0	41.9±16.9	44.6±17.5
Face	60.8±16.0	31.5±17.0	52.8±9.8	49.3±5.6	12.6±3.3	32.9±14.6	43.9±4.5	46.3±5.5
Speech	61.8±12.8	26.6±17.6	43.8±34.5	34.2±20.6	14.4±4.8	12.9±9.6	40.9±16.0	43.1±16.1
Concat	68.6±8.6	24.9±19.9	39.7±19.5	39.3±19.8	10.8±2.8	18.3±18.1	42.2±12.3	43.2±14.2
MMC-PCFG	67.9±9.8	52.3±9.0	63.5±8.6	60.7±10.8	34.7±17.0	50.4±8.3	55.0±3.7	58.9±3.4
Table 3: Performance Comparison on DiDeMo.
Method	NP	VP	PP	SBAR	ADJP	ADVP	C-F1	S-F1
LBranch	1.7	42.8	0.4	8.1	1.5	0.0	6.8	5.9
RBranch	35.6	47.5	67.0	88.9	33.9	65.0	35.0	41.6
Random	27.2±0.3	27.1±1.4	29.9±0.5	31.3±5.2	26.9±7.7	26.2±11.9	21.2±0.2	24.0±0.2
C-PCFG	47.4±18.4	49.4±11.9	58.0±22.6	45.7±6.0	27.7±15.1	36.2±7.4	37.8±6.7	41.4±6.6
VC-PCFG	ResNeXt	46.5±13.7	40.8±9.8	67.9±12.7	50.5±13.3	22.3±6.7	38.8±21.3	38.2±8.3	42.8±8.4
SENet	48.3±14.4	40.7±9.2	73.6±11.2	45.5±17.0	26.9±13.6	41.2±17.5	39.9±8.7	44.9±8.3
I3D	48.1±10.7	39.0±8.0	79.4±8.4	50.0±14.9	18.5±7.0	41.2±4.1	40.6±3.6	45.7±3.2
R2P1D	52.4±10.9	33.7±16.4	66.7±10.7	49.5±13.8	25.8±10.6	33.8±12.4	39.4±8.1	44.4±8.3
S3DG	50.4±13.1	32.6±16.3	71.7±7.5	33.3±5.9	30.8±17.5	40.0±7.1	39.3±6.5	44.1±6.6
Audio	51.2±3.1	42.0±7.2	61.5±18.0	51.0±14.8	23.5±16.8	48.8±8.2	39.2±4.7	43.3±4.9
OCR	48.6±8.1	41.5±4.1	65.5±17.4	39.9±4.4	18.5±6.6	53.8±14.7	38.6±5.5	43.2±5.6
Concat	50.3±10.3	42.3±2.9	81.6±8.7	40.1±3.9	17.7±8.2	52.5±5.6	42.3±5.7	47.0±5.6
MMC-PCFG	62.7±9.8	45.3±2.8	63.4±17.7	43.9±4.8	26.2±7.5	35.0±3.5	44.7±5.2	48.9±5.7