SlowGuess/ACL_2020_OCR huggingface dataset:
In the test phase, our method takes a multimedia document with sentences  $S = \{s_1,s_2,\ldots \}$  and images  $M = \{m_{1},m_{2},\dots ,\}$  as input. We first generate the structured common embedding for each sentence and each image, and then compute pairwise similarities  $\langle s,m\rangle$ . We pair each sentence  $s$  with the closest image  $m$ , and aggregate the features of each word of  $s$  with the aligned representation from  $m$  by weighted averaging:
$$
\boldsymbol {w} _ {i} ^ {\prime \prime} = (1 - \gamma) \boldsymbol {w} _ {i} + \gamma \boldsymbol {w} _ {i} ^ {\prime}, \tag {5}
$$
where  $\gamma = \exp (-\langle s,m\rangle)$  and  $\pmb{w}_i^{\prime}$  is derived from  $m$  using Equation 4. We use  $\pmb{w}_i^{\prime \prime}$  to classify each
Training	Model	Text-Only Evaluation	Image-Only Evaluation	Multimedia Evaluation
Event Mention	Argument Role	Event Mention	Argument Role	Event Mention	Argument Role
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Text	JMEE	42.5	58.2	48.7	22.9	28.3	25.3	-	-	-	-	42.1	34.6	38.1	21.1	12.6	15.8
GAIL	43.4	53.5	47.9	23.6	29.2	26.1	-	-	-	-	44.0	32.4	37.3	22.7	12.8	16.4
WASET	42.3	58.4	48.2	21.4	30.1	24.9	-	-	-	-	41.2	33.1	36.7	20.1	13.0	15.7
Image	WASEIatt	-	-	-	-	-	-	29.7	61.9	40.1	9.1	10.2	9.6	28.3	23.0	25.4	2.9	6.1	3.8
WASEIobj	-	-	-	-	-	-	28.6	59.2	38.7	13.3	9.8	11.2	26.1	22.4	24.1	4.7	5.0	4.9
Multimedia	VSE-C	33.5	47.8	39.4	16.6	24.7	19.8	30.3	48.9	26.4	5.6	6.1	5.7	33.3	48.2	39.3	11.1	14.9	12.8
Flatatt	34.2	63.2	44.4	20.1	27.1	23.1	27.1	57.3	36.7	4.3	8.9	5.8	33.9	59.8	42.2	12.9	17.6	14.9
Flatobj	38.3	57.9	46.1	21.8	26.6	24.0	26.4	55.8	35.8	9.1	6.5	7.6	34.1	56.4	42.5	16.3	15.9	16.1
WASEatt	37.6	66.8	48.1	27.5	33.2	30.1	32.3	63.4	42.8	9.7	11.1	10.3	38.2	67.1	49.1	18.6	21.6	19.9
WASEobj	42.8	61.9	50.6	23.5	30.3	26.4	43.1	59.2	49.9	14.5	10.1	11.9	43.0	62.1	50.8	19.5	18.9	19.2
Table 3: Event and argument extraction results (\%). We compare three categories of baselines in three evaluation settings. The main contribution of the paper is joint training and joint inference on multimedia data (bottom right).
word into an event type and to classify each entity into a role with multimedia classifiers in Equation 2. To this end, we define  $\pmb{t}_i^{\prime \prime}$  similar to  $\pmb{w}_i^{\prime \prime}$  but using  $\pmb{t}_i$  and  $\pmb{t}_i^{\prime}$ . Similarly, for each image  $m$  we find the closest sentence  $s$ , compute the aggregated multimedia features  $\pmb{m}^{\prime \prime}$  and  $\pmb{o}_i^{\prime \prime}$ , and feed into the shared classifiers (Equation 3) to predict visual event and argument roles. Finally, we corefer the cross-media events of the same event type if the similarity  $\langle s, m \rangle$  is higher than a threshold.
Evaluation Metrics We conduct evaluation on text-only, image-only, and multimedia event mentions in  $\mathbf{M}^2\mathbf{E}^2$  dataset in Section 2.2. We adopt the traditional event extraction measures, i.e., Precision, Recall and  $F_{1}$ . For text-only event mentions, we follow (Ji and Grishman, 2008; Li et al., 2013): a textual event mention is correct if its event type and trigger offsets match a reference trigger; and a textual event argument is correct if its event type, offsets, and role label match a reference argument. We make a similar definition for image-only event mentions: a visual event mention is correct if its event type and image match a reference visual event mention; and a visual event argument is correct if its event type, localization, and role label match a reference argument. A visual argument is correctly localized if the Intersection over Union (IoU) of the predicted bounding box with the ground truth bounding box is over 0.5. Finally, we define a multimedia event mention to be correct if its event type and trigger offsets (or the image) match the reference trigger (or the reference image). The arguments of multimedia
events are either textual or visual arguments, and are evaluated accordingly. To generate bounding boxes for the attention-based model, we threshold the heatmap using the adaptive value of  $0.75 * p$ , where  $p$  is the peak value of the heatmap. Then we compute the tightest bounding box that encloses all of the thresholded region. Examples are shown in Figure 7 and Figure 8.
Baselines The baselines include: (1) Text-only models: We use the state-of-the-art model JMEE (Liu et al., 2018a) and GAIL (Zhang et al., 2019) for comparison. We also evaluate the effectiveness of cross media joint training by including a version of our model trained only on ACE, denoted as  $\mathrm{WASE^T}$ . (2) Image-only models: Since we are the first to extract newsworthy events, and the most similar work situation recognition can not localize arguments in images, we use our model trained only on image corpus as baselines. Our visual branch has two versions, object-based and attention-based, denoted as  $\mathrm{WASE}^{\mathbb{I}}_{\mathrm{obj}}$  and  $\mathrm{WASE}^{\mathbb{I}}_{\mathrm{att}}$ . (3) Multimedia models: To show the effectiveness of structured embedding, we include a baseline by removing the text and image GCNs from our model, which is denoted as Flat. The Flat baseline ignores edges and treats images and sentences as sets of vectors. We also compare to the state-of-the-art cross-media common representation model, Contrastive Visual Semantic Embedding VSE-C (Shi et al., 2018), by training it the same way as WASE.
Parameter Settings The common space dimension is 300. The dimension is 512 for image position embedding and feature map, and 50 for word position embedding, entity type embedding, and POS tag embedding. The layer of GCN is 3.
4.2 Quantitative Performance
As shown in Table 3, our complete methods  $\mathrm{(WASE_{att}}$  and  $\mathrm{WASE}_{\mathrm{obj}}$  ) outperform all baselines in the three evaluation settings in terms of  $\mathbf{F}_1$ . The comparison with other multimedia models demonstrates the effectiveness of our model architecture and training strategy. The advantage of structured embedding is shown by the better performance over the flat baseline. Our model outperforms its text-only and image-only variants on multimedia events, showing the inadequacy of single-modal information for complex news understanding. Furthermore, our model achieves better performance on text-only and image-only events, which demonstrates the effectiveness of multimedia training framework in knowledge transfer between modalities.
$\mathrm{WASE}_{\mathrm{obj}}$  and  $\mathrm{WASE}_{\mathrm{att}}$ , are both superior to the state of the art and each has its own advantages.  $\mathrm{WASE}_{\mathrm{obj}}$  predicts more accurate bounding boxes since it is based on a Fa  $\mathrm{F}$  bounding box annotation argument precision. W higher argument recall as it predefined object
Model
rule_based
VSE
Flatatt	35.1	75.5	43.6
Flatobj	34.3	76.4	47.3
WASEatt	39.5	73.5	51.5
WASEobj	40.1	75.4	52.4
Furthermore, to evaluate the cross-media event coreference performance, we pair textual and visual event mentions in the same document, and calculate Precision, Recall and  $F_{1}$  to compare with ground truth event mention pairs<sup>7</sup>. As shown in Table 4,  $\mathrm{WASE}_{\mathrm{obj}}$  outperforms all multimedia embedding models, as well as the rule-based baseline using event type matching. This demonstrates the effectiveness of our cross-media soft alignment.
Our cross-media joint training approach successfully boosts both event extraction and argument role labeling performance. For example, in Figure 5 (a), the text-only model can not extract  $Jus$ -
tice.Arrest event, but the joint model can use the image as background to detect the event type. In Figure 5 (b), the image-only model detects the image as Conflict.Demonstration, but the sentences in the same document help our model not to label it as Conflict.Demonstration. Compared with multimedia flat embedding in Figure 6, WASE can learn structures such as Artifact is on top of Vehicle, and the person in the middle of Justice.Arrest is Entity instead of Agent.
Iraqi security forces search [Justice.Arrest] a civilian in the city of Mosul.
People celebrate Supreme Court ruling on Same Sex Marriage in front of the Supreme Court in Washington.
Figure 5: Image helps textual event extraction, and surrounding sentence helps visual event extraction.
Table 4: Cross-media event coreference performance.
Flat	Event	Movement. Transport
Role	Artifact = none
Ours	Event	Movement. Transport
Role	Artifact = man
Flat	Event	Justice:ArrestJail
Role	Agent = man
Ours	Event	Conflictattack
Role	Entity = man
Figure 6: Comparison with multimedia flat embedding.
based method is not able to precisely localize the objects for each argument, since there is no supervision on attention extraction during training. For example, in Figure 7, the Entity argument in the Conflict.Demonstrate event is correctly predicted as troops, but its localization is incorrect because Place argument share similar attention. When one argument targets at too many instances, attention heatmaps tend to lose focus and cover the whole image, as shown in Figure 8.
Text Event Extraction Text event extraction has been extensively studied for general news do
<sup>7</sup>We do not use coreference clustering metrics because we only focus on mention-level cross-media event coreference instead of the full coreference in all documents.
Figure 7: Argument labeling error examples: correct entity name but wrong localization.
Figure 8: Attention heatmaps lose focus due to large instance candidate number.
main (Ji and Grishman, 2008; Liao and Grishman, 2011; Huang and Riloff, 2012; Li et al., 2013; Chen et al., 2015; Nguyen et al., 2016; Hong et al., 2018; Liu et al., 2018b; Chen et al., 2018; Zhang et al., 2019; Liu et al., 2018a; Wang et al., 2019; Yang et al., 2019; Wadden et al., 2019). Multimedia features has been proven to effectively improve text event extraction (Zhang et al., 2017).
Visual Event Extraction "Events" in NLP usually refer to complex events that involve multiple entities in a large span of time (e.g. protest), while in CV (Chang et al., 2016; Zhang et al., 2007; Ma et al., 2017) events are less complex single-entity activities (e.g. washing dishes) or actions (e.g. jumping). Visual event ontologies focus on daily life domains, such as "dogshow" and "wedding ceremony" (Perera et al., 2012). Moreover, most efforts ignore the structure of events including arguments. There are a few methods that aim to localize the agent (Gu et al., 2018; Li et al., 2018; Duarte et al., 2018), or classify the recipient (Sigurdsson et al., 2016; Kato et al., 2018; Wu et al., 2019a) of events, but neither detects the complete set of arguments for an event. The most similar to our work is Situation Recognition (SR) (Yatskar et al., 2016; Mallya and Lazebnik, 2017) which predicts an event and multiple arguments from an input image, but does not localize the arguments. We use SR as an auxiliary task for training our visual branch, but exploit object detection and attention to enable localization of arguments. Silberer and Pinkal redefine the problem of visual argument role labeling with event types and bounding boxes as input. Different from their work, we extend the problem scope to including event identification and coreference, and further
advance argument localization by proposing an attention framework which does not require bounding boxes for training nor testing.
Multimedia Representation Multimedia common representation has attracted much attention recently (Toselli et al., 2007; Weegar et al., 2015; Hewitt et al., 2018; Chen et al., 2019; Liu et al., 2019; Su et al., 2019a; Sarafianos et al., 2019; Sun et al., 2019b; Tan and Bansal, 2019; Li et al., 2019a,b; Lu et al., 2019; Sun et al., 2019a; Rahman et al., 2019; Su et al., 2019b). However, previous methods focus on aligning images with their captions, or regions with words and entities, but ignore structure and semantic roles. UniVSE (Wu et al., 2019b) incorporates entity attributes and relations into cross-media alignment, but does not capture graph-level structures of images or text.
In this paper we propose a new task of multimedia event extraction and setup a new benchmark. We also develop a novel multimedia structured common space construction method to take advantage of the existing image-caption pairs and single-modal annotated data for weakly supervised training. Experiments demonstrate its effectiveness as a new step towards semantic understanding of events in multimedia data. In the future, we aim to extend our framework to extract events from videos, and make it scalable to new event types. We plan to expand our annotations by including event types from other text event ontologies, as well as new event types not in existing text ontologies. We will also apply our extraction results to downstream applications including cross-media event inference, timeline generation, etc.
This research is based upon work supported in part by U.S. DARPA AIDA Program No. FA8750-18-2-0014 and U.S. DARPA KAIROS Program No. FA8750-19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970.
Xiaojun Chang, Zhigang Ma, Yi Yang, Zhiqiang Zeng, and Alexander G Hauptmann. 2016. Bi-level semantic representation analysis for multimedia event detection. IEEE transactions on cybernetics, 47(5):1180-1197.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proc. ACL-IJCNLP2015.
Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yantao Jia. 2018. Collective event detection via a hierarchical and bias tagging networks with gated multilevel attention mechanisms. In Proc. EMNLP2018.
Kevin Duarte, Yogesh Rawat, and Mubarak Shah. 2018. Videocapsulenet: A simplified network for action detection. In Advances in Neural Information Processing Systems, pages 7610-7619.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives.
Charles J Fillmore, Christopher R Johnson, and Miriam RL Petruck. 2003. Background to framenet. International journal of lexicography, 16(3):235-250.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645–6649. IEEE.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6047-6056.
John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch. 2018. Learning translations via images with a massively multilingual image dataset. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2566-2576.
Yu Hong, Wenxuan Zhou, jingli zhang jingli, Guodong Zhou, and Qiaoming Zhu. 2018. Self-regulation: Employing a generative adversarial network to improve event detection. In Proc. ACL2018.
Ruihong Huang and Ellen Riloff. 2012. Bootstrapped training of event extraction classifiers. In Proc. EACL2012.
Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL-08: HLT, pages 254-262.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137.
Keizo Kato, Yin Li, and Abhinav Gupta. 2018. Compositional learning for human object interaction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 234-251.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. 2018. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982.
Dong Li, Zhaofan Qiu, Qi Dai, Ting Yao, and Tao Mei. 2018. Recurrent tubelet proposal and recognition networks for action detection. In Proceedings of the European conference on computer vision (ECCV), pages 303-318.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2019a. Unicoder-vl: A universal encoder for vision and language by cross-modal pretraining. arXiv preprint arXiv:1908.06066.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019b. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proc. ACL2013.
Shasha Liao and Ralph Grishman. 2011. Acquiring topic features to improve event extraction: in pre-selected and balanced collections. In Proc. RANLP2011.
Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In Proc. The 57th Annual Meeting of the Association for Computational Linguistics (ACL2019).
Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, and Yongdong Zhang. 2019. Focus your attention: A bidirectional focal attention network for image-text matching. In Proceedings of the 27th ACM International Conference on Multimedia, pages 3-11. ACM.
Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018a. Jointly multiple events extraction via attention-based graph information aggregation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1247-1256.
Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018b. Jointly multiple events extraction via attention-based graph information aggregation. In Proc. EMNLP2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. In H. Wallach, H. Larochelle, A. Beygelz-imer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13-23.
Zhigang Ma, Xiaojun Chang, Zhongwen Xu, Nicu Sebe, and Alexander G Hauptmann. 2017. Joint attributes and event analysis for multimedia event detection. IEEE transactions on neural networks and learning systems, 29(7):2921-2930.
Arun Mallya and Svetlana Lazebnik. 2017. Recurrent models for situation recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 455-463.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proc. NAACL-HLT2016.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for
word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
AG Amitha Perera, Sangmin Oh, P Megha, Tianyang Ma, Anthony Hoogs, Arash Vahdat, Kevin Cannons, Greg Mori, Scott Mccloskey, Ben Miller, et al. 2012. Trecvid 2012 genie: Multimedia event detection and recounting. In In TRECVID Workshop. CiteSeer.
Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Morency, and Mohammed Ehsan Hoque. 2019. M-bert: Injecting multimodal information in the bert structure. arXiv preprint arXiv:1908.05787.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99.
Nikolaos Sarafianos, Xiang Xu, and Ioannis A. Kakadiaris. 2019. Adversarial representation learning for text-to-image matching. In The IEEE International Conference on Computer Vision (ICCV).
Haoyue Shi, Jiayuan Mao, Tete Xiao, Yuning Jiang, and Jian Sun. 2018. Learning visually-grounded semantics from contrastive adversarial samples. arXiv preprint arXiv:1806.10348.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer.
Carina Silberer and Manfred Pinkal. 2018. Grounding semantic roles in images. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2616-2626.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402.
Mitchell Stephens. 1998. The Rise of the Image, The Fall of the Word. New York: Oxford University Press.
Shupeng Su, Zhisheng Zhong, and Chao Zhang. 2019a. Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval. In The IEEE International Conference on Computer Vision (ICCV).
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019b. Vl-bert: Pretraining of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530.