Venue: ACL 2020

# Cross-media Structured Common Space for Multimedia Event Extraction

Manling Li1, Alireza Zareian2, Qi Zeng1, Spencer Whitehead1, Di Lu3, Heng Ji1, Shih-Fu Chang2

###### Abstract

We introduce a new task, **M**ulti**Me**dia **E**vent **E**xtraction (M\({}^{2}\)E\({}^{2}\)), which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments.1 We propose a novel method, **W**eakly **A**ligned **S**tructured **E**mbedding (**WASE**), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to unimodal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state-of-the-art multimedia unstructured representations, we achieve 8.3% and 5.0% absolute F-score gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract 21.4% more event mentions than traditional text-only methods.

Footnote 1: Our data and code are available at [http://blender.cs.illinois.edu/software/m2e2](http://blender.cs.illinois.edu/software/m2e2)

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2557–2568

## 1 Introduction

Traditional event extraction methods target a single modality, such as text Wadden et al. (2019), images Yatskar et al. (2016) or videos Ye et al. (2015); Caba Heilbron et al. (2015); Soomro et al. (2012). However, the practice of contemporary journalism Stephens (1998) distributes news via multimedia. By randomly sampling 100 multimedia news articles from the Voice of America (VOA), we find that 33% of images in the articles contain visual objects that serve as event arguments and are not mentioned in the text. Take Figure 1 as an example, we can extract the _Agent_ and _Person_ arguments of the _Movement.Transport_ event from text, but can extract the _Vehicle_ argument only from the image. Nevertheless, event extraction is independently studied in Computer Vision (CV) and Natural Language Processing (NLP), with major differences in task definition, data domain, methodology, and terminology. Motivated by the complementary and holistic nature of multimedia data, we propose **M**ulti**Me**dia **E**vent **E**xtraction (**M\({}^{2}\)E\({}^{2}\)**), a new task that aims to jointly extract events and arguments from multiple modalities. We construct the first benchmark and evaluation dataset for this task, which consists of 245 fully annotated news articles.

We propose the first method, **W**eakly **A**ligned **S**tructured **E**mbedding (**WASE**), for extracting events and arguments from multiple modalities. Complex event structures have not been covered by existing multimedia representation methods Wu et al. (2019b); Faghri et al. (2017); Karpathy and Fei-Fei (2015), so we propose to learn a _structured_ multimedia embedding space. More specifically, given a multimedia document, we represent each image or sentence as a graph, where each node represents an e

Figure 1: An example of Multimedia Event Extraction. An event mention and some event arguments (_Agent_ and _Person_) are extracted from text, while the vehicle arguments can only be extracted from the image.



edge represents an argument role. The node and edge embeddings are represented in a multimedia common semantic space, as they are trained to resolve event co-reference across modalities and to match images with relevant sentences. This enables us to jointly classify events and argument roles from both modalities. A major challenge is the lack of multimedia event argument annotations, which are costly to obtain due to the annotation complexity. Therefore, we propose a weakly supervised framework, which takes advantage of annotated uni-modal corpora to separately learn visual and textual event extraction, and uses an image-caption dataset to align the modalities.

We evaluate WASE on the new task of M\({}^{2}\)E\({}^{2}\). Compared to the state-of-the-art uni-modal methods and multimedia flat representations, our method significantly outperforms on both event extraction and argument role labeling tasks in all settings. Moreover, it extracts 21.4% more event mentions than text-only baselines. The training and evaluation are done on heterogeneous data sets from multiple sources, domains and data modalities, demonstrating the scalability and transferability of the proposed model. In summary, this paper makes the following contributions:

* We propose a new task, MultiMedia Event Extraction, and construct the first annotated news dataset as a benchmark to support deep analysis of cross-media events.
* We develop a weakly supervised training framework, which utilizes existing single-modal annotated corpora, and enables joint inference without cross-modal annotation.
* Our proposed method, WASE, is the first to leverage structured representations and graph-based neural networks for multimedia common space embedding.

## 2 Task Definition

### Problem Formulation

Each input document consists of a set of images \(\mathcal{M}=\left\{m_{1},m_{2},\dots\right\}\) and a set of sentences \(\mathcal{S}=\left\{s_{1},s_{2},\dots\right\}\). Each sentence \(s\) can be represented as a sequence of tokens \(s=\left(w_{1},w_{2},\dots\right)\), where \(w_{i}\) is a token from the document vocabulary \(\mathcal{W}\). The input also includes a set of entities \(\mathcal{T}=\left\{t_{1},t_{2},\dots\right\}\) extracted from the document text. An entity is an individually unique object in the real world, such as a person, an organization, a facility, a location, a geopolitical entity, a weapon, or a vehicle. The objective of M\({}^{2}\)E\({}^{2}\)is twofold:

**Event Extraction**: Given a multimedia document, extract a set of event mentions, where each event mention \(e\) has a type \(y_{e}\) and is grounded on a text trigger word \(w\) or an image \(m\) or both, i.e.,

\[e=(y_{e},\left\{w,m\right\}).\]

Note that for an event, \(w\) and \(m\) can both exist, which means the visual event mention and the textual event mention refer to the same event. For example in Figure 1, _deploy_ indicates the same _Movement.Transport_ event as the image. We consider the event \(e\) as **text-only** event if it only has textual mention \(w\), and as **image-only** event if it only contains visual mention \(m\), and as **multimedia** event if both \(w\) and \(m\) exist.

**Argument Extraction**: The second task is to extract a set of arguments of event mention \(e\). Each argument \(a\) has an argument role type \(y_{a}\), and is grounded on a text entity \(t\) or an image object \(o\) (represented as a bounding box), or both,

\[a=\left(y_{a},\left\{t,o\right\}\right).\]

The arguments of visual and textual event mentions are merged if they refer to the same real-world event, as shown in Figure 1.

### The M\({}^{2}\)E\({}^{2}\) Dataset

We define multimedia newsworthy event types by exhaustively mapping between the event ontology in NLP community for the news domain (ACE2) and the event ontology in CV community for general domain (imSitu Yatskar et al., 2016)). They cover the largest event training resources in each community. Table 1 shows the selected complete intersection, which contains 8 ACE types (i.e., 24% of all ACE types), mapped to 98 imSitu types (i.e., 20% of all imSitu types). We expand the ACE event role set by adding visual arguments from imSitu, such as _instrument,_ bolded in Table 1. This set encompasses 52% ACE events in a news corpus, which indicates that the selected eight types are salient in the news domain. We reuse these existing ontologies because they enable us to train event and argument classifiers for both modalities without requiring joint multimedia event annotation as training data.

 

We collect 108,693 multimedia news articles from the Voice of America (VOA) website 32 2006-2017, covering a wide range of newsworthy topics such as military, economy and health. We select 245 documents as the annotation set based on three criteria: (1) Informativeness: articles with more event mentions; (2) Illustration: articles with more images (\(>4\)); (3) Diversity: articles that balance the event type distribution regardless of true frequency. The data statistics are shown in Table 2. Among all of these events, 192 textual event mentions and 203 visual event mentions can be aligned as 309 cross-media event mention pairs. The dataset can be divided into 1,105 text-only event mentions, 188 image-only event mentions, and 395 multimedia event mentions.

Footnote 3: [https://www.voanews.com/](https://www.voanews.com/)

We follow the ACE event annotation guidelines Walker et al. (2006) for textual event and argument annotation, and design an annotation guideline 4 for multimedia events annotation.

Footnote 4: [http://blender.cs.illinois.edu/software/mgeq/ACL2020_M2E2_annotation.pdf](http://blender.cs.illinois.edu/software/mgeq/ACL2020_M2E2_annotation.pdf)

One unique challenge in multimedia event annotation is to localize visual arguments in complex scenarios, where images include a crowd of people or a group of object. It is hard to delineate each of them using a bounding box. To solve this problem, we define two types of bounding boxes: (1) _union bounding box_: for each role, we annotate the smallest bounding box covering all constituents; and (2) _instance bounding box_: for each role, we annotate a set of bounding boxes, where each box is the smallest region that covers an individual participant (e.g., one person in the crowd), following the VOC2011 Annotation Guidelines5. Figure 2 shows an example. Eight NLP and CV researchers complete the annotation work with two independent passes and reach an Inter-Annotator Agreement (IAA) of 81.2%. Two expert annotators perform adjudication.

Footnote 5: [http://host.robots.ox.ac.uk/pascal/VOC/voc2011/guidelines.html](http://host.robots.ox.ac.uk/pascal/VOC/voc2011/guidelines.html)

## 3 Method

### Approach Overview

As shown in Figure 3, the training phase contains three tasks: text event extraction (Section 3.2), visual situation recognition (Section 3.3), and cross-media alignment (Section 3.4). We learn a cross-media shared encoder, a shared event classifier, and a shared argument classifier. In the testing phase (Section 3.5), given a multimedia news article, we encode the sentences and images into the structured common space, and jointly extract textual and visual events and arguments, followed by cross-modal coreference resolution.

### Text Event Extraction

**Text Structured Representation:** As shown in Figure 4, we choose Abstract Meaning Representation (AMR) Banarescu et al. (2013) to represent text because it includes a rich set of 150 fine-grained semantic roles. To encode each text sentence, we run the CAMR parser Wang et al. (2015b, a, 2016) to generate an AMR graph, based on the named entity recognition and part-of-speech (POS) tagging results from Stanford CoreNLP Manning et al. (2014). To represent each word \(w\) in a sentence \(s\), we concatenate its

\begin{table}
\begin{tabular}{|l|l|} \hline
**Event Type** & **Argument Role** \\ \hline Movement.Transport (223][53] & Agent (46|64), Artifact (179|103), Vehicle (24|51), Destination (120|0), Origin (66||0) \\ \hline Conflict.Attack (326|27) & Attacker (192|12), Target (207|19), Instrument (37|15), Place (121|0) \\ \hline Conflict.Demonstrate (151|69) & Entity (102|184), **Police** (326), **Instrument** (0|118), Place (86|25) \\ Justice.ArrestJail (160|56) & Agent (64|119), Person (147|99), **Instrument** (0|11), Place (43|0) \\ Contact.PhoneWrite (33|37) & Entity (33|46), **Instrum** (0|43), Place (80) \\ Contact.Meet (127|79) & Participant (119|321), Place (68|0) \\ Life.Die (244|64) & Agent (39|0), Instrument (4|2), Victim (165|155), Place (54|0) \\ Transaction. & Giver (19|3), Recipient (19|5), Money (0|8) \\ \hline \end{tabular}
\end{table}
Table 1: Event types and argument roles in M\({}^{2}\)E\({}^{2}\), with expanded ones in bold. Numbers in parentheses represent the counts of textual and visual events/arguments.

Figure 2: Example of bounding boxes.

 

pre-trained GloVe word embedding (Pennington et al., 2014), POS embedding, entity type embedding and position embedding. We then input the word sequence to a bi-directional long short term memory (Bi-LSTM) (Graves et al., 2013) network to encode the word order and get the representation of each word \({\mathbf{w}}\). Given the AMR graph, we apply a Graph Convolutional Network (GCN) (Kipf and Welling, 2016) to encode the graph contextual information following Liu et al. (2018a):

\[{\mathbf{w}}_{i}^{(k+1)}=f(\sum_{j\in\mathcal{N}(i)}g_{ij}^{(k)}({\mathbf{W}}_{E(i, j)}{\mathbf{w}}_{j}^{(k)}+{\mathbf{b}}_{E(i,j)}^{(k)})), \tag{1}\]

where \(\mathcal{N}(i)\) is the neighbour nodes of \(w_{i}\) in the AMR graph, \(E(i,j)\) is the edge type between \(w_{i}\) and \(w_{j}\), \(g_{ij}\) is the gate following Liu et al. (2018a), \(k\) represents GCN layer number, and \(f\) is the Sigmoid function. \({\mathbf{W}}\) and \({\mathbf{b}}\) denote parameters of neural layers in this paper. We take the hidden states of the last GCN layer for each word as the common-space representation \({\mathbf{w}}^{\mathbb{C}}\), where \(\mathbb{C}\) stands for the common (multimedia) embedding space. For each entity \(t\), we obtain its representation \({\mathbf{t}}^{\mathbb{C}}\) by averaging the embeddings of its tokens.

**Event and Argument Classifier:** We classify each word \(w\) into event types \(y_{e}\)6 and classify each entity \(t\) into argument role \(y_{a}\):

Footnote 6: We use BIO tag schema to decide trigger word boundary, i.e., adding prefix _B_- to the type label to mark the beginning of a trigger, _I_- for inside, and \(O\) for none.

\[P(y_{e}|w) =\frac{\exp\left({\mathbf{W}}_{e}{\mathbf{w}}^{\mathbb{C}}+{\mathbf{b}}_{e} \right)}{\sum_{e^{\prime}}\exp\left({\mathbf{W}}_{e^{\prime}}{\mathbf{w}}^{\mathbb{ C}}+{\mathbf{b}}_{e^{\prime}}\right)}, \tag{2}\] \[P(y_{a}|t) =\frac{\exp({\mathbf{W}}_{a}[{\mathbf{t}}^{\mathbb{C}};{\mathbf{w}}^{ \mathbb{C}}]+{\mathbf{b}}_{a})}{\sum_{a^{\prime}}\exp({\mathbf{W}}_{a^{\prime}}[{\mathbf{ t}}^{\mathbb{C}};{\mathbf{w}}^{\mathbb{C}}]+{\mathbf{b}}_{a^{\prime}})}.\]

We take ground truth text entity mentions as input following Ji and Grishman (2008) during training, and obtain testing entity mentions using a named entity extractor Lin et al. (2019).

### Image Event Extraction

**Image Structured Representation:** To obtain image structures similar to AMR graphs, and inspired by _situation recognition_Yatskar et al. (2016), we represent each image with a _situation graph_, that is a star-shaped graph as shown in Figure 4, where the central node is labeled as a verb \(v\) (e.g., _destroying_), and the neighbor nodes are arguments labeled as \(\{(n,r)\}\), where \(n\) is a noun (e.g., _ship_) derived from WordNet synsets Miller (1995) to indicate the entity type, and \(r\) indicates the role (e.g., _item_) played by the entity in the event, based on FrameNet Fillmore et al. (2003). We develop two methods to construct situation graphs from images and train them using the imSitu dataset Yatskar et al. (2016) as follows.

**(1) Object-based Graph:** Similar to extracting entities to get candidate arguments, we employ the

Figure 3: Approach overview. During training (left), we jointly train three tasks to establish a cross-media structured embedding space. During test (right), we jointly extract events and arguments from multimedia articles.

 

most similar task in CV, object detection, and obtain the object bounding boxes detected by a Faster R-CNN (Ren et al., 2015) model trained on Open Images (Kuznetsova et al., 2018) with 600 object types ( classes).We employ a VGG-16 CNN (Simonyan and Zisserman, 2014) to extract visual features of an image \({\mathbf{m}}\) and and another VGG-16 to encode the bounding boxes \(\{{\mathbf{o}_{i}}\}\). Then we apply a Multi-Layer Perceptron (MLP) to predict a verb embedding from \({\mathbf{m}}\) and another MLP to predict a noun embedding for each \({\mathbf{o}_{i}}\).

\[{\mathbf{\hat{m}}}=\mathrm{MLP_{m}}({\mathbf{m}})~{},~{}{\mathbf{\dot{o}_{i}}}=\mathrm{ MLP_{o}}({\mathbf{o}_{i}}).\]

We compare the predicted verb embedding to all verbs \(v\) in the imSitu taxonomy in order to classify the verb, and similarly compare each predicted noun embedding to all imSitu nouns \(n\) which results in probability distributions:

\[P(v|m) =\frac{\exp{({\mathbf{\dot{m}}}{\mathbf{v}})}}{\sum_{v^{\prime}}\exp{({\bm {\ddot{m}}}{\mathbf{v}^{\prime}})}},\] \[P(n|o_{i}) =\frac{\exp({\mathbf{\dot{o}_{i}}{\mathbf{n}}})}{\sum_{n^{\prime}}\exp({ \mathbf{\dot{o}_{i}}{\mathbf{n}^{\prime}}})},\]

where \({\mathbf{v}}\) and \({\mathbf{n}}\) are word embeddings initialized with GloVE (Pennington et al., 2014). We use another MLP with one hidden layer followed by Softmax (\(\sigma\)) to classify role \(r_{i}\) for each object \(o_{i}\):

\[P(r_{i}|o_{i})=\sigma\big{(}\mathrm{MLP_{r}}({\mathbf{\dot{o}_{i}}})\big{)}.\]

Given verb \(v^{*}\) and role-noun \((r_{i}^{*},n_{i}^{*})\) annotations for an image (from the imSitu corpus), we define the situation loss functions:

\[\mathcal{L}_{v} =-\log P(v^{*}|m),\] \[\mathcal{L}_{r} =-\log(P(r_{i}^{*}|o_{i})+P(n_{i}^{*}|o_{i})).\]

**(2) Attention-based Graph:** State-of-the-art object detection methods only cover a limited set of object types, such as 600 types defined in Open Images. Many salient objects such as _bomb_, _stone_ and _stretcher_ are not covered in these ontologies. Hence, we propose an open-vocabulary alternative to the object-based graph construction model. To this end, we construct a role-driven attention graph, where each argument node is derived by a spatially distributed attention (heatmap) conditioned on a role \(r\). More specifically, we use a VGG-16 CNN to extract a \(7\times 7\) convolutional feature map for each image \(m\), which can be regarded as attention _keys_\({\mathbf{k}_{i}}\) for \(7\times 7\) local regions. Next, for each role \(r\) defined in the situation recognition ontology (e.g., _agent_), we build an attention _query_ vector \({\mathbf{q}_{r}}\) by concatenating role embedding \({\mathbf{r}}\) with the image feature \({\mathbf{m}}\) as context and apply a fully connected layer:

\[{\mathbf{q}_{r}}={\mathbf{W}_{q}}[{\mathbf{r}};{\mathbf{m}}]+{\mathbf{b}_{q}}.\]

Then, we compute the dot product of each query with all keys, followed by Softmax, which forms a heatmap \({\mathbf{h}}\) on the image, i.e.,

\[h_{i}=\frac{\exp({\mathbf{q}_{r}}{\mathbf{k}_{i}})}{\sum_{j\in 7\times 7}\exp({\mathbf{ q}_{r}}{\mathbf{k}_{j}})}.\]

Figure 4: Multimedia structured common space construction. Red pixels stands for attention heatmap.

 

We use the heatmap to obtain a weighted average of the feature map to represent the argument \(o_{r}\) of each role \(r\) in the visual space:

\[\mathbf{o}_{r}=\sum_{i}h_{i}\mathbf{m}_{i}.\]

Similar to the object-based model, we embed \(\mathbf{o}_{r}\) to \(\hat{\mathbf{o}}_{r}\), compare it to the imSitu noun embeddings to define a distribution, and define a classification loss function. The verb embedding \(\hat{\mathbf{m}}\) and the verb prediction probability \(P(v|m)\) and loss are defined in the same way as in the object-based method.

**Event and Argument Classifier:** We use either the object-based or attention-based formulation and pre-train it on the imSitu dataset Yatskar et al. (2016). Then we apply a GCN to obtain the structured embedding of each node in the common space, similar to Equation 1. This yields \(\mathbf{m}^{\mathbb{C}}\) and \(\mathbf{o}_{i}^{\mathbb{C}}\). We use the same classifiers as defined in Equation 2 to classify each visual event and argument using the common space embedding:

\[\begin{split} P(y_{e}|m)&=\frac{\exp(\mathbf{W}_{e}\mathbf{ m}^{\mathbb{C}}+\mathbf{b}_{e})}{\sum_{e^{\prime}}\exp(\mathbf{W}_{e^{\prime}}\mathbf{m}^{ \mathbb{C}}+\mathbf{b}_{e^{\prime}})},\\ P(y_{a}|o)&=\frac{\exp(\mathbf{W}_{a}[\mathbf{o}^{\mathbb{ C}};\mathbf{m}^{\mathbb{C}}]+\mathbf{b}_{a})}{\sum_{a^{\prime}}\exp(\mathbf{W}_{a^{\prime}}[ \mathbf{o}^{\mathbb{C}};\mathbf{m}^{\mathbb{C}}]+\mathbf{b}_{a^{\prime}})}.\end{split} \tag{3}\]

### Cross-Media Joint Training

In order to make the event and argument classifier shared across modalities, the image and text graph should be encoded to the same space. However, it is extremely costly to obtain the parallel text and image event annotation. Hence, we use event and argument annotations in separate modalities (i.e., ACE and imSitu datasets) to train classifiers, and simultaneously use VOA news image and caption pairs to align the two modalities. To this end, we learn to embed the nodes of each image graph close to the nodes of the corresponding caption graph, and far from those in irrelevant caption graphs. Since there is no ground truth alignment between the image nodes and caption nodes, we use image and caption pairs for weakly supervised training, to learn a soft alignment from each words to image objects and vice versa.

\[\alpha_{ij}=\frac{\exp{(\mathbf{w}_{i}^{\mathbb{C}}\mathbf{o}_{j}^{\mathbb{C}})}}{\sum _{j^{\prime}}\exp{(\mathbf{w}_{i}^{\mathbb{C}}\mathbf{o}_{j^{\prime}}^{\mathbb{C}})}} ,\beta_{ji}=\frac{\exp{(\mathbf{w}_{i}^{\mathbb{C}}\mathbf{o}_{j}^{\mathbb{C}})}}{\sum _{i^{\prime}}\exp{(\mathbf{w}_{i^{\prime}}^{\mathbb{C}}\mathbf{o}_{j}^{\mathbb{C}})}},\]

where \(w_{i}\) indicates the \(i^{th}\) word in caption sentence \(s\) and \(o_{j}\) represents the \(j^{th}\) object of image \(m\). Then, we compute a weighted average of softly aligned nodes for each node in other modality, i.e.,

\[\mathbf{w}_{i}^{\prime}=\sum_{j}\alpha_{ij}\mathbf{o}_{j}^{\mathbb{C}}~{},~{}\mathbf{o}_{ j}^{\prime}=\sum_{i}\beta_{ji}\mathbf{w}_{i}^{\mathbb{C}}. \tag{4}\]

We define the alignment cost of the image-caption pair as the Euclidean distance between each node to its aligned representation,

\[\langle s,m\rangle=\sum_{i}||\mathbf{w}_{i}-\mathbf{w}_{i}^{\prime}||_{2}^{2}+\sum_{j} ||\mathbf{o}_{j}-\mathbf{o}_{j}^{\prime}||_{2}^{2}\]

We use a triplet loss to pull relevant image-caption pairs close while pushing irrelevant ones apart:

\[\mathcal{L}_{c}=\max(0,1+\langle s,m\rangle-\langle s,m^{-}\rangle),\]

where \(m^{-}\) is a randomly sampled negative image that does not match \(s\). Note that in order to learn the alignment between the image and the trigger word, we treat the image as a special object when learning cross-media alignment.

The common space enables the event and argument classifiers to share weights across modalities, and be trained jointly on the ACE and imSitu datasets, by minimizing the following objective functions:

\[\begin{split}\mathcal{L}_{e}&=-\sum_{w}\log P(y_{ e}|w)-\sum_{m}\log P(y_{e}|m),\\ \mathcal{L}_{a}&=-\sum_{t}\log P(y_{a}|t)-\sum_{o} \log P(y_{a}|o),\end{split}\]

All tasks are jointly optimized:

\[\mathcal{L}=\mathcal{L}_{v}+\mathcal{L}_{r}+\mathcal{L}_{e}+\mathcal{L}_{a}+ \mathcal{L}_{c}\]

### Cross-Media Joint Inference

In the test phase, our method takes a multimedia document with sentences \(S=\{s_{1},s_{2},\dots\}\) and images \(M=\{m_{1},m_{2},\dots,\}\) as input. We first generate the structured common embedding for each sentence and each image, and then compute pairwise similarities \(\langle s,m\rangle\). We pair each sentence \(s\) with the closest image \(m\), and aggregate the features of each word of \(s\) with the aligned representation from \(m\) by weighted averaging:

\[\mathbf{w}_{i}^{\prime\prime}=(1-\gamma)\mathbf{w}_{i}+\gamma\mathbf{w}_{i}^{\prime}, \tag{5}\]

where \(\gamma=\exp(-\langle s,m\rangle)\) and \(\mathbf{w}_{i}^{\prime}\) is derived from \(m\) using Equation 4. We use \(\mathbf{w}_{i}^{\prime\prime}\) to classify each 

word into an event type and to classify each entity into a role with multimedia classifiers in Equation 2. To this end, we define \({\mathbf{t}^{\prime\prime}_{i}}\) similar to \({\mathbf{w}^{\prime\prime}_{i}}\) but using \({\mathbf{t}_{i}}\) and \({\mathbf{t}^{\prime}_{i}}\). Similarly, for each image \(m\) we find the closest sentence \(s\), compute the aggregated multimedia features \({\mathbf{m}^{\prime\prime}}\) and \({\mathbf{o}^{\prime\prime}_{i}}\), and feed into the shared classifiers (Equation 3) to predict visual event and argument roles. Finally, we core-fer the cross-media events of the same event type if the similarity \(\langle s,m\rangle\) is higher than a threshold.

## 4 Experiments

### Evaluation Setting

**Evaluation Metrics** We conduct evaluation on text-only, image-only, and multimedia event mentions in M\({}^{2}\)E\({}^{2}\) dataset in Section 2.2. We adopt the traditional event extraction measures, i.e., _Precision_, _Recall_ and \(F_{1}\). For text-only event mentions, we follow Ji and Grishman (2008); Li et al. (2013): a textual event mention is correct if its event type and trigger offsets match a reference trigger; and a textual event argument is correct if its event type, offsets, and role label match a reference argument. We make a similar definition for image-only event mentions: a visual event mention is correct if its event type and image match a reference visual event mention; and a visual event argument is correct if its event type, localization, and role label match a reference argument. A visual argument is correctly localized if the Intersection over Union (IoU) of the predicted bounding box with the ground truth bounding box is over 0.5. Finally, we define a multimedia event mention to be correct if its event type and trigger offsets (or the image) match the reference trigger (or the reference image). The arguments of multimedia events are either textual or visual arguments, and are evaluated accordingly. To generate bounding boxes for the attention-based model, we threshold the heatmap using the adaptive value of \(0.75*p\), where \(p\) is the peak value of the heatmap. Then we compute the tightest bounding box that encloses all of the thresholded region. Examples are shown in Figure 7 and Figure 8.

**Baselines** The baselines include: (1) **Text-only** models: We use the state-of-the-art model JMEE Liu et al. (2018a) and GAIL Zhang et al. (2019) for comparison. We also evaluate the effectiveness of cross media joint training by including a version of our model trained only on ACE, denoted as WASE\({}^{\mathbb{T}}\). (2) **Image-only** models: Since we are the first to extract newsworthy events, and the most similar work _situation recognition_ can not localize arguments in images, we use our model trained only on image corpus as baselines. Our visual branch has two versions, object-based and attention-based, denoted as WASE\({}^{\mathbb{I}}_{\text{obj}}\) and WASE\({}^{\mathbb{I}}_{\text{att}}\). (3) **Multimedia** models: To show the effectiveness of structured embedding, we include a baseline by removing the text and image GCNs from our model, which is denoted as Flat. The Flat baseline ignores edges and treats images and sentences as sets of vectors. We also compare to the state-of-the-art cross-media common representation model, Contrastive Visual Semantic Embedding VSE-C Shi et al. (2018), by training it the same way as WASE.

**Parameter Settings** The common space dimension is \(300\). The dimension is \(512\) for image position embedding and feature map, and \(50\) for word position embedding, entity type embedding, and POS tag embedding. The layer of GCN is \(3\).

\begin{table}
\begin{tabular}{|c|l|c c c|c c c|c c c|c c c|c c c|c c c|} \hline  & & & \multicolumn{6}{c|}{**Text-Only Evaluation**} & \multicolumn{6}{c|}{**Image-Only Evaluation**} & \multicolumn{6}{c|}{**Multimedia Evaluation**} \\ \cline{3-14}  & & \multirow{2}{*}{**Model**} & \multicolumn{3}{c|}{**Event Menton**} & \multicolumn{3}{c|}{**Argument Role**} & \multicolumn{3}{c|}{**Event Menton**} & \multicolumn{3}{c|}{**Argument Role**} & \multicolumn{3}{c|}{**Event Menton**} & \multicolumn{3}{c|}{**Argument Role**} \\ \cline{3-14}  & & & \(P\) & \(R\) & \(F_{1}\) & \(P\) & \(R\) & \(F_{1}\) & \(P\) & \(R\) & \(F_{1}\) & \(P\) & \(R\) & \(F_{1}\) & \(P\) & \(R\) & \(F_{1}\) & \(P\) & \(R\) & \(F_{1}\) \\ \cline{3-14}  & & JMEE & 42.5 & 58.2 & 48.7 & 22.9 & 28.3 & 25.3 & - & - & - & - & - & - & 42.1 & 34.6 & 38.1 & 21.1 & 12.6 & 15.8 \\  & GAIL & 43.4 & 53.5 & 47.9 & 23.6 & 29.2 & 26.1 & - & - & - & - & - & - & 44.0 & 32.4 & 37.3 & 22.7 & 12.8 & 16.4 \\  & WASE\({}^{\mathbb{T}}\) & 42.3 & 58.4 & 48.2 & 21.4 & 30.1 & 24.9 & - & - & - & - & - & - & 41.2 & 33.1 & 36.7 & 20.1 & 13.0 & 15.7 \\ \hline \multirow{3}{*}{**Baseline**} & WASE\({}^{\mathbb{M}}\) & ant & - & - & - & - & - & 29.7 & 61.9 & 40.1 & 9.1 & 10.2 & 9.6 & 28.3 & 23.0 & 25.4 & 2.9 & 6.1 & 3.8 \\  & WASE\({}^{\mathbb{S}}_{\text{obj}}\) & - & - & - & - & - & 28.6 & 59.2 & 38.7 & 13.3 & 9.8 & 11.2 & 26.1 & 22.4 & 24.1 & 4.7 & 5.0 & 4.9 \\ \cline{1-1} \cline{2-14}  & VSE-C & 33.5 & 47.8 & 39.4 & 16.6 & 24.7 & 19.8 & 30.3 & 48.9 & 26.4 & 5.6 & 6.1 & 5.7 & 33.3 & 48.2 & 39.3 & 11.1 & 14.9 & 12.8 \\  & Flata\({}^{\mathbb{N}}_{\text{att}}\) & 34.2 & 63.2 & 44.4 & 20.1 & 27.1 & 23.1 & 27.1 & 57.3 & 36.7 & 4.3 & 8.9 & 5.8 & 33.9 & 59.8 & 42.2 & 12.9 & 17.6 & 14.9 \\  & Flata\({}^{\mathbb{N}}_{\text{att}}\) & 38.3 & 57.9 & 46.1 & 21.8 & 26.6 & 24.0 & 26.4 & 55.8 & 35.8 & 9.1 & 6.5 & 7.6 & 34.1 & 56.4 & 42.5 & 16.3 & 15.9 & 16.1 \\  & WASE\({}^{\mathbb{S}}_{\text{att}}\) & 37.6 & 66.8 & 48.1 & 27.5 & 33.2 & **30.1** & 32.3 & 63.4 & 42.8 & 9.7 & 11.1 & 10.3 & 38.2 & 67.1 & 49.1 & 18.6 & 21.6 & **19.9** \\  & WASE\({}^{\mathbb{S}}_{\text{obj}}\) & 42.8 & 61.9 & **50.6** & 23.5 & 30.3 & 26.4 & 43.1 & 59.2 & **49.9** & 14.5 & 10.1 & **11.9** & 43.0 & 62.1 & **50.8** & 19.5 & 18.9 & 19.2 \\ \hline \end{tabular}
\end{table}
Table 3: Event and argument extraction results (%). We compare three categories of baselines in three evaluation settings. The main contribution of the paper is joint training and joint inference on multimedia data (bottom right).

 



### Quantitative Performance

As shown in Table 3, our complete methods (WASE\({}_{\text{att}}\) and WASE\({}_{\text{obj}}\)) outperform all baselines in the three evaluation settings in terms of F\({}_{1}\). The comparison with other multimedia models demonstrates the effectiveness of our model architecture and training strategy. The advantage of structured embedding is shown by the better performance over the flat baseline. Our model outperforms its text-only and image-only variants on multimedia events, showing the inadequacy of single-modal information for complex news understanding. Furthermore, our model achieves better performance on text-only and image-only events, which demonstrates the effectiveness of multimedia training framework in knowledge transfer between modalities.

WASE\({}_{\text{obj}}\) and WASE\({}_{\text{att}}\), are both superior to the state of the art and each has its own advantages. WASE\({}_{\text{obj}}\) predicts more accurate bounding boxes since it is based on a Faster R-CNN pretrained on bounding box annotations, resulting in a higher argument precision. While WASE\({}_{\text{att}}\) achieves a higher argument recall as it is not limited by the predefined object classes of the Faster R-CNN.

Furthermore, to evaluate the cross-media event coreference performance, we pair textual and visual event mentions in the same document, and calculate _Precision_, _Recall_ and \(F_{1}\) to compare with ground truth event mention pairs7. As shown in Table 4, WASE\({}_{\text{obj}}\) outperforms all multimedia embedding models, as well as the rule-based baseline using event type matching. This demonstrates the effectiveness of our cross-media soft alignment.

Footnote 7: We do not use coreference clustering metrics because we only focus on mention-level cross-media event coreference instead of the full coreference in all documents.

### Qualitative Analysis

Our cross-media joint training approach successfully boosts both event extraction and argument role labeling performance. For example, in Figure 5 (a), the text-only model can not extract _JusticeArrest_ event, but the joint model can use the image as background to detect the event type. In Figure 5 (b), the image-only model detects the image as _Conflict.Demonstration_, but the sentences in the same document help our model not to label it as _Conflict.Demonstration_. Compared with multimedia flat embedding in Figure 6, WASE can learn structures such as _Artifact_ is on top of _Vehicle_, and the person in the middle of _Justice.Arrest_ is _Entity_ instead of _Agent_.

### Remaining Challenges

One of the biggest challenges in M2E2is localizing arguments in images. Object-based models suffer from the limited object types. Attention-based method is not able to precisely localize the objects for each argument, since there is no supervision on attention extraction during training. For example, in Figure 7, the _Entity_ argument in the _Conflict.Demonstrate_ event is correctly predicted as _troops_, but its localization is incorrect because _Place_ argument share similar attention. When one argument targets at too many instances, attention heatmaps tend to lose focus and cover the whole image, as shown in Figure 8.

## 5 Related Work

**Text Event Extraction** Text event extraction has been extensively studied for general news do

Figure 5: Image helps textual event extraction, and surrounding sentence helps visual event extraction.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Model** & \(P\) (\%) & \(R\) (\%) & \(F_{1}\) (\%) \\ \hline rule\_based & 10.1 & 100 & 18.2 \\ VSE & 31.2 & 74.5 & 44.0 \\ Flat\({}_{\text{att}}\) & 33.1 & 73.5 & 45.6 \\ Flat\({}_{\text{obj}}\) & 34.3 & 76.4 & 47.3 \\ \hline WASE\({}_{\text{att}}\) & 39.5 & 73.5 & 51.5 \\ WASE\({}_{\text{obj}}\) & 40.1 & 75.4 & 52.4 \\ \hline \end{tabular}
\end{table}
Table 4: Cross-media event coreference performance.

Figure 6: Comparison with multimedia flat embedding.



main (Ji and Grishman, 2008; Liao and Grishman, 2011; Huang and Riloff, 2012; Li et al., 2013; Chen et al., 2015; Nguyen et al., 2016; Hong et al., 2018; Liu et al., 2018b; Chen et al., 2018; Zhang et al., 2019; Liu et al., 2018a; Wang et al., 2019; Yang et al., 2019; Wadden et al., 2019). Multimedia features has been proven to effectively improve text event extraction (Zhang et al., 2017).

**Visual Event Extraction** "Events" in NLP usually refer to complex events that involve multiple entities in a large span of time (e.g. protest), while in CV Chang et al. (2016); Zhang et al. (2007); Ma et al. (2017) events are less complex single-entity activities (e.g. washing dishes) or actions (e.g. jumping). Visual event ontologies focus on daily life domains, such as "dogshow" and "wedding ceremony" Perera et al. (2012). Moreover, most efforts ignore the structure of events including arguments. There are a few methods that aim to localize the agent Gu et al. (2018); Li et al. (2018); Duarte et al. (2018), or classify the recipient Sigurdsson et al. (2016); Kato et al. (2018); Wu et al. (2019) of events, but neither detects the complete set of arguments for an event. The most similar to our work is Situation Recognition (SR) Yatskar et al. (2016); Mallya and Lazebnik (2017) which predicts an event and multiple arguments from an input image, but does not localize the arguments. We use SR as an auxiliary task for training our visual branch, but exploit object detection and attention to enable localization of arguments. Silberer and Pinkal redefine the problem of visual argument role labeling with event types and bounding boxes as input. Different from their work, we extend the problem scope to including event identification and coreference, and further advance argument localization by proposing an attention framework which does not require bounding boxes for training nor testing.

**Multimedia Representation** Multimedia common representation has attracted much attention recently Toselli et al. (2007); Weegar et al. (2015); Hewitt et al. (2018); Chen et al. (2019); Liu et al. (2019); Su et al. (2019); Sarafianos et al. (2019); Sun et al. (2019); Tan and Bansal (2019); Li et al. (2019); Lu et al. (2019); Sun et al. (2019); Rahman et al. (2019); Su et al. (2019). However, previous methods focus on aligning images with their captions, or regions with words and entities, but ignore structure and semantic roles. UniVSE Wu et al. (2019) incorporates entity attributes and relations into cross-media alignment, but does not capture graph-level structures of images or text.

## 6 Conclusions and Future Work

In this paper we propose a new task of multimedia event extraction and setup a new benchmark. We also develop a novel multimedia structured common space construction method to take advantage of the existing image-caption pairs and single-modal annotated data for weakly supervised training. Experiments demonstrate its effectiveness as a new step towards semantic understanding of events in multimedia data. In the future, we aim to extend our framework to extract events from videos, and make it scalable to new event types. We plan to expand our annotations by including event types from other text event ontologies, as well as new event types not in existing text ontologies. We will also apply our extraction results to downstream applications including cross-media event inference, timeline generation, etc.

## Acknowledgement

This research is based upon work supported in part by U.S. DARPA AIDA Program No. FA8750-18-2-0014 and U.S. DARPA KAIROS Program No. FA8750-19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Figure 8: Attention heatmaps lose focus due to large instance candidate number.

Figure 7: Argument labeling error examples: correct entity name but wrong localization.

 



## References

* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 178-186.
* Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 961-970.
* Chang et al. (2016) Xiaojun Chang, Zhigang Ma, Yi Yang, Zhiqiang Zeng, and Alexander G Hauptmann. 2016. Bilevel semantic representation analysis for multimedia event detection. _IEEE transactions on cybernetics_, 47(5):1180-1197.
* Chen et al. (2019) Yen-Chun Chen, Linijie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-text representations. _arXiv preprint arXiv:1909.11740_.
* Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In _Proc. ACL-IJCNLP2015_.
* Chen et al. (2018) Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yantao Jia. 2018. Collective event detection via a hierarchical and bias tagging networks with gated multilevel attention mechanisms. In _Proc. EMNLP2018_.
* Duarte et al. (2018) Kevin Duarte, Yogesh Rawat, and Mubarak Shah. 2018. Videocapsulenet: A simplified network for action detection. In _Advances in Neural Information Processing Systems_, pages 7610-7619.
* Faghri et al. (2017) Fardash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives.
* Fillmore et al. (2003) Charles J Fillmore, Christopher R Johnson, and Miram RL Petruck. 2003. Background to framenet. _International journal of lexicography_, 16(3):235-250.
* Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In _2013 IEEE international conference on acoustics, speech and signal processing_, pages 6645-6649. IEEE.
* Gu et al. (2018) Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6047-6056.
* Hewitt et al. (2018) John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch. 2018. Learning translations via images with a massively multilingual image dataset. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2566-2576.
* Hong et al. (2018) Yu Hong, Wenxuan Zhou, jingli zhang jingli, Guodong Zhou, and Qiaoming Zhu. 2018. Self-regulation: Employing a generative adversarial network to improve event detection. In _Proc. ACL2018_.
* Huang and Rikoff (2012) Ruihong Huang and Ellen Rikoff. 2012. Bootstrapped training of event extraction classifiers. In _Proc. EACL2012_.
* Ji and Grishman (2008) Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In _Proceedings of ACL-08: HLT_, pages 254-262.
* Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 3128-3137.
* Kato et al. (2018) Keizo Kato, Yin Li, and Abhinav Gupta. 2018. Compositional learning for human object interaction. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 234-251.
* Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. _arXiv preprint arXiv:1609.02907_.
* Kuznetsova et al. (2018) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. 2018. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. _arXiv preprint arXiv:1811.00982_.
* Li et al. (2018) Dong Li, Zhaofan Qiu, Qi Dai, Ting Yao, and Tao Mei. 2018. Recurrent tubelet proposal and recognition networks for action detection. In _Proceedings of the European conference on computer vision (ECCV)_, pages 303-318.
* Li et al. (2019a) Gen Li, Nan Duan, Yuejian Fang, Davin Jiang, and Ming Zhou. 2019a. Unicoder-vl: A universal encoder for vision and language by cross-modal pretraining. _arXiv preprint arXiv:1908.06066_.
* Li et al. (2019b) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019b. Visualbert: A simple and performant baseline for vision and language. _arXiv preprint arXiv:1908.03557_.
* Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In _Proc. ACL2013_

* Liao and Grishman (2011) Shasha Liao and Ralph Grishman. 2011. Acquiring topic features to improve event extraction: in pre-selected and balanced collections. In _Proc. RANLP2011_.
* Lin et al. (2019) Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In _Proc. The 57th Annual Meeting of the Association for Computational Linguistics (ACL2019)_.
* Liu et al. (2019) Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, and Yongdong Zhang. 2019. Focus your attention: A bidirectional focal attention network for image-text matching. In _Proceedings of the 27th ACM International Conference on Multimedia_, pages 3-11. ACM.
* Liu et al. (2018a) Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018a. Jointly multiple events extraction via attention-based graph information aggregation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1247-1256.
* Liu et al. (2018b) Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018b. Jointly multiple events extraction via attention-based graph information aggregation. In _Proc. EMNLP2018_.
* Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlche-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_, pages 13-23.
* Ma et al. (2017) Zhigang Ma, Xiaojun Chang, Zhongwen Xu, Nicu Sebe, and Alexander G Hauptmann. 2017. Joint attributes and event analysis for multimedia event detection. _IEEE transactions on neural networks and learning systems_, 29(7):2921-2930.
* Mallya and Lazebnik (2017) Arun Mallya and Svetlana Lazebnik. 2017. Recurrent models for situation recognition. In _Proceedings of the IEEE International Conference on Computer Vision_, pages 455-463.
* Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Association for Computational Linguistics (ACL) System Demonstrations_, pages 55-60.
* Miller (1995) George A Miller. 1995. Wordnet: a lexical database for english. _Communications of the ACM_, 38(11):39-41.
* Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In _Proc. NAACL-HLT2016_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Perera et al. (2012) AG Amitha Pereira, Sangmin Oh, P Megha, Tianyang Ma, Anthony Hoogs, Arash Vahdat, Kevin Cannons, Greg Mori, Scott Mccloskey, Ben Miller, et al. 2012. Trecvid 2012 genie: Multimedia event detection and recounting. In _In TRECVID Workshop_. Citeseer.
* Rahman et al. (2019) Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Morency, and Mohammed Ehsan Hoque. 2019. M-bert: Injecting multimodal information in the bert structure. _arXiv preprint arXiv:1908.05787_.
* Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In _Advances in neural information processing systems_, pages 91-99.
* Sarafianos et al. (2019) Nikolaos Sarafianos, Xiang Xu, and Ioannis A. Kakadiaris. 2019. Adversarial representation learning for text-to-image matching. In _The IEEE International Conference on Computer Vision (ICCV)_.
* Shi et al. (2018) Haoyue Shi, Jiayuan Mao, Tete Xiao, Yuning Jiang, and Jian Sun. 2018. Learning visually-grounded semantics from contrastive adversarial samples. _arXiv preprint arXiv:1806.10348_.
* Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In _European Conference on Computer Vision_, pages 510-526. Springer.
* Silberer and Pinkal (2018) Carina Silberer and Manfred Pinkal. 2018. Grounding semantic roles in images. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2616-2626.
* Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. _arXiv preprint arXiv:1409.1556_.
* Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. _arXiv preprint arXiv:1212.0402_.
* Stephens (1998) Mitchell Stephens. 1998. _The Rise of the Image, The Fall of the Word_. New York: Oxford University Press.
* Su et al. (2019a) Shupeng Su, Zhisheng Zhong, and Chao Zhang. 2019a. Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval. In _The IEEE International Conference on Computer Vision (ICCV)_.
* Su et al. (2019b) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019b. Vl-bert: Pre-training of generic visual-linguistic representations. _arXiv preprint arXiv:1908.08530_ 

Cited by: SS1.
* C. Sun, F. Baradel, K. Murphy, and C. Schmid (2019)Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743. Cited by: SS1.
* A. Myers, C. Vondrick, K. Murphy, and C. Schmid (2019)Videobert: a joint model for video and language representation learning. In The IEEE International Conference on Computer Vision (ICCV), Cited by: SS1.
* H. Tan and M. Bansal (2019)LXmert: learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Cited by: SS1.
* A. Ioselli, V. Romero, and E. Vidal (2007)Vietorb based alignment between text images and their transcripts. In Proceedings of the Workshop on Language Technology for Cultural Heritage Data (LaTeCH 2007), pp. 9-16. Cited by: SS1.
* D. Wadden, U. Wennberg, Y. Luan, and H. Hajishirzi (2019)Entity, relation, and event extraction with contextualized span representations. arXiv preprint arXiv:1909.03546. Cited by: SS1.
* C. Walker, S. Strassel, J. Medero, and Kazuaki Maeda (2006)ACE 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia57. Cited by: SS1.
* C. Wang, S. Pradhan, X. Pan, H. Ji, and N. Xue (2016)Camr at semeval-2016 task 8: an extended transition-based amr parser. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), Vol. , pp. 1173-1178. External Links: Document, Link Cited by: SS1.
* C. Wang, N. Xue, and S. Pradhan (2015a)Boosting transition-based amr parsing with refined actions and auxiliary analyzers. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Beijing, China, pp. 857-862. Cited by: SS1.
* C. Wang, N. Xue, and S. Pradhan (2015b)A transition-based algorithm for amr parsing. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, pp. 366-375. Cited by: SS1.
* R. Wang, D. Zhou, and Y. He (2019)Open event extraction from online text using a generative adversarial network. arXiv preprint arXiv:1908.09246. Cited by: SS1.
* R. Weegar, K. Astrom, and P. Nugues (2015)Linking entities across images and text. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pp. 185-193. Cited by: SS1.
* C. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick (2019)Long-term feature banks for detailed video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 284-293. Cited by: SS1.
* H. Wu, J. Mao, Y. Zhang, Y. Jiang, L. Li, W. Sun, and W. Ma (2019)Univsse: robust visual semantic embeddings via structured semantic representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Cited by: SS1.
* S. Yang, D. Feng, L. Qiao, Z. Kan, and D. Li (2019)Exploring pre-trained language models for event extraction and generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 5284-5294. Cited by: SS1.
* M. Yatskar, L. Zettlemoyer, and A. Farhadi (2016)Situation recognition: visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5534-5542. Cited by: SS1.
* G. Ye, Y. Li, H. Xu, D. Liu, and S. Chang (2015)Eventnet: a large scale structured concept library for complex event detection in video. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 471-480. Cited by: SS1.
* T. Zhang, H. Ji, and A. Sil (2019)Joint entity and event extraction with generative adversarial imitation learning. Data Intelligence Vol 1 (2): 99-120. Cited by: SS1.
* T. Zhang, S. Whitehead, H. Zhang, H. Li, J. Ellis, L. Huang, W. Liu, H. Ji, and S. Chang (2017)Improving event extraction via multimodal integration. In Proceedings of the 25th ACM international conference on Multimedia, pp. 270-278. Cited by: SS1.
* Y. Zhang, C. Xu, Y. Rui, J. Wang, and H. Lu (2007)Semantic event extraction from basketball games using multi-modal analysis. In 2007 IEEE International Conference on Multimedia and Expo, pp. 2190-2193. Cited by: SS1.