SlowGuess/ACL_2020_OCR huggingface dataset:
Manling Li $^{1*}$ , Alireza Zareian $^{2*}$ , Qi Zeng $^{1}$ , Spencer Whitehead $^{1}$ , Di Lu $^{3}$ , Heng Ji $^{1}$ , Shih-Fu Chang $^{2}$
<sup>1</sup>University of Illinois at Urbana-Champaign, <sup>2</sup>Columbia University <sup>3</sup>Dataminr {manling2,hengji}@illinois.edu,{az2407,sc250}@columbia.edu
We introduce a new task, MultiMedia Event Extraction  $(\mathbf{M}^2\mathbf{E}^2)$ , which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments.<sup>1</sup> We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to unimodal state-of-the-art methods, our approach achieves  $4.0\%$  and  $9.8\%$  absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state-of-the-art multimedia unstructured representations, we achieve  $8.3\%$  and  $5.0\%$  absolute F-score gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract  $21.4\%$  more event mentions than traditional text-only methods.
Traditional event extraction methods target a single modality, such as text (Wadden et al., 2019), images (Yatskar et al., 2016) or videos (Ye et al., 2015; Caba Heilbron et al., 2015; Soomro et al., 2012). However, the practice of contemporary journalism (Stephens, 1998) distributes news via multimedia. By randomly sampling 100 multimedia news articles from the Voice of America (VOA), we find that  $33\%$  of images in the articles contain visual objects that serve as event arguments and are not mentioned in the text. Take
Figure 1: An example of Multimedia Event Extraction. An event mention and some event arguments (Agent and Person) are extracted from text, while the vehicle arguments can only be extracted from the image.
Figure 1 as an example, we can extract the Agent and Person arguments of the Movement.Transform event from text, but can extract the Vehicle argument only from the image. Nevertheless, event extraction is independently studied in Computer Vision (CV) and Natural Language Processing (NLP), with major differences in task definition, data domain, methodology, and terminology. Motivated by the complementary and holistic nature of multimedia data, we propose MultiMedia Event Extraction  $(\mathbf{M}^2\mathbf{E}^2)$ , a new task that aims to jointly extract events and arguments from multiple modalities. We construct the first benchmark and evaluation dataset for this task, which consists of 245 fully annotated news articles.
We propose the first method, Weakly Aligned Structured Embedding (WASE), for extracting events and arguments from multiple modalities. Complex event structures have not been covered by existing multimedia representation methods (Wu et al., 2019b; Faghri et al., 2017; Karpathy and Fei-Fei, 2015), so we propose to learn a structured multimedia embedding space. More specifically, given a multimedia document, we represent each image or sentence as a graph, where each node represents an event or entity and each
*These authors contributed equally to this work.  
<sup>1</sup>Our data and code are available at http://blender.cs.illinois.edu/software/m2e2
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2557-2568 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
edge represents an argument role. The node and edge embeddings are represented in a multimedia common semantic space, as they are trained to resolve event co-reference across modalities and to match images with relevant sentences. This enables us to jointly classify events and argument roles from both modalities. A major challenge is the lack of multimedia event argument annotations, which are costly to obtain due to the annotation complexity. Therefore, we propose a weakly supervised framework, which takes advantage of annotated uni-modal corpora to separately learn visual and textual event extraction, and uses an image-caption dataset to align the modalities.
We evaluate WASE on the new task of  $\mathrm{M}^2\mathrm{E}^2$ . Compared to the state-of-the-art uni-modal methods and multimedia flat representations, our method significantly outperforms on both event extraction and argument role labeling tasks in all settings. Moreover, it extracts  $21.4\%$  more event mentions than text-only baselines. The training and evaluation are done on heterogeneous data sets from multiple sources, domains and data modalities, demonstrating the scalability and transferability of the proposed model. In summary, this paper makes the following contributions:
- We propose a new task, MultiMedia Event Extraction, and construct the first annotated news dataset as a benchmark to support deep analysis of cross-media events.
- We develop a weakly supervised training framework, which utilizes existing single-modal annotated corpora, and enables joint inference without cross-modal annotation.
- Our proposed method, WASE, is the first to leverage structured representations and graph-based neural networks for multimedia common space embedding.
2 Task Definition
Each input document consists of a set of images  $\mathcal{M} = \{m_1, m_2, \ldots\}$  and a set of sentences  $S = \{s_1, s_2, \ldots\}$ . Each sentence  $s$  can be represented as a sequence of tokens  $s = (w_1, w_2, \ldots)$ , where  $w_i$  is a token from the document vocabulary  $\mathcal{W}$ . The input also includes a set of entities  $\mathcal{T} = \{t_1, t_2, \ldots\}$  extracted from the document text. An entity is an individually unique object in
the real world, such as a person, an organization, a facility, a location, a geopolitical entity, a weapon, or a vehicle. The objective of  $\mathrm{M}^2\mathrm{E}^2$  is twofold:
Event Extraction: Given a multimedia document, extract a set of event mentions, where each event mention  $e$  has a type  $y_{e}$  and is grounded on a text trigger word  $w$  or an image  $m$  or both, i.e.,
$$
e = \left(y _ {e}, \{w, m \}\right).
$$
Note that for an event,  $w$  and  $m$  can both exist, which means the visual event mention and the textual event mention refer to the same event. For example in Figure 1, deploy indicates the same Movement.Transform event as the image. We consider the event  $e$  as text-only event if it only has textual mention  $w$ , and as image-only event if it only contains visual mention  $m$ , and as multimedia event if both  $w$  and  $m$  exist.
Argument Extraction: The second task is to extract a set of arguments of event mention  $e$ . Each argument  $a$  has an argument role type  $y_{a}$ , and is grounded on a text entity  $t$  or an image object  $o$  (represented as a bounding box), or both,
$$
a = \left(y _ {a}, \{t, o \}\right).
$$
The arguments of visual and textual event mentions are merged if they refer to the same real-world event, as shown in Figure 1.
2.2 The  $\mathbf{M}^2\mathbf{E}^2$  Dataset
We define multimedia newsworthy event types by exhaustively mapping between the event ontology in NLP community for the news domain  $(\mathrm{ACE}^2)$  and the event ontology in CV community for general domain (imSitu (Yatskar et al., 2016)). They cover the largest event training resources in each community. Table 1 shows the selected complete intersection, which contains 8 ACE types (i.e.,  $24\%$  of all ACE types), mapped to 98 imSitu types (i.e.,  $20\%$  of all imSitu types). We expand the ACE event role set by adding visual arguments from imSitu, such as instrument, bolded in Table 1. This set encompasses  $52\%$  ACE events in a news corpus, which indicates that the selected eight types are salient in the news domain. We reuse these existing ontologies because they enable us to train event and argument classifiers for both modalities without requiring joint multimedia event annotation as training data.
$^{2}$ https://catalog.ldc.upenn.edu/1dc2006T06
Event Type	Argument Role
MovementTRANSPORT(223|53)	Agent (46|64), Artifact (179|103), Vehicle (24|51), Destination (120|0), Origin (66|0)
ConflictAttack(326|27)	Attacker (192|12), Target (207|19), Instrument (37|15), Place (121|0)
Conflict.Demonstrate(151|69)	Entity (102|184), Police (3|26),In-strument (0|118), Place (86|25)
Justice.ArrestJail(160|56)	Agent (64|119), Person (147|99),Instrument (0|11), Place (43|0)
Contact.PhoneWrite(33|37)	Entity (33|46), Instrument (0|43),Place (8|0)
Contact.Meet (127|79)	Participant (119|321), Place (68|0)
Life.Die(244|64)	Agent (39|0), Instrument (4|2),Victim (165|155), Place (54|0)
Transaction.TransformMoney (33|6)	Giver (19|3), Recipient (19|5), Money (0|8)
We collect 108,693 multimedia news articles from the Voice of America (VOA) website  $^{3}$  2006-2017, covering a wide range of newsworthy topics such as military, economy and health. We select 245 documents as the annotation set based on three criteria: (1) Informativeness: articles with more event mentions; (2) Illustration: articles with more images  $(>4)$ ; (3) Diversity: articles that balance the event type distribution regardless of true frequency. The data statistics are shown in Table 2. Among all of these events, 192 textual event mentions and 203 visual event mentions can be aligned as 309 cross-media event mention pairs. The dataset can be divided into 1,105 text-only event mentions, 188 image-only event mentions, and 395 multimedia event mentions.
Table 1: Event types and argument roles in  $\mathbf{M}^2\mathbf{E}^2$  , with expanded ones in bold. Numbers in parentheses represent the counts of textual and visual events/arguments.
Source	Event Mention	Argument Role
sentence	image	textual	visual	textual	visual
6,167	1,014	1,297	391	1,965	1,429
Table 2:  ${\mathrm{M}}^{2}{\mathrm{E}}^{2}$  data statistics.
We follow the ACE event annotation guidelines (Walker et al., 2006) for textual event and argument annotation, and design an annotation guideline for multimedia events annotation.
One unique challenge in multimedia event annotation is to localize visual arguments in complex scenarios, where images include a crowd of people or a group of object. It is hard to delineate
Union bounding box
Figure 2: Example of bounding boxes.
Instance bounding box
each of them using a bounding box. To solve this problem, we define two types of bounding boxes: (1) union bounding box: for each role, we annotate the smallest bounding box covering all constituents; and (2) instance bounding box: for each role, we annotate a set of bounding boxes, where each box is the smallest region that covers an individual participant (e.g., one person in the crowd), following the VOC2011 Annotation Guidelines<sup>5</sup>. Figure 2 shows an example. Eight NLP and CV researchers complete the annotation work with two independent passes and reach an Inter-Annotator Agreement (IAA) of  $81.2\%$ . Two expert annotators perform adjudication.
3.1 Approach Overview
As shown in Figure 3, the training phase contains three tasks: text event extraction (Section 3.2), visual situation recognition (Section 3.3), and cross-media alignment (Section 3.4). We learn a cross-media shared encoder, a shared event classifier, and a shared argument classifier. In the testing phase (Section 3.5), given a multimedia news article, we encode the sentences and images into the structured common space, and jointly extract textual and visual events and arguments, followed by cross-modal coreference resolution.
3.2 Text Event Extraction
Text Structured Representation: As shown in Figure 4, we choose Abstract Meaning Representation (AMR) (Banarescu et al., 2013) to represent text because it includes a rich set of 150 fine-grained semantic roles. To encode each text sentence, we run the CAMR parser (Wang et al., 2015b,a, 2016) to generate an AMR graph, based on the named entity recognition and part-of-speech (POS) tagging results from Stanford CoreNLP (Manning et al., 2014). To represent each word  $w$  in a sentence  $s$ , we concatenate its
<sup>5</sup>http://hostrobots.ox.ac.uk/pascal/VOC/voc2011/guidelines.html
<sup>3</sup>https://www.voanews.com/
<sup>4</sup>http://blender.cs.illinois.edu/software/
m2e2/ACL2020_M2E2_annotation.pdf
Figure 3: Approach overview. During training (left), we jointly train three tasks to establish a cross-media structured embedding space. During test (right), we jointly extract events and arguments from multimedia articles.
pre-trained GloVe word embedding (Pennington et al., 2014), POS embedding, entity type embedding and position embedding. We then input the word sequence to a bi-directional long short term memory (Bi-LSTM) (Graves et al., 2013) network to encode the word order and get the representation of each word  $\boldsymbol{w}$ . Given the AMR graph, we apply a Graph Convolutional Network (GCN) (Kipf and Welling, 2016) to encode the graph contextual information following (Liu et al., 2018a):
$$
\boldsymbol {w} _ {i} ^ {(k + 1)} = f \left(\sum_ {j \in \mathcal {N} (i)} g _ {i j} ^ {(k)} \left(\boldsymbol {W} _ {E (i, j)} \boldsymbol {w} _ {j} ^ {(k)} + \boldsymbol {b} _ {E (i, j)} ^ {(k)}\right)\right), \tag {1}
$$
where  $\mathcal{N}(i)$  is the neighbour nodes of  $w_{i}$  in the AMR graph,  $E(i,j)$  is the edge type between  $w_{i}$  and  $w_{j}$ ,  $g_{ij}$  is the gate following (Liu et al., 2018a),  $k$  represents GCN layer number, and  $f$  is the Sigmoid function.  $\pmb{W}$  and  $\pmb{b}$  denote parameters of neural layers in this paper. We take the hidden states of the last GCN layer for each word as the common-space representation  $\pmb{w}^{\mathbb{C}}$ , where  $\mathbb{C}$  stands for the common (multimedia) embedding space. For each entity  $t$ , we obtain its representation  $\pmb{t}^{\mathbb{C}}$  by averaging the embeddings of its tokens. Event and Argument Classifier: We classify each word  $w$  into event types  $y_{e}^{6}$  and classify each
entity  $t$  into argument role  $y_{a}$
$$
\begin{array}{l} P (y _ {e} | w) = \frac {\exp \left(\boldsymbol {W} _ {e} \boldsymbol {w} ^ {\mathrm {C}} + \boldsymbol {b} _ {e}\right)}{\sum_ {e ^ {\prime}} \exp \left(\boldsymbol {W} _ {e ^ {\prime}} \boldsymbol {w} ^ {\mathrm {C}} + \boldsymbol {b} _ {e ^ {\prime}}\right)}, \\ P (y _ {a} | t) = \frac {\exp \left(\boldsymbol {W} _ {a} [ \boldsymbol {t} ^ {\mathbb {C}} ; \boldsymbol {w} ^ {\mathbb {C}} ] + \boldsymbol {b} _ {a}\right)}{\sum_ {a ^ {\prime}} \exp \left(\boldsymbol {W} _ {a ^ {\prime}} [ \boldsymbol {t} ^ {\mathbb {C}} ; \boldsymbol {w} ^ {\mathbb {C}} ] + \boldsymbol {b} _ {a ^ {\prime}}\right)}. \tag {2} \\ \end{array}
$$
We take ground truth text entity mentions as input following (Ji and Grishman, 2008) during training, and obtain testing entity mentions using a named entity extractor (Lin et al., 2019).
3.3 Image Event Extraction
Image Structured Representation: To obtain image structures similar to AMR graphs, and inspired by situation recognition (Yatskar et al., 2016), we represent each image with a situation graph, that is a star-shaped graph as shown in Figure 4, where the central node is labeled as a verb  $v$  (e.g., destroying), and the neighbor nodes are arguments labeled as  $\{(n,r)\}$ , where  $n$  is a noun (e.g., ship) derived from WordNet synsets (Miller, 1995) to indicate the entity type, and  $r$  indicates the role (e.g., item) played by the entity in the event, based on FrameNet (Fillmore et al., 2003). We develop two methods to construct situation graphs from images and train them using the im-Situ dataset (Yatskar et al., 2016) as follows.
(1) Object-based Graph: Similar to extracting entities to get candidate arguments, we employ the
<sup>6</sup>We use BIO tag schema to decide trigger word boundary, i.e., adding prefix  $B$ - to the type label to mark the beginning of a trigger,  $I$ - for inside, and  $O$  for none.
Figure 4: Multimedia structured common space construction. Red pixels stands for attention heatmap.
most similar task in CV, object detection, and obtain the object bounding boxes detected by a Faster R-CNN (Ren et al., 2015) model trained on Open Images (Kuznetsova et al., 2018) with 600 object types (classes). We employ a VGG-16 CNN (Simonyan and Zisserman, 2014) to extract visual features of an image  $m$  and another VGG-16 to encode the bounding boxes  $\{o_i\}$ . Then we apply a Multi-Layer Perceptron (MLP) to predict a verb embedding from  $m$  and another MLP to predict a noun embedding for each  $o_i$ .
$$
\hat {\pmb {m}} = \mathrm {M L P} _ {\mathrm {m}} (\pmb {m}), \hat {\pmb {o}} _ {i} = \mathrm {M L P} _ {\mathrm {o}} (\pmb {o} _ {i}).
$$
We compare the predicted verb embedding to all verbs  $v$  in the imSitu taxonomy in order to classify the verb, and similarly compare each predicted noun embedding to all imSitu nouns  $n$  which results in probability distributions:
$$
P (v | m) = \frac {\exp (\hat {m} v)}{\sum_ {v ^ {\prime}} \exp (\hat {m} v ^ {\prime})},
$$
$$
P (n | o _ {i}) = \frac {\exp (\hat {\boldsymbol {o}} _ {i} \boldsymbol {n})}{\sum_ {n ^ {\prime}} \exp (\hat {\boldsymbol {o}} _ {i} \boldsymbol {n} ^ {\prime})},
$$
where  $\pmb{v}$  and  $\pmb{n}$  are word embeddings initialized with GloVE (Pennington et al., 2014). We use another MLP with one hidden layer followed by Softmax  $(\sigma)$  to classify role  $r_i$  for each object  $o_i$ :
$$
P (r _ {i} | o _ {i}) = \sigma \left(\mathrm {M L P} _ {\mathrm {r}} (\hat {o} _ {i})\right).
$$
Given verb  $v^{*}$  and role-noun  $(r_{i}^{*}, n_{i}^{*})$  annotations for an image (from the imSitu corpus), we define
the situation loss functions:
$$
\mathcal {L} _ {v} = - \log P (v ^ {*} | m),
$$
$$
\mathcal {L} _ {r} = - \log \left(P \left(r _ {i} ^ {*} \mid o _ {i}\right) + P \left(n _ {i} ^ {*} \mid o _ {i}\right)\right).
$$
(2) Attention-based Graph: State-of-the-art object detection methods only cover a limited set of object types, such as 600 types defined in Open Images. Many salient objects such as bomb, stone and stretcher are not covered in these ontologies. Hence, we propose an open-vocabulary alternative to the object-based graph construction model. To this end, we construct a role-driven attention graph, where each argument node is derived by a spatially distributed attention (heatmap) conditioned on a role  $r$ . More specifically, we use a VGG-16 CNN to extract a  $7 \times 7$  convolutional feature map for each image  $m$ , which can be regarded as attention keys  $k_{i}$  for  $7 \times 7$  local regions. Next, for each role  $r$  defined in the situation recognition ontology (e.g., agent), we build an attention query vector  $q_{r}$  by concatenating role embedding  $r$  with the image feature  $m$  as context and apply a fully connected layer:
$$
\boldsymbol {q} _ {r} = \boldsymbol {W} _ {q} [ \boldsymbol {r}; \boldsymbol {m} ] + \boldsymbol {b} _ {q}.
$$
Then, we compute the dot product of each query with all keys, followed by Softmax, which forms a heatmap  $h$  on the image, i.e.,
$$
h _ {i} = \frac {\exp (\boldsymbol {q} _ {r} \boldsymbol {k} _ {i})}{\sum_ {j \in 7 \times 7} \exp (\boldsymbol {q} _ {r} \boldsymbol {k} _ {j})}.
$$
We use the heatmap to obtain a weighted average of the feature map to represent the argument  $o_r$  of each role  $r$  in the visual space:
$$
\boldsymbol {o} _ {r} = \sum_ {i} h _ {i} \boldsymbol {m} _ {i}.
$$
Similar to the object-based model, we embed  $\pmb{o}_r$  to  $\hat{o}_r$ , compare it to the imSitu noun embeddings to define a distribution, and define a classification loss function. The verb embedding  $\hat{m}$  and the verb prediction probability  $P(v|m)$  and loss are defined in the same way as in the object-based method.
Event and Argument Classifier: We use either the object-based or attention-based formulation and pre-train it on the imSitu dataset (Yatskar et al., 2016). Then we apply a GCN to obtain the structured embedding of each node in the common space, similar to Equation 1. This yields  $m^{\mathbb{C}}$  and  $o_i^{\mathbb{C}}$ . We use the same classifiers as defined in Equation 2 to classify each visual event and argument using the common space embedding:
$$
P \left(y _ {e} \mid m\right) = \frac {\exp \left(\boldsymbol {W} _ {e} \boldsymbol {m} ^ {\mathbb {C}} + \boldsymbol {b} _ {e}\right)}{\sum_ {e ^ {\prime}} \exp \left(\boldsymbol {W} _ {e ^ {\prime}} \boldsymbol {m} ^ {\mathbb {C}} + \boldsymbol {b} _ {e ^ {\prime}}\right)}, \tag {3}
$$
$$
P (y _ {a} | o) = \frac {\exp (\boldsymbol {W} _ {a} [ o ^ {\mathbb {C}} ; \boldsymbol {m} ^ {\mathbb {C}} ] + \boldsymbol {b} _ {a})}{\sum_ {a ^ {\prime}} \exp (\boldsymbol {W} _ {a ^ {\prime}} [ o ^ {\mathbb {C}} ; \boldsymbol {m} ^ {\mathbb {C}} ] + \boldsymbol {b} _ {a ^ {\prime}})}.
$$
3.4 Cross-Media Joint Training
In order to make the event and argument classifier shared across modalities, the image and text graph should be encoded to the same space. However, it is extremely costly to obtain the parallel text and image event annotation. Hence, we use event and argument annotations in separate modalities (i.e., ACE and imSitu datasets) to train classifiers, and simultaneously use VOA news image and caption pairs to align the two modalities. To this end, we learn to embed the nodes of each image graph close to the nodes of the corresponding caption graph, and far from those in irrelevant caption graphs. Since there is no ground truth alignment between the image nodes and caption nodes, we use image and caption pairs for weakly supervised training, to learn a soft alignment from each words to image objects and vice versa.
$$
\alpha_ {i j} = \frac {\exp \left(\boldsymbol {w} _ {i} ^ {\mathbb {C}} \boldsymbol {o} _ {j} ^ {\mathbb {C}}\right)}{\sum_ {j ^ {\prime}} \exp \left(\boldsymbol {w} _ {i} ^ {\mathbb {C}} \boldsymbol {o} _ {j ^ {\prime}} ^ {\mathbb {C}}\right)}, \beta_ {j i} = \frac {\exp \left(\boldsymbol {w} _ {i} ^ {\mathbb {C}} \boldsymbol {o} _ {j} ^ {\mathbb {C}}\right)}{\sum_ {i ^ {\prime}} \exp \left(\boldsymbol {w} _ {i ^ {\prime}} ^ {\mathbb {C}} \boldsymbol {o} _ {j} ^ {\mathbb {C}}\right)},
$$
where  $w_{i}$  indicates the  $i^{th}$  word in caption sentence  $s$  and  $o_{j}$  represents the  $j^{th}$  object of image
$m$ . Then, we compute a weighted average of softly aligned nodes for each node in other modality, i.e.,
$$
\boldsymbol {w} _ {i} ^ {\prime} = \sum_ {j} \alpha_ {i j} \boldsymbol {o} _ {j} ^ {\mathbb {C}}, \boldsymbol {o} _ {j} ^ {\prime} = \sum_ {i} \beta_ {j i} \boldsymbol {w} _ {i} ^ {\mathbb {C}}. \tag {4}
$$
We define the alignment cost of the image-caption pair as the Euclidean distance between each node to its aligned representation,
$$
\langle s, m \rangle = \sum_ {i} | | \boldsymbol {w} _ {i} - \boldsymbol {w} _ {i} ^ {\prime} | | _ {2} ^ {2} + \sum_ {j} | | \boldsymbol {o} _ {j} - \boldsymbol {o} _ {j} ^ {\prime} | | _ {2} ^ {2}
$$
We use a triplet loss to pull relevant image-caption pairs close while pushing irrelevant ones apart:
$$
\mathcal {L} _ {c} = \max  (0, 1 + \langle s, m \rangle - \langle s, m ^ {-} \rangle),
$$
where  $m^{-}$  is a randomly sampled negative image that does not match  $s$ . Note that in order to learn the alignment between the image and the trigger word, we treat the image as a special object when learning cross-media alignment.
The common space enables the event and argument classifiers to share weights across modalities, and be trained jointly on the ACE and im-Situ datasets, by minimizing the following objective functions:
$$
\mathcal {L} _ {e} = - \sum_ {w} \log P (y _ {e} | w) - \sum_ {m} \log P (y _ {e} | m),
$$
$$
\mathcal {L} _ {a} = - \sum_ {t} \log P (y _ {a} | t) - \sum_ {o} \log P (y _ {a} | o),
$$
All tasks are jointly optimized:
$$
\mathcal {L} = \mathcal {L} _ {v} + \mathcal {L} _ {r} + \mathcal {L} _ {e} + \mathcal {L} _ {a} + \mathcal {L} _ {c}
$$
3.5 Cross-Media Joint Inference