SlowGuess/NAACL_2022_OCR huggingface dataset:
Luis F. Guzman-Nateras, Minh Van Nguyen and Thien Huu Nguyen
{lfguzman,minhny,thien}@cs.uoregon.edu
In this work, we focus on Cross-Lingual Event Detection where a model is trained on data from a source language but its performance is evaluated on data from a second, target, language. Most recent works in this area have harnessed the language-invariant qualities displayed by pre-trained Multi-lingual Language Models. Their performance, however, reveals there is room for improvement as the crosslingual setting entails particular challenges. We employ Adversarial Language Adaptation to train a Language Discriminator to discern between the source and target languages using unlabeled data. The discriminator is trained in an adversarial manner so that the encoder learns to produce refined, language-invariant representations that lead to improved performance. More importantly, we optimize the adversarial training process by only presenting the discriminator with the most informative samples. We base our intuition about what makes a sample informative on two disparate metrics: sample similarity and event presence. Thus, we propose leveraging Optimal Transport as a solution to naturally combine these two distinct information sources into the selection process. Extensive experiments on 8 different language pairs, using 4 languages from unrelated families, show the flexibility and effectiveness of our model that achieves state-of-the-art results.
Event Detection (ED) is an important sub-task within the broader Information Extraction (IE) task. Event detection consists of being able to identify the words, commonly referred to as triggers, that denote the occurrence of events in a sentence, and classify them into a discrete set of event types. For example, in the sentence "Jamie bought a car yesterday.", bought is considered the trigger of a TRANSACTION:TRANSFER-OWNERSHIP<sup>1</sup>
event type. It is a very well studied task in which there have been lots of previous research efforts that have recently been primarily deep learning-based (Nguyen and Grishman, 2015; Chen et al., 2015; Nguyen et al., 2016a,b; Sha et al., 2018; Wadden et al., 2019; Zhang et al., 2019a; Yang et al., 2019; Nguyen and Nguyen, 2019; Zhang et al., 2020; Liu et al., 2020).
Nonetheless, ED remains quite a challenging task as the context in which a trigger occurs can change its corresponding type completely. Furthermore, the same event might also be expressed by entirely different words/phrases. Additionally, the vast majority of the aforementioned efforts are limited to a monolingual setting — performing ED on text belonging to a single language.
Alternatively, Cross-Lingual ED (CLED) proposes the scenario of creating models that effectively perform ED on data belonging to more than one language, which brings about additional challenges. For instance, trigger words present in one language might not exist in another one. An frequent example of this phenomenon are verb conjugations where some tenses only exist in some languages. Accurate verb handling is of particular importance for the ED task as event triggers are usually related to the verbs in a sentence. Some recent work (Majewska et al., 2021) has attempted to address this issue by injecting external verb knowledge into the training process. Another similar problematic issue for CLED are triggers with different meanings that are each distinct words in different languages. For instance, the word "juicio" in Spanish can either mean "judgement" or "trial" in English, depending on the context.
A compelling approach to creating a cross-lingual model is to use transfer learning which carries the performance of a model trained on a source language over onto a second target language. The general idea is leveraging the existing high-quality annotated data available for a high
Event type taken from the ACE05 dataset.
Human Language Technologies, pages 5588 - 5599
resource language to train a model in a way that allows it to learn the language-invariant characteristics of the task at hand, ED in this case, so that it also performs effectively on text from a second language. Prior works on transfer learning for CLED have relied on pre-trained Multilingual Language Models (MLMs), such as multilingual BERT (mBERT) (Devlin et al., 2019), to take advantage of their innate language-invariant qualities. Yet, their performance still shows room for improvement as they sometimes struggle to handle the difficult instances, unique to cross-lingual settings, mentioned earlier. We identify a significant shortcoming of previous CLED efforts in that they do not exploit the abundant supply of unlabeled data: even though MLMs are trained on immense amounts of it, unlabeled data is not used when fine-tuning for the ED task. It is our intuition that by integrating unlabeled target-language data into the training process, the model is exposed to more language context which should help deal with issues such as verb variation and multiple connotations.
As such, we propose making use of Adversarial Language Adaptation (ALA) (Joty et al., 2017; Chen et al., 2018) to train a CLED model. The key idea is to generate language-invariant representations that are not indicative of language but remain informative for the ED task. Unlabeled data from both the source and target languages is used to train a Language Discriminator (LD) network that learns to discern between the two. The adversarial part comes from the fact that the encoder and discriminator are trained with opposing objectives: as the LD becomes better at distinguishing between languages, the encoder learns to generate more language-invariant representations in an attempt to fool the LD. To the best of our knowledge, our work is the first one proposing the use of ALA for the CLED task.
Nonetheless, contrary to past uses of ALA where the same importance is given to all unlabeled samples, we recognize that such course of action is suboptimal as certain samples are bound to be more informative for the discriminator than others. For example, we would like to present the LD with the samples that allow it to learn the fine-grained distinctions between the source and target languages, instead of relying on syntactic differences. Moreover, in the context of ED, we suggest it would be beneficial for the LD to be trained with examples containing events, instead of non-event samples, as
the presence of an event can then be incorporated into the generated representations.
Hence, we propose refining the adversarial training process by only keeping the most informative examples while disregarding less useful ones. Our intuition as to what makes samples more informative for CLED is two-fold: First, we presume that presenting the LD with examples that are too different makes the discrimination task too simple. As mentioned previously, we would like the LD to learn a fine-grained distinction between the source and target languages which, in turn, improves the language-invariance of the encoder's representations. Thus, we suggest presenting the LD with examples that have similar contextual semantics, i.e., similar contextualized representations. Second, we consider that sentences containing events should provide an ED system with additional task-relevant information when compared against non-event samples. Accordingly, we argue that event-containing sentences should have a larger probability of being selected for ALA training.
With these intuitions in mind, we propose Optimal Transport (OT) (Villani, 2008) as a natural solution to simultaneously incorporate both the similarity between sample representations and the likelihood of the samples containing an event into a single framework. Therefore, we cast sample selection as an OT problem in which we attempt to find the best alignment between the samples from the source and target languages.
For our experiments, we focus on the widely used ACE05 (Walker et al., 2006) and ERE (Song et al., 2015) datasets which, in conjunction, contain event-annotations in 4 different languages: English, Spanish, Chinese, and Arabic. We work on 8 different language pairs by selecting different languages as the source and target. Our proposed model obtains new state-of-the-art results with considerable performance improvements  $(+2 - 3\%)$  in F1 scores) over competitive baselines and previously published results (M'hamdi et al., 2019). We believe these results demonstrate our model's efficacy and applicability at creating CLED systems.
The rest of this paper is organized as follows: section 2 provides an thorough description of our proposed model, section 3 presents and analyses the results from our experiments, section 4 provides a brief review of related work, and section 5 includes our conclusions.
Following prior works (M'hamdi et al., 2019; Majewska et al., 2021), we treat ED as a sequence labeling problem. Given a set  $\mathcal{D}$  of word sequences  $w_{i} = \{w_{i1}, w_{i2}, \dots, w_{in-1}, w_{in}\}$  and their corresponding label sequences  $y_{i} = \{y_{i1}, y_{i2}, \dots, y_{in-1}, y_{in}\}$ , we use an encoder network  $E$  to obtain a contextualized vector representation of the words in the input sequence  $\mathbf{h}_{\mathbf{i}} = E(w_{i}) = \{h_{i1}, h_{i2}, \dots, h_{in-1}, h_{in}\}$ . Using such representations as input, a prediction network  $P$  computes a distribution over the set of possible labels and is trained in a supervised manner using the negative log-likelihood function  $\mathcal{L}_P$ :
$$
\mathcal {L} _ {P} = - \sum_ {i = 1} ^ {| D |} \sum_ {j = 1} ^ {n} \log P \left(y _ {i j} \mid h _ {i j}\right) \tag {1}
$$
In the cross-lingual transfer-learning setting, the data used to train the model and the data on which the model is tested come from different languages known as the source and target, respectively. As such, we deal with two datasets  $\mathcal{D}_{src}$  and  $\mathcal{D}_{tgt}$ . We assume that we do not have access to the gold labels of the target language  $y_{tgt}$ , other than to evaluate our CLED model at testing time.
Our goal is to define a model able to generate language-invariant word representations that are refined enough so that cross-lingual issues, such as the ones described in section 1, are properly handled.
2.2 Baseline Model
Here, we briefly describe the BERT-CRF model proposed by M'hamdi et al. (2019) which was the previous state-of-the-art and serves as our main baseline. Using multilingual BERT (mBERT, (Devlin et al., 2019)) as its encoder, BERT-CRF generates robust, contextualized representations for words from different languages. For words that are split into multiple word-pieces, the average of the representation vectors for all comprising sub-pieces is used as the representation of the full word.
For classification purposes, instead of assigning the labels of each token independently, BERT-CRF uses a Conditional Random Field (CRF) (Lafferty et al., 2001) layer on top of the prediction network to better capture the interactions between the label
sequences. In summary, the contextualized representation vectors  $h_i$  generated by the mBERT encoder from the words in the sequence are then fed to a CRF layer which finds the optimal label sequence.
2.3 Adversarial Language Adaptation
The pre-trained versions of MLMs like mBERT or XLM-RoBERTa (Conneau et al., 2019) generate contextualized representations with a certain degree of language-invariance. This can be confirmed by their successful application in cross-lingual settings (M'hamdi et al., 2019; Majewska et al., 2021). However, a lingering issue is the difficulty of learning the nuances of the target language such as verb variations that do not exist in the source language used to train them. Majewska et al. (2021), for instance, propose to address this issue by injecting external verb knowledge into the encoder via task-specific adapter modules (Pfeiffer et al., 2020).
It is our intuition, however, that these issues can be mitigated by achieving a more refined level of language-invariance in the word representations. As such, we propose using Adversarial Language Adaptation (ALA) (Joty et al., 2017), a technique used to create language-invariant models. The ALA framework consists in including a Language Discriminator (LD) whose purpose is to learn language-dependent features and be able to differentiate between the samples from either the source or the target languages.
A fundamental characteristic of the ALA approach is its lack of requirements for annotated data in the target language. As such, we can use data from both  $\mathcal{D}_{src}$  and  $\mathcal{D}_{tgt}$ . An auxiliary dataset  $D_{aux} = \{(w_1, l_1), \ldots, (w_{2m}, l_{2m})\}$  is created where  $w_i$  is a text sequence from either  $\mathcal{D}_{src}$  or  $\mathcal{D}_{tgt}$ , and  $l_i$  is a language label. The cardinality of  $D_{aux}$  is  $|D_{aux}| = 2m$ , where  $m$  is equal to the batch size. Text samples  $w_1 \ldots w_m \in \mathcal{D}_{src}$  and samples  $w_{m+1} \ldots w_{2m} \in \mathcal{D}_{tgt}$ . As described earlier, the encoder E receives the text sequences and produces a sequence of contextualized representations  $E(w_i) = h_i = \{h_{i0}, h_{i1}, h_{i2}, \ldots, h_{in}\}$  where  $h_{i0}$  is the representation of the [CLS] token added at the beginning of every input sequence.
In our work, the LD is a simple Multi-Layer Perceptron(MLP) network that takes  $h_{i0}$  as input and produces a single sigmoid output. It's trained with the usual binary cross-entropy loss function objective:  $LD_{loss} = \arg \min_{LD} \mathcal{L}(LD(h_{i0}), l_i)$ .
As the LD learns to distinguish between the source and target languages, we concurrently train the encoder to "foo" the discriminator. In other words, the encoder must learn to generate representations that are language-invariant enough that the LD is unable to classify them while still remaining predictive for event-trigger classification. We optimize the following loss:
$$
\arg \min  _ {E, C} \sum_ {j = 1} ^ {n} \left(\mathcal {L} \left(C \left(h _ {i j}\right), y _ {i j}\right)\right) - \lambda \mathcal {L} \left(L D \left(h _ {i 0}, l _ {i}\right)\right) \tag {2}
$$
Where C refers to the CRF-based classifier network and  $\lambda$  is a hyperparameter.
Equation 2 is implemented by using a Gradient-Reversal Layer (GRL) (Ganin and Lempitsky, 2015) which acts as the identity during the forward pass, but reverses the direction of the gradients during the backward pass. The first term in Equation 2 can, of course, only be applied for annotated data from the source language.
The GRL is applied to the input vectors,  $h_{i0}$ , of the LD. This way, the LD is being trained to differentiate between the two languages while the encoder is trained in the opposite direction, i.e. to generate sequence representations that are harder to discriminate.
2.4 Adversarial Training Optimization
ALA has already been shown to be effective at generating language-invariant models (Joty et al., 2017; Chen et al., 2018). However, in regular ALA training, all samples in a batch, from both the source and target domains, are treated equally. That is, all samples are used as examples for the discriminator to learn how to better discern between the two domains. We propose that ALA effectiveness can be further improved by carefully selecting the samples with which to train the discriminator. We argue that some samples might be more informative than others and that, by only using such informative samples during training, better adaptation results can be achieved.
We base our notion as to what makes a sample more informative on two factors. First, we argue that presenting the LD with examples from the source and target language that are too dissimilar makes its task easier which, in turn, leads to the LD not learning the fine-grained distinctions between the languages. Instead, we propose using samples whose vector representations  $h_{i0}$  are
close to each other in the embedding space. The intuition for this being that, as representations capture the contextual semantics of the samples, closer representations correspond to more similar examples. Second, we suggest that presenting the LD with samples containing events should make the encoder incorporate task-specific information into its representations.
2.4.1 Optimal Transport
One challenge of using the two mentioned criteria for the ALA sample selection process is that they come with two different measures which are hard to combine. To address this, we propose using Optimal Transport (OT) (Villani, 2008) as a natural way to combine these two metrics into a single framework for sample selection. Optimal transport is, in broad terms, the problem of finding out the cheapest transformation between two discrete probability distributions. It requires a cost function to determine the cost of transforming a data point in one distribution into a data point in the second distribution. When the cost function is based on a valid distance function, the minimum cost is known as the Wasserstein distance. Formally, it solves the following optimization problem:
$$
\pi^ {*} (s, t) = \min  _ {\pi \in \prod (s, t)} \sum_ {s \in \mathcal {S}} \sum_ {t \in \mathcal {T}} \pi (s, t) C (s, t) d s d t \tag {3}
$$
$$
\mathbf {s . t .} s \sim p (s) \text {a n d} t \sim q (t)
$$
where  $S$  and  $\mathcal{T}$  are the two domains to be transformed;  $p(s)$  and  $q(t)$  are the probability distributions of  $S$  and  $\mathcal{T}$ , respectively;  $C$  is a cost function for mapping  $S$  to  $\mathcal{T}$ ,  $C(s,t): S \times \mathcal{T} \longrightarrow \mathbb{R}_{+}$ ; and finally,  $\pi^{*}(s,t)$  is the optimal joint distribution over the set of all joint distributions  $\prod (s,t)$ . The problem described by Equation 3 is, of course, intractable. Therefore, we use instead the Sinkhorn algorithm (Cuturi, 2013) which is an entropy-based relaxation of the discrete OT problem.
2.4.2 Problem Formulation
We formulate the OT problem as follows: the domains  $S$  and  $\mathcal{T}$  are defined as the representation vectors of the text samples in either the source  $h_{i0}^{s}$  or the target  $h_{j0}^{t}$  languages. We use the L2 distance between these representations as the cost function:
$$
C \left(h _ {i 0} ^ {s}, h _ {j 0} ^ {t}\right) = \left\| h _ {i 0} ^ {s} - h _ {j 0} ^ {t} \right\| _ {2} ^ {2} \tag {4}
$$
To define the marginal probability distributions  $p(s)$  and  $q(t)$  for the  $S$  and  $\mathcal{T}$  domains, we propose including an Event-Presence (EP) prediction module and use its normalized likelihood scores as the probability distributions for  $S$  and  $\mathcal{T}$ . Thus, the auxiliary dataset  $D_{aux}$  is augmented to include an event-presence label  $e_i$  for each sample. Of course, this can only be done for samples in the source language as the labels for the target-language data are unavailable:
$$
\begin{array}{l} D _ {a u x} = \left\{\left(w _ {1}, l _ {1}, e _ {1}\right), \dots , \left(w _ {m}, l _ {m}, e _ {m}\right), \right. \\ \left. \left(w _ {m + 1}, l _ {m + 1}\right), \dots , \left(w _ {2 m}, l _ {2 m}\right) \right\} \\ \end{array}
$$
The EP module is then trained to optimize the following loss:
$$
E P _ {l o s s} = \arg \min  _ {E P} \mathcal {L} (E P \left(h _ {i 0}\right), e _ {i}) \tag {5}
$$
where  $i \leq m$ , i.e., only using samples from the source language.
The probability distributions  $p(s)$  and  $p(t)$  are the computed as follows:
$$
\begin{array}{l} p (s) = \operatorname {S o f t m a x} \left(E P \left(h _ {i 0} ^ {s}\right) \mid l _ {i} = = s\right) (6) \\ p (t) = \operatorname {S o f t m a x} \left(E P \left(h _ {i 0} ^ {t}\right) \mid l _ {i} = = t\right) (7) \\ \end{array}
$$
2.4.3 Sample Selection
We use the OT solution matrix  $\pi^{*}$ , where an entry  $\pi^{*}(s,t)$  represents the optimal cost of transforming data point  $s\in S$  into  $t\in \mathcal{T}$ , to compute an the overall similarity score  $v_{i}$  of a sample  $h_{i0}\in S$  to the samples in the target domain  $\mathcal{T}$  by using the average distance:
$$
v _ {i} = \frac {\sum_ {j} ^ {m} \pi^ {*} \left(h _ {i 0} ^ {s} , h _ {j 0} ^ {t}\right)}{m} \tag {8}
$$
Correspondingly, we compute an overall similarity score  $v_{j}$  of each sample  $h_{j0} \in \mathcal{T}$  to the samples in the source domain  $S$ :
$$
v _ {j} = \frac {\sum_ {i} ^ {m} \pi^ {*} \left(h _ {i 0} ^ {s} , h _ {j 0} ^ {t}\right)}{m} \tag {9}
$$
Lastly, we select a fraction, hyperparameter  $\gamma$ , of samples with the best similarity scores from both the source and target languages, and only use these selected samples during ALA training.
2.5 OACLED Model
We train our Optimized Adversarial Cross-Linguual Event Detection (OACLED) model end-to-end with the following loss objective:
$$
L _ {f u l l} = C R F _ {l o s s} + \alpha L D _ {l o s s} + \beta E P _ {l o s s} \tag {10}
$$
where  $\alpha$  and  $\beta$  are trade-off hyperparameters.
We evaluate our model on the ACE05 (Walker et al., 2006) dataset which includes annotated event-trigger data in 3 languages: English, Chinese and Arabic. To include an additional language in our experiments, we also evaluate on the ERE dataset which has annotated data in English and Spanish. Note that the ACE05 and ERE datasets do not share the same label set: ACE05 involves 33 distinct event types while ERE involves 38 event types. We follow the same data pre-processing and splits as in previous work (M'hamdi et al., 2019) to ensure a fair comparison. Table 1 presents the data statistics.
Dataset	Language	Split	Sentences	Events
ACE05	English	Train	19,240	4,419
Dev	902	468
Test	676	424
Chinese	Train	6,841	2,926
Dev	526	217
Test	547	190
Arabic	Train	2,555	1,793
Dev	301	230
Test	262	247
ERE	English	Train	14,219	6,419
Dev	1,162	552
Test	1,129	559
Spanish	Train	7,067	3,272
Dev	556	210
Test	546	269
3.2 Hyper-parameters
We fine-tune the hyper-parameters for our OA-CLED model using the development data. We apply the following values based on the fine-tuning process:
- AdamW as the optimizer.
- 5 warm up epochs.
- A learning rate of  $1e^{-5}$  for the transformer parameters and of  $1e^{-4}$  for the rest of the parameters.
- A batch size of 16.
- 300 for the dimensionality of the layers in feed-forwards networks.
- A  $\gamma = 0.5$  for the percentage of samples used in adversarial training.
- A  $\lambda = 0.001$  as the scaling factor of the GRL layer.
- An  $\alpha = 1$  and  $\beta = 0.001$  as the trade-off parameters of the LD loss and ED loss, respectively.
- A dropout of  $10\%$  for added regularization during training.
In our experiments, we work with 8 distinct language pairs by selecting each of the available languages as either the source or target language: English-Chinese, Chinese-English, English-Arabic, Arabic-English, Chinese-Arabic, Arabic-Chinese, English-Spanish, and Spanish-English. The Chinese-Spanish, Spanish-Chinese, Arabic-Spanish, and Spanish-Arabic language combinations are unavailable due the previously mentioned incompatibility between the event type sets in ACE05 and ERE.
We compare our OACLED model against 3 relevant baselines. First, the previous state-of-the-art CLED model BERT-CRF (M'hamdi et al., 2019) as described in section 2.2. Second, the mBERT2TA model (Majewska et al., 2021) which aims at improving cross-lingual performance by incorporating language-independent verb knowledge via task-specific adapters. And third, XLM-R-CRF which is equivalent in all regards to BERT-CRF except that it uses XLM-RoBERTa (Conneau et al., 2019) as the encoder.
Table 2 and Table 3 show the results of our experiments on the ACE05 and ERE datasets, respectively. In all our experiments, we use the base transformer versions bert-base-cased and xlm-roberta-base as the encoders, parameters are tuned on the development data of the source language, and all entries are the average of five runs.