SlowGuess/ACL_2020_OCR huggingface dataset:
Cross-Linguual Semantic Role Labeling with High-Quality Translated Training Corpus
Hao Fei<sup>1</sup> and Meishan Zhang<sup>2*</sup> and Donghong Ji<sup>1</sup>
1. Department of Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China
2. School of New Media and Communication, Tianjin University, China {hao.fei, dhji}@whu.edu.cn and mason.zms@gmail.com
Many efforts of research are devoted to semantic role labeling (SRL) which is crucial for natural language understanding. Supervised approaches have achieved impressing performances when large-scale corpora are available for resource-rich languages such as English. While for the low-resource languages with no annotated SRL dataset, it is still challenging to obtain competitive performances. Cross-lingual SRL is one promising way to address the problem, which has achieved great advances with the help of model transferring and annotation projection. In this paper, we propose a novel alternative based on corpus translation, constructing high-quality training datasets for the target languages from the source gold-standard SRL annotations. Experimental results on Universal Proposition Bank show that the translation-based method is highly effective, and the automatic pseudo datasets can improve the target-language SRL performances significantly.
Semantic role labeling (SRL), which aims to capture the high-level meaning of a sentence, such as who did what to whom, is an underlying task for facilitating a broad range of natural language processing (NLP) tasks (Shen and Lapata, 2007; Liu and Gildea, 2010; Genest and Lapalme, 2011; Gao and Vogel, 2011; Wang et al., 2015; Khan et al., 2015). Currently, the majority of research work on SRL is dedicated to the English language, due to the availability of large quantity of labeled data. With this regard, cross-lingual SRL, especially the one transferring the advantage of the source language with affluent amount of resources (e.g., English) to the target language where the labeled data is scarce or even not available, is of great importance
Figure 1: Illustration of cross-lingual SRL methods. Our method is in the dotted blue box.
(Kozhevnikov and Titov, 2013; He et al., 2019; Aminian et al., 2019).
Previous work on cross-lingual SRL can generally be divided into two categories: model transferring and annotation projection. The former builds cross-lingual models on language-independent features such as cross-lingual word representations and universal POS tags which can be transferred into target languages directly (McDonald et al., 2013; Swayamdipta et al., 2016; Daza and Frank, 2019). The latter bases on a large-scale parallel corpus between the source and target languages where the source-side sentences are annotated with SRL tags automatically by a source SRL labeler, and then the source annotations are projected onto the target-side sentences in accordance of word alignments (Yarowsky et al., 2001; Hwa et al., 2005; van der Plas et al., 2011; Kozhevnikov and Titov, 2013; Pado and Lapata, 2014; Akbik et al., 2015). In addition, the annotation projection can be combined with model transferring naturally.
Particularly, the projected SRL tags in annotation projection could contain much noise because of the source-side automatic annotations. A straightforward solution is the translation-based approach, which has been demonstrated effective for cross-lingual dependency parsing (Täckström et al., 2012; Rasooli and Collins, 2015; Guo et al., 2016; Zhang et al., 2019). The key idea is to translate the gold-standard source training data into target
7014
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7014-7026 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
language side by translation directly, avoiding the problem of the low-quality source annotations. Fortunately, due to recent great advances of neural machine translation (NMT) (Bahdanau et al., 2015; Wu et al., 2016), this approach could have great potentials for cross-lingual transferring.
To this end, in this paper, we study the translation-based method for cross-lingual SRL. Figure 1 illustrates the differences between previous approaches. Sentences of the source language training corpus are translated into the target language, and then the source SRL annotations are projected into the target side, resulting in a set of high-quality target language SRL corpus, which is used to train the target SRL model. Further, we merge the gold-standard source corpus and the translated target together, which can be regarded as a combination of the translation-based method and the model transferring. Our baseline is a simple BiLSTM CRF model by using multilingual contextualized word representations (Peters et al., 2018; Devlin et al., 2019). For a better exploration of the blended corpus, we adopt a parameter generation network (PGN) to enhance the BiLSTM module, which can capture the language differences effectively (Platanios et al., 2018; Jia et al., 2019).
We conduct experiments based on Universal Proposition Bank corpus (v1.0) (Akbik et al., 2015; Akbik and Li, 2016) over seven languages. First, we verify the effectiveness of our method on the single-source SRL transferring, where the English language is adopted as the source language and the remaining are used as the target languages. Results show that the translation-based method is highly effective for cross-lingual SRL, and the performances are further improved when PGN-BiLSTM is used. Further, we conduct experiments on the multi-source SRL transferring, where for each target language all the remaining six languages are used as the source languages. The same tendencies as the single-source setting can be observed. We conduct detailed analysis work for both settings to understand our proposed method comprehensively.
In summary, we make the following two main contributions in this work:
- We present the first work of the translation-based approach for unsupervised cross-lingual SRL. We build a high-quality of pseudo training corpus for a target language, and then verify the effectiveness of the corpus under a range of settings.
- We take advantage of the multilingual contextualized word representations, and strengthen the multilingual model training with PGN-BiLSTM model.
All codes and datasets are released publicly available for the research purpose<sup>1</sup>.
There exists extensive work for cross-lingual transfer learning (van der Plas et al., 2011; Kozhevnikov and Titov, 2013; Pado and Lapata, 2014; Rasooli and Collins, 2015; Tiedemann and Agic, 2016; Zhao et al., 2018; Chen et al., 2018, 2019; Aminian et al., 2019). Model transferring and annotation projection are two mainstream categories for the goal. The first category aims to build a model based on the source language corpus, and then adapt it to the target languages (Yarowsky et al., 2001; Hwa et al., 2005; Tiedemann, 2015). The second category attempts to produce a set of automatic training instances for the target language by a source language model and a number of parallel sentences, and then train a target model on the dataset (Björkelund et al., 2009; McDonald et al., 2013; Lei et al., 2015; Swayamdipta et al., 2016; Mulcaire et al., 2018; Daza and Frank, 2019).
For cross-lingual SRL, annotation projection has received the most attention (Pado and Lapata, 2014). A range of strategies have been proposed to enhance the SRL performance of the target language, including improving the projection quality (Tiedemann, 2015), joint learning of syntax and semantics (Kozhevnikov and Titov, 2013), iterative bootstrapping to reduce the influence of noise target corpus (Akbik et al., 2015), and joint translation and SRL (Aminian et al., 2019).
Our work is mainly inspired by the recent work of treebank translation of cross-lingual dependency parsing (Tiedemann et al., 2014; Tiedemann, 2015; Rasooli and Collins, 2015; Guo et al., 2016; Tiedemann and Agic, 2016; Conneau et al., 2018; Zhang et al., 2019), which is referred to as the translation-based approaches. These approaches directly project the gold-standard annotation into the target side, alleviating the problem of erroneous source annotations in standard annotation projection. In addition, we combine the approach with model transferring, which has been concerned little for cross-lingual SRL. The model transferring
$^{1}$ https://github.com/scofield7419/ XSRL-ACL under Apache License 2.0.
benefits much from the recent advance of cross-lingual contextualized word representations (He et al., 2019).
The development of universal annotation schemes for a variety of NLP tasks can greatly facilitate cross-lingual SRL, including POS tagging (Petrov et al., 2012), dependency parsing (McDonald et al., 2013; Przepiörkowski and Patejuk, 2018), morphology (Sylak-Glassman et al., 2015) and SRL (Aminian et al., 2019). Our work makes use of the publicly available Universal Proposition Bank (UPB) (Akbik et al., 2015; Akbik and Li, 2016), which annotates the predicate and semantic roles following the frame and role schemes of the English Proposition Bank 3.0 (Kingsbury and Palmer, 2002; Palmer et al., 2005).
Supervised SRL models are also closely related to our work (He et al., 2017, 2018a; Xia et al., 2019). A great deal of work attempts for an end-to-end solution with sophistical neural networks, detecting the predicates as well as the corresponding argument roles in one shot (He et al., 2017; Tan et al., 2018; Li et al., 2019). Also there exist a number of studies which aims for adapting various powerful features for the task (Strubell et al., 2018; Li et al., 2018). In this work, we exploit a multilingual PGN-BiLSTM model (Jia et al., 2019) with contextualized word representations (He et al., 2019), which can obtain state-of-the-art performance for cross-lingual SRL.
3 SRL Translation
We induce automatic target data from the gold-standard source data by full translation and then project the SRL predicates and arguments into their corresponding words by aligning, producing the final translated SRL corpus for the target language automatically. The method has been demonstrated effective for cross-lingual dependency parsing (Tiedemann et al., 2014; Tiedemann, 2015; Tiedemann and Agic, 2016; Zhang et al., 2019). Compared with annotation projection, we can ensure the annotation quality at the source side, thus higher quality target corpus is also expected. In addition, dependency-based SRL could benefit more by this method, as only predicate words and their arguments are required to be projected into the target side, while dependency parsing should concern all sentential words. The overall process is accomplished by two steps: translating and projecting.
(a) One-to-one projection.
(b) Predicate-argument collision. Only keep predicate.
(c) Argument-argument collision. Only keep the one with higher confidence.
Figure 2: Examples of SRL projection from English (upper) to German (lower), where the thick blue solid arrows indicate successful projections, and the thin red dotted arrows indicate invalid projections.
Translating. First, we use a state-of-the-art translation system to produce the target translations for the sentences of the source SRL data. Give a source sentence  $e_1 \cdots e_n$ , we translate it into  $f_1 \cdots f_m$  of the target language. It is worth noting that the recent impressive advances in NMT (Bahdanau et al., 2015; Wu et al., 2016) facilitate our work greatly, which enables our method to have high-quality translations.
Projecting. Then we incrementally project the corresponding predicates or arguments of a source sentence  $e_1 \cdots e_n$  to its target  $f_1 \cdots f_m$ . We adopt two kinds of information to assist the projection: (1) the alignment probabilities  $a(f_j | e_i)$  from the source word  $e_i$  into  $f_j$ , which can be calculated by a word-alignment tool, and (2) the POS tag distributions  $p(t_* | f_j)$  of the target sentential words, which can be derived from a supervised target language POS tagger, where  $i \in [1, n]$ ,  $j \in [1, m]$ , and  $t_*$  denotes an arbitrary POS tag.
We focus on SRL-related words of the source sentence only, and perform the process gradually at the predicate level. For each predicate in a sentence, we collect the predicate word as well as its role words, and then project their role labels into the target sentence. Formally, for each of these words (i.e.,  $e_i$ ), we have the SRL role tag  $r_{e_i}$  as
7016
well as its POS tag  $t_{e_i}$ , both of which have been already annotated in the UPB. First, we find its target word  $f_j$  with the highest alignment probability, regarding the word  $f_j$  as the corresponding projection carrying the semantic role  $r_{e_i}$ . Then we calculate the confidence score of this projection by the following formula:
$$
\operatorname {s c o r e} \left(e _ {i} \rightarrow f _ {j}, r _ {e _ {i}}\right) = a \left(f _ {j} \mid e _ {i}\right) p \left(t _ {e _ {i}} \mid f _ {j}\right), \tag {1}
$$
which is a joint probability of word alignment corresponding and POS tag consistency.
The one-one target-source alignment 2(a) is the ideal condition of the projection. However, there could be many-to-one cases for the given words, leading to semantic role conflicts at the target language words. For these cases, we take precedence for the predicate projections, and otherwise keep only the highest confidence projections. Figure 2(b) shows a predicate-argument conflict example, where the predicate projection is reserved, and Figure 2(c) shows an argument-argument conflict example where the projection with the higher confidence score is reserved.
Finally, we set a threshold value  $\alpha$  to remove low confidence projections. If the confidence score of a predicate projection is below  $\alpha$ , all the roles of this predicate are removed as well. For the argument projections whose confidence is below  $\alpha$ , we remove the single arguments directly, with no influence on the other projections.
4 The SRL Model
In this work, we focus on dependency-based SRL, recognizing semantic roles for a given predicate (He et al., 2017). The task can be treated as a standard sequence labeling problem, and a simple multi-layer BiLSTM-CRF model is exploited here, which has archived state-of-the-art performance with contextualized word representations (He et al., 2018b; Xia et al., 2019; He et al., 2019). In particular, we adapt the model to better support multilingual inputs by using a PGN module on the BiLSTM (Hochreiter and Schmidhuber, 1997). Figure 3 shows the overall architecture.
4.1 Word Representation
Given an input sentence  $s = w_{1} \cdots w_{n}$  of a specific language  $\mathcal{L}$  and  $w_{p}$  ( $p$  denotes the position) is the predicate word, we use three sources of features to represent each word: (1) the word form, (2) the
Figure 3: The overall architecture of the SRL model.
POS tag and (3) the predicate indicator:
$$
\boldsymbol {x} _ {i} = \boldsymbol {v} _ {w _ {i}} \oplus \boldsymbol {v} _ {t _ {i}} \oplus \boldsymbol {v} _ {(i = = p)}, \tag {2}
$$
where  $t_1 \cdots t_n$  is the universal POS tag sequence for the input sentence. For the POS tags and the predicate indicators, we use the embedding method to obtain their vectorial representations.
We compare three kinds of word form representations for cross-lingual SRL: (1) multilingual word embeddings, (2) multilingual ELMo representation (Peters et al., 2018), and (3) multilingual BERT representation (Devlin et al., 2019). Note that we use the averaged vector of the inner-word piece representations from BERT outputs as the full word representation.
4.2 Encoding Layer
We employ the PGN-BiLSTM (Platanios et al., 2018; Jia et al., 2019) to encode the input sequence  $x_{1} \cdots x_{n}$ , which is first introduced for cross-domain transfer learning to capture domain difference. Here we use it for the multilingual setting aiming to model the language characteristics.
Compared with the vanilla BiLSTM module, PGN-BiLSTM dynamically selects the language-aware parameters for BiLSTM. Let  $V$  be the flattened vector of all the parameters of a BiLSTM cell, the language-aware  $V_{\mathcal{L}}$  is produced by:
$$
\boldsymbol {V} _ {\mathcal {L}} = \boldsymbol {W} _ {\mathrm {P G N}} \times \boldsymbol {e} _ {\mathcal {L}}, \tag {3}
$$
where  $W_{\mathrm{PGN}}$  denotes the parameters of vanilla BiLSTM part in the PGN-BiLSTM, including the weights of the input, forget, output gates and the cell modules, and  $e_{\mathcal{L}}$  is the embedding representation of language  $\mathcal{L}$ . The mechanism of parameter generation of PGN-BiLSTM is illustrated in Figure 4. Following, we derive module parameters from
7017
Figure 4: The mechanism of the PGN-BiLSTM.
$V_{\mathcal{L}}$  to compute the BiLSTM outputs. The overall process can be formalized as:
$$
\begin{array}{l} \boldsymbol {h} _ {1} \dots \boldsymbol {h} _ {n} = \operatorname {P G N - B i L S T M} \left(\boldsymbol {x} _ {1} \dots \boldsymbol {x} _ {n}, e _ {\mathcal {L}}\right) \tag {4} \\ = \operatorname {B i L S T M} _ {\mathbf {V} _ {\mathcal {L}}} \left(\boldsymbol {x} _ {1} \dots \boldsymbol {x} _ {n}\right) \\ \end{array}
$$
which differs from the vanilla BiLSTM in that  $e_{\mathcal{L}}$  is one extra input to obtain BiLSTM parameters. Specifically, we adopt a 3-layer bidirectional PGN-LSTM as the encoder.
4.3 Output Layer
Given the encoder output  $\pmb{h}_1 \cdots \pmb{h}_n$  for sentence  $s = w_1 \cdots w_n$ , we use CRFs (Lafferty et al., 2001) to compute the probability of each candidate output  $y = y_1 \cdots y_n$ :
$$
\begin{array}{l} \boldsymbol {o} _ {i} = \boldsymbol {W h} _ {n}, i \in [ 1, n ] \\ p (y \mid s) = \frac {\exp \left\{\sum_ {i} \left(\boldsymbol {o} _ {i} + \boldsymbol {T} _ {y _ {i - 1} , y _ {i}}\right) \right\}}{Z} \tag {5} \\ \end{array}
$$
where  $W$  and  $\pmb{T}$  are the parameters of CRFs, and  $Z$  is a normalization factor for probability calculation. The Viterbi algorithm is used to search for the highest-probability output SRL tag sequence.
5.1 Universal Proposition Bank
Our experiments are based on the Universal Proposition Bank (UPB, v1.0)  $^{2}$ , which is built upon Universal Dependency Treebank (UDT, v1.4) $^{3}$  and Proposition Bank (PB, v3.0) $^{4}$ . In UPB, consistent dependency-based universal SRL annotations are constructed across all languages. In particular, we assemble the English SRL dataset based on the English EWT subset from the UDT v1.4 and the English corpus in PB v3.0. Finally, we choose a total of seven languages as our datasets, including English (EN) and German (DE) of the IE.German
Fam.	Lang.	Train	Dev	Test	Pred.	Arg.
IE.Ge	EN	10,907	1,631	1,633	41,359	100,170
DE	14,118	799	977	23,256	58,319
IE.Ro	FR	14,554	1,596	298	26,934	44,007
IT	12,837	489	489	26,576	56,893
ES	28,492	3,206	1,995	81,318	177,871
PT	7,494	938	936	19,782	41,449
Ura	FI	12,217	716	648	27,324	60,502
Table 1: Statistics of the UPB, where Fam. indicates the language famaily, IE.Ge refers to the Indo-European Germanic, IE.Ro refers to the Indo-European Romance,and Ura represents Uralic.
family, French (FR), Italian (IT), Spanish  $(\mathrm{ES})^{5}$  and Portuguese (PT) of the IE.Romance family, and Finnish (FI) of the Uralic family. Table 1 shows the data statistics in detail.
5.2 SRL Translation
We focus on unsupervised cross-lingual SRL, assuming that no gold-standard target-language SRL corpus is available. Our goal is to construct pseudo training datasets by corpus translation from the gold-standard source-language SRL datasets. The Google Translation System  $^{6}$  is adopted for sentence translation, and the fastAlign toolkit (Dyer et al., 2013) is used to obtain word alignments. In order to obtain accurate word alignment, we collect a set of parallel corpora to augment the training dataset of fastAlign. $^{7}$  The universal POS tags of the translated sentences are produced by supervised monolingual POS taggers, which are trained on the corresponding UDT v1.4 datasets, respectively. $^{8}$
5.3 Settings
Multi-lingual word representations. As mentioned in Section 4.1, we investigate three kinds of multilingual word representations: (1) Word Embedding (Emb): MUSE is exploited to align all monolingual fastText word embeddings into a universal space (Lample et al., 2018). (2) ELMo: A blended dataset of the seven languages is used to train multilingual ELMo (Mulcaire et al., 2019).
$^{2}$ https://github.com/System-T/  
UniversalPropositions  
 $^{3}$ https://lindat.mff.cuni.cz/  
repository/xmlui/handle/11234/1-1827  
 $^{4}$ http://propbank.github.io/
5We merge the Spanish and Spanish-AnCora as one. 6https://translate.google.com/, Oct. 1 2019 7http://opus.nlpl.eu/, Europarl v8. A simple BiLSTM-CRF POS tagging model with monolingual ELMo representations is used, which can achieve accuracies of  $96.54\% (\mathrm{EN})$ $97.15\% (\mathrm{DE})$ $94.42\% (\mathrm{FR})$ $97.21\% (\mathrm{IT})$ $94.12\% (\mathrm{ES})$ $95.86\% (\mathrm{PT})$  and  $92.16\% (\mathrm{FI})$  , respectively. 9https://github.com/facebookresearch/MUSE 10CoNLL2017 corpus: https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1989
7018
Figure 5: Performances with the translated target data under varying projection threshold  $\alpha$ .
(3) BERT: the official released multilingual BERT (base, cased version) is used directly (Devlin et al., 2019).<sup>11</sup>
Hyperparameters. For SRL translation, there is only one hyperparameter, the projection confidence threshold  $\alpha$ , for filtering low-quality translated SRL sentences. Figure 5 shows the performances in the preliminary experiments for each languages under different  $\alpha$ . Accordingly, we set  $\alpha$  universally for all languages to 0.4. For the neural SRL models, the dimension sizes of multilingual word embeddings, ELMo and BERT are 300, 1024 and 768, respectively. The POS tag, predicate-indicator and language ID embedding sizes are 100, 100 and 32, respectively. The hidden size of LSTM is set to 650. We exploit online training with a batch size of 50, and the model parameters are optimized by using the Adam algorithm with an initial rate of 0.0005. The training is performed over the whole training dataset without early-stopping for 80 iterations on bilingual transfer, and 300 iterations on multilingual transfer.
Baselines. In order to test the effectiveness of our  $PGN$  model, we compare it with several baselines as well. First, we denote our model by using the vanilla BiLSTM instead as BASIC, and in particular, this model is exploited for all monolingual training all through this work. Further, we adopt two much stronger baselines, the  $MoE$  model proposed by Guo et al. (2018) and the MAN-MoE model proposed by Chen et al. (2019), respectively. Both the two models are designed to train a model effectively based on corpora from multiple languages, similar to our  $PGN$  model.
Evaluation. We use the F1 score as the major metric to measure the model performance for each
Model	DE	FR	IT	ES	PT	FI	Avg
SRC
Emb	42.7	51.0	42.6	40.1	43.9	30.0	41.7
BERT	43.2	53.1	44.4	41.2	44.2	31.6	43.0
ELMo	46.8	54.6	43.0	42.1	46.1	33.9	44.4
TGT
Emb	49.4	51.3	45.5	48.4	46.9	38.7	46.7
BERT	53.0	54.3	49.1	51.3	48.8	41.1	49.6
ELMo	54.6	55.3	49.7	53.6	49.8	43.9	51.1
SRC & TGT (ELMo)
BASIC	59.2	61.7	55.1	58.3	53.7	47.6	55.8
PGN	65.0	64.8	58.7	62.5	56.0	54.5	60.3
MoE	63.2	63.3	56.7	60.3	55.0	50.6	58.2
MAN-MoE	64.3	65.3	57.1	62.8	55.2	52.3	59.4
Table 2: Results of cross-lingual transfer from English.
target language. Each model is trained five times and the averaged value is reported. We conduct significance tests by using the Dan Bikel's randomized parsing evaluation comparator<sup>12</sup>.
5.4 Cross-Linguual Transfer from English