SlowGuess/NAACL_2021_OCR huggingface dataset:
Simone Conia Andrea Bacciu Roberto Navigli
{first这个名字}@uniromal.it
While cross-lingual techniques are finding increasing success in a wide range of Natural Language Processing tasks, their application to Semantic Role Labeling (SRL) has been strongly limited by the fact that each language adopts its own linguistic formalism, from PropBank for English to AnCora for Spanish and PDT-Vallex for Czech, inter alia. In this work, we address this issue and present a unified model to perform cross-lingual SRL over heterogeneous linguistic resources. Our model implicitly learns a high-quality mapping for different formalisms across diverse languages without resorting to word alignment and/or translation techniques. We find that, not only is our cross-lingual system competitive with the current state of the art but that it is also robust to low-data scenarios. Most interestingly, our unified model is able to annotate a sentence in a single forward pass with all the inventories it was trained with, providing a tool for the analysis and comparison of linguistic theories across different languages. We release our code and model at https://github.com/SapienzaNLP/unify-srl.
Semantic Role Labeling (SRL) – a long-standing open problem in Natural Language Processing (NLP) and a key building block of language understanding (Navigli, 2018) – is often defined as the task of automatically addressing the question "Who did what to whom, when, where, and how?" (Gildea and Jurafsky, 2000; Márquez et al., 2008). While the need to manually engineer and fine-tune complex feature templates severely limited early work (Zhao et al., 2009), the great success of neural networks in NLP has resulted in impressive progress in SRL, thanks especially to the ability of recurrent networks to better capture relations over sequences (He et al., 2017; Marcheggiani et al., 2017). Owing to the recent wide availability of
robust multilingual representations, such as multilingual word embeddings (Grave et al., 2018) and multilingual language models (Devlin et al., 2019; Conneau et al., 2020), researchers have been able to shift their focus to the development of models that work on multiple languages (Cai and Lapata, 2019b; He et al., 2019; Lyu et al., 2019).
A robust multilingual representation is nevertheless just one piece of the puzzle: a key challenge in multilingual SRL is that the task is tightly bound to linguistic formalisms (Márquez et al., 2008) which may present significant structural differences from language to language (Hajic et al., 2009). In the recent literature, it is standard practice to sidestep this issue by training and evaluating a model on each language separately (Cai and Lapata, 2019b; Chen et al., 2019; Kasai et al., 2019; He et al., 2019; Lyu et al., 2019). Although this strategy allows a model to adapt itself to the characteristics of a given formalism, it is burdened by the non-negligible need for training and maintaining one model instance for each language, resulting in a set of monolingual systems.
Instead of dealing with heterogeneous linguistic theories, another line of research consists in actively studying the effect of using a single formalism across multiple languages through annotation projection or other transfer techniques (Akbik et al., 2015, 2016; Daza and Frank, 2019; Cai and Lapata, 2020; Daza and Frank, 2020). However, such approaches often rely on word aligners and/or automatic translation tools which may introduce a considerable amount of noise, especially in low-resource languages. More importantly, they rely on the strong assumption that the linguistic formalism of choice, which may have been developed with a specific language in mind, is also suitable for other languages.
In this work, we take the best of both worlds and propose a novel approach to cross-lingual SRL. Our contributions can be summarized as follows:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 338-351 June 6-11, 2021. ©2021 Association for Computational Linguistics
- We introduce a unified model to perform cross-lingual SRL with heterogeneous linguistic resources;
- We find that our model is competitive against state-of-the-art systems on all the 6 languages of the CoNLL-2009 benchmark;
- We show that our model is robust to low-resource scenarios, thanks to its ability to generalize across languages;
- We probe our model and demonstrate that it implicitly learns to align heterogeneous linguistic resources;
- We automatically build and release a crosslingual mapping that aligns linguistic formalisms from diverse languages.
We hope that our unified model will further advance cross-lingual SRL and represent a tool for the analysis and comparison of linguistic theories across multiple languages.
End-to-end SRL. The SRL pipeline is usually divided into four steps: predicate identification, predicate sense disambiguation, argument identification, and argument classification. While early research focused its efforts on addressing each step individually (Xue and Palmer, 2004; Björkelund et al., 2009; Zhao et al., 2009), recent work has successfully demonstrated that tackling some of these subtasks jointly with multitask learning (Caruana, 1997) is beneficial. In particular, He et al. (2018) and, subsequently, Cai et al. (2018), Li et al. (2019) and Conia et al. (2020), indicate that predicate sense signals aid the identification of predicate-argument relations. Therefore, we follow this line and propose an end-to-end system for cross-lingual SRL.
Multilingual SRL. Current work in multilingual SRL revolves mainly around the development of novel neural architectures, which fall into two broad categories, syntax-aware and syntax-agnostic ones. On one hand, the quality and diversity of the information encoded by syntax is an enticing prospect that has resulted in a wide range of contributions: Marcheggiani and Titov (2017) made use of Graph Convolutional Networks (GCNs) to better capture relations between neighboring nodes in syntactic dependency trees; Strubell et al. (2018)
demonstrated the effectiveness of linguistically-informed self-attention layers in SRL; Cai and Lapata (2019b) observed that syntactic dependencies often mirror semantic relations and proposed a model that jointly learns to perform syntactic dependency parsing and SRL; He et al. (2019) devised syntax-based pruning rules that work for multiple languages. On the other hand, the complexity of syntax and the noisy performance of automatic syntactic parsers have deterred other researchers who, instead, have found methods to improve SRL without syntax: Cai et al. (2018) took advantage of an attentive biaffine layer (Dozat and Manning, 2017) to better model predicate-argument relations; Chen et al. (2019) and Lyu et al. (2019) obtained remarkable results in multiple languages by capturing predicate-argument interactions via capsule networks and iteratively refining the sequence of output labels, respectively; Cai and Lapata (2019a) proposed a semi-supervised approach that scales across different languages.
While we follow the latter trend and develop a syntax-agnostic model, we underline that both the aforementioned syntax-aware and syntax-agnostic approaches suffer from a significant drawback: they require training one model instance for each language of interest. Their two main limitations are, therefore, that i) the number of trainable parameters increases linearly with the number of languages, and ii) the information available in one language cannot be exploited to make SRL more robust in other languages. In contrast, one of the main objectives of our work is to develop a unified cross-lingual model which can mitigate the paucity of training data in some languages by exploiting the information available in other, resource-richer languages.
Cross-lingual SRL. A key challenge in performing cross-lingual SRL with a single unified model is the dissimilarity of predicate sense and semantic role inventories between languages. For example, the multilingual dataset distributed as part of the CoNLL-2009 shared task (Hajic et al., 2009) adopts the English Proposition Bank (Palmer et al., 2005) and NomBank (Meyers et al., 2004) to annotate English sentences, the Chinese Proposition Bank (Xue and Palmer, 2009) for Chinese, the AnCora (Taulé et al., 2008) predicate-argument structure inventory for Catalan and Spanish, the German Proposition Bank which, differently from the other PropBanks, is derived from FrameNet (Hajic et al., 2009), and
PDT-Vallex (Hajic et al., 2003) for Czech. Many of these inventories are not aligned with each other as they follow and implement different linguistic theories which, in turn, may pose different challenges.
Padó and Lapata (2009), and Akbik et al. (2015, 2016) worked around these issues by making the English PropBank act as a universal predicate sense and semantic role inventory and projecting PropBank-style annotations from English onto non-English sentences by means of word alignment techniques applied to parallel corpora such as Europarl (Koehn, 2005). These efforts resulted in the creation of the Universal PropBank, a multilingual collection of semi-automatically annotated corpora for SRL, which is actively in use today to train and evaluate novel cross-lingual methods such as word alignment techniques (Aminian et al., 2019). In the absence of parallel corpora, annotation projection techniques can still be applied by automatically translating an annotated corpus and then projecting the original labels onto the newly created silver corpus (Daza and Frank, 2020; Fei et al., 2020), whereas Daza and Frank (2019) have recently found success in training an encoder-decoder architecture to jointly tackle SRL and translation.
While the foregoing studies have greatly advanced the state of cross-lingual SRL, they suffer from an intrinsic downside: using translation and word alignment techniques may result in a considerable amount of noise, which automatically puts an upper bound to the quality of the projected labels. Moreover, they are based on the strong assumption that the English PropBank provides a suitable formalism for non-English languages, and this may not always be the case. Among the numerous studies that adopt the English PropBank as a universal predicate-argument structure inventory for cross-lingual SRL, the work of Mulcaire et al. (2018) stands out for proposing a bilingual model that is able to perform SRL according to two different inventories at the same time, although with significantly lower results compared to the state of the art at the time. With our work, we go beyond current approaches to cross-lingual SRL and embrace the diversity of the various representations made available in different languages. In particular, our model has three key advantages: i) it does not rely on word alignment or machine translation tools; ii) it learns to perform SRL with multiple linguistic inventories; iii) it learns to link resources that would otherwise be disconnected from each other.
In the wake of recent work in SRL, our model falls into the broad category of end-to-end systems as it learns to jointly tackle predicate identification, predicate sense disambiguation, argument identification and argument classification. The model architecture can be roughly divided into the following components:
- A universal sentence encoder whose parameters are shared across languages and which produces word encodings that capture predicate-related information (Section 3.2);
- A universal predicate-argument encoder whose parameters are also shared across languages and which models predicate-argument relations (Section 3.3);
- A set of language-specific decoders which indicate whether words are predicates, select the most appropriate sense for each predicate, and assign a semantic role to every predicateargument couple, according to several different SRL inventories (Section 3.4).
Unlike previous work, our model does not require any preexisting cross-resource mappings, word alignment techniques, translation tools, other annotation transfer techniques, or parallel data, to perform high-quality cross-lingual SRL, as it relies solely on implicit cross-lingual knowledge transfer.
3.1 Input representation
Pretrained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020), inter alia, are becoming the de facto input representation method, thanks to their ability to encode vast amounts of knowledge. Following recent studies (Hewitt and Manning, 2019; Kuznetsov and Gurevych, 2020; Conia and Navigli, 2020), which show that different layers of a language model capture different syntactic and semantic characteristics, our model builds a contextual representation for an input word by concatenating the corresponding hidden states of the four top-most inner layers of a language model. More formally, given a word  $w_{i}$  in a sentence  $\mathbf{w} = \langle w_0, w_1, \dots, w_i, \dots, w_{n-1} \rangle$  of  $n$  words and its hidden state  $\mathbf{h}_i^k = l^k(w_i|\mathbf{w})$  from the  $k$ -th inner layer  $l^k$  of a language model with  $K$  layers, the model computes the word encoding  $\mathbf{e}_i$
$$
\mathbf {h} _ {i} = \mathbf {h} _ {i} ^ {K} \oplus \mathbf {h} _ {i} ^ {K - 1} \oplus \mathbf {h} _ {i} ^ {K - 2} \oplus \mathbf {h} _ {i} ^ {K - 3}
$$
$$
\mathbf {e} _ {i} = \mathrm {S w i s h} (\mathbf {W} ^ {w} \mathbf {h} _ {i} + \mathbf {b} ^ {w})
$$
where  $\mathbf{x} \oplus \mathbf{y}$  is the concatenation of the two vectors  $\mathbf{x}$  and  $\mathbf{y}$ , and  $\operatorname{Swish}(x) = x \cdot \operatorname{sigmoid}(x)$  is a non-linear activation which was found to produce smoother gradient landscapes than the more traditional ReLU (Ramachandran et al., 2018).
3.2 Universal sentence encoder
Expanding on the seminal intuition of Fillmore (1968), who suggests the existence of deep semantic relations between a predicate and other sentential constituents, we argue that such semantic relations may be preserved across languages. With this reasoning in mind, we devise a universal sentence encoder whose parameters are shared across languages. Intuitively, the aim of our universal sentence encoder is to capture sentence-level information that is not formalism-specific and spans across languages, such as information about predicate positions and predicate senses. In our case, we implement this universal sentence encoder as a stack of BiLSTM layers (Hochreiter and Schmidhuber, 1997), similarly to Marcheggiani et al. (2017), Cai et al. (2018) and He et al. (2019), with the difference that we concatenate the output of each layer to its input in order to mitigate the problem of vanishing gradients. More formally, given a sequence of word encodings  $\mathbf{e} = \langle \mathbf{e}_0, \mathbf{e}_1, \dots, \mathbf{e}_{n-1} \rangle$ , the model computes a sequence of timestep encodings  $\mathbf{t}$  as follows:
$$
\begin{array}{l} \mathbf {t} _ {i} ^ {j} = \left\{ \begin{array}{l l} \mathbf {e} _ {i} & \text {i f} j = 0 \\ \mathbf {t} _ {i} ^ {j - 1} \oplus \operatorname {B i L S T M} _ {i} ^ {j} (\mathbf {t} ^ {j - 1}) & \text {o t h e r w i s e} \end{array} \right. \\ \mathbf {t} = \langle \mathbf {t} _ {0} ^ {K ^ {\prime}}, \mathbf {t} _ {1} ^ {K ^ {\prime}}, \dots , \mathbf {t} _ {n - 1} ^ {K ^ {\prime}} \rangle \\ \end{array}
$$
where  $\mathrm{BiLSTM}_i^j (\cdot)$  is the  $i$ -th timestep of the  $j$ -th BiLSTM layer and  $K^{\prime}$  is the total number of layers in the stack. Starting from each timestep encoding  $\mathbf{t}_i$ , the model produces a predicate representation  $\mathbf{p}_i$ , which captures whether the corresponding word  $w_i$  is a predicate, and a sense representation  $\mathbf{s}_i$  which encodes information about the sense of a predicate at position  $i$ :
$$
\mathbf {p} _ {i} = \operatorname {S w i s h} \left(\mathbf {W} ^ {p} \mathbf {t} _ {i} + \mathbf {b} ^ {p}\right)
$$
$$
\mathbf {s} _ {i} = \mathrm {S w i s h} (\mathbf {W} ^ {s} \mathbf {t} _ {i} + \mathbf {b} ^ {s})
$$
We stress that the vector representations obtained for each timestep, each predicate and each sense lie
in three spaces that are shared across the languages and formalisms used to perform SRL.
3.3 Universal predicate-argument encoder
In the same vein, and for the same reasoning that motivated the design of the above universal sentence encoder, our model includes a universal predicate-argument encoder whose parameters are also shared across languages. The objective of this second encoder is to capture the relations between each predicate-argument couple that appears in a sentence, independently of the input language. Similarly to the universal sentence encoder, we implement this universal predicate-argument encoder as a stack of BiLSTM layers. More formally, let  $w_{p}$  be a predicate in the input sentence  $\mathbf{w} = \langle w_0, w_1, \dots, w_p, \dots, w_{n - 1} \rangle$ , then the model computes a sequence of predicate-specific argument encodings a as follows:
$$
\begin{array}{l} \mathbf {a} _ {i} ^ {j} = \left\{ \begin{array}{l l} \mathbf {t} _ {p} \oplus \mathbf {t} _ {i} & \text {i f} j = 0 \\ \mathbf {a} _ {i} ^ {j - 1} \oplus \operatorname {B i L S T M} _ {i} ^ {j} (\mathbf {a} ^ {j - 1}) & \text {o t h e r w i s e} \end{array} \right. \\ \mathbf {a} = \langle \mathbf {a} _ {0} ^ {K ^ {\prime \prime}}, \mathbf {a} _ {1} ^ {K ^ {\prime \prime}}, \dots , \mathbf {a} _ {n - 1} ^ {K ^ {\prime \prime}} \rangle \\ \end{array}
$$
where  $\mathbf{t}_i$  is the  $i$ -th timestep encoding from the universal sentence encoder and  $K''$  is the total number of layers in the stack. Starting from each predicate-specific argument encoding  $\mathbf{a}_i$ , the model produces a semantic role representation  $\mathbf{r}_i$  for word  $w_i$ :
$$
\mathbf {r} _ {i} = \mathrm {S w i s h} (\mathbf {W} ^ {r} \mathbf {a} _ {i} + \mathbf {b} ^ {r})
$$
Similarly to the predicate and sense representations  $\mathbf{p}$  and  $\mathbf{s}$ , since the predicate-argument encoder is one and the same for all languages, the semantic role representation  $\mathbf{r}$  obtained must draw upon cross-lingual information in order to abstract from language-specific peculiarities.
3.4 Language-specific decoders
The aforementioned predicate encodings  $\mathbf{p}$ , sense encodings  $\mathbf{s}$  and semantic role encodings  $\mathbf{r}$  are shared across languages, forcing the model to learn from semantics rather than from surface-level features such as word order, part-of-speech tags and syntactic rules, all of which may differ from language to language. Ultimately, however, we want our model to provide semantic role annotations according to an existing predicate-argument structure inventory, e.g., PropBank, AnCora, or PDT-Vallex. Our model, therefore, includes a set of linear decoders that indicate whether a word  $w_{i}$  is
a predicate, what the most appropriate sense for a predicate  $w_{p}$  is, and what the semantic role of a word  $w_{r}$  with respect to a specific predicate  $w_{p}$  is, for each language  $l$ :
$$
\sigma^ {p} (w _ {i} | l) = \mathbf {W} ^ {p | l} \mathbf {p} _ {i} + \mathbf {b} ^ {p | l}
$$
$$
\sigma^ {s} (w _ {p} | l) = \mathbf {W} ^ {s | l} \mathbf {s} _ {i} + \mathbf {b} ^ {s | l}
$$
$$
\sigma^ {r} (w _ {r} | w _ {p}, l) = \mathbf {W} ^ {r | l} \mathbf {r} _ {i} + \mathbf {b} ^ {r | l}
$$
Although we could have opted for more complex decoding strategies, in our case linear decoders have two advantages: 1) they keep the language-specific part of the model as simple as possible, pushing the model into learning from its universal encoders; 2) they can be seen as linear probes, providing an insight into the quality of the crosslingual knowledge that the model can capture.
3.5 Training objective
The model is trained to jointly minimize the sum of the categorical cross-entropy losses on predicate identification, predicate sense disambiguation and argument identification/classification over all the languages in a multitask learning fashion. More formally, given a language  $l$  and the corresponding predicate identification loss  $\mathcal{L}^{p|l}$ , predicate sense disambiguation loss  $\mathcal{L}^{s|l}$  and argument identification/classification loss  $\mathcal{L}^{r|l}$ , the cumulative loss  $\mathcal{L}$  is:
$$
\mathcal {L} = \sum_ {l \in L} \left(\mathcal {L} ^ {p | l} + \mathcal {L} ^ {s | l} + \mathcal {L} ^ {r | l}\right)
$$
where  $L$  is the set of languages - and the corresponding formalisms - in the training set.
We evaluate our model in dependency-based multilingual SRL. The remainder of this Section describes the experimental setup (Section 4.1), provides a brief overview of the multilingual dataset we use for training, validation and testing (Section 4.2), and shows the results obtained on each language (Section 4.3).
We implemented the model in PyTorch<sup>1</sup> and PyTorch Lightning<sup>2</sup>, and used the pretrained language models for multilingual BERT (m-BERT) and XLM-RoBERTa (XLM-R) made available by the Transformers library (Wolf et al., 2020). We
trained each model configuration for 30 epochs using Adam (Kingma and Ba, 2015) with a "slanted triangle" learning rate scheduling strategy which linearly increases the learning rate for 1 epoch and then linearly decreases the value for 15 epochs. We did not perform hyperparameter tuning and opted instead for standard values used in the literature; we provide more details about our model configuration and its hyperparameter values in Appendix A. In the remainder of this Section, we report the  $\mathrm{F_1}$  scores of the best models selected according to the highest  $\mathrm{F_1}$  score obtained on the validation set at the end of a training epoch.
To the best of our knowledge, the dataset provided as part of the CoNLL-2009 shared task (Hajic et al., 2009) is the largest and most diverse collection of human-annotated sentences for multilingual SRL. It comprises 6 languages $^4$ , namely, Catalan, Chinese, Czech, English, German and Spanish, which belong to different linguistic families and feature significantly varying amounts of training samples, from 400K predicate instances in Czech to only 17K in German; we provide an overview of the statistics of each language in Appendix B. CoNLL-2009 is the ideal testbed for evaluating the ability of our unified model to generalize across heterogeneous resources since each language adopts its own linguistic formalism, from English PropBank to PDT-Vallex, from Chinese PropBank to AnCora. We also include VerbAtlas (Di Fabio et al., 2019), a recently released resource for SRL $^5$ , with the aim of understanding whether our model can learn to align inventories that are based on "distant" linguistic theories; indeed, VerbAtlas is based on clustering WordNet synsets into frames that share similar semantic behavior, whereas PropBank-based resources enumerate and define the possible senses of a lexeme.
As a final note, we did not evaluate our model on Universal PropBank since it was semiautomatically generated through annotation pro
3Hereafter, all the results of our experiments are computed by the official scorer of the CoNLL-2009 shared task, available at https://ufal.mff.cuni.cz/conl12009-st/scorer.html.
4The CoNLL-2009 shared task originally included a seventh language, Japanese, which is not available anymore on LDC due to licensing issues.
We build a training set for VerbAtlas using the mapping from PropBank available at http://verbatlas.org.
$^{6}$ https://github.com/System-T/UniversalPropositions
'https://pytorch.org
$^{2}$ https://www.pytorchlightning.ai