SlowGuess/NAACL_2021_OCR huggingface dataset:
CONLL-2009 - MULTILINGUAL - IN DOMAIN	CA	CZ	DE	EN	ES	ZH
CoNLL-2009 ST best	⊙	80.3	85.4	79.7	85.6	80.5	78.6
Marcheggiani et al. (2017)	⊙	—	86.0	—	87.7	80.3	81.2
Chen et al. (2019)	⊙	81.7	88.1	76.4	91.1	81.3	81.7
Cai and Lapata (2019b)	⊙	—	—	82.7	90.0	81.8	83.6
Cai and Lapata (2019a)	⊙	—	—	83.8	91.2	82.9	85.0
Lyu et al. (2019)	⊙	80.9	87.5	75.8	90.1	80.5	83.3
He et al. (2019)	⊙	86.0	89.7	81.1	90.9	85.2	86.9
This work m-BERT frozen / monolingual	⊙	86.2	90.0	85.2	90.5	85.0	86.4
This work m-BERT / monolingual	⊙	86.8	90.3	85.8	90.7	85.3	86.9
This work m-BERT / cross-lingual	⊙	87.1	90.8	86.5	91.0	85.6	87.3
This work XLM-R frozen / monolingual	⊙	86.8	90.4	86.5	90.8	85.2	86.9
This work XLM-R / monolingual	⊙	87.8	91.6	87.6	91.6	86.0	87.5
This work XLM-R / cross-lingual	⊙	88.0	91.5	88.0	91.8	86.3	87.7
Table 1:  $\mathrm{F}_1$  scores on the in-domain evaluation CoNLL-2009 with gold pre-identified predicates. "CoNLL-2009 ST best" refers to the best results obtained (by different systems) during the Shared Task. We include all the systems that reported results in at least 4 languages.  $\odot$ : syntax-aware system.  $\varnothing$ : syntax-agnostic system.
CoNLL-2009 - OOD	CZ	DE	EN
CoNLL-2009 ST best	85.4	65.9	73.3
Zhao et al. (2009)	82.7	67.8	74.6
Marcheggiani et al. (2017)	87.2	—	77.7
Li et al. (2019)	—	—	81.5
Chen et al. (2019)	—	—	82.7
Lyu et al. (2019)	86.0	65.7	82.2
This work m-BERT / mono	90.4	72.6	84.6
This work m-BERT / cross	91.0	73.0	85.0
This work XLM-R / mono	90.8	73.9	83.7
This work XML-R / cross	91.1	74.2	84.3
Table 2:  $F_1$  scores on the out-of-domain evaluation of CoNLL-2009 with gold pre-identified predicates.
jection techniques, and ii) it uses the English PropBank for all languages, which goes against our interest in capturing cross-lingual knowledge over heterogeneous inventories.
Cross-lingual SRL. Table 1 compares the results obtained by our unified cross-lingual model against the state of the art in multilingual SRL, including both syntax-agnostic and syntax-aware architectures, on the in-domain test sets of CoNLL-2009 when using gold pre-identified predicates, rather than the predicates identified by the model itself, as standard in the CoNLL-2009 shared task. While
proposing a state-of-the-art architecture is not the focus of this work, we believed it was important to build our cross-lingual approach starting from a strong and consistent baseline. For this reason, Table 1 includes the results obtained when training a separate instance of our model for each language, using the same strategy adopted by current multilingual systems (Cai and Lapata, 2019a; He et al., 2019; Lyu et al., 2019) and showing results that are competitive with He et al. (2019), inter alia. Remarkably, thanks to its universal encoders shared across languages and formalisms, our unified cross-lingual model outperforms our state-of-the-art baseline in all the 6 languages at a fraction of the cost in terms of number of trainable parameters (a single cross-lingual model against six monolingual models, each trained on a different language). Similar results can be seen in Table 2 where our cross-lingual approach improves over the state of the art on the out-of-domain evaluation of CoNLL2009, especially in the German and English test sets which were purposely built to include predicates that do not appear in the training set. These results confirm empirically our initial hunch that semantic role labeling relations are deeply rooted beyond languages, independently of their surface realization and their predicate-argument structure inventories.
Finally, for completeness, Appendix E includes the results of our system on the individual subtasks, namely, predicate identification and predicate sense
CONLL-2009 - IN DOMAIN	CA	CZ	DE	EN	ES	ZH
This work XLM-R / monolingual / 10% training	52.7	79.9	60.2	81.7	49.2	72.9
This work XLM-R / cross-lingual / 10% training	78.2	84.0	69.9	84.3	76.1	78.6
This work XLM-R / monolingual / 1-shot learning	44.5	21.8	40.9	67.4	46.5	72.1
This work XLM-R / cross-lingual / 1-shot learning	63.2	28.9	50.1	70.2	62.6	73.6
This work XLM-R / cross-lingual / 1-shot learning / 100% EN	66.4	29.6	55.5	91.6*	64.3	76.7
Table 3:  $\mathrm{F}_{1}$  scores on the in-domain evaluation CoNLL-2009 with gold pre-identified predicates for low-resource (top) and one-shot learning (bottom) scenarios. *: the result in EN on the last line is not directly comparable with those above as we use the full English training set.
disambiguation.
Low-resource cross-lingual SRL. We evaluate the robustness of our model in low-resource cross-lingual SRL by artificially reducing the training set of each language to  $10\%$  of its original size. Table 3 (top) reports the results obtained by our model when trained separately on the reduced training set of each language (monolingual), and the results obtained by the same model when trained on the union of the reduced training sets (cross-lingual). The improvements of our cross-lingual approach compared to the more traditional monolingual baseline are evident, especially in lower-resource scenarios, with absolute improvements in  $\mathrm{F}_1$  score of  $25.5\%$ ,  $9.7\%$  and  $26.9\%$  on the Catalan, German and Spanish test sets, respectively. This is thanks to the ability of the model to use the knowledge from a language to improve its performance on other languages.
One-shot cross-lingual SRL. An interesting open question in SRL is whether a system can learn to model the semantic relations between a predicate sense  $s$  and its arguments, given a limited number of training samples in which  $s$  appears. In particular in our case, we are interested in understanding how the model fares in a synthetic scenario where each sense appears at most once in the training set, that is, we evaluate our model in a one-shot learning setting. As we can see from Table 3 (bottom), our cross-lingual approach outperforms its monolingual counterpart trained on each synthetic dataset separately by a wide margin, once again providing strong absolute improvements –  $18.7\%$  in Catalan,  $9.2\%$  in German and  $16.1\%$  in Spanish in terms of  $F_1$  score – for languages where the number of training instances is smaller.
It is not uncommon for supervised cross-lingual tasks to feature different amounts of data for each
language, depending on how difficult it is to get manual annotations for each language of interest. We simulate this setting in SRL by training our model on  $100\%$  of the training data available for the English language, while keeping the one-shot learning setting for all the other languages. As Table 3 (bottom) shows, non-English languages exhibit further improvements as the number of English training samples increases, lending further credibility to the idea that SRL can be learnt across languages even when using heterogeneous resources. Not only do these results suggest that a cross-lingual/cross-resource approach might mitigate the need for a large training set in each language, but also that reasonable cross-lingual results may be obtained by maintaining a single large dataset for a high-resource language, together with several small datasets for low-resource languages.
Cross-formalism SRL. In contrast to existing multilingual systems, a key benefit of our unified cross-lingual model is its ability to provide annotations for predicate senses and semantic roles in any linguistic formalism. As we can see from Figure 1 (left), given the English sentence "the cat threw its ball out of the window", our language-specific decoders produce predicate sense and semantic role labels not only according to the English PropBank inventory, but also for all the other resources, as it correctly identifies the agentive and patientive constituents independently of the formalism of interest. And this is not all, our model may potentially work on any of the 100 languages supported by the underlying language model (m-BERT or XLM-RoBERTa), e.g., in Italian, as shown in Figure 1 (right). This is vital for those languages for which a predicate-argument structure inventory has not yet been developed - an endeavor that may take
Figure 1: Thanks to its universal encoders, our unified cross-lingual model is able to provide predicate sense and semantic role labels according to several linguistic formalisms. Left: SRL labels for an English input sentence. Right: SRL labels for an Italian input sentence, which can be translated into English as "The president refuses the help of the opponents". Notice that Italian is not among the languages in the training set.
Figure 2: Visualization of the cross-resource mapping learnt by our model. Left: Mapping from Chinese PropBank, German PropBank and AnCora (both Catalan and Spanish) to English PropBank. Right: Mapping from English PropBank, German PropBank, Chinese PropBank and AnCora (both Spanish and Catalan) to VerbAtlas.
years to come to fruition – and, therefore, manually annotated data are unavailable. Thus, as long as a large amount of pretraining data is openly accessible, our system provides a robust cross-lingual tool to compare and analyze different linguistic theories and formalisms across a wide range of languages, on the one hand, and to overcome the issue of performing SRL on languages where no inventory is available, on the other.
Aligning heterogeneous resources. As briefly mentioned previously, the universal encoders in the model architecture force our system to learn cross-lingual features that are important across different formalisms. A crucial consequence of this approach is that the model learns to implicitly align the resources it is trained on, without the aid of word aligners and translation tools, even
when these resources may be designed around specific languages and, therefore, present significant differences. In order to bring to light what our model implicitly learns to align in its shared cross-lingual space (see Sections 3.2 and 3.3), we exploit its language-specific decoders to build a mapping from any source inventory, e.g., AnCora, to a target inventory, e.g., the English PropBank. In particular, we use our cross-lingual model to label a training set originally tagged with a source inventory to produce silver annotations according to a target inventory, similarly to what is shown in Figure 1. While producing the silver annotations, we keep track of the number of times each predicate sense in the source inventory is associated by the model with a predicate sense of the target inventory. As a result, we produce a weighted directed graph in which the nodes are predicate senses and an edge
$(a, b)$  with weight  $w$  indicates that our model maps the source predicate sense  $a$  to the target predicate sense  $b$  at least  $w$  times. A portion of this graph is displayed in Figure 2 where, for visualization purposes, we show the most frequent alignments for each language, i.e., the top-3 edges with largest weight from the nodes of each inventory to the nodes of the English PropBank (Figure 2, left) and to the nodes of VerbAtlas (Figure 2, right).<sup>7</sup>
For example, Figure 2 (left) shows that our model learns to map the Spanish AnCora sense empezar.c1 and the German PropBank sense starten.2 to the English PropBank sense start.01, but also that, depending on the context, the Chinese PropBank sense开始.01 can correspond to both start.01 and begin.01. Figure 2 (right) also shows that our model learns to map senses from different languages and formalisms to the coarse-grained senses of VerbAtlas, even though the latter formalism is quite distant from the others as its frames are based on clustering WordNet synsets – sets of synonymous words – that share similar semantic behavior, rather than enumerating and defining all the possible senses of a lexeme as in the English and Chinese PropBanks. To the best of our knowledge, our unified model is the first transfer-based tool to automatically align diverse linguistic resources across languages without relying on human supervision.
On one hand, recent research in multilingual SRL has focused mainly on proposing novel model architectures that achieve state-of-the-art results, but require a model instance to be trained on and for each language of interest. On the other hand, the latest developments in cross-lingual SRL have revolved around using the English PropBank inventory as a universal resource for other languages through annotation transfer techniques. Following our hunch that semantic relations may be deeply rooted beyond the surface realizations that distinguish one language from another, we propose a new approach to cross-lingual SRL and present a model which learns from heterogeneous linguistic resources in order to obtain a deeper understanding of sentence-level semantics. To achieve this objective, we equip our model architecture with "universal" encoders which share their weights across
languages and are, therefore, forced to learn knowledge that spans across varying formalisms.
Our unified cross-lingual model, evaluated on the gold multilingual benchmark of CoNLL-2009, outperforms previous state-of-the-art multilingual systems over 6 diverse languages, ranging from Catalan to Czech, from German to Chinese, and, at the same time, also considerably reduces the amount of trainable parameters required to support different linguistic formalisms. And this is not all. We find that our approach is robust to low-resource scenarios where the model is able to exploit the complementary knowledge contained in the training set of different languages.
Most importantly, our model is able to provide predicate sense and semantic role labels according to 7 predicate-argument structure inventories in a single forward pass, facilitating comparisons between different linguistic formalisms and investigations about interlingual phenomena. Our analysis shows that, thanks to the prior knowledge encoded in recent pretrained language models and our focus on learning from cross-lingual features, our model can be used on languages that were never seen at training time, opening the door to alignment-free cross-lingual SRL on languages where a predicate-argument structure inventory is not yet available. Finally, we show that our model implicitly learns to align heterogeneous resources, providing useful insights into inter-resource relations. We leave an in-depth qualitative and quantitative analysis of the learnt inter-resource mappings for future work.
We hope that our work can set a stepping stone for future developments towards the unification of heterogeneous SRL. We release the code to reproduce our experiments and the checkpoints of our best models at https://github.com/SapienzaNLP/unify-srl.
7We release the full alignment and the corresponding graph at https://github.com/SapienzaNLP/unify-srl.
Alan Akbik, Laura Chiticariu, Marina Danilevsky, Yun-yao Li, Shivakumar Vaithyanathan, and Huaiyu Zhu. 2015. Generating high quality proposition banks for multilingual Semantic Role Labeling. In Proceedings of ACL.
Alan Akbik, Vishwajeet Kumar, and Yunyao Li. 2016. Towards semi-automatic generation of proposition banks for low-resource languages. In Proceedings of EMNLP.
Maryam Aminian, Mohammad Sadegh Rasooli, and Mona Diab. 2019. Cross-lingual transfer of semantic roles: From raw text to semantic roles. In Proceedings of IWCS.
Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual Semantic Role Labeling. In Proceedings of CoNLL.
Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018. A full end-to-end semantic role labeler, syntactic-agnostic over syntactic-aware? In Proceedings COLING.
Rui Cai and Mirella Lapata. 2019a. Semi-supervised Semantic Role Labeling with cross-view training. In Proceedings of EMNLP.
Rui Cai and Mirella Lapata. 2019b. Syntax-aware Semantic Role Labeling without parsing. Transactions of ACL (TACL), 7:343-356.
Rui Cai and Mirella Lapata. 2020. Alignment-free cross-lingual Semantic Role Labeling. In Proceedings of EMNLP.
Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41-75.
Xinchi Chen, Chunchuan Lyu, and Ivan Titov. 2019. Capturing argument interaction in Semantic Role Labeling with capsule networks. In Proceedings of EMNLP.
Simone Conia, Fabrizio Brignone, Davide Zanfardino, and Roberto Navigli. 2020. InVeRo: Making Semantic Role Labeling accessible with intelligible verbs and roles. In Proceedings of EMNLP.
Simone Conia and Roberto Navigli. 2020. Bridging the gap in multilingual semantic role labeling: a language-agnostic approach. In Proceedings of COLING.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL.
Angel Daza and Anette Frank. 2019. Translate and label! an encoder-decoder approach for cross-lingual semantic role labeling. In Proceedings of EMNLP, pages 603-615.
Angel Daza and Anette Frank. 2020. X-SRL: A parallel cross-lingual Semantic Role Labeling dataset. In Proceedings of EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of NAACL.
Andrea Di Fabio, Simone Conia, and Roberto Navigli. 2019. VerbAtlas: a novel large-scale verbal semantic resource and its application to Semantic Role Labeling. In Proceedings of EMNLP.
Timothy Dozat and Christopher D. Manning. 2017. Deep bioaffine attention for neural dependency parsing. In Proceedings of ICLR.
Hao Fei, Meishan Zhang, and Donghong Ji. 2020. Cross-lingual Semantic Role Labeling with high-quality translated training corpus. In Proceedings of ACL.
Charles J. Fillmore. 1968. The case for case. Universals in Linguistic Theory.
Daniel Gildea and Daniel Jurafsky. 2000. Automatic labeling of semantic roles. In Proceedings of ACL.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of LREC.
Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antonia Martí, Lluis Márquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Stepanek, Pavel Stranák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of CoNLL.
Jan Hajic, J. Panevova, Zdenka Uresová, Alevtina Bémová, V. Kolárová, and P. Pajas. 2003. PDT-Vallex: Creating a large-coverage valency lexicon for treebank annotation.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural Semantic Role Labeling. In Proceedings of ACL.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettle-moyer. 2017. Deep Semantic Role Labeling: What works and what's next. In Proceedings of ACL.
Shexia He, Zuchao Li, and Hai Zhao. 2019. Syntax-aware multilingual Semantic Role Labeling. In Proceedings of EMNLP.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of NAACL.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural Comput.*, 9(8).
Jungo Kasai, Dan Friedman, Robert Frank, Dragomir R. Radev, and Owen Rambow. 2019. Syntax-aware neural Semantic Role Labeling with supertags. In Proceedings of NAACL.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit, volume 5, pages 79-86.
Ilia Kuznetsov and Iryna Gurevych. 2020. A matter of framing: The impact of linguistic formalism on probing results. In Proceedings of EMNLP.
Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of AAAI.
Chunchuan Lyu, Shay B. Cohen, and Ivan Titov. 2019. Semantic Role Labeling with iterative structure refinement. In Proceedings of EMNLP.
Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based Semantic Role Labeling. In Proceedings of CoNLL.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for Semantic Role Labeling. In Proceedings of EMNLP.
Lluis Marquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Semantic Role Labeling: An introduction to the special issue. Computational Linguistics, 34(2):145-159.
Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The NomBank project: An interim report. In Proceedings of the Workshop Frontiers in Corpus Annotation.
Phoebe Mulcaire, Swabha Swayamdipta, and Noah A. Smith. 2018. Polyglot Semantic Role Labeling. In Proceedings of ACL.
Roberto Navigli. 2018. Natural Language Understanding: Instructions for (present and future) use. In Proceedings of IJCAI.
Sebastian Padó and Mirella Lapata. 2009. Crosslingual annotation projection for semantic roles. J. Artif. Intell. Res., 36:307-340.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2018. Searching for activation functions. In Proceedings of ICLR.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for Semantic Role Labeling. In Proceedings of EMNLP.
Mariona Taule, Maria Antônia Martí, and Marta Recasens. 2008. Ancora: Multilevel annotated corpora for catalan and spanish. In Proceedings of LREC.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP.
Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of EMNLP.
Nianwen Xue and Martha Palmer. 2009. Adding semantic roles to the chinese treebank. Nat. Lang. Eng., 15(1):143-172.
Hai Zhao, Wenliang Chen, Jun'ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In Proceedings of CoNLL.
Table 4 reports the hyperparameter values we choose for our model configuration and experiments.
Hyperparameter	Value
dw	Size of ei	512
K'	Universal sentence encoder layers	3
dt	Size of ti	512
K''	Universal pred.-arg. encoder layers	1
dz	Size of ai	256
dsp	Size of pi	32
dss	Size of si	512
dsa	Size of ri	512
Batch size	32
Batch size when fine-tuning	128
Max learning rate	10-3
Min learning rate	10-5
Max lr for LM fine-tuning	10-5
Min lr for LM fine-tuning	10-6
Warmup epochs	1
Cooldown epochs	15
Training epochs	30
Table 4: Hyperparameter values for our model architecture. We use the same hyperparameter values for our monolingual and cross-lingual experiments.
Tables 5, 6 and 7 provide an overview of the training sets provided as part of the CoNLL-2009 shared task, with statistics about sentences, predicates and arguments.
C Hardware Infrastructure
All the experiments were performed on a x86-64 architecture with 64GB of RAM, an 8-core CPU running at  $3.60\mathrm{GHz}$ , and a single Nvidia RTX 2080Ti with 11GB of VRAM.
Training was performed using half-precision via Apex. Training times varied considerably depending on the experiment setting: the shorter experiment lasted 26 minutes (training m-BERT on  $10\%$  of the Catalan training set), whereas the longest
experiment lasted for 46 hours (training XLM-RoBERTa on the union of all the datasets of all the languages).
E Other Results
Predicate identification. In Table 8 we report the results of our model on predicate identification.