SlowGuess/ACL_2020_OCR huggingface dataset:
We first conduct experiments on cross-lingual transfer from the English source to the rest of the other six target languages, respectively, which has been a typical setting for cross-lingual investigations (Wang et al., 2019). The results are summarized in Table 2. We list the F-scores by using only the source corpus (SRC), only the translated target corpus (TGT) and the mixture corpus of source and target (SRC & TGT), comparing the performances of different multilingual word representations as well as different multilingual SRL models.
Multilingual word representations. First, we evaluate the effectiveness of the three different multilingual word representations exploited. We compare their performances under two settings, by using SRC and TGT corpus, respectively. According to the results, we find that the multilingual contextualized word representations (i.e. BERT and ELMo) are better in both two settings, which is consistent with previous studies (Mulcaire et al., 2019; Schuster et al., 2019). Interestingly, the multilingual BERT performs worse than the ELMo, which can be explained by that the ELMo representation is pre-trained based on the corpus which involves in the focused seven languages. This indicates that the official released multilingual BERT can be further improved, since monolingual BERT has been demonstrated to produce better performances than
12http://www.cis.upenn.edu/\~dbikel/software.html#comparator
11https://github.com/google-research/bert
7019
ELMo (Tenney et al., 2019).
Translated target. Next, We consider taking the translated target as only the training data to examine the effectiveness of the pseudo datasets. As shown in Table 2, we find that the translated datasets can bring significantly better performances than the source baseline overall languages, resulting in an averaged F1 score increase of  $51.1 - 44.4 = 6.7$ . The results demonstrate that corpus translation is one effective way for cross-lingual SRL. The observation is in line with the previous work for cross-lingual dependency parsing (Tiedemann and Agic, 2016; Zhang et al., 2019). By direct gold-standard corpus translation, the produced pseudo training data can not only remain high-quality SRL annotations but also capture the language divergences effectively, which leads to better performance than the source baseline model.
Combining source and pseudo target. Further, we combine the pseudo translated target corpus with the source language corpus together to train the target SRL models. According to the numbers in Table 2, we see that further gains can be achieved for all languages, where the averaged improvement is  $55.8 - 51.1 = 4.7$  (BASIC is used for a fair comparison). Note that since several source sentences are filtered during translation which might be the reason for the gains, we make a fairer comparison off-the-line by setting  $\alpha = 0$  (i.e., no sentence filtering). Similar gains can be achieved still. Considering that the translated sentences are semantically equal to their counterparts in the gold-standard source, the possible reasons could be two hands: (1) the translated sentences may be biased in linguistic expression due to the data-driven translation models, (2) the discarded conflicted annotations in corpus translation are important, which are complementary to our model.
Language-aware encoder. Finally, we investigate the effectiveness of PGN-BiLSTM module, which is exploited to capture language-specific information when the mixture corpus of both source and target datasets are used for training. As shown in Table 2, we can see that the language-aware encoder by  $PGN$  can boost the F1 scores significantly, achieving an averaged improvement by 60.3-55.8=4.5. In addition, we report the results of  $MoE$  and  $MAN-MoE$ , respectively, which also exploit the language information. All the results demonstrate the usefulness of language-specific informa
Model	EN	DE	FR	IT	ES	PT	FI	Avg
SRC
Emb	50.3	49.2	52.4	44.9	46.7	51.0	36.4	47.3
BERT	51.8	50.6	54.0	45.3	51.3	51.8	38.1	49.0
ELMo	53.6	51.6	56.7	51.3	57.4	52.6	39.7	51.8
TGT
Emb	56.5	51.6	55.2	47.1	50.0	53.2	40.4	50.6
BERT	59.8	55.5	57.0	52.6	54.3	56.6	44.0	54.3
ELMo	60.7	57.8	59.9	54.8	56.7	58.8	46.9	56.5
SRC & TGT (ELMo)
BASIC	61.9	64.8	60.3	56.4	61.1	63.1	50.7	59.8
PGN	65.7	68.8	66.1	64.8	68.7	69.2	58.6	66.0
MoE	63.2	67.8	63.1	62.6	65.2	67.5	54.2	63.4
MAN-MoE	64.0	68.5	67.2	65.7	67.5	69.0	57.5	65.6
Table 3: Cross-lingual transfer with multiple sources.
Source	EN	DE	FR	IT	ES	PT	FI
EN	65.0	64.8	58.7	62.5	56.0	54.5
DE	63.2	63.9	60.4	65.8	53.4	50.5
FR	60.1	53.7	63.3	63.6	62.1	51.3
IT	60.2	58.9	65.3	65.1	58.6	48.6
ES	60.1	57.3	64.9	64.1	67.0	50.7
PT	57.3	58.6	65.1	63.5	67.8	40.9
FI	50.7	52.1	64.6	53.6	60.3	51.6
ALL	65.7	68.8	66.1	64.8	68.7	69.2	58.6
Table 4: The results of bilingual transferring.
tion, and our  $PGN$  model is most effective.
5.5 Multi-Source Transfer
Further, we investigate the setting of multi-source transfer learning, where all other languages except a given target language are used as the source languages, aiming to study the effectiveness of our translation-based method comprehensively.
Overall performances. The results on multiple source SRL transferring are shown in Table 3. Generally, the results share similar tendencies with the single-source cross-lingual transfer from the source English, where the multilingual ELMo performs the best, the SRL models trained on the translated target datasets show better performances than those trained with the source datasets, and the mixture corpus with both source and target language datasets bring the best performances, which can be further improved by our final PGN model with language-aware encoders. We compare the PGN model with the MoE and MAN-MoE as well, showing slightly better performances, which indicates the effectiveness of the PGN-BiLSTM module. In addition, we can see that multi-source models outperform the single-source models in all cases, which is intuitive and consistent with previous studies (Lin et al., 2019).
Figure 6: Similarity heatmap of language embeddings for source languages to target languages. Deeper color indicates higher similarity.
Fine-grained bilingual transfer. Following, we investigate the individual bilingual SRL transferring by examining the performance of each source-target language pair, aiming to uncover which language benefits a target most and trying to answer whether all source languages are useful for a target language. Table 4 shows the results, where the cross-lingual models are trained on the mixture corpus of the source and translated target datasets. First, we can see that the languages belonging to a single family can benefit each other greatly, bringing better performances than the other languages in the majority of cases (i.e., EN-DE, FR-IT-ES-PT). Second, the multi-source transfer as indicated by All is able to obtain better performances across all languages, which further demonstrates its advantages over the single-source transfer.
Further, we look into the  $PGN$  model in detail, aiming to understand their capabilities of modeling linguistic-specific information. We examine it by simply visualizing the language ID embeddings  $e_{\mathcal{L}}$  of each source-target language pair, respectively, where their Euclidean distances are depicted. Intuitively, better performance can be achieved if the distance between the target and the source languages is closer. Figure 6 shows the heatmap matrix. We can see the overall tendency is highly similar to the results in Table 4, which is consistent with our intuition.
5.6 Analysis
Here we conduct detailed analysis to understand the gains from the translated target datasets. We select three representative languages for analysis, including German (DE), French (FR) and Finnish (FI), one language for each family, and compare
Figure 7: Performances on different argument label.
four models mainly, including three models (i.e., SRC, TGT and SRC & TGT with  $PGN$ ) of the single-source transfer from English and the final  $PGN$  model of multi-source transfer.
Performances by the SRL roles. First, we investigate the cross-lingual SRL performances in terms of SRL Roles. We select four representative roles for comparison, including A0 (Agent), A1 (Patient), A2 (Instrument, Benefactive, Attribute) and AM-TMP (Temporal), and report their F1 scores. Figure 7 shows the results. As a whole, the role A0 achieves the best F1 scores across all languages and all models, A1 ranks the second, and A2 and AM-TMP are slightly worse. The tendency could be accounted for by the distribution of these labels, where A0 is the most frequent and A2 and AM-TMP have lower frequencies than A0 and A1. The second possible reason could be due to that the majority of the A0 and A1 words are notional words which could be more easily transferred by cross-lingual models.
In addition, we can see that the tendencies across different models for all three languages and all labels are identical, where multi-source transfer performs the best, single-source  $\mathrm{SRC + TGT}$  ranks the second and our baseline model is the last. The observation is consistent with the overall tendency, demonstrating the stability and also further verifying the effectiveness of our proposed models.
Performances by the distances to the predicate. Second, we study the SRL performances in terms of the distance to the predicate word. Intuitively, long-distance relations are more difficult, thus we expect that the SRL performance would decrease as the distance increases, as SRL actually detects the relationship between the role words and their
7021
Figure 8: Performances by surface distance between predicates and arguments.
predicates. Figure 8 shows the F1 scores. First, for all the settings we can see that the SRL performance drops by longer distances, which confirms our intuition. In addition, the tendency between different models is the same as the overall results, demonstrating the effectiveness of our method.
We proposed a translation-based alternative for cross-lingual SRL. The key idea is to construct high-quality datasets for the target languages by corpus translation from the gold-standard SRL annotations of the source languages. In addition, we combined the gold-standard source SRL corpora and the pseudo translated target corpora together to enhance the cross-lingual SRL models. We investigated cross-lingual SRL models with different kinds of multilingual word representations. Further, we presented a PGN-BiLSTM encoder to better exploit the mixture corpora of different languages. Experimental results on the UPB v1.0 dataset show that the translation-based method is an effective method for cross-lingual SRL transferring. Significant improvements can be achieved by using the translated datasets for all selected languages, including both single-source and multi-source transfer. Experiment analysis is offered to understand the proposed method in depth.
This work is supported by the National Natural Science Foundation of China (No.61772378 and 61602160), the National Key Research and Development Program of China (No.2017YFC1200500),
the Research Foundation of Ministry of Education of China (No.18JZD015), and the Major Projects of the National Social Science Foundation of China (No.11&ZD189).
Alan Akbik, Laura Chiticariu, Marina Danilevsky, Yun-yao Li, Shivakumar Vaithyanathan, and Huaiyu Zhu. 2015. Generating high quality proposition banks for multilingual semantic role labeling. In Proceedings of the ACL, pages 397-407.
Alan Akbik and Yunyao Li. 2016. Polyglot: Multilingual semantic role labeling with unified labels. In Proceedings of ACL, pages 1-6.
Maryam Aminian, Mohammad Sadegh Rasooli, and Mona Diab. 2019. Cross-lingual transfer of semantic roles: From raw text to semantic roles. In Proceedings of the IWCS, pages 200-210.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the ICLR.
Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the CoNLL, pages 43-48.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the ACL, pages 3098-3112.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Q. Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the EMNLP, pages 2475–2485.
Angel Daza and Anette Frank. 2019. Translate and label! an encoder-decoder approach for cross-lingual semantic role labeling. In Proceedings of the EMNLP, pages 603-615.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL, pages 4171-4186.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the NAACL, pages 644-648.
7022
Qin Gao and Stephan Vogel. 2011. Corpus expansion for statistical machine translation with semantic role label substitution rules. In Proceedings of the ACL, pages 294-298.
Pierre-Etienne Genest and Guy Lapalme. 2011. Framework for abstractive summarization using text-to-text generation. In Proceedings of the Workshop on Monolingual Text-To-Text Generation, pages 64-73.
Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proceedings of the AAAI, pages 2734-2740.
Jiang Guo, Darsh Shah, and Regina Barzilay. 2018. Multi-source domain adaptation with mixture of experts. In Proceedings of the EMNLP, pages 4694-4703.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018a. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the ACL, pages 364-369.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettle-moyer. 2017. Deep semantic role labeling: What works and what's next. In Proceedings of the ACL, pages 473-483.
Shexia He, Zuchao Li, and Hai Zhao. 2019. Syntax-aware multilingual semantic role labeling. In Proceedings of the EMNLP, pages 5353-5362.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018b. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the ACL, pages 2061-2071.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara I. Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(3):311-325.
Chen Jia, Xiaobo Liang, and Yue Zhang. 2019. Cross-domain NER using cross-domain language modeling. In Proceedings of the ACL.
Atif Khan, Naomie Salim, and Yogan Jaya Kumar. 2015. A framework for multi-document abstractive summarization based on semantic role labelling. Appl. Soft Comput., 30:737-747.
Paul R. Kingsbury and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the LREC, Spain.
Mikhail Kozhevnikov and Ivan Titov. 2013. Crosslingual transfer of semantic role labeling models. In Proceedings of the ACL, pages 1190-1200.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the ICML, pages 282-289.
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceedings of the ICLR.
Tao Lei, Yuan Zhang, Lluís Márquez, Alessandro Moschitti, and Regina Barzilay. 2015. High-order low-rank tensors for semantic role labeling. In Proceedings of the NAACL, pages 1150-1160.
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the EMNLP, pages 2401-2411.
Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of the AAAI, pages 6730-6737.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the ACL, pages 3125-3135.
Ding Liu and Daniel Gildea. 2010. Semantic role features for machine translation. In Proceedings of the COLING, pages 716-724.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Tackstrom, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the ACL, pages 92-97.
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019. Low-resource parsing with crosslingual contextualized representations. CoRR, abs/1909.08744.
Phoebe Mulcaire, Swabha Swayamdipta, and Noah A. Smith. 2018. Polyglot semantic role labeling. In Proceedings of the ACL, pages 667-672.
Sebastian Pado and Mirella Lapata. 2014. Crosslingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36(36):307-340.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
7023
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *Proceedings of the NAACL*, pages 2227-2237.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the LREC, pages 2089-2096.
Lonneke van der Plas, Paola Merlo, and James Henderson. 2011. Scaling up automatic cross-lingual semantic role annotation. In Proceedings of the ACL, pages 299-304.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the EMNLP, pages 425-435.
Adam Przepiórkowski and Agnieszka Patejuk. 2018. Arguments and adjuncts in universal dependencies. In Proceedings of the COLING, pages 3837-3852.
Mohammad Sadegh Rasooli and Michael Collins. 2015. Density-driven cross-lingual transfer of dependency parsers. In Proceedings of the EMNLP, pages 328-338.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the NAACL, pages 1599–1613.
Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of the EMNLP, pages 12-21.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the EMNLP, pages 5027-5038.
Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Greedy, joint syntactic-semantic parsing with stack LSTMs. In Proceedings of the CoNLL, pages 187-197.
John Sylak-Glassman, Christo Kirov, David Yarowsky, and Roger Que. 2015. A language-independent feature schema for inflectional morphology. In Proceedings of the ACL, pages 674-680.
Oscar Täckström, Ryan T. McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of the NAACL, pages 477-487.
Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In Proceedings of the AAAI, pages 4929-4936.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In Proceedings of the ICLR.
Jörg Tiedemann. 2015. Improving the cross-lingual projection of syntactic dependencies. In Proceedings of the Nordic Conference of Computational Linguistics, pages 191-199.
Jörg Tiedemann and Zeljko Agic. 2016. Synthetic treebanking for cross-lingual dependency parsing. Journal of Artificial Intelligence Research, 55:209-248.
Jörg Tiedemann, Željko Agić, and Joakim Nivre. 2014. Treebank translation for cross-lingual parser induction. In Proceedings of the CoNLL, pages 130-140.
Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Proceedings of the ACL, pages 700-706.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the EMNLP, pages 5720-5726.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Qingrong Xia, Zhenghua Li, Min Zhang, Meishan Zhang, Guohong Fu, Rui Wang, and Luo Si. 2019. Syntax-aware neural semantic role labeling. In Proceedings of the AAAI, pages 7305-7313.
David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the International Conference on Human Language Technology Research.
Meishan Zhang, Yue Zhang, and Guohong Fu. 2019. Cross-lingual dependency parsing using code-mixed TreeBank. In Proceedings of the EMNLP, pages 996-1005.
Han Zhao, Shanghang Zhang, Guanhang Wu, João Paulo Costeira, José M. F. Moura, and Geoffrey J. Gordon. 2018. Multiple source domain adaptation with adversarial learning. In Proceedings of the ICLR.
7024